Foreign object detection teacher data generation device, trained model manufacturing device, foreign object detection device, foreign object detection teacher data generation method, and program

Abstract

To provide a foreign substance detection teacher data generating device capable of generating teacher data for detecting foreign substances without requiring labeling work.SOLUTION: A foreign substance detection teacher data generating device according to the present invention includes a detection target image acquisition unit, a foreign substance image acquisition unit, an image combining unit, and a teacher data output unit. The detection target image acquisition unit acquires a detection target image capturing a foreign substance detection target after a foreign substance is removed. The foreign substance image acquisition unit acquires a foreign substance image capturing the foreign substance. The image combining unit combines the foreign substance image with the detection target image to generate a foreign substance composite image. The teacher data output unit outputs, as teacher data with foreign substances, a pair of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a foreign object detection training data generation device, a trained model production device, a foreign object detection device, a foreign object detection training data generation method, and a program. [Background technology]

[0002] As a method for detecting and classifying foreign matter contained in an object to be detected, a foreign matter inspection method in which image data is processed by supervised machine learning is known (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-14652 Summary of the Invention [Problem to be solved by the invention]

[0004] However, supervised classification requires a large amount of training data, which requires labeling the positions and types of impurities in images containing them, which poses a problem of requiring a great deal of time and effort to do manually.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a foreign object detection training data generating device that can generate training data for foreign object detection without requiring labeling work. [Means for solving the problem]

[0006] In order to achieve the above object, the foreign object detection training data generating device of the present invention comprises: The apparatus includes a detection object image acquisition unit, a foreign substance image acquisition unit, an image synthesis unit, and a training data output unit, the detection target image acquisition unit acquires a detection target image obtained by capturing an image of the foreign object detection target after the foreign object has been removed, the foreign substance image acquisition unit acquires a foreign substance image by capturing an image of the foreign substance; the image composition unit combines the foreign substance image with the detection target image to generate a foreign substance composite image; The teacher data output unit A set of the foreign substance composite image and the composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance.

[0007] The trained model production device of the present invention comprises: The system includes a training data acquisition unit and a trained model generation unit, the teacher data acquisition unit acquires, as the teacher data for foreign object detection, the teacher data for foreign object detection output by the teacher data generation device for foreign object detection of the present invention; The trained model generation unit generates, through machine learning using the foreign object detection training data, a foreign object detection model that, when inputted, outputs whether or not the foreign object detection target contains a foreign object, as a trained model.

[0008] The foreign object detection device of the present invention comprises: a foreign object detection target image acquisition unit and a foreign object detection unit, the foreign object detection target image acquisition unit acquires a foreign object detection target image by capturing an image of the foreign object detection target, the foreign object detection unit inputs the foreign object detection target image into a foreign object detection model and detects whether or not the foreign object detection target includes a foreign object; The foreign object detection model is a trained model generated by machine learning using training data generated by the foreign object detection training data generation device of the present invention, so that when an image of the foreign object detection target is input, it outputs whether or not the foreign object detection target contains a foreign object, or it is a trained model manufactured by the trained model manufacturing device of the present invention.

[0009] The method for generating training data for foreign object detection according to the present invention includes the steps of: The method includes a detection object image acquisition step, a foreign substance image acquisition step, an image synthesis step, and a training data output step, the detection target image acquisition step acquires a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed; the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance; the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image; The teacher data output step includes: A set of the foreign substance composite image and the composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance.

[0010] The trained model production method of the present invention includes: The method includes a training data acquisition step and a trained model generation step. the teacher data acquisition step acquires, as the teacher data for foreign object detection, the teacher data for foreign object detection output by the method for generating teacher data for foreign object detection of the present invention; The trained model generation process uses machine learning using the foreign object detection training data to generate a trained model of a foreign object detection model that, when an image of a foreign object detection target is input, outputs whether or not the foreign object detection target contains a foreign object.

[0011] The foreign object detection method of the present invention comprises: The method includes a foreign object detection image acquisition step and a foreign object detection step, the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target, the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model and detecting whether or not a foreign object is included in the foreign object detection target; The foreign object detection model is a trained model generated by machine learning using training data generated by the foreign object detection training data generation method of the present invention, so that when a foreign object detection target image of the foreign object detection target is input, it outputs whether or not the foreign object detection target contains a foreign object, or is a trained model produced by the trained model production method of the present invention.

[0012] The first program of the present invention includes a detection object image acquisition procedure, a foreign substance image acquisition procedure, an image synthesis procedure, and a training data output procedure, the detection target image acquisition step includes acquiring a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed; the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance; the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image; The teacher data output procedure includes: outputting a set of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image as training data indicating the presence of a foreign substance; This is a program for causing a computer to execute each of the above procedures.

[0013] A second program of the present invention includes a training data acquisition procedure and a trained model generation procedure, the teacher data acquisition step acquires teacher data for foreign object detection output by the first program of the present invention as teacher data for foreign object detection; the trained model generation step generates, as a trained model, a foreign object detection model that outputs whether or not a foreign object is included in a foreign object detection target when an image of the foreign object detection target obtained by capturing the foreign object detection target is input, by machine learning using the foreign object detection training data; This is a program for causing a computer to execute each of the above procedures.

[0014] A third program of the present invention includes a foreign object detection image acquisition procedure and a foreign object detection procedure, the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target; the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model, and detecting whether or not the foreign object detection target includes a foreign object; the foreign object detection model is a trained model generated by machine learning using training data generated by the first program of the present invention so as to output whether or not a foreign object is included in a foreign object detection target when an image of the foreign object detection target obtained by capturing the foreign object detection target is input, or is a trained model produced by the second program of the present invention; This is a program for causing a computer to execute each of the above procedures. [Effects of the Invention]

[0015] According to the present invention, training data for foreign object detection can be generated without the need for labeling work. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a foreign object detection training data generation device according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of the foreign object detection training data generation device according to the first embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of processing in the foreign object detection training data generation device of the first embodiment. [Figure 4] FIG. 4A is a schematic diagram showing a foreign substance composite image, and FIG. 4B is a schematic diagram for explaining the composite position of a foreign substance in the foreign substance composite image. [Figure 5] FIG. 5 is a block diagram showing an example of the configuration of a foreign object detection training data generation device according to the second embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of processing in the foreign object detection training data generation device of the second embodiment. [Figure 7] FIG. 7 is a block diagram showing an example of the configuration of a foreign object detection training data generation device according to the third embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of processing in the foreign object detection training data generation device of the third embodiment. [Figure 9]FIG. 9 is a block diagram showing an example of the configuration of a trained model production device according to the fourth embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of the hardware configuration of a trained model production device according to the fourth embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of processing in the trained model production device of the fourth embodiment. [Figure 12] FIG. 12 is a block diagram showing an example of the configuration of a foreign object detection device according to the fifth embodiment. [Figure 13] FIG. 13 is a block diagram showing an example of the hardware configuration of a foreign object detection device according to the fifth embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of processing in the foreign object detection device of the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Next, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments. In the following drawings, the same parts are denoted by the same reference numerals. Furthermore, the descriptions of the embodiments can be mutually incorporated unless otherwise specified, and the configurations of the embodiments can be combined unless otherwise specified.

[0018] [Embodiment 1] The foreign object detection training data generation device of this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of a foreign object detection training data generation device 10 of this embodiment. As shown in Fig. 1, the foreign object detection training data generation device 10 (hereinafter also referred to as "the device 10") includes a detection object image acquisition unit 11, a foreign object image acquisition unit 12, an image synthesis unit 13, and a training data output unit 14. Although not shown, the device 10 may also include, for example, a memory unit.

[0019] The device 10 may be, for example, a single device including the above-described units, or a device in which the units can be connected via a communication network. The device 10 can also be connected to an external device (described later) via the communication network. The communication network is not particularly limited, and any known network can be used, for example, wired or wireless. Examples of communication networks include the Internet, the World Wide Web (WWW), a telephone line, a Local Area Network (LAN), a Storage Area Network (SAN), a Delay Tolerant Networking (DTN), a Low Power Wide Area Network (LPWA), and a Local 5G (L5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), Local 5G, and LPWA. Examples of wireless communication include direct communication between devices (Ad Hoc communication), infrastructure communication, and indirect communication via an access point. The device 10 may be incorporated into a server as a system. The device 10 may be, for example, a personal computer (PC, for example, a desktop or notebook type) on which the program of the present invention is installed, a smartphone, a tablet terminal, etc. Furthermore, the device 10 may be in the form of cloud computing or edge computing, for example, in which at least one of the units is located on a server and the other units are located on a terminal.

[0020] 2 shows a block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a CPU 101, a memory 102, a bus 103, a storage device 104, an input device 106, an output device 107, and a communication device (communication unit) 108. The components of the device 10 are connected to each other via the bus 103 and their respective interfaces (I / F).

[0021] The CPU 101 cooperates with other components, for example, via a controller (such as a system controller or an I / O controller), and is responsible for overall control of the device 10. In the device 10, the CPU 101 executes, for example, the program 105 of the present invention and other programs, and also reads and writes various types of information. Specifically, for example, the CPU 101 functions as a detection object image acquisition unit 11, a foreign substance image acquisition unit 12, an image synthesis unit 13, and a training data output unit 14. The device 10 includes a CPU as a computing device, but may also include other computing devices such as a GPU (Graphics Processing Unit) or an APU (Accelerated Processing Unit), or may include a combination of the CPU and these.

[0022] The bus 103 can also be connected to, for example, external devices. Examples of the external devices include a trained model production device (described later), a foreign object detection device, an external storage device (external database, etc.), a printer, an external input device, an external output device, an audio output device such as a speaker, an external imaging device such as a camera, and various sensors such as an acceleration sensor, a geomagnetic sensor, and a direction sensor. The device 10 can be connected to an external network (the communication line network) by, for example, a communication device 108 connected to the bus 103, and can also be connected to other devices via the external network.

[0023] The memory 102 may be, for example, a main memory (primary storage device). When the CPU 101 performs processing, the memory 102 reads various operation programs, such as a program 105 of the present invention stored in a storage device 104 (described later), and the CPU 101 receives data from the memory 102 and executes the program. The main memory may be, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read only memory).

[0024] The storage device 104 is also referred to as an auxiliary storage device, for example, in contrast to the main memory (primary storage device). As described above, the storage device 104 stores an operating program 105 including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive for reading and writing data from and to the recording medium. The recording medium is not particularly limited and may be, for example, an internal or external type, such as a hard disk (HD), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, or memory card. The storage device 104 may be, for example, a hard disk drive (HDD) or a solid-state drive (SSD) in which the recording medium and drive are integrated. When the device 10 includes, for example, the storage unit, the storage device 104 functions as the storage unit. The storage device 104 may store, for example, a foreign object detection model, which will be described later.

[0025] In the present device 10, the memory 102 and the storage device 104 can also store various information such as log information, information acquired from an external database (not shown) or an external device, information generated by the present device 10, and information used when the present device 10 executes processing. Note that at least a portion of the information may be stored, for example, in an external server other than the memory 102 and the storage device 104, or may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

[0026] The device 10 further includes, for example, an input device 106 and an output device 107. Examples of the input device 106 include a pointing device such as a touch panel, track pad, or mouse; a keyboard; an imaging device such as a camera or scanner; a card reader such as an IC card reader or a magnetic card reader; and an audio input device such as a microphone. Examples of the output device 107 include a display device such as an LED display or a liquid crystal display; an audio output device such as a speaker; a printer; etc. In the first embodiment, the input device 106 and the output device 107 are configured separately, but the input device 106 and the output device 107 may also be configured as an integrated device, such as a touch panel display.

[0027] Next, an example of the method for generating training data for foreign object detection according to this embodiment will be described with reference to the flowchart of FIG. 3. The method for generating training data for foreign object detection according to this embodiment can be implemented as follows, for example, using the device for generating training data for foreign object detection 10 shown in FIG. 1 or FIG. 2. Note that the method for generating training data for foreign object detection according to this embodiment is not limited to use with the device for generating training data for foreign object detection 10 shown in FIG. 1 or FIG. 2. The object of foreign object detection in this invention is not particularly limited as long as it is an object that can be detected as a foreign object, and examples include food waste, products on a production line, agricultural products, etc. In the following description, an example will be given in which the object of foreign object detection is food waste at a biogas plant, but the present invention is not limited in any way to the following example.

[0028] First, the detection target image acquisition unit 11 acquires a detection target image capturing an image of the foreign object detection target after the foreign object has been removed (S1, detection target image acquisition step). The detection target image may be, for example, an image of the foreign object detection target from which the foreign object has been completely removed, or an image of the foreign object detection target from which the foreign object remains. However, it is preferable that the detection target image be an image of the foreign object detection target from which the foreign object has been completely removed. The detection target image may be, for example, a still image, a video, or a still image extracted from a video. The detection target image acquisition unit 11 may, for example, acquire images continuously or intermittently. In the latter case, the images may be acquired every predetermined time period or at any timing. The detection target image acquisition unit 11 may acquire the detection target image by capturing an image of the foreign object detection target using the imaging device, which is the input device 106, or may acquire the detection target image from an external imaging device via the communication network using the communication device 108. As a specific example of the latter, the detection target image can be acquired by capturing an image taken by a camera capturing an image of a site where foreign objects are being removed from food waste at a food waste sorting site in a biogas plant. The detection target image acquisition unit 11 may store the acquired detection target image in the memory 102 or the storage device 104, for example.

[0029] Furthermore, the detection target image acquisition unit 11 may include, for example, a detection target imaging unit and a difference target image extraction unit. In this case, the detection target imaging unit captures an image of the detection target, for example, at predetermined time intervals. The difference target image extraction unit extracts the difference between, for example, a first target image of the detection target captured the nth time and a second target image of the detection target captured the n-1th time, thereby generating a difference target image. The second target image of the detection target captured the n-1th time may be stored, for example, in the storage device 104 or in an external storage device outside the device. In the latter case, for example, the difference target image extraction unit acquires the second target image from the external storage device and generates the difference target image. The extraction of the difference can be performed, for example, by a difference extraction method in a known image analysis technique. The difference target image extraction unit acquires the first target image as the detection target image, for example, when the area of ​​the difference target image is equal to or greater than a threshold. The threshold is not particularly limited and may be, for example, the size of the difference target image or the ratio of the area of ​​the difference target image to the area of ​​the first target image. For example, if the area is equal to or greater than the threshold, the difference target image extraction unit may store the first target image as the detection target image in storage device 104 or memory 102, and if the area is less than the threshold, discard the first target image without storing it. According to this aspect, for example, it is possible to acquire only images with large differences, i.e., images in which the detection target has changed, thereby generating more diverse training data and, for example, saving storage capacity of the device.

[0030] Next, the foreign substance image acquisition unit 12 acquires a foreign substance image by capturing an image of the foreign substance (S2, foreign substance image acquisition step). The foreign substance image is, for example, an image of a foreign substance removed from the detection target, and may be an image of one type of foreign substance or an image including multiple types of foreign substances. The foreign substance image may be, for example, an image of a foreign substance actually removed from the detection target, or may include images of foreign substances expected to be included in the detection target. However, an image of a foreign substance actually removed from the detection target is preferable. The foreign substance image may be, for example, a still image, a video, or a still image extracted from a video. The foreign substance image acquisition unit 12 may, for example, acquire images continuously or intermittently. In the latter case, the image may be acquired at predetermined time intervals or at any timing. The foreign object image acquisition unit 11 may acquire the foreign object image by, for example, capturing an image of the foreign object using the imaging device that is the input device 106, or may acquire the foreign object image from an external imaging device via the communication network using the communication device 108. As a specific example of the latter, the foreign object image can be acquired by capturing an image captured by a camera that captures images of foreign objects removed from food waste at a food waste sorting area in a biogas plant. The foreign object image acquisition unit 12 may store the acquired foreign object image in the memory 102 or the storage device 104, for example.

[0031] The foreign substance image acquisition unit may include, for example, a foreign substance imaging unit, a differential foreign substance image extraction unit, and a foreign substance determination unit. In this case, for example, the foreign substance imaging unit images the foreign substance at predetermined time intervals. The differential foreign substance image extraction unit generates a differential foreign substance image by extracting the difference between a first foreign substance image captured the nth time and a second foreign substance image captured the n-1th time. The difference may be extracted, for example, by a difference extraction method in a known image analysis technique. For example, if the area of ​​the differential foreign substance image is within a threshold range, the foreign substance determination unit determines the differential foreign substance image to be an image of a foreign substance and acquires the differential foreign substance image as the foreign substance image. The threshold range is not particularly limited and may be, for example, a range of the size of the differential foreign substance image or a range of the ratio of the area of ​​the differential foreign substance image to the area of ​​the first foreign substance image. The threshold range may be set appropriately depending on, for example, the size and type of foreign substances that may be contained in the detection target. For example, if the area is within a threshold range, the foreign object determination unit may store the difference image as the foreign object image in storage device 104 or memory 102, and further store the first foreign object image in storage device 104 or memory 102 as a second foreign object image for subsequent determinations. Alternatively, if the area is outside the threshold range, the foreign object determination unit may discard the first foreign object image and the difference threshold image rather than store them. If the detection target is food waste and the foreign object image is an image of a foreign object removed from the food waste captured at a food waste sorting area, if the area of ​​the difference foreign object image is below a certain level, the image captured by the foreign object image capture unit is likely to not contain a new foreign object. Alternatively, if the area of ​​the difference foreign object image is above a certain level, the image captured by the foreign object image capture unit is likely to contain an object other than a foreign object (e.g., a part of the body of a worker removing the foreign object). According to this aspect, for example, such images can be removed from the acquired foreign object images, thereby generating higher quality training data.

[0032] The camera that captures the detection target and the camera that captures the foreign object may be, for example, the same camera or different cameras.

[0033] Next, the image composition unit 13 combines the foreign substance image with the detection target image to generate a foreign substance composite image (S3, image composition step). Specifically, the image composition unit 13 can generate the foreign substance composite image by, for example, randomly selecting the detection target image and the foreign substance image and combining the selected detection target image with the foreign substance image. The image composition unit 13 can combine, for example, one foreign substance image with one detection target image, or two or more foreign substance images. For example, the position (combining position) at which the foreign substance image is combined in the detection target image is not particularly limited, and the foreign substance image can be combined at any position. The image composition unit 13 can generate multiple foreign substance composite images by changing the combining position for a pair of the detection target image and the foreign substance image. Furthermore, the image composition unit 13 can perform processing such as changing the angle or size of the foreign substance image or flipping it, and combine the processed foreign substance image with the detection target image.

[0034] The composition position can be specified, for example, by dividing the foreign substance composite image into a plurality of pixel elements and then specifying the composition position as a two-dimensional matrix having the same number and arrangement as the pixel elements. FIG. 4 shows a specific example of a foreign substance composite image 131 and its composition position. FIG. 4(A) is a schematic diagram showing the foreign substance composite image 131, showing that a foreign substance image 131b is composed in the lower right corner of a detection target image 131a. As shown in FIG. 4(B), the composition position can be specified by dividing the foreign substance composite image 131 into a plurality of pixel elements 131c and then specifying a two-dimensional matrix having the same number and arrangement as the pixel elements 131c, by recording "0" for matrix elements 131d corresponding to pixel elements 131c where the foreign substance image 131b does not exist and "1" for matrix elements 131d corresponding to pixel elements 131c where the foreign substance image 131b exists. For ease of explanation, in FIG. 4, matrix elements corresponding to pixel elements where no foreign matter is present are shown as "0" and matrix elements corresponding to pixel elements where a foreign matter is present are shown as "1". However, the present invention is not limited to this, and it is sufficient if, for example, the matrix elements corresponding to positions where a foreign matter is present and positions where no foreign matter is present have different numerical values.

[0035] Then, the training data output unit 14 outputs a pair of the superimposed position of the foreign substance image in the foreign substance composite image as training data for the presence of a foreign substance (S4, training data output step). Alternatively, the training data output unit 14 may output, for example, the detection target image as training data for the absence of a foreign substance. The output may be, for example, output (storage) to the memory 102 or storage device 104 of the device 10, or output to an external device via a communication network. The external device may be, for example, an external storage device or a device that uses the training data generated by the device 10, specifically, a trained model production device or a foreign substance detection device of the present invention, which will be described later.

[0036] In the method for generating training data for foreign object detection according to this embodiment, an example has been described in which S1 and S2 are executed, followed by S3 and S4, but the present invention is not limited to this. Specifically, in the present invention, S1 and S2 may be executed in a process upstream of S3, and S1 and S2 may be executed simultaneously or separately, and in the latter case, the order of execution is not particularly limited and is arbitrary.

[0037] According to the foreign object detection training data generation device of this embodiment, by separately acquiring a detection target image of the detection target and a foreign object image of a foreign object, and then combining these, it is possible to generate foreign object detection training data that includes a large number of pairs of foreign object composite images and foreign object composite positions, i.e., the positions of foreign objects in the foreign object composite images. This allows the foreign object detection training data generation device of this embodiment to generate a large number of training data for foreign object detection without manual labeling.

[0038] [Embodiment 2] The second embodiment is another example of the foreign object detection training data generating device of the present invention.

[0039] The foreign object detection teacher data generation device of this embodiment is similar to the foreign object detection teacher data generation device 10 of the first embodiment, except that it includes a change information acquisition unit in addition to the configuration of the foreign object detection teacher data generation device 10 of the first embodiment, and the description thereof can be used. The foreign object detection teacher data generation device 10A of this embodiment includes, for example, a change information acquisition unit, which acquires change information that detects a change in at least one of the detection object and the foreign object, the detection object image acquisition unit acquires the detection object image when it acquires the change information of the detection object, and the foreign object image acquisition unit acquires the foreign object image when it acquires the change information of the foreign object.

[0040] Fig. 5 is a block diagram showing an example of the configuration of a foreign object detection teacher data generation device 10A of this embodiment. As shown in Fig. 5, the foreign object detection teacher data generation device 10A includes a change information acquisition unit 15 in addition to the configuration of the foreign object detection teacher data generation device 10 of embodiment 1. The hardware configuration of the foreign object detection teacher data generation device 10A is the same as that of the foreign object detection teacher data generation device 10 of Fig. 2, except that the CPU 101 includes the configuration of the foreign object detection teacher data generation device 10A of Fig. 5 instead of the configuration of the foreign object detection teacher data generation device 10 of Fig. 1.

[0041] Next, a method for generating teacher data for foreign object detection according to this embodiment will be described with reference to the flowchart of Fig. 6. The method for generating teacher data for foreign object detection according to this embodiment can be implemented, for example, using the device 10A for generating teacher data for foreign object detection according to this embodiment shown in Fig. 5. Note that the method for generating teacher data for foreign object detection according to the present invention is not limited to use with the device 10A for generating teacher data for foreign object detection.

[0042] First, the change information acquisition unit 15 acquires change information, for example, detecting a change in at least one of the detection target and the foreign object (S6, change information acquisition step). The change information is information indicating that there has been a change in the state of the detection target or the foreign object. Specific examples of the change information include, for example, weight change information of the detection target or the foreign object, distance change information of the detection target or the foreign object, volume change information of the detection target or the foreign object, and acceleration information of a container containing the detection target or the foreign object. However, the change information is not particularly limited as long as it indicates that there has been a change in the state of the detection target or the foreign object. The change information can be acquired by, for example, a sensor that continuously measures the state of the detection target or the foreign object, such as a weigh scale, a distance sensor, a volume measuring device, or an acceleration sensor. Hereinafter, a case where weight change information is acquired will be described as a specific example of the change information, but the present invention is not limited thereto. The change information acquisition unit 15 may, for example, measure the weight of the detection object or the foreign object over time, and if there is a change in the measured value, acquire the changed measured value as the change information, or may acquire, as the change information, a signal notifying that there has been a change in weight of at least one of the detection object and the foreign object from a weight scale connected to the device via a wired or wireless connection.As a specific example, in a food waste sorting area of ​​a biogas plant, weight scales are installed in each of the trash bins for disposing of food waste (detection object) and the trash bins for disposing of foreign objects sorted from the food waste, and change information of the detection object and the foreign object can be acquired by acquiring weight information measured by the weight scales via communication from the weight scales.

[0043] When the detection target image acquisition unit 11 acquires change information of the detection target, it acquires the detection target image in the same manner as S1 in the first embodiment. Furthermore, an image captured immediately after a change in the weight of the detection target is likely to include, for example, the cause of the change in the state of the detection target (for example, a part of the body of the person who disposed of the food waste). For this reason, the detection target image acquisition unit 11 may acquire the detection target image, for example, after a predetermined time has elapsed since acquiring the change information.

[0044] When the foreign object image acquisition unit 12 acquires the change information of the foreign object, it acquires the foreign object image in the same manner as in S2 of the first embodiment. Furthermore, an image captured immediately after a change in the weight of the foreign object is likely to include, for example, the cause of the change in the state of the foreign object (for example, a part of the body of the person who discarded the foreign object). For this reason, the foreign object image acquisition unit 12 may acquire the foreign object image, for example, after a predetermined time has elapsed since acquiring the change information.

[0045] Then, S3 and S4 are carried out in the same manner as S3 and S4 in the first embodiment, and the process ends (END).

[0046] In the foreign object detection training data generation device of this embodiment, for example, a change information acquisition unit acquires change information that detects changes in at least one of the detection target and the foreign object, the detection target image acquisition unit acquires the detection target image when it acquires the change information of the detection target, and the foreign object image acquisition unit acquires the foreign object image when it acquires the change information of the foreign object. Therefore, with the foreign object detection training data generation device of this embodiment, for example, it is possible to more efficiently acquire detection target images and foreign object images for generating training data.

[0047] [Embodiment 3] The third embodiment is another example of the foreign object detection training data generating device of the present invention.

[0048] The foreign object detection teacher data generation device of this embodiment is similar to the foreign object detection teacher data generation device 10 of the first embodiment, except that it includes an image processing unit in addition to the configuration of the foreign object detection teacher data generation device 10 of the first embodiment, and the description thereof can be used. The foreign object detection teacher data generation device 10B of this embodiment includes, for example, an image processing unit that processes the detection target image to generate a processed detection target image and a processed foreign object image by processing the foreign object image, and the image composition unit generates a foreign object composite image by combining at least one of the processed detection target image and the processed foreign object image.

[0049] Fig. 7 is a block diagram showing an example of the configuration of a foreign object detection teacher data generation device 10B of this embodiment. As shown in Fig. 7, the foreign object detection teacher data generation device 10B includes an image processing unit 16 in addition to the configuration of the foreign object detection teacher data generation device 10 of embodiment 1. The hardware configuration of the foreign object detection teacher data generation device 10B is the same as that of the foreign object detection teacher data generation device 10 of Fig. 2, except that the CPU 101 includes the configuration of the foreign object detection teacher data generation device 10B of Fig. 7 instead of the configuration of the foreign object detection teacher data generation device 10 of Fig. 1.

[0050] Next, a method for generating teacher data for foreign object detection according to this embodiment will be described with reference to the flowchart of Fig. 8. The method for generating teacher data for foreign object detection according to this embodiment can be implemented, for example, using the device for generating teacher data for foreign object detection 10B according to this embodiment shown in Fig. 7. Note that the method for generating teacher data for foreign object detection according to the present invention is not limited to use with the device for generating teacher data for foreign object detection 10B.

[0051] First, steps S1 and S2 are performed in the same manner as in the first embodiment to obtain an image of the detection object and an image of the foreign substance.

[0052] The image processing unit 16 processes the detection target image to generate a processed detection target image and the foreign substance image to generate a processed foreign substance image. The processing can be, for example, an image data extension method used in creating training data using general image recognition, and specific examples include changing the color, size, tilt, perspective, etc. of the image, horizontal shifting, random shifting, horizontal flipping, vertical flipping, shearing, RGB channel conversion, background cutting, etc. When the detection target is food waste and the foreign matter is a foreign matter removed from the food waste, the image processing unit 16 preferably generates a processed detection target image and a processed foreign matter image by performing color adjustment on the detection target image and the foreign matter image using a filter with a color tone specified for each time period. Also, the image processing unit 16 may generate a processed foreign matter image by performing oblique deformation on the foreign matter image using a shear transformation process.

[0053] Next, the image composition unit 13 generates a foreign substance composite image by combining at least one of the processed detection target image and the processed foreign substance image (S3, image composition step). The composition can be performed in the same manner as S3 in the first embodiment, except that an image including at least one of the processed target image and the processed foreign substance image is used in addition to the detection target image and the foreign substance image.

[0054] Then, S4 is carried out in the same manner as S4 in the first embodiment, and the process ends (END).

[0055] The foreign object detection training data generation device of this embodiment can, for example, use an image processing unit to generate a processed detection target image by processing the detection target image and a processed foreign object image by processing the foreign object image. Therefore, according to the foreign object detection training data generation device of this embodiment, for example, it is possible to generate training data using not only actually acquired images but also images expanded by processing for the detection target image and foreign object image used to generate training data, thereby making it possible to generate training data for foreign object detection more efficiently.

[0056] [Embodiment 4] Embodiment 4 is an example of a trained model production device of the present invention.

[0057] The trained model production device of this embodiment will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of the configuration of the trained model production device 20 of this embodiment. As shown in Fig. 9, the trained model production device 20 includes a teacher data acquisition unit 21 and a trained model generation unit 22. Although not shown, the trained model production device 20 may also include, for example, a storage unit.

[0058] The trained model production device 20 may be, for example, a single device including the above-mentioned units, or a device in which the above-mentioned units can be connected via a communication network. The trained model production device 20 can also be connected to an external device (described later) via the communication network. The communication network is not particularly limited, and any known network can be used, for example, wired or wireless. Examples of communication networks include the Internet, the World Wide Web (WWW), telephone lines, local area networks (LANs), storage area networks (SANs), delay tolerant networking (DTNs), low power wide area networks (LPWAs), and local 5G (L5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. Examples of wireless communication include direct communication between devices (ad hoc communication), infrastructure communication, and indirect communication via an access point. The trained model production device 20 may be incorporated into a server as a system. The trained model production device 20 may be, for example, a personal computer (PC, for example, a desktop or laptop) on which the program of the present invention is installed, a smartphone, a tablet terminal, etc. Furthermore, the trained model production device 20 may be in the form of cloud computing or edge computing, for example, in which at least one of the above-mentioned units is located on a server and the other units are located on a terminal.

[0059] FIG. 10 illustrates a block diagram of the hardware configuration of the trained model manufacturing device 20. As shown in FIG. 10, the trained model manufacturing device 20 includes, for example, a CPU 201, a memory 202, a bus 203, a storage device 204, an input device 206, an output device 207, a communication device 208, and the like. The description of each component of the trained model manufacturing device 20 can be made using the description of each component of the foreign object detection training data generation device 10. Each unit of the trained model manufacturing device 20 is connected via the bus 203 by its respective interface (I / F). In the trained model manufacturing device 20, the CPU 201 functions as a training data acquisition unit 21 and a trained model generation unit 22.

[0060] Next, an example of a method for manufacturing a trained model according to this embodiment will be described with reference to the flowchart in Fig. 11. The method for manufacturing a trained model according to this embodiment is carried out as follows, for example, using the trained model manufacturing device 20 of Figs. 9 and 10. Note that the method for manufacturing a trained model according to this embodiment is not limited to use of the trained model manufacturing device 20 of Figs. 9 and 10.

[0061] First, the teacher data acquisition unit 21 acquires the teacher data for foreign object detection output from the device for generating teacher data for foreign object detection of the present invention as the teacher data for foreign object detection (S21, teacher data acquisition step). The teacher data acquisition unit 21 may acquire the teacher data for foreign object detection from the device for generating teacher data for foreign object detection of the present invention via the communication network, for example, or may acquire the teacher data for foreign object detection from an external storage device in which the data for foreign object detection is stored.

[0062] Next, the trained model generation unit 21 generates, as a trained model, a foreign object detection model that outputs whether or not a foreign object is included in a foreign object detection target when a foreign object detection target image obtained by capturing an image of the foreign object detection target is input, through machine learning using the foreign object detection training data (S22, training step). The machine learning is not particularly limited, and may be, for example, machine learning using a neural network such as a convolutional neural network (CNN), a support vector machine (SVM), a Bayesian network, a regression tree, or the like. The machine learning is preferably, for example, machine learning using a convolutional neural network, particularly semantic segmentation. Furthermore, the trained model generation unit 21 may, for example, generate a re-trained trained model (derived model) using the foreign object detection training data and an already generated trained model. Furthermore, the trained model generation unit 21 may generate a trained model obtained by transfer learning using a trained model generated using the training data for foreign object detection, or may generate the trained model by model compression of the trained model generated using the training data for foreign object detection.

[0063] The trained model generated by this embodiment is used, for example, in a foreign object detection device (described later), which enables foreign object detection within a foreign object detection target using a foreign object detection target image obtained by capturing an image of the foreign object detection target.

[0064] [Embodiment 5] The fifth embodiment is an example of a foreign object detection device of the present invention.

[0065] The foreign object detection device of this embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the configuration of a foreign object detection device 30 of this embodiment. As shown in Fig. 12, the foreign object detection device 30 includes a foreign object detection target image acquisition unit 31 and a foreign object detection unit 32. Although not shown, the foreign object detection device 30 may also include, for example, a storage unit.

[0066] The foreign object detection device 30 may be, for example, a single device including the above-described units, or a device in which the units can be connected via a communication network. The foreign object detection device 30 can also be connected to an external device (described later) via the communication network. The communication network is not particularly limited, and any known network can be used, for example, a wired or wireless network. Examples of communication networks include the Internet, the World Wide Web (WWW), a telephone line, a Local Area Network (LAN), a Storage Area Network (SAN), a Delay Tolerant Networking (DTN), a Low Power Wide Area Network (LPWA), and a Local 5G (L5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), Local 5G, and LPWA. Examples of wireless communication include direct communication between devices (Ad Hoc communication), infrastructure communication, and indirect communication via an access point. The foreign object detection device 30 may be incorporated into a server as a system. Furthermore, the foreign object detection device 30 may be, for example, a personal computer (PC, for example, a desktop or notebook type) on which the program of the present invention is installed, a smartphone, a tablet terminal, etc. Furthermore, the foreign object detection device 30 may be in the form of cloud computing or edge computing, for example, in which at least one of the above-mentioned units is located on a server and the other units are located on a terminal.

[0067] 13 is a block diagram illustrating an example of the hardware configuration of the foreign object detection device 30. As shown in FIG. 13, the foreign object detection device 30 includes, for example, a CPU 301, a memory 302, a bus 303, a storage device 304, an input device 306, an output device 307, and a communication device 308. The description of each component of the foreign object detection device 30 can be made using the description of each component of the foreign object detection training data generation device 10. The components of the foreign object detection device 30 are connected via their respective interfaces (I / F) via the bus 303. In the foreign object detection device 30, the CPU 301 functions as a foreign object detection target image acquisition unit 31 and a foreign object detection unit 32.

[0068] Next, an example of a method for manufacturing a trained model of this embodiment will be described with reference to the flowchart in Fig. 14. The method for manufacturing a trained model of this embodiment is carried out as follows, for example, using the foreign object detection device 30 of Figs. 12 and 13. Note that the method for manufacturing a trained model of this embodiment is not limited to using the foreign object detection device 30 of Figs. 12 and 13.

[0069] First, the foreign object detection target image acquisition unit 31 acquires a foreign object detection target image by capturing an image of the foreign object detection target (S31, foreign object detection target image acquisition step). The foreign object detection target may, for example, contain a foreign object, may not contain a foreign object, or it may be unclear whether the foreign object is present. The foreign object detection target image may, for example, be a still image, a video, or a still image extracted from a video. The foreign object detection target image acquisition unit 31 may, for example, acquire images continuously or intermittently. In the latter case, the images may be acquired every predetermined time period or at any timing. The foreign object detection target image acquisition unit 31 may, for example, acquire the foreign object detection target image by capturing an image of the foreign object detection target using the imaging device that is the input device 306. Alternatively, the foreign object detection target image may be acquired from an external imaging device via the communication network using the communication device 308. As a specific example of the latter, for example, the foreign object detection target image can be acquired by capturing an image of food waste captured by a camera at a food waste disposal site in a biogas plant. The foreign object detection target image acquisition unit 31 may store the acquired foreign object detection target image in the memory 302 or the storage device 304, for example.

[0070] The foreign object detection unit 32 inputs the foreign object detection target image into a foreign object detection model and detects whether the foreign object detection target contains a foreign object (S32, foreign object detection step). The foreign object detection model is a trained model generated, for example, by machine learning using training data generated by the foreign object detection training data generation device of the present invention, so as to output a foreign object detection result, i.e., a foreign object detection result, when the foreign object detection target image is input. Note that the foreign object detection model may also be a trained model generated, for example, by the trained model generation device of the fourth embodiment.

[0071] The foreign object detection model may include, for example, an input layer that receives an image of a foreign object to be detected, an output layer that outputs the foreign object detection result, and at least one intermediate layer disposed between the input layer and the output layer. The foreign object detection model may be a program module that is part of artificial intelligence software. An example of the multilayer network is a neural network. An example of the neural network is a convolution neural network (CNN), but it is not limited to CNN and may be a trained model constructed using other learning algorithms such as a neural network other than CNN, a support vector machine (SVM), a Bayesian network, or a regression tree.

[0072] The foreign object detection model can be generated, for example, by machine learning using the training data generated by the foreign object detection training data generation device of the present invention. The foreign object detection model may be, for example, a trained model generated in advance. The trained model may also be a trained model (derived model) retrained using the foreign object detection training data and an already generated trained model. Furthermore, the trained model may be a trained model obtained by transfer learning using a trained model generated using foreign object detection training data, or a trained model generated by model compression of a trained model generated using foreign object detection training data.

[0073] Foreign object detection device 30 may include, for example, an output unit. In this case, the output unit may, for example, output the foreign object detection result. The output unit may, for example, output the foreign object detection result to a terminal outside the device via the communication network, or may output the foreign object detection result to output device 307. Furthermore, the output foreign object detection result may be stored in, for example, memory 302 or storage device 304.

[0074] In the method for generating training data for foreign object detection according to this embodiment, the case where steps S31 and S32 are executed sequentially has been described as an example, but the present invention is not limited to this. Specifically, in the present invention, steps S31 and S32 may be executed simultaneously or separately, and in the latter case, the order of execution is not particularly limited and is arbitrary.

[0075] The foreign object detection device of this embodiment makes it possible to detect foreign objects using a foreign object detection model generated by machine learning, for example.

[0076] [Embodiment 6] The first program of this embodiment is a program for causing a computer to execute each step of the above-mentioned foreign object detection training data generation method. Specifically, the first program of this embodiment is a program for causing a computer to execute a detection object image acquisition procedure, a foreign object image acquisition procedure, an image synthesis procedure, and a training data output procedure.

[0077] the detection target image acquisition step includes acquiring a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed; the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance; the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image; The teacher data output procedure includes: A set of the foreign substance composite image and the composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance.

[0078] The first program of this embodiment can also be said to be a program that causes a computer to function as a detection object image acquisition procedure, a foreign substance image acquisition procedure, an image synthesis procedure, and a training data output procedure.

[0079] The first program of this embodiment can be implemented by invoking the descriptions of the foreign object detection training data generation device and foreign object detection training data generation method of the present invention. For example, the "procedure" in each of the steps can be read as "processing." The program of this embodiment may also be recorded on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, and floppy disk (FD).

[0080] [Embodiment 7] The second program of this embodiment is a program for causing a computer to execute each step of the trained model production method described above. Specifically, the second program of this embodiment is a program for causing a computer to execute a training data acquisition procedure and a trained model generation procedure.

[0081] the teacher data acquisition step acquires teacher data for foreign object detection output from the first program as teacher data for foreign object detection; The trained model generation procedure uses machine learning using the foreign object detection training data to generate a foreign object detection model as a trained model that, when an image of a foreign object detection target captured as an image of the foreign object detection target is input, outputs whether the foreign object detection target contains a foreign object.

[0082] The second program of this embodiment can also be said to be a program that causes a computer to function as a training data acquisition procedure and a trained model generation procedure.

[0083] The second program of this embodiment can cite the descriptions of the trained model production device and trained model production method of the present invention. For each of the steps, for example, "step" can be read as "processing." The program of this embodiment may also be recorded on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, and floppy disk (FD).

[0084] [Embodiment 8] The third program of this embodiment is a program for causing a computer to execute each step of the foreign object detection method described above. Specifically, the third program of this embodiment is a program for causing a computer to execute a foreign object detection target image acquisition procedure and a foreign object detection procedure.

[0085] the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target; the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model, and detecting whether or not the foreign object detection target includes a foreign object; The foreign object detection model is a trained model generated by machine learning using training data generated by the first program so as to output whether or not a foreign object is contained in a foreign object detection target when an image of the foreign object detection target is input, or is a trained model produced by the second program.

[0086] The second program of this embodiment can also be said to be a program that causes a computer to function as a foreign object detection target image acquisition procedure and a foreign object detection procedure.

[0087] The third program of this embodiment can be implemented by incorporating the descriptions of the foreign object detection device and foreign object detection method of the present invention. For each of the steps, for example, "step" can be replaced with "processing." The program of this embodiment may be recorded on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, and floppy disk (FD).

[0088] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0089] <Additional Notes> Some or all of the above embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) The apparatus includes a detection object image acquisition unit, a foreign substance image acquisition unit, an image synthesis unit, and a training data output unit, the detection target image acquisition unit acquires a detection target image obtained by capturing an image of the foreign object detection target after the foreign object has been removed, the foreign substance image acquisition unit acquires a foreign substance image by capturing an image of the foreign substance; the image composition unit combines the foreign substance image with the detection target image to generate a foreign substance composite image; The teacher data output unit a pair of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance; A device for generating training data for foreign object detection. (Appendix 2) 2. The foreign object detection training data generating device according to claim 1, wherein the training data output unit outputs the detection target image as training data without a foreign object. (Appendix 3) the detection target image acquisition unit includes a detection target imaging unit and a difference target image extraction unit; the detection target imaging unit images the detection target at predetermined time intervals; The foreign object detection training data generation device described in Appendix 1 or 2, wherein the differential target image extraction unit extracts the difference between a first target image of the detection target captured the nth time and a second target image of the detection target captured the n-1th time, generates a differential target image, and acquires the first target image as the detection target image if the area of ​​the differential target image is greater than or equal to a threshold. (Appendix 4) the foreign substance image acquisition unit includes a foreign substance imaging unit, a differential foreign substance image extraction unit, and a foreign substance determination unit; the foreign object image capturing unit captures an image of the foreign object at predetermined time intervals; the differential impurity image extracting unit extracts a difference between a first impurity image of the impurity captured the nth time and a second impurity image of the impurity captured the (n-1)th time to generate a differential impurity image; The foreign substance detection training data generation device according to any one of appendices 1 to 3, wherein the foreign substance determination unit determines that the differential foreign substance image is an image of a foreign substance when the area of ​​the differential foreign substance image is within a threshold range, and acquires the differential foreign substance image as the foreign substance image. (Appendix 5) A change information acquisition unit is included, the change information acquisition unit acquires change information obtained by detecting a change in at least one of the detection target and the foreign matter; the detection target image acquisition unit acquires the detection target image when change information of the detection target is acquired; 5. The foreign substance detection training data generation device according to claim 1, wherein the foreign substance image acquisition unit acquires the foreign substance image when it acquires change information of the foreign substance. (Appendix 6) Includes an image processing section, the image processing unit processes the detection target image to generate a processed detection target image and a processed foreign substance image to generate a processed foreign substance image, The foreign matter detection training data generation device according to any one of appendices 1 to 5, wherein the image synthesis unit generates a foreign matter composite image by synthesizing at least one of the processed detection target image and the processed foreign matter image. (Appendix 7) The system includes a training data acquisition unit and a trained model generation unit, the teacher data acquisition unit acquires, as the teacher data for foreign object detection, teacher data for foreign object detection output by a teacher data generation device for foreign object detection described in any one of Supplementary Notes 1 to 6; The trained model generation unit generates, as a trained model, a foreign object detection model that outputs whether or not a foreign object is contained in a foreign object detection target when an image of the foreign object detection target is input, through machine learning using the foreign object detection training data. (Appendix 8) a foreign object detection target image acquisition unit and a foreign object detection unit, the foreign object detection target image acquisition unit acquires a foreign object detection target image by capturing an image of the foreign object detection target, the foreign object detection unit inputs the foreign object detection target image into a foreign object detection model and detects whether or not the foreign object detection target includes a foreign object; A foreign object detection device, wherein the foreign object detection model is a trained model generated by machine learning using training data generated by a foreign object detection training data generation device described in any one of Supplementary Notes 1 to 6, so as to output whether or not a foreign object is contained in a foreign object detection target when an image of the foreign object detection target is input, or a trained model manufactured by a trained model manufacturing device described in Supplementary Note 7. (Appendix 9) The method includes a detection object image acquisition step, a foreign substance image acquisition step, an image synthesis step, and a training data output step, the detection target image acquisition step acquires a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed; the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance; the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image; The teacher data output step includes: a pair of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance; A method for generating training data for foreign object detection. (Appendix 10) 10. The method for generating training data for foreign substance detection according to claim 9, wherein the training data output step outputs the detection target image as training data without any foreign substance. (Appendix 11) the detection target image acquisition step includes a detection target imaging step and a difference target image extraction step; the detection target imaging step images the detection target at predetermined time intervals; The method for generating training data for foreign object detection described in Appendix 9 or 10, wherein the differential object image extraction process extracts the difference between a first object image of the detection object captured the nth time and a second object image of the detection object captured the n-1th time, generates a differential object image, and acquires the first object image as the detection object image if the area of ​​the differential object image is equal to or greater than a threshold. (Appendix 12) the foreign substance image acquisition step includes a foreign substance imaging step, a differential foreign substance image extraction step, and a foreign substance determination step; the foreign matter imaging step images the foreign matter at predetermined time intervals; the differential impurity image extraction step extracts a difference between a first impurity image of the impurity captured the nth time and a second impurity image of the impurity captured the (n-1)th time to generate a differential impurity image; 12. The method for generating training data for foreign substance detection according to any one of appendices 9 to 11, wherein the foreign substance determination step determines that the differential foreign substance image is an image of a foreign substance if the area of ​​the differential foreign substance image is within a threshold range, and acquires the differential foreign substance image as the foreign substance image. (Appendix 13) A change information acquisition step is included, the change information acquisition step acquires change information obtained by detecting a change in at least one of the detection object and the foreign substance; the detection object image acquisition step acquires the detection object image when change information of the detection object is acquired; 13. The method for generating training data for foreign substance detection according to any one of appendices 9 to 12, wherein the foreign substance image acquisition step acquires the foreign substance image when change information of the foreign substance is acquired. (Appendix 14) Including image processing process, the image processing step processes the detection target image and the foreign substance image to generate a processed detection target image and a processed foreign substance image, The foreign matter detection training data generation method according to any one of appendices 9 to 13, wherein the image synthesis step generates a foreign matter composite image by synthesizing at least one of the processed detection object image and the processed foreign matter image. (Appendix 15) The method includes a training data acquisition step and a trained model generation step. the teacher data acquisition step acquires, as the teacher data for foreign object detection, teacher data for foreign object detection output by a teacher data generation method for foreign object detection described in any one of Supplementary Notes 9 to 14, The trained model generation process is a trained model manufacturing method in which, through machine learning using the foreign object detection training data, a foreign object detection model is generated as a trained model that, when an image of a foreign object detection target captured as a foreign object detection target is input, outputs whether the foreign object detection target contains a foreign object. (Appendix 16) The method includes a foreign object detection image acquisition step and a foreign object detection step, the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target, the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model and detecting whether or not a foreign object is included in the foreign object detection target; A foreign object detection method, wherein the foreign object detection model is a trained model generated by machine learning using training data generated by a foreign object detection training data generation method described in any one of Supplementary Notes 9 to 14, so as to output whether or not a foreign object is contained in a foreign object detection target when an image of the foreign object detection target is input, or is a trained model produced by a trained model production method described in Supplementary Note 15. (Appendix 17) The method includes a detection target image acquisition procedure, a foreign substance image acquisition procedure, an image synthesis procedure, and a training data output procedure. the detection target image acquisition step includes acquiring a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed; the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance; the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image; The teacher data output procedure includes: outputting a set of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image as training data indicating the presence of a foreign substance; A program for causing a computer to execute each of the above procedures. (Appendix 18) The program according to claim 17, wherein the teacher data output step outputs the detection target image as teacher data without foreign matter. (Appendix 19) the detection target image acquisition step includes a detection target image capture step and a difference target image extraction step; the step of capturing an image of the detection target includes capturing an image of the detection target every predetermined time period; The program described in Appendix 17 or 18, wherein the differential target image extraction procedure extracts the difference between a first target image of the detection target captured the nth time and a second target image of the detection target captured the (n-1)th time, generates a differential target image, and acquires the first target image as the detection target image if the area of ​​the differential target image is equal to or greater than a threshold. (Appendix 20) the foreign substance image acquisition step includes a foreign substance imaging step, a differential foreign substance image extraction step, and a foreign substance determination step; the foreign object imaging step images the foreign object at predetermined time intervals; the step of extracting a differential impurity image includes extracting a difference between a first impurity image of the impurity captured the nth time and a second impurity image of the impurity captured the (n-1)th time to generate a differential impurity image; 20. The program according to any one of appendices 17 to 19, wherein the foreign substance determination step determines that the differential foreign substance image is an image of a foreign substance if the area of ​​the differential foreign substance image is within a threshold range, and acquires the differential foreign substance image as the foreign substance image. (Appendix 21) A procedure for acquiring change information is included. the change information acquisition step acquires change information obtained by detecting a change in at least one of the detection object and the foreign substance; the detection target image acquisition step includes acquiring the detection target image when change information of the detection target object is acquired; 21. The program according to any one of appendices 17 to 20, wherein the foreign substance image acquisition step acquires the foreign substance image when change information of the foreign substance is acquired. (Appendix 22) Includes image processing procedures, the image processing step includes processing the detection target image to generate a processed detection target image and processing the foreign substance image to generate a processed foreign substance image, 22. The program according to claim 17, wherein the image synthesis step generates a foreign matter composite image by synthesizing at least one of the processed foreign matter detection image and the processed foreign matter image. (Appendix 23) This includes a procedure for acquiring training data and a procedure for generating a trained model. The teacher data acquisition step acquires, as the teacher data for foreign object detection, teacher data for foreign object detection output by a program described in any one of Supplementary Notes 17 to 22; the trained model generation step generates, as a trained model, a foreign object detection model that outputs whether or not a foreign object is included in a foreign object detection target when an image of the foreign object detection target obtained by capturing the foreign object detection target is input, by machine learning using the foreign object detection training data; A program for causing a computer to execute each of the above procedures. (Appendix 24) The method includes a foreign object detection image acquisition step and a foreign object detection step, the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target; the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model, and detecting whether or not the foreign object detection target includes a foreign object; the foreign object detection model is a trained model generated by machine learning using training data generated by a program described in any one of Supplementary Notes 17 to 22, so as to output whether or not a foreign object is included in a foreign object detection target when an image of the foreign object detection target obtained by capturing the foreign object detection target is input, or is a trained model produced by a program described in Supplementary Note 23; (Appendix 25) A computer-readable recording medium having recorded thereon a program according to any one of appendices 17 to 24. [Industrial Applicability]

[0090] According to the present invention, training data for foreign object detection can be generated without the need for labeling work, and therefore the present invention is widely useful in fields related to foreign object detection. [Explanation of symbols]

[0091] 10, 10A, 10B Foreign object detection training data generation device 11. Detection target image acquisition unit 12 Foreign object image acquisition unit 13 Image synthesis unit 14 Teacher data output section 15 Change information acquisition unit 16 Image Processing Department 101 CPU 102 memory 103 Bus 104 Storage device 105 Programs 106 Input Device 107 Output Device 108 Communication Devices 20 Trained model production device 21 Teacher data acquisition unit 22 Trained model generation unit 201 CPU 202 memory 203 Bus 204 Storage device 205 Programs 206 Input Device 207 Output Device 208 Communication Devices 30 Foreign object detection device 31 Foreign object detection target image acquisition unit 32 Foreign object detection unit 301 CPU 302 memory 303 Bus 304 Storage device 305 Program 306 Input Device 307 Output Device 308 Communication Devices

Claims

1. A detection object imaging unit, a foreign object imaging unit, a change information acquisition unit, a detection object image acquisition unit, a foreign object image acquisition unit, an image synthesis unit, and a teacher data output unit, the detection target imaging unit intermittently images the detection target after the foreign matter has been removed, the foreign object image capturing unit intermittently captures images of the foreign object; the change information acquisition unit acquires change information obtained by detecting a change in at least one of the detection target and the foreign matter; the detection target image acquisition unit acquires a detection target image when change information of the detection target is acquired; the foreign substance image acquisition unit acquires a foreign substance image when acquiring information about a change in the foreign substance; the image composition unit combines the foreign substance image with the detection target image to generate a foreign substance composite image; The teacher data output unit a pair of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance; A device for generating training data for foreign object detection.

2. 2. The foreign substance detection training data generating device according to claim 1, wherein the training data output unit outputs the detection target image as training data without a foreign substance.

3. the detection target image acquisition unit includes a difference target image extraction unit, the detection target imaging unit images the detection target at predetermined time intervals; The foreign object detection training data generation device of claim 1 or 2, wherein the difference target image extraction unit extracts the difference between a first target image of the detection target captured the nth time and a second target image of the detection target captured the (n-1)th time, generates a difference target image, and acquires the first target image as the detection target image if the area of ​​the difference target image is equal to or greater than a threshold.

4. the foreign substance image acquisition unit includes a differential foreign substance image extraction unit and a foreign substance determination unit, the foreign object image capturing unit captures an image of the foreign object at predetermined time intervals; the differential foreign substance image extracting unit extracts a difference between a first foreign substance image of the foreign substance captured the nth time and a second foreign substance image of the foreign substance captured the (n-1)th time to generate a differential foreign substance image; 4. The foreign substance detection training data generation device according to claim 1, wherein the foreign substance determination unit determines that the differential foreign substance image is an image of a foreign substance when an area of ​​the differential foreign substance image is within a threshold range, and acquires the differential foreign substance image as the foreign substance image.

5. Includes an image processing section, the image processing unit processes the detection target image to generate a processed detection target image and a processed foreign substance image to generate a processed foreign substance image, The foreign matter detection training data generating device according to claim 1 , wherein the image combining unit generates a foreign matter composite image by combining at least one of the processed detection object image and the processed foreign matter image.

6. A method for generating training data for foreign object detection, in which each step including a detection object imaging step, a foreign object imaging step, a change information acquisition step, a detection object image acquisition step, a foreign object image acquisition step, an image synthesis step, and a training data output step is executed by a computer, the detection target imaging step intermittently images the detection target after the foreign matter has been removed, the foreign matter imaging step intermittently images the foreign matter, the change information acquisition step acquires change information obtained by detecting a change in at least one of the detection target and the foreign matter; the detection object image acquisition step acquires a detection object image when change information of the detection object is acquired; the foreign substance image acquisition step acquires a foreign substance image when the change information of the foreign substance is acquired, The image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesis image, and the training data output step a pair of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance; A method for generating training data for foreign object detection.

7. A method for detecting a target object, a foreign object, a change information acquisition procedure, a detection target image acquisition procedure, a foreign object image acquisition procedure, an image synthesis procedure, and a training data output procedure, the step of capturing an image of the detection target includes intermittently capturing an image of the detection target after the foreign object has been removed; The foreign object imaging step includes intermittently imaging the foreign object, the change information acquisition step acquires change information obtained by detecting a change in at least one of the detection target and the foreign object; the detection target image acquisition step includes acquiring a detection target image when change information of the detection target is acquired; the foreign substance image acquisition step includes acquiring a foreign substance image when the change information of the foreign substance is acquired; The image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image, and the training data output step outputting a set of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image as training data indicating the presence of a foreign substance; A program for causing a computer to execute each of the above procedures.

Images