Model generation method, foreign object detection method, foreign object detection system, and foreign object detection program

Abstract

To detect a foreign matter accurately.SOLUTION: A model generation method for generating an inference model to be used for detecting a foreign matter included in an image of an object includes: a training acquisition step (S01) which acquires, as training images, a first normal image including a training object, a second normal image including the training object, a second foreign matter image formed by adding a foreign matter assumed as a detection target to the second normal image, a third normal image including the training object, and a third foreign matter image formed by adding a foreign matter not assumed as a detection target to the third normal image; and a model generation step (S02) which generates an inference model by performing training using the acquired training images.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a model generation method, a model generation system, and a model generation program for generating an inference model used to detect foreign objects included in an image showing an object, a foreign object detection method, a foreign object detection system, and a foreign object detection program for detecting foreign objects included in an image showing an object using the generated inference model, and the generated inference model. [Background technology]

[0002] Patent Document 1 shows that an inference model generated by machine learning is used to generate a restored image from an inspection target image of the appearance of the inspection target, and the inspection target is inspected based on the difference between the inspection target image and the restored image. In Patent Document 1, the inference model used for the inspection is generated from a good product image of the appearance of the inspection target determined to be a good product, and a pseudo-defect image obtained by combining an image showing a defect with the good product image. This is said to improve the inspection accuracy. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-205163 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method disclosed in Patent Document 1 does not necessarily perform appropriate detection of foreign objects in, for example, an image of a packaged object captured by X-ray. In particular, when the target object contains multiple objects such as pasta, and the foreign objects are smaller than the individual objects, the inference model generated by the method disclosed in Patent Document 1 does not necessarily perform appropriate detection of foreign objects.

[0005] One embodiment of the present invention has been made in consideration of the above, and aims to provide a model generation method, a model generation system, a model generation program, a foreign object detection method, a foreign object detection system, a foreign object detection program, and an inference model that are capable of detecting foreign objects with high accuracy. [Means for solving the problem]

[0006] In order to achieve the above object, a model generation method according to one embodiment of the present invention is a model generation method for generating an inference model used to detect a foreign object included in an image showing an object, the model generation method including a training acquisition step of acquiring, as training images, a first normal image showing a training object, a second normal image showing the training object, a second foreign object image in which a foreign object assumed to be a detection target is added to the second normal image, a third normal image showing the training object, and a third foreign object image in which a foreign object not assumed to be a detection target is added to the third normal image, and a model generation step of performing training using the training images acquired in the training acquisition step to generate an inference model, the training including training in which information based on the first normal image is used as an input to the inference model and information based on the first normal image is used as an output from the inference model, training in which information based on the second foreign object image is used as an input to the inference model and information based on the second normal image is used as an output from the inference model, and training in which information based on the third foreign object image is used as an input to the inference model and information based on the third normal image is used as an output from the inference model.

[0007] In a model generation method according to an embodiment of the present invention, in addition to a normal image, two different types of foreign object images are used for training to generate an inference model. The inference model generated in this manner reflects the target object, among those captured in the input image, in the output, but does not reflect the foreign object in the output. Therefore, by detecting a foreign object using the generated inference model, it is possible to detect the foreign object with high accuracy.

[0008] The method further includes the step of generating an inference model based on the detected foreign object, the step of generating an inference model based on the detected foreign object, and the step of generating an inference model based on the detected foreign object.

[0009] The second normal image and the third normal image may be the first normal image, and in the training acquisition step, a foreign object expected to be detected may be added to the first normal image to generate and acquire the second foreign object image, and a foreign object not expected to be detected may be added to the first normal image to generate and acquire the third foreign object image. With this configuration, the second foreign object image and the third foreign object image can be easily and reliably acquired. As a result, an inference model can be easily and reliably generated.

[0010] The object may be a specific type of object, and the first normal image, the second normal image, the second foreign object image, the third normal image, and the third foreign object image acquired in the training acquisition step may be images in which the specific type of object is captured as the training object. With this configuration, an inference model for detecting foreign objects with high accuracy can be generated for the specific type of object.

[0011] The ratio of the number of the first normal images, the number of combinations of the second normal images and the second foreign object images, and the number of combinations of the third normal images and the third foreign object images acquired in the training acquisition step may be a preset ratio. With this configuration, it is possible to generate an inference model more appropriately and reliably.

[0012] The method may further include generating an inference model based on the calculated image of the foreign object that is not expected to be detected.

[0013] The third foreign object image may be an image in which a foreign object not expected to be detected is added to the third normal image by at least one of transparent addition and replacement addition. With this configuration, it is possible to generate an inference model more appropriately and reliably.

[0014] The inference model may include a neural network having multiple layers, have a structure for connecting layers, and add an image after the connections to an input image. With this configuration, an inference model for detecting foreign objects with high accuracy can be generated.

[0015] In the model generation step, a new second inference model may be generated by performing new training, in which a part for outputting information indicating the degree of a foreign object for each position of the image input to the inference model is added to the output side of the generated inference model. With this configuration, a second inference model for easily and reliably detecting foreign objects can be generated.

[0016] One embodiment of the present invention can be described not only as a model generation method invention as described above, but also as a model generation system and a model generation program invention as described below. These are essentially the same invention, just in different categories, and have similar functions and effects.

[0017] That is, a model generation system according to one embodiment of the present invention is a model generation system that generates an inference model used to detect a foreign object included in an image showing an object, and includes an acquisition means that acquires as training images a first normal image showing a training object, a second normal image showing the training object, a second foreign object image in which a foreign object assumed to be a detection target is added to the second normal image, a third normal image showing the training object, and a third foreign object image in which a foreign object not assumed to be a detection target is added to the third normal image, and a model generation means that performs training using the training images acquired by the acquisition means to generate an inference model, and the training includes training in which information based on the first normal image is used as an input to the inference model and information based on the first normal image is used as an output from the inference model, training in which information based on the second foreign object image is used as an input to the inference model and information based on the second normal image is used as an output from the inference model, and training in which information based on the third foreign object image is used as an input to the inference model and information based on the third normal image is used as an output from the inference model.

[0018] A model generation program according to one embodiment of the present invention is a model generation program that causes a computer to operate as a model generation system that generates an inference model used to detect a foreign object included in an image showing an object, and causes the computer to function as an acquisition means that acquires, as training images, a first normal image showing a training object, a second normal image showing the training object, a second foreign object image in which a foreign object assumed to be a detection target has been added to the second normal image, a third normal image showing the training object, and a third foreign object image in which a foreign object not assumed to be a detection target has been added to the third normal image, and a model generation means that generates an inference model by training using the training images acquired by the acquisition means, and the training includes training in which information based on the first normal image is used as input to the inference model and information based on the first normal image is used as output from the inference model, training in which information based on the second foreign object image is used as input to the inference model and information based on the second normal image is used as output from the inference model, and training in which information based on the third foreign object image is used as input to the inference model and information based on the third normal image is used as output from the inference model.

[0019] In addition, in order to achieve the above-mentioned object, a foreign object detection method according to one embodiment of the present invention is a foreign object detection method that uses an inference model generated by a model generation method to detect foreign objects contained in an image depicting an object, and includes a detection acquisition step of acquiring a target image that is to be detected for foreign objects, a calculation step of inputting information based on the target image acquired in the detection acquisition step into the inference model and performing a calculation to obtain an output from the inference model, and a detection step of calculating the difference between the information related to the input to the inference model in the calculation step and the information related to the output from the inference model, and detecting a foreign object contained in the target image from the calculated difference.

[0020] In the foreign object detection method according to one embodiment of the present invention, the above-mentioned inference model is used to detect foreign objects, and therefore, according to the foreign object detection method according to one embodiment of the present invention, foreign objects can be detected with high accuracy.

[0021] In addition, in order to achieve the above-mentioned object, a foreign object detection method according to one embodiment of the present invention is a foreign object detection method that uses a second inference model generated by a model generation method to detect foreign objects contained in an image depicting an object, and includes a detection acquisition step of acquiring a target image that is to be subject to foreign object detection, a calculation step of inputting information based on the target image acquired in the detection acquisition step into the second inference model and performing a calculation to obtain an output from the second inference model, and a detection step of detecting foreign objects contained in the target image from the output from the second inference model obtained in the calculation step.

[0022] In the foreign object detection method according to one embodiment of the present invention, the second inference model is used to detect foreign objects. Therefore, according to the foreign object detection method according to one embodiment of the present invention, foreign objects can be detected with high accuracy.

[0023] One embodiment of the present invention can be described not only as a foreign object detection method invention as described above, but also as a foreign object detection system and a foreign object detection program invention as described below. These are essentially the same invention, just in different categories, and have similar functions and effects.

[0024] That is, a foreign object detection system according to one embodiment of the present invention is a foreign object detection system that uses an inference model generated by a model generation method to detect foreign objects contained in an image depicting an object, and includes a detection acquisition means that acquires a target image that is to be detected for foreign objects, a calculation means that inputs information based on the target image acquired by the detection acquisition means into the inference model and performs a calculation to obtain an output from the inference model, and a detection means that calculates the difference between the information input to the inference model by the calculation means and the information output from the inference model, and detects foreign objects contained in the target image from the calculated difference.

[0025] A foreign object detection system according to one embodiment of the present invention is a foreign object detection system that detects foreign objects contained in an image depicting an object using a second inference model generated by a model generation method, and includes a detection acquisition means that acquires a target image that is to be detected for foreign objects, a calculation means that inputs information based on the target image acquired by the detection acquisition means into the second inference model to perform a calculation and obtain an output from the second inference model, and a detection means that detects foreign objects contained in the target image from the output from the second inference model obtained by the calculation means.

[0026] A foreign object detection program according to one embodiment of the present invention is a foreign object detection program that causes a computer to operate as a foreign object detection system that uses an inference model generated by a model generation method to detect foreign objects contained in an image depicting an object, and causes the computer to function as a detection acquisition means that acquires a target image from which foreign objects are to be detected, a calculation means that inputs information based on the target image acquired by the detection acquisition means into the inference model, performs calculations, and obtains output from the inference model, and a detection means that calculates the difference between the information related to the input to the inference model by the calculation means and the information related to the output from the inference model, and detects foreign objects contained in the target image from the calculated difference.

[0027] A foreign object detection program according to one embodiment of the present invention is a foreign object detection program that causes a computer to operate as a foreign object detection system that uses an inference model generated by a model generation method to detect foreign objects contained in an image depicting an object, and causes the computer to function as a detection acquisition means that acquires a target image from which foreign objects are to be detected, a calculation means that inputs information based on the target image acquired by the detection acquisition means into a second inference model to perform a calculation and obtain an output from the second inference model, and a detection means that detects foreign objects contained in the target image from the output from the second inference model obtained by the calculation means.

[0028] The inference model generated by the model generation method according to one embodiment of the present invention is itself an invention having a novel configuration. That is, the inference model according to one embodiment of the present invention is an inference model for inputting image-based information, performing calculations according to the input, and functioning a computer to output information, and is generated by the model generation method. Effect of the Invention

[0029] According to one embodiment of the present invention, foreign matter can be detected with high accuracy. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a model generation system and a foreign object detection system according to an embodiment of the present invention. [Diagram 2] A figure showing an example of a target image used to detect foreign objects and an example of an image output when the target image is input into an inference model. [Diagram 3] FIG. 13 is a diagram showing an example of an abnormality degree map generated as a result of detection of a foreign object; [Figure 4] FIG. 1 is a diagram illustrating an inference model. [Diagram 5] FIG. 2 is a diagram showing an example of a first normal image used for training. [Figure 6] FIG. 13 is a diagram showing an example of a combination of a second normal image and a second foreign object image used for training. [Figure 7] FIG. 13 is a diagram showing an example of a combination of a third normal image and a third foreign object image used for training. [Figure 8] 13A and 13B are diagrams illustrating examples of images used to generate a third foreign object image. [Figure 9] 4 is a flowchart showing a model generation method which is a process executed in the model generation system according to the embodiment of the present invention. [Figure 10] 4 is a flowchart showing a foreign object detection method which is a process executed in the foreign object detection system according to the embodiment of the present invention. [Figure 11]11A and 11B are diagrams showing examples of detection results according to a comparative example and an embodiment of the present invention. [Figure 12] A figure showing examples of images used for training, and images and anomaly maps output from the inference model. [Figure 13] A figure showing examples of images used for training, and images and anomaly maps output from the inference model. [Figure 14] This is an example of an image in which rice is the object. [Figure 15] 13 is an example of an image of a target object, granola. [Figure 16] FIG. 2 is a diagram showing configurations of a model generation program and a foreign object detection program according to an embodiment of the present invention, together with a recording medium. [Figure 17] FIG. 13 is a diagram showing an example of a combination of a third normal image and a third foreign object image used for training. [Figure 18] FIG. 13 is a diagram showing an example of a combination of a third normal image and a third foreign object image used for training. [Figure 19] 13A and 13B are diagrams illustrating examples of images used to generate a third foreign object image. [Figure 20] 13A and 13B are diagrams illustrating examples of images used to generate a third foreign object image. [Figure 21] 13A and 13B are diagrams illustrating examples of images used to generate a third foreign object image. [Figure 22] 13A and 13B are diagrams illustrating examples of images used to generate a third foreign object image. [Figure 23] 11A and 11B are diagrams showing examples of detection results according to a comparative example and an embodiment of the present invention. [Figure 24] 11A and 11B are diagrams showing examples of detection results according to a comparative example and an embodiment of the present invention. [Diagram 25] FIG. 13 is a diagram showing an example of a third foreign object image used for training. [Figure 26] FIG. 13 is a diagram showing an example of a combination of a third normal image and a third foreign object image used for training. [Figure 27]FIG. 13 is a diagram showing an example of a combination of a third normal image and a third foreign object image used for training. [Figure 28] FIG. 11 is a diagram showing an example of a detection result according to an embodiment of the present invention. [Figure 29] FIG. 11 is a diagram showing an example of a detection result according to an embodiment of the present invention. [Diagram 30] FIG. 1 is a diagram illustrating an example of an inference model. [Diagram 31] A diagram showing a schematic example of a second inference model. [Diagram 32] A figure showing example images used for training to generate a second inference model. [Diagram 33] A figure showing an example of detection results according to an embodiment of the present invention using a second inference model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Hereinafter, embodiments of a model generation method, a model generation system, a model generation program, a foreign object detection method, a foreign object detection system, a foreign object detection program, and an inference model according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated descriptions will be omitted.

[0032] Fig. 1(a) shows a model generation system 10 according to this embodiment. Fig. 1(b) shows a foreign object detection system 20 according to this embodiment. The model generation system 10 is a system (apparatus) that performs machine learning training to generate an inference model (learning model, trained model) used to detect a foreign object included in an image showing an object. The foreign object detection system 20 is a system (apparatus) that uses the inference model generated by the model generation system 10 to detect (detect) a foreign object included in an image showing an object.

[0033] FIG. 2 shows an example of a target image 30 used for detecting foreign objects. In this embodiment, the detection of foreign objects is the detection of foreign objects in food packaged as a product, for example, food in a plastic packaging bag. The detection of foreign objects is performed, for example, in the manufacturing process of food products, to determine whether the manufactured product is of good quality or not. If no foreign object is detected in the image of the product to be detected, the product is determined to be of good quality, and if a foreign object is detected in the image of the product to be detected, the product is determined to be defective. The images used for detecting foreign objects are captured by X-rays for each product to be detected. The parts surrounded by dashed lines in the figure are parts where foreign objects are present.

[0034] The assumed foreign object may be, for example, an object that may be included in the manufacturing process. In this embodiment, an example is shown in which a ball (a SUS (Steel Use Stainless) ball, a glass ball) smaller than the food is used as the foreign object. The product to be detected for the foreign object is, for example, pasta such as penne. The product to be detected for the foreign object may be something other than pasta, such as rice or granola. The foreign object does not need to be smaller than the target object such as food, and may be about the same size as the target object or larger than the target object. As shown in the target image 30 in FIG. 2, a large number of similarly shaped objects are usually included in one package of a product such as pasta. In the conventional method, it is difficult to accurately detect the foreign object in such a case, but in this embodiment, the foreign object can be accurately detected.

[0035] Note that the object containing the foreign object (a food product in this embodiment), the foreign object, and the image (an X-ray image in this embodiment) are not limited to those in this embodiment, and may be any object to which this embodiment is applicable. Furthermore, the foreign object to be detected does not necessarily have to be something separate from the object (food in the above example), and may be a defect such as a flaw or flaw that appears on the object. In other words, detection of a foreign object may be detection of whether the object is a defective product.

[0036] Each of the model generation system 10 and the foreign object detection system 20 includes a conventional computer including hardware such as a processor, a memory, and a communication module, such as a CPU (Central Processing Unit). The functions of the model generation system 10 and the foreign object detection system 20, which will be described later, are realized by these components operating through programs or the like. In FIG. 1, the model generation system 10 and the foreign object detection system 20 are shown as separate systems (devices), but they may be realized by the same system (device). The computer that constitutes the model generation system 10 and the foreign object detection system 20 may be a computer system including multiple computers. The computer may also be configured using cloud computing or edge computing.

[0037] Next, a description will be given of functions of the model generation system 10 and the foreign object detection system 20 according to this embodiment. As shown in FIG.

[0038] Before describing each function of the model generation system 10, an inference model generated by the model generation system 10 will be described. Additionally, an overview of foreign object detection by the foreign object detection system 20 using the inference model will be described. The inference model is a model that inputs an image of a target for foreign object detection and outputs (infers) an image in which the foreign object portion of the input image has been removed (it is inferred). FIG. 2 shows an example of a target image 30 that is input to the inference model and is a target for foreign object detection, and an example of an image 40 in which the foreign object portion has been removed (it is inferred) that is output when the target image 30 is input to the inference model. A foreign object is captured in the portion surrounded by a dashed line in the target image 30, but no foreign object is captured in the same position in the image 40 output from the inference model.

[0039] By taking the difference between the image input to the inference model and the image output from the inference model, it is possible to obtain an image (information) showing only the foreign objects removed by the inference model. Fig. 3 shows an anomaly map 50, which is an image of the difference between the target image 30 input to the inference model in Fig. 2 and the image 40 output from the inference model. The anomaly map 50 makes it possible to grasp foreign objects. Detection of foreign objects by the foreign object detection system 20 using the inference model corresponds to, for example, generation of an anomaly map.

[0040] The inference model includes, for example, a neural network. The neural network may be multi-layered. That is, the inference model may be generated by deep learning. The neural network may be a convolutional neural network (CNN).

[0041] The inference model of this embodiment is shown diagrammatically in Fig. 4. The inference model includes an encoder that encodes an input image into features, and a decoder that decodes the features output from the encoder into an image.

[0042] In the inference model, neurons are provided in the input layer of the encoder for inputting information based on an image. For example, the information input to the inference model is the pixel value (brightness value) of each pixel of the image. In this case, the input layer is provided with neurons equal to the number of pixels in the image, and the pixel value of the corresponding pixel is input to each neuron. Note that the information input to the inference model may be other than the pixel value of each pixel as long as it is based on the image.

[0043] In the inference model, neurons for outputting an image are provided in the output layer of the decoder. For example, the information output from the inference model is the pixel value of each pixel of the image. In this case, the output layer is provided with neurons equal to the number of pixels in the image, and each neuron outputs the pixel value of the corresponding pixel. Note that the information output from the inference model may be other than the pixel value of each pixel, as long as it is capable of generating the above-mentioned output image.

[0044] In addition, the inference model may be constructed using something other than a neural network, so long as it is generated by machine learning training and performs the input and output described above.

[0045] The inference model is expected to be used as a program module that is a part of artificial intelligence software. The inference model is used, for example, in a computer having a processor and a memory, and the processor of the computer operates according to instructions from the model stored in the memory. For example, the processor of the computer operates to input information to the model according to the instructions, perform calculations according to the model, and output results from the model. Specifically, the processor of the computer operates to input information to the input layer of the neural network according to the instructions, perform calculations based on parameters such as weighting coefficients for learning in the neural network, and output results from the output layer of the neural network. The above is an overview of the inference model generated by the model generation system 10 and the detection of a foreign object by the foreign object detection system 20 using the inference model.

[0046] The training acquisition unit 11 is a training acquisition means for acquiring training images. The training images are a first normal image in which a training object is captured, a second normal image in which a training object is captured, a second foreign object image in which a foreign object assumed to be a detection target is added to the second normal image, a third normal image in which a training object is captured, and a third foreign object image in which a foreign object not assumed to be a detection target is added to the third normal image. The foreign object not assumed to be a detection target may be a natural image.

[0047] The second normal image and the third normal image may be the first normal image, and the training acquisition unit 11 may add a foreign object expected to be detected to the first normal image to generate and acquire a second foreign object image, and add a foreign object not expected to be detected to the first normal image to generate and acquire a third foreign object image. The object may be a specific type of object, and the first normal image, the second normal image, the second foreign object image, the third normal image, and the third foreign object image acquired by the training acquisition unit 11 may be images in which a specific type of object is shown as a training object. The ratio of the number of the first normal images acquired by the training acquisition unit 11, the number of combinations of the second normal image and the second foreign object image, and the number of combinations of the third normal image and the third foreign object image may be a preset ratio.

[0048] The inference model is generated by machine learning training using training images acquired by the training acquisition unit 11. The training of the inference model includes three types of training: training using a first normal image, training using a combination of a second normal image and a second foreign object image, and training using a combination of a third normal image and a third foreign object image.

[0049] FIG. 5 shows an example of the first normal image 61. The first normal image 61 is an image that includes a training object and does not include a foreign object. The first normal image 61 is, for example, an image obtained by capturing an image of a product that has been confirmed to contain no foreign objects under the same conditions as those used to detect foreign objects. Products that have been confirmed to contain no foreign objects are training objects. In addition, the product is of the same type as the object for which a foreign object is to be detected. That is, if the object for which a foreign object is to be detected is a pasta product, the first normal image 61 is an image of the same type of pasta product.

[0050] FIG. 6 shows examples of the second normal image 62 and the second foreign object image 72. The second normal image 62 is an image in which a training object is captured and no foreign object is captured, and is the same as the first normal image 61. The first normal image 61 may be used as the second normal image 62. The second foreign object image 72 is an image in which (an image of) a foreign object assumed to be a detection target (for example, the above-mentioned small ball) is added to the second normal image 62. The second foreign object image 72 is, for example, an image obtained by superimposing an image obtained by capturing only a foreign object assumed to be a detection target on the second normal image 62. Alternatively, the second foreign object image 72 may be generated by performing image processing on the second normal image 62 assuming a foreign object assumed to be a detection target, without using an image obtained by capturing only a foreign object. The image processing may be performed using a simulation technique (for example, a simulation related to image processing of an X-ray image). In training the inference model, a combination of the second foreign object image 72 and the second normal image 62 used to generate the second foreign object image 72 is used.

[0051] 7 shows examples of a third normal image 63 and a third foreign object image 73. The third normal image 63 is an image in which a training object is captured and no foreign object is captured, and is the same as the first normal image 61. The first normal image 61 may be used as the third normal image 63. The third foreign object image 73 is an image in which a foreign object not expected to be a detection target is added to the third normal image 63. The third foreign object image 73 is an image obtained by, for example, superimposing an image of a foreign object not expected to be a detection target on the third normal image 63.

[0052] Fig. 8 shows an example of a foreign object image 80 used to generate the third foreign object image 73. As shown in Fig. 8, the foreign object added to the third normal image 63 to generate the third foreign object image 73 is a natural image. The natural image here is an image of a landscape, a person, or the like, which is completely different from the foreign object assumed to be the detection target. The third foreign object image 73 shown in Fig. 7 is obtained by superimposing the image of the tank in the upper right of Fig. 8 on the third normal image 63.

[0053] Alternatively, the third foreign object image 73 may be generated by performing image processing on the third normal image 63 without considering foreign objects that are expected to be detected, without using an image of a foreign object. The image processing may be performed using a simulation technique. In training the inference model, a combination of the third foreign object image 73 and the third normal image 63 used to generate the third foreign object image 73 is used.

[0054] The training acquisition unit 11 acquires a combination of a first normal image 61, a second normal image 62, and a second foreign object image 72, and a combination of a third normal image 63 and a third foreign object image 73. For example, the training acquisition unit 11 acquires each image by accepting each image input to the model generation system 10 by a user of the model generation system 10. In addition, the training acquisition unit 11 may acquire each image by any other method.

[0055] Alternatively, the training acquisition unit 11 acquires only the first normal image 61 from among the above images in the same manner as described above. Note that, if only the first normal image 61 is acquired, it may be acquired by receiving an image transmitted from an imaging device that acquires an image by imaging (for example, an X-ray imaging device that acquires an X-ray image). The training acquisition unit 11 sets the acquired first normal image 61 as the second normal image 62 and the third normal image 63. The training acquisition unit 11 generates and acquires the second foreign object image 72 from the second normal image 62 (first normal image 61). For example, the training acquisition unit 11 acquires an image obtained by imaging only a foreign object assumed to be a detection target, and generates the second foreign object image 72 by superimposing the acquired image and the second normal image 62.

[0056] The training acquisition unit 11 also generates and acquires a third foreign object image 73 from the third normal image 63 (first normal image 61). For example, the training acquisition unit 11 acquires an image 80 of a foreign object not expected to be detected (for example, a natural image shown in FIG. 8 ) and superimposes the acquired image on the third normal image 63 to generate the third foreign object image 73. Images used to generate the second foreign object image 72 and the third foreign object image 73 may be acquired by the same method as the first normal image 61, or by any other method.

[0057] Alternatively, the training acquisition unit 11 may store in advance a method of image processing by simulation for generating the second foreign object image 72 and the third foreign object image 73, and may perform image processing on the second normal image 62 and the third normal image 63 by the method to generate the second foreign object image 72 and the third foreign object image 73. In the image processing by simulation, for example, processing may be performed to change the shape and contrast of the foreign object, or to add blurring related to the scintillator and blurring related to the afterglow. In addition to the above, image processing may be performed according to changes in the imaging conditions. The image processing by simulation can be realized by a conventional method. In addition, the method used to generate the second foreign object image 72 and the third foreign object image 73 may be a combination of a method of superimposing images and a method of performing image processing by simulation.

[0058] The training acquisition unit 11 acquires a number of training images that allows the model generation unit 12 to perform training appropriately. The training acquisition unit 11 acquires training images such that the ratio of the number of the first normal images 61, the number of combinations of the second normal images 62 and the second foreign object images 72, and the number of combinations of the third normal images 63 and the third foreign object images 73 is a preset ratio. For example, the training acquisition unit 11 acquires training images such that the numbers are the same, that is, the numbers are each 1 / 3 of the total number. This is because the images output by the inference model correspond to this ratio. The training acquisition unit 11 outputs the acquired training images 61, 62, 63, 72, and 73 to the model generation unit 12.

[0059] The model generation unit 12 is a model generation means for generating an inference model by performing training using training images acquired by the training acquisition unit 11. The training for generating the inference model includes training in which information based on the first normal image 61 is used as an input to the inference model and information based on the first normal image 61 is used as an output from the inference model, training in which information based on the second foreign object image 72 is used as an input to the inference model and information based on the second normal image 62 is used as an output from the inference model, and training in which information based on the third foreign object image 73 is used as an input to the inference model and information based on the third normal image 63 is used as an output from the inference model.

[0060] The model generation unit 12 generates an inference model, for example, as follows: The model generation unit 12 inputs training images 61, 62, 63, 72, and 73 from the training acquisition unit 11. The model generation unit 12 performs training for generating an inference model for each first normal image 61, each combination of a corresponding second normal image 62 and a corresponding second foreign object image 72, and each combination of a corresponding third normal image 63 and a corresponding third foreign object image 73.

[0061] In the following, an example will be described in which the inference model inputs an image itself and outputs the image itself. When training is performed using a first normal image 61, the model generation unit 12 performs training by inputting the first normal image 61 to the inference model and outputting the first normal image 61 from the inference model, as shown in FIG. 5. When training is performed using a combination of a second normal image 62 and a second foreign object image 72 that correspond to each other, the model generation unit 12 performs training by inputting the second foreign object image 72 to the inference model and outputting the second normal image 62 that corresponds to the second foreign object image 72 from the inference model, as shown in FIG. 6. When training is performed using a combination of a third normal image 63 and a third foreign object image 73 that correspond to each other, the model generation unit 12 performs training by inputting the third foreign object image 73 to the inference model and outputting the third normal image 63 that corresponds to the third foreign object image 73 from the inference model, as shown in FIG. 7.

[0062] The above-mentioned training, i.e., the update of the parameters of the inference model, may be performed in the same manner as conventional machine learning training. In addition, the training may be performed for each type of image (the first normal image 61, the combination of the corresponding second normal image 62 and the corresponding second foreign object image 72, and the combination of the corresponding third normal image 63 and the corresponding third foreign object image 73) collectively, or may be performed by changing the type of image each time.

[0063] If the inference model is one that inputs information based on images other than the images themselves, the model generation unit 12 may generate information based on the images 61, 72, and 73 from each of the images 61, 72, and 73 that correspond to the input to the inference model among the training images, and perform training using the generated information as an input to the inference model. Also, if the inference model is one that outputs information corresponding to an image other than the image itself, the model generation unit 12 may generate information corresponding to the images 61, 62, and 63 (information based on the images 61, 62, and 63) from each of the images 61, 62, and 63 that correspond to the output from the inference model among the training images, and perform training using the generated information as an output from the inference model.

[0064] The model generation unit 12 generates an inference model by, for example, using all of the training images 61, 62, 63, 72, and 73 input from the training acquisition unit 11 for training. Alternatively, the model generation unit 12 may generate an inference model by performing training until a preset condition for ending training other than the above is satisfied. The generated inference model is used in the foreign object detection system 20. The model generation unit 12 outputs the generated inference model. For example, the model generation unit 12 transmits the inference model to the foreign object detection system 20. Note that the inference model may be input to the foreign object detection system 20 by a method other than the output from the model generation unit 12. For example, the inference model may be input to the foreign object detection system 20 by operating the model generation system 10 or the foreign object detection system 20. The above is the function of the model generation system 10 according to this embodiment.

[0065] As shown in FIG. 1, the foreign object detection system 20 includes an acquisition unit for detection 21, a calculation unit 22, and a detection unit .

[0066] The acquisition unit for detection 21 is a detection acquisition means for acquiring a target image 30 that is to be detected for a foreign object. The acquisition unit for detection 21 receives and acquires, for example, an image transmitted from an imaging device that acquires an image by imaging (for example, an X-ray imaging device that acquires an X-ray image) as the target image 30. The acquisition unit for training 11 may acquire the target image 30 by any other method. The acquisition unit for detection 21 outputs the acquired target image 30 to the calculation unit 22.

[0067] The calculation unit 22 is a calculation unit that inputs information based on the target image 30 acquired by the acquisition unit for detection 21 to an inference model, performs calculations, and obtains an output from the inference model. The calculation unit 22 inputs and stores the inference model generated by the model generation system 10. The calculation unit 22 inputs the target image 30 from the acquisition unit for detection 21.

[0068] The calculation unit 22 inputs information based on the input target image 30 into the stored inference model, performs calculations, and obtains an output from the inference model. When the inference model corresponds to the type of object, the calculation unit 22 uses the inference model corresponding to the type of object related to the target image 30 for calculations. For example, when the object shown in the target image 30 is pasta, the inference model for pasta is used.

[0069] The information input to the inference model corresponds to the inference model, and is, for example, the target image 30 itself as described above. The information input to the inference model may also be information based on the target image 30 other than the target image 30 itself. In this case, the calculation unit 22 generates information to be input to the inference model from the target image 30. The information output from the inference model corresponds to the inference model, and is, for example, an image 40 from which the foreign object portion of the target image 30 has been removed (as inferred) as described above. The information output from the inference model may also be information other than the image 40 and corresponding to the image 40.

[0070] The calculation unit 22 outputs information related to the input / output to the inference model to the detection unit 23. For example, the calculation unit 22 outputs the target image 30 and the image 40 output from the inference model to the detection unit 23 as information related to the input / output to the inference model. In addition, the calculation unit 22 may output information corresponding to the target image 30 and the image 40 output from the inference model other than the image 30 and the image 40 output from the inference model to the detection unit 23.

[0071] The detection unit 23 is a detection means that calculates the difference between information input to the inference model by the calculation unit 22 and information output from the inference model, and detects foreign objects contained in the target image 30 from the calculated difference.

[0072] The detection unit 23 detects a foreign object, for example, as follows. The detection of a foreign object by the detection unit 23 is, for example, as described above, the generation of an anomaly map 50 that is information indicating the detection result as shown in Fig. 3. The detection unit 23 inputs information related to input and output from the calculation unit 22 to the inference model, for example, the target image 30 and the image 40 output from the inference model.

[0073] The detection unit 23 takes these differences to generate the anomaly map 50. Specifically, the detection unit 23 takes the difference in pixel value between the target image 30 and the image 40 output from the inference model for each corresponding pixel to generate the anomaly map 50. In the anomaly map 50, a portion having a pixel value other than 0, i.e., a portion where there is a difference in pixel value between the images 30 and 40, is a portion where a foreign object is detected (estimated) to exist.

[0074] The detection unit 23 outputs an abnormality degree map 50, which is information indicating the detection result. The information indicating the detection result may be output, for example, to another system (device) or another module, or may be output in a form that can be recognized by the user of the foreign object detection system 20 (for example, a display or audio output).

[0075] Furthermore, the detection unit 23 may store a criterion for detecting a foreign object in advance and determine whether or not a foreign object is included in the target image 30 based on the criterion from the anomaly degree map 50. The detection unit 23 may also perform other processes as long as it calculates the difference between information related to the input to the inference model and information related to the output from the inference model and detects a foreign object included in the target image 30 from the calculated difference. The above are the functions of the foreign object detection system 20 according to this embodiment.

[0076] Next, the processes executed by the model generation system 10 and the foreign object detection system 20 according to this embodiment (the operation methods performed by the model generation system 10 and the foreign object detection system 20) will be described with reference to the flowcharts of FIGS.

[0077] First, a model generation method, which is a process executed by the model generation system 10 according to the present embodiment, will be described with reference to the flowchart of FIG. 9. In this process, first, the training acquisition unit 11 acquires training images 61, 62, 63, 72, and 73 (S01, training acquisition step). The training images are a combination of a first normal image 61, a second normal image 62, and a second foreign object image 72, and a combination of a third normal image 63 and a third foreign object image 73. The first to third normal images 61 to 63 are images in which training objects are shown. The second foreign object image 72 is an image in which a foreign object assumed to be a detection target is added to the second normal image. The third foreign object image 73 is an image in which a foreign object not assumed to be a detection target is added to the third normal image.

[0078] Next, the model generating unit 12 performs training using the training images 61, 62, 63, 72, and 73 to generate an inference model (S02, model generating step). The training includes training in which information based on the first normal image 61 is input to the inference model and information based on the first normal image 61 is output from the inference model. The training includes training in which information based on the second foreign object image 72 is input to the inference model and information based on the second normal image 62 is output from the inference model. The training includes training in which information based on the third foreign object image 73 is input to the inference model and information based on the third normal image 63 is output from the inference model.

[0079] The generated inference model is output from the model generation unit 12 (S03). The inference model output from the model generation system 10 is stored in the foreign object detection system 20. The above is the model generation method which is the processing executed by the model generation system 10 according to this embodiment.

[0080] Next, a foreign object detection method, which is a process executed by the foreign object detection system 20 according to this embodiment, will be described with reference to the flowchart of FIG. 10. In this process, first, the detection acquisition unit 21 acquires the target image 30 (S11, detection acquisition step). Next, the calculation unit 22 inputs information based on the target image 30 to an inference model, performs calculations, and obtains an output from the inference model (S12, calculation step). Next, the detection unit 23 calculates the difference between information related to the input to the inference model and information related to the output from the inference model as a process for detecting a foreign object contained in the target image 30, and generates an abnormality degree map (S13, detection step). Next, the generated abnormality degree map is output from the detection unit 23 as information indicating the detection result of the foreign object (S14). The above is the foreign object detection method, which is a process executed by the foreign object detection system 20 according to this embodiment.

[0081] Conventional methods may have difficulty detecting foreign objects from images that contain many objects of roughly similar shapes (e.g., pasta), such as the above-mentioned food product images. For example, if an inference model is generated by training using only the combination of the second normal image 62 and the second foreign object image 72, the output may reflect foreign objects or may be missing parts of the objects. This is believed to be due to the structure of the above-mentioned images.

[0082] In the model generation method according to the present embodiment, in addition to the normal images 61 to 63, two different foreign object images 72 and 73 are used for training to generate an inference model. Specifically, as described above, three types of training are performed to generate an inference model: training using the first normal image 61 (training shown in FIG. 5), training using the second normal image 62 and the second foreign object image 72 (training shown in FIG. 6), and training using the third normal image 63 and the third foreign object image 73 (training shown in FIG. 7). The inference model generated in this way reflects the target object, among those captured in the input image, in the output, but does not reflect the foreign object in the output. That is, the inference model with pasta as the target object is a pasta pass filter that passes the structure of pasta and does not pass the structure of foreign object.

[0083] Therefore, even when detecting small foreign objects in an image containing multiple objects of roughly the same shape, such as pasta, as in this embodiment, the foreign objects can be removed from the output image. Therefore, according to this embodiment, foreign objects can be detected with high accuracy by detecting foreign objects using the generated inference model.

[0084] Also, as in this embodiment, the foreign object not expected to be a detection target used to generate the third foreign object image 73 may be a natural image. With this configuration, it is possible to generate an inference model more appropriately and reliably. However, the foreign object not expected to be a detection target may be an image of various textures other than a natural image.

[0085] Also, as in this embodiment, the second normal image 62 and the third normal image 63 may be the first normal image 61. Alternatively, a foreign object expected to be detected may be added to the first normal image 61 to generate and acquire the second foreign object image 72, and a foreign object not expected to be detected may be added to the first normal image 61 to generate and acquire the third foreign object image 73. According to this configuration, if the first normal image 61 can be acquired as the normal images 61 to 63, the second foreign object image 72 and the third foreign object image 73 can be easily and reliably acquired. As a result, all the training images 61, 62, 63, 72, and 73 can be acquired, and an inference model can be easily and reliably generated. However, the acquisition of the training images 61, 62, 63, 72, and 73 does not have to be performed as described above, and may be performed in any manner.

[0086] Also, as in this embodiment, the target object may be a specific type of object, and the training images 61, 62, 63, 72, and 73 may be images in which a specific type of object is depicted as the training target object. For example, the target object may be a pasta product. Inference models for each type of food may also be generated and used. With this configuration, an inference model for detecting foreign objects with high accuracy for a specific type of object can be generated. However, training images 61, 62, 63, 72, and 73 in which the target object is not limited to a specific type of object may also be used.

[0087] Also, as in this embodiment, the ratio of the number of the first normal images 61 acquired and used for training, the number of combinations of the second normal images 62 and the second foreign object images 72, and the number of combinations of the third normal images 63 and the third foreign object images 73 may be a preset ratio. For example, as described above, these numbers may be the same. By setting this ratio appropriately, an inference model can be generated more appropriately and reliably. However, this ratio does not need to be preset.

[0088] The model generation system 10 and the foreign object detection system 20 may be provided by the same entity or by different entities, and the model generation system 10 and the foreign object detection system 20 may be used by the same user or by different users.

[0089] FIG. 11 shows an example of the result of actually detecting a foreign object by this embodiment. Here, the results of detecting a foreign object by two conventional methods are compared with the results according to this embodiment as comparative examples. The first method among the conventional methods is as follows. A non-defective image (for example, normal images 61 to 63 used in this embodiment) that is an image of an object without a foreign object is prepared in advance. For the target image for detecting a foreign object, a feature amount (for example, a feature vector represented by a vector) is calculated for each part of the image. For the non-defective image, a feature amount for each part of the image is also calculated. The feature amount of the target image is compared with the feature amount of the non-defective image, and whether or not a foreign object exists for each part is detected based on the comparison. For example, the distance between the feature vectors is calculated, and if the distance between them is equal to or greater than a preset threshold, it is determined that a foreign object exists, and if the distance is less than the threshold, it is determined that a foreign object does not exist.

[0090] The second method among the conventional methods is as follows. Good product images (for example, normal images 61 to 63 used in this embodiment) that are images of an object without foreign objects are prepared in advance. An autoencoder is generated from the good product images by training machine learning. Using the generated autoencoder as an inference model, foreign objects are detected in the same manner as in the method by foreign object detection system 20 of this embodiment.

[0091] The results shown in FIG. 11 are the results of detecting foreign objects by putting five SUS balls and five glass balls as foreign objects into a pasta product. The table (a) in FIG. 11 shows the results by the first method, the table (b) shows the results by the second method, and the table (c) shows the results by this embodiment. The first method detected two SUS balls and two glass balls, with three false positives (foreign objects detected even though there were no foreign objects). The second method detected three SUS balls and no glass balls, with no false positives. In this embodiment, four SUS balls and four glass balls were detected, with no false positives. As shown by these results, this embodiment can detect foreign objects with higher accuracy than the conventional method.

[0092] 12 and 13 show examples of images used for training and images and anomaly maps output from the inference model. The images and anomaly maps shown in Figs. 12(a)-(f) and Figs. 13(a) and (b) are comparative examples, and the images and anomaly maps shown in Figs. 13(c) and (d) are examples according to this embodiment.

[0093] 12(a) and (b) show an image and an anomaly map output from an inference model when the inference model is generated by training using only the first normal image 61 (i.e., when the inference model is an autoencoder). In this case, the image output from the inference model does not erase foreign objects, and foreign objects are not detected even in the areas surrounded by ellipses in the anomaly map where foreign objects should be.

[0094] 12(c) and (d) show the image and anomaly map output from the inference model when the inference model is generated by training using only the combination of the second normal image 62 and the second foreign object image 72. In this case, the image output from the inference model has the foreign object removed, but the pasta portion has also been removed in excess. Foreign objects are detected even in the portion of the anomaly map surrounded by an ellipse where there are no foreign objects.

[0095] 12(e) and (f) show the image and the anomaly map output from the inference model when the inference model is generated by training using only the combination of the third normal image 63 and the third foreign object image 73. In this case, the image output from the inference model tends to have the structure of the pasta portion generally removed.

[0096] 13(a) and (b) show images and anomaly maps output from an inference model generated by training an inference model using an equal number of first normal images 61 and combinations of second normal images 62 and second foreign object images 72. In this case, the image output from the inference model has foreign objects removed. However, there is some overlearning of foreign objects, and some parts of the pasta structure have been removed as foreign objects. Foreign objects are detected even in the parts of the anomaly map that are surrounded by ellipses and do not contain foreign objects.

[0097] 13(c) and (d) show images and anomaly maps output from an inference model when an inference model is generated by training using the same number of combinations of the first normal image 61, the second normal image 62 and the second foreign object image 72, and the third normal image 63 and the third foreign object image 73 (i.e., in the case of this embodiment). In this case, the image output from the inference model has foreign objects appropriately removed compared to other examples, and the anomaly map also shows that appropriate foreign objects have been detected.

[0098] In the above explanation, an image of pasta was used as an example, but an example will be shown in which another type of object is used as the target. FIG. 14 shows an example in which rice is used as the target. FIG. 14(a) shows an example of a target image used to detect foreign objects, FIG. 14(b) shows an example of an image output from the inference model when the target image is input to the inference model, and FIG. 14(c) shows an example of an anomaly map generated from these images. When rice is used as the target, training images 61, 62, 63, 72, and 73 in which rice is shown as the target are used to perform training and generate an inference model. This inference model is a rice-pass filter that passes the structure of rice but does not pass the structure of foreign objects.

[0099] FIG. 15 shows an example of a case where granola with added dried fruit is used as the target object. FIG. 15(a) shows an example of a target image used to detect foreign objects, FIG. 15(b) shows an example of an image output from the inference model when the target image is input to the inference model, and FIG. 15(c) shows an example of an anomaly map generated from those images. The part surrounded by the dashed line in the figure is the part where the foreign object exists. When granola is used as the target object, training images 61, 62, 63, 72, and 73 in which granola is shown as the target object are used to perform training and generate an inference model. This inference model is a granola pass filter that passes the structure of granola but does not pass the structure of foreign objects.

[0100] Next, a description will be given of a model generation program and a foreign object detection program for executing the above-mentioned series of processes by the model generation system 10 and the foreign object detection system 20. As shown in Fig. 16(a), the model generation program 100 is stored in a program storage area 111 formed in a computer-readable recording medium 110 that is inserted into a computer and accessed, or that is provided in the computer. The recording medium 110 may be a non-transitory recording medium.

[0101] The model generation program 100 includes a training acquisition module 101 and a model generation module 102. Functions realized by executing the training acquisition module 101 and the model generation module 102 are similar to the functions of the training acquisition unit 11 and the model generation unit 12 of the model generation system 10 described above, respectively.

[0102] 16(b), the foreign object detection program 200 is stored in a program storage area 211 formed in a computer-readable recording medium 210 that is inserted into a computer and accessed, or that is provided in the computer. The recording medium 210 may be a non-transitory recording medium. Note that when the model generation program 100 and the foreign object detection program 200 are executed by the same computer, the recording medium 210 may be the same as the recording medium 110.

[0103] The foreign object detection program 200 includes an acquisition module for detection 201, a calculation module 202, and a detection module 203. Functions realized by executing the acquisition module for detection 201, the calculation module 202, and the detection module 203 are similar to the functions of the acquisition unit for detection 21, the calculation unit 22, and the detection unit 23 of the foreign object detection system 20 described above, respectively.

[0104] The model generation program 100 and the foreign object detection program 200 may be configured such that a part or all of them are transmitted via a transmission medium such as a communication line and are received and recorded (including installed) by other equipment. Also, each module of the model generation program 100 and the foreign object detection program 200 may be installed in any one of a plurality of computers, not just one computer. In that case, the above-mentioned series of processes are performed by a computer system including the plurality of computers.

[0105] Next, further modifications and examples of the embodiment of the present invention will be described. Note that the modifications described below may be implemented by replacing or adding to part of the embodiment described above.

[0106] FIG. 17 shows another example of the third normal image 63 and the third foreign object image 73. As in the third foreign object image 73 shown in FIG. 17, the third foreign object image 73 may be one in which a plurality of partial natural images are superimposed on the third normal image 63. Here, the partial natural images are natural images smaller in size than the third normal image 63. Even when such a third foreign object image 73 is used, it is possible to generate an appropriate inference model for detecting foreign objects. That is, classification is sufficiently possible based on the characteristics of the natural image, and it is possible to handle any foreign object structure (unexpected foreign object).

[0107] The foreign object added to the third foreign object image 73 may be an image drawn based on a calculation formula. FIG. 18 shows examples of the third normal image 63 and the third foreign object image 73 in this case. Like the third foreign object image 73 shown in FIG. 18, the image relating to the foreign object is, for example, an image generated by simulation based on a calculation formula prepared in advance. The image may be generated by a conventional method. The image relating to the foreign object added to the third normal image 63 may be generated by the model generation system 10 (the training acquisition unit 11), or may be generated by a system other than the model generation system 10 and acquired by the model generation system 10 (the training acquisition unit 11).

[0108] The image related to the foreign substance added to the third foreign substance image 73 and drawn based on a formula may be a geometric pattern image 81 as shown in Fig. 19. The image may also be a procedural texture image (a texture image such as a texture generated based on a formula) 82 as shown in Fig. 20. The image may also be a frequency image 83, which is an image generated based on a formula related to frequency, as shown in Fig. 21, or a combination of multiple frequency images 83. The image may also be an image combining the above images, for example, an image obtained by cutting out a frequency image 83 based on a procedural texture image 82 as shown in Fig. 22.

[0109] A more appropriate and reliable inference model can also be generated by using a configuration in which the third foreign object image 73 is based on the above image.

[0110] Next, an example of detecting foreign objects from an image of coffee beans as the object will be shown. FIG. 23(a) shows a case where foreign objects are detected from a target image 30 used for detecting foreign objects by a conventional simple binarization of an image, and the detected foreign objects are superimposed on the target image 30. As shown in FIG. 23(a), in the conventional method, two foreign objects were detected in the upper part of the image, three in the middle part of the image, and two in the lower part of the image. FIG. 23(b) shows an anomaly map obtained from the target image 30 using the inference model of this embodiment, and FIG. 23(c) shows a case where foreign objects detected from the anomaly map are superimposed on the target image 30. As shown in FIG. 23(c), in the method according to this embodiment, four foreign objects were detected in the upper part of the image, two in the middle part of the image, and six in the lower part of the image.

[0111] FIG. 24(a) shows a case where foreign objects are detected from a target image 30 (an image different from the target image shown in FIG. 23) used for detecting foreign objects by a conventional simple binarization of an image, and the detected foreign objects are superimposed on the target image 30. As shown in FIG. 24(a), in the conventional method, two foreign objects were detected in the upper part of the image, three in the middle part of the image, and three in the lower part of the image. FIG. 24(b) shows an anomaly map obtained from the target image 30 using the inference model of this embodiment, and FIG. 24(c) shows a case where foreign objects detected from the anomaly map are superimposed on the target image 30. As shown in FIG. 24(c), in the method according to this embodiment, three foreign objects were detected in the upper part of the image, six in the middle part of the image, and five in the lower part of the image. As shown in the examples of FIG. 23 and FIG. 24, the method according to this embodiment detects actual foreign objects that could not be detected by the conventional method. Thus, the method according to this embodiment allows for highly accurate detection of foreign objects.

[0112] In the third foreign object image 73, the portion to which the foreign object is added does not need to be the entire third foreign object image 73, and may be a part of the third foreign object image 73. In addition, the position of the portion to which the foreign object is added in the third foreign object image 73 may be an irregular position (random position). In addition, the size of each foreign object in the third foreign object image 73 may be various sizes (multi-size). This makes it possible to make the inference model capable of appropriately detecting foreign objects even if the foreign objects detected from the target image 30 are located in irregular positions or have various sizes. FIG. 25 shows an example of the third foreign object image 73 to which foreign objects are added in irregular positions.

[0113] The third foreign object image 73 may be an image in which a foreign object not expected to be detected is added to the third normal image 63 by at least one of transparent addition and replacement addition. Fig. 26 shows the third foreign object image 73 by transparent addition of a foreign object, and the corresponding third normal image 63. Fig. 27 shows the third foreign object image 73 by replacement addition of a foreign object, and the corresponding third normal image 63.

[0114] Transparent addition of a foreign object (mix-up) means adding a foreign object so that both the foreign object and the third normal image 63 before the addition are visible in the portion of the third foreign object image 73 to which the foreign object has been added. That is, transparent addition of a foreign object means superimposing a semi-transparent foreign object on the third normal image 63 before the addition in a state in which the third normal image 63 is visible through the foreign object in the portion to which the foreign object is added (or in a state in which the foreign object is visible through the third normal image 63). When transparent addition of a foreign object is performed, the ratio of the size of the portion to which the foreign object has been added in the entire third foreign object image 73 may be 10% to 100%. By increasing this ratio, the foreign object detection performance of the inference model can be improved.

[0115] Fig. 28 shows an example in which a foreign object is detected from a target image 30 of a coffee bean as an object when an inference model is generated using a third foreign object image 73 obtained by transparently adding a foreign object. Fig. 28(a) shows an anomaly map obtained from the target image 30, and Fig. 28(b) shows the target image 30 with a foreign object detected from the anomaly map superimposed thereon.

[0116] When the third foreign body image 73 with the transparent addition of the foreign body is used, it is possible to detect particularly localized foreign bodies (e.g., texture changes (flaws or abnormalities)) with high accuracy. Furthermore, when the target image 30 is an image in which the foreign body and other objects are transparently overlapped, as in the case of the transparent addition of the foreign body described above, it is possible to detect foreign bodies with high accuracy. For example, when the target image 30 is an X-ray image, it is possible to detect foreign bodies with high accuracy.

[0117] Addition by replacement of a foreign object (mix-out) refers to removing the third normal image 63 and adding the foreign object in a portion of the third foreign object image 73 to which the foreign object has been added. That is, addition by replacement of a foreign object refers to superimposing the foreign object on the third normal image 63 before the addition, in a state in which the third normal image 63 is shielded by the foreign object in the portion to be added. When addition by replacement of a foreign object is performed, the proportion of the size of the portion to which the foreign object has been added in the entire third foreign object image 73 may be 10% to 60%. By increasing this proportion, the foreign object detection performance of the inference model can be improved.

[0118] Fig. 29 shows an example in which a foreign object is detected from a target image 30, which is a coffee bean, when an inference model is generated using a third foreign object image 73 obtained by adding a foreign object by replacing it with another foreign object. Fig. 29(a) shows an anomaly map obtained from the target image 30, and Fig. 29(b) shows the target image 30 with the foreign object detected from the anomaly map superimposed thereon.

[0119] When the third foreign object image 73 obtained by replacing and adding a foreign object is used, it is possible to detect foreign objects (e.g., objects with different colors, missing objects, or bent objects) with high accuracy. In this case, the inference model that is generated is obtained by restoring the occluded portion of the third normal image 63 (e.g., restoring it to a normal product without foreign objects) and learning the positional relationship of the objects captured in the image. For example, when the target image is an image similar to MVTecAD, which is a dataset used to evaluate anomaly detection methods, it is possible to detect foreign objects with high accuracy.

[0120] Even when the third foreign object image 73 is at least one of transparent addition and replacement addition, the training acquisition unit 11 may acquire the third foreign object image 73 in the same manner as the above-mentioned method. When the training acquisition unit 11 generates the third foreign object image 73, the third foreign object image 73 may be generated using a conventional technique of transparent image addition and image replacement. The third foreign object image 73 may be one of transparent addition, replacement addition, and both. The multiple third foreign object images 73 may be composed of any one of the above types, or may include multiple types of the above types.

[0121] As described above, a configuration in which the third foreign object image 73 is generated by at least one of transparent addition and replacement addition can also generate an inference model more appropriately and reliably.

[0122] Furthermore, when training an inference model, an image related to information to be output from the inference model may be an image (normal image) related to input to the inference model that has been subjected to image processing other than the above. The image processing is preset, and is, for example, any of rotation, inversion, change in pixel value (brightness distribution), gamma correction, edge enhancement, and smoothing processing. The image processing may be performed on the second foreign object image 72 or the third foreign object image 73. Furthermore, images that have been subjected to the image processing other than the above-mentioned training images 61, 62, 63, 72, and 73 may be used for training the inference model.

[0123] The inference model generated by the model generation system 10 and used by the foreign object detection system 20 may be a model that includes a neural network having multiple layers, has a structure for performing concatenation (merging) between layers, and adds an image after concatenation to an input image. For example, the inference model is a neural network having the above configuration. The neural network is shown in FIG. 30.

[0124] As shown in FIG. 30, the neural network includes an encoder that encodes an input image into features, and a decoder that decodes the features output from the encoder into an image. The encoder has a plurality of layers (Conv2D Layer, Activation Layer) in which two-dimensional convolution and activation are performed. Pooling is performed between the plurality of layers of the encoder. The decoder has a plurality of layers (Conv2D Layer, Activation Layer) in which two-dimensional convolution and activation are performed. Unpooling is performed between the plurality of layers of the decoder. The encoder layer is concatenated (Concat) with a decoder layer of the same size as the encoder layer (Concatenation Layer). The two concatenated layers are not adjacent to each other. In this way, the neural network is a U-Net type model with a pooling layer.

[0125] The image input to the encoder is added to the image output from the last layer of the decoder where two-dimensional convolution and activation are performed, and the image obtained by the addition is output from the output layer (regression layer) of the decoder. The above image addition is the addition of pixel values ​​for each corresponding pixel.

[0126] In the above neural network, the image output from the last layer of the decoder where two-dimensional convolution and activation are performed can be an image of a foreign object contained in the input image (more precisely, an image obtained by subtracting the pixel value of a pixel related to a foreign object, which can be removed by adding it to the input image). With this structure, the inference model can be made to output an image from which the foreign object portion has been appropriately removed. As a result, foreign objects can be detected with high accuracy.

[0127] The model generation system 10 may generate a second inference model by performing new training based on the above-mentioned inference model. That is, the model generation system 10 may perform two-stage training, namely, training to generate an inference model and then training to generate a second inference model. The foreign object detection system 20 may detect a foreign object contained in an image showing an object by using the second inference model generated by the model generation system 10 instead of the above-mentioned inference model. In the following description, when simply referring to an inference model, it refers to the inference model of the above-mentioned embodiment (the inference model generated in the first stage).

[0128] In this case, the model generation unit 12 performs new training to generate a new second inference model in which a part for outputting information indicating the degree of foreign matter for each position of the image input to the inference model is added to the output side of the generated inference model. That is, the model generation unit 12 performs transfer learning based on the inference model to generate the second inference model.

[0129] The second inference model is a model (discrimination model, classification model) that inputs an image of a target for foreign object detection and outputs (infers) information indicating the degree of a foreign object at each position of the input image. For example, the second inference model outputs a value of the probability (probability of pass / fail, class classification value) that the pixel is related to a foreign object for each pixel of the target image 30 of the target for foreign object detection. In this case, the second inference model may output a value of 0 to 1 for each pixel as the probability. The closer the output value is to 1, the higher the degree to which the pixel is related to a foreign object, and the closer the output value is to 0, the lower the degree to which the pixel is related to a foreign object.

[0130] In foreign object detection using an inference model, foreign objects are detected by taking the difference between the image input to the inference model and the image output from the inference model. In foreign object detection using the second inference model, it is not necessary to take the difference between images as in foreign object detection using an inference model.

[0131] 31 is a schematic diagram showing an example of the second inference model of the present embodiment. For example, the second inference model is a neural network. The second inference model is generated by adding a new layer to the output side of the inference model, which is a neural network, and performing new training.

[0132] In the example shown in FIG. 31, the inference model is a neural network including an encoder that encodes an input image into a feature and a decoder that decodes the feature output from the encoder into an image. The encoder has a plurality of layers (Conv2D Layer, Activation Layer) in which two-dimensional convolution and activation are performed. Pooling is performed between the plurality of layers of the encoder. The decoder has a plurality of layers (Conv2D Layer, Activation Layer) in which two-dimensional convolution and activation are performed. Unpooling is performed between the plurality of layers of the decoder. The layer of the encoder is concatenated (Concat) with a layer of the decoder having the same size as the layer of the encoder (Concatenation Layer). The two concatenated layers are not adjacent to each other. The last layer of the plurality of layers in which two-dimensional convolution and activation are performed in the decoder is the output layer of the inference model. Note that the inference model used for the second inference model does not necessarily have to be the one shown in FIG. 31, and may be any one that can configure the second inference model.

[0133] The parts added in the second inference model are multiple layers of a neural network. For example, as shown in FIG. 31, three layers are added in which adjacent layers are connected to each other. The first layer from the inference model side is a layer (Conv+Relu, Conv2D Layer, Activation Layer) in which convolution and calculations using the Relu function are performed. This layer is connected to the output layer of the first inference model. The second layer is a layer (softmax) in which calculations using the softmax function are performed. The third layer is an output layer (Pixel Classification) that outputs the above probability values.

[0134] The input layer of the second inference model is the same as the input layer of the inference model. In the second inference model, the output layer is provided with neurons for outputting information indicating the degree of a foreign object at each position of the image related to the information input to the input layer. For example, the information output from the inference model is the probability value of whether each pixel of the image is related to a foreign object as described above. In this case, the output layer is provided with neurons equal to the number of pixels of the image, and each neuron outputs the probability value of the corresponding pixel. Note that the information output from the inference model may be other than the probability value of each pixel as long as it is information indicating the degree of a foreign object at each position of the image described above.

[0135] In addition, the second inference model may be configured by something other than a neural network, so long as it is generated by machine learning training and performs the above-mentioned input and output. The second inference model is also expected to be used as a program module that is part of artificial intelligence software, similar to the first inference model.

[0136] The second inference model may be based on the type of object, similar to the inference model. In this case, the second inference model may be treated similarly to the inference model in this respect. Furthermore, the second inference model may be treated similarly to the inference model in respects that can be similar in other respects as well.

[0137] Next, the functions of the model generation system 10 and the foreign object detection system 20 related to the second inference model will be described. The training acquisition unit 11 also acquires information for training the second inference model. The information for training the second inference model is a combination of an image for training the second inference model and information indicating the degree of a foreign object at each position of the image. FIG. 32 shows an example of an image 91 for training the second inference model and information 92 indicating the degree of a foreign object at each position of the image 91. The image 91 for training the second inference model can be at least any one of the first normal image 61, the second foreign object image 72, and the third foreign object image 73 described above. Therefore, the training acquisition unit 11 does not need to acquire the image 91 for training the second inference model separately from the image for training the inference model. However, the training acquisition unit 11 may acquire the image 91 for training the second inference model separately from the image for training the inference model. In this case, the training acquisition unit 11 may acquire the image 91 in a manner similar to the method of acquiring the image for training the inference model.

[0138] The information 92 indicating the degree of a foreign object for each position of the image 91 for training the second inference model, which is a part of the information for training the second inference model, is, for example, a value indicating whether or not each pixel of the image 91 for training the second inference model is related to a foreign object. As in the above example, when the output from the second inference model is a value between 0 and 1, and the closer the output value is to 1, the higher the degree to which the pixel is related to a foreign object, the value of the information 92 is 1 if the pixel is related to a foreign object, and 0 if the pixel is not related to a foreign object. The information 92 in FIG. 32 indicates a value for each image 91 for training the second inference model, where white parts are 1 (i.e., white parts are parts of foreign objects) and black parts are 0 (i.e., black parts are parts that are not foreign objects). Note that the value of the information 92 does not necessarily have to be the above, and may be any value that corresponds to the output from the second inference model.

[0139] The training acquisition unit 11 may generate and acquire the above information 92. For example, in the case of a training image 91 for the second inference model, a first normal image 61, a second foreign object image 72, and a third foreign object image 73, the training acquisition unit 11 generates information in which the foreign object portion is set to 1 and the non-foreign object portion is set to 0 for each pixel of these images 61, 72, and 73 as the above information 92. The foreign object portion is set to, for example, a portion of an image to be added to the normal image as a foreign object. Alternatively, a foreign object portion of an image to be added to the normal image as a foreign object may be detected by an existing detection technology, and the detected portion may be set to the foreign object portion in the above information 92. That is, the above information 92 may be acquired without annotation by a user, that is, annotation-free. The training acquisition unit 11 may also acquire the above information 92 by accepting the above information 92 input to the model generation system 10 by a user of the model generation system 10.

[0140] The training acquisition unit 11 acquires a sufficient number of pieces of training information for the second inference model to enable the model generation unit 12 to appropriately train the second estimation model. The training acquisition unit 11 outputs the acquired training information for the second inference model to the model generation unit 12.

[0141] The model generation unit 12 generates the second inference model, for example, as follows. The model generation unit 12 inputs training information for the second inference model from the training acquisition unit 11. The model generation unit 12 performs training for generating the second inference model for each of the above combinations of training information for the second inference model. The training for generating the second inference model is performed after the first inference model is generated by training.

[0142] In the following, an example will be described in which the second inference model inputs an image itself and outputs the above-mentioned probability value. As shown in Fig. 32, the model generation unit 12 performs training by inputting an image 91 for training the second inference model to the second inference model and outputting information 92 of the probability value corresponding to the image 91 from the second inference model.

[0143] When training the second inference model, the encoder part of the inference model in the second inference model is not updated by training. That is, the first encoder part of the second inference model is updated only in the training of the inference model (training in the first stage), and the learning rate in the training of the second inference model (training in the second stage) is set to 0. When training in the second stage, the decoder part of the inference model in the second inference model is set to have a lower learning rate than the added part. For example, the learning rate of the decoder part is set to 1 / 100 of the learning rate of the added part. In addition, for example, the cross entropy error is used as the loss function during learning. The training of the second inference model may be performed in a manner other than the above. Each of the above trainings themselves, that is, the update of the parameters of the second inference model, may be performed in the same manner as conventional machine learning training.

[0144] If the second inference model inputs information based on an image other than the image itself, the model generation unit 12 generates information based on image 91 from image 91 corresponding to the input to the second inference model, and performs training using the generated information as input to the inference model.

[0145] The model generation unit 12 generates the second inference model, for example, by using all of the training information for the second inference model input from the training acquisition unit 11 for training. Alternatively, the model generation unit 12 may perform training until a preset condition for ending training other than the above is satisfied to generate the second inference model. The generated second inference model is used in the foreign object detection system 20. The model generation unit 12 outputs the generated second inference model. Input and output of the second inference model may be performed in the same manner as the input and output of the above inference model. Furthermore, when the second inference model is used to detect a foreign object, output of the inference model is not necessary.

[0146] When the second inference model is used, foreign object detection in foreign object detection system 20 is performed as follows. Acquisition unit for detection 21 acquires target image 30 which is to be detected for foreign objects. Acquisition unit for detection 21 acquires target image 30 in the same manner as when the inference model is used, and outputs it to calculation unit 22.

[0147] The calculation unit 22 inputs information based on the target image 30 acquired by the acquisition unit for detection 21 into a second inference model, performs calculations, and obtains output from the second inference model. The calculation unit 22 inputs and stores the second inference model generated by the model generation system 10. The calculation unit 22 inputs the target image 30 from the acquisition unit for detection 21.

[0148] The calculation unit 22 inputs the information based on the input target image 30 into the stored second inference model and performs calculation to obtain an output from the second inference model. The information input to the second inference model corresponds to the second inference model, and is, for example, the target image 30 itself as described above. The information input to the second inference model may also be information based on the target image 30 other than the target image 30 itself. In that case, the calculation unit 22 generates information to be input to the second inference model from the target image 30. The information output from the second inference model corresponds to the second inference model, and is, for example, a probability value (class map) for each pixel of the target image 30 as described above. The information output from the second inference model may also be information indicating the degree of a foreign object for each position of the target image 30 other than the above. The calculation unit 22 outputs the information output from the second inference model to the detection unit 23.

[0149] The detection unit 23 detects a foreign object contained in the target image 30 from the output from the second inference model obtained by the calculation unit 22. The detection unit 23 detects a foreign object, for example, as follows. The detection unit 23 inputs information indicating the degree of a foreign object at each position of the target image 30, which is an output from the second inference model, for example, a probability value for each pixel of the target image 30, from the calculation unit 22. The detection unit 23 stores a criterion for detecting a foreign object, for example, a threshold value for detection (for example, 0.5), in advance. The detection unit 23 compares the probability value, which is an output from the second inference model, with the threshold value for each pixel of the target image 30. For a pixel whose probability is equal to or greater than the threshold value, the detection unit 23 determines that the part of the pixel is a foreign object (a foreign object is captured in the part of the pixel). For a pixel whose probability is not equal to or greater than the threshold value, the detection unit 23 determines that the part of the pixel is not a foreign object (no foreign object is captured in the part of the pixel and the pixel is normal). Furthermore, the detection unit 23 may detect a foreign object in the target image 30 using a method other than the above, as long as the detection unit 23 detects a foreign object in the target image 30 using the output from the second inference model obtained by the calculation unit 22.

[0150] The detection unit 23 outputs information indicating the detection result. The information indicating the detection result may be output in the same manner as described above. FIG. 33(a) shows an example of the output (class map) from the second inference model. FIG. 33(b) shows a target image on which foreign objects detected using this output are superimposed. In FIG. 33, the multiple circular parts lined up horizontally are the foreign objects.

[0151] By generating the second inference model as described above and using it to detect a foreign object, a foreign object can be easily and reliably detected. Moreover, the information output from the second inference model has smaller variations depending on various conditions and objects (samples) related to the target image 30 than, for example, the above-mentioned anomaly map. Therefore, the criterion (for example, the above-mentioned threshold value) used when detecting a foreign object does not need to be made depending on various conditions and objects related to the target image 30, and can be easily set. Therefore, by using the second inference model, a foreign object can be stably and appropriately detected even if a uniform criterion is used regardless of various conditions and objects related to the target image 30.

[0152] The model generation method, model generation system, model generation program, foreign object detection method, foreign object detection system, foreign object detection program, and inference model disclosed herein have the following configuration. [1] A model generation method for generating an inference model used to detect foreign objects contained in an image of an object, comprising: a training acquisition step of acquiring, as training images, a first normal image showing a training object, a second normal image showing the training object, a second foreign object image in which a foreign object expected to be a detection target is added to the second normal image, a third normal image showing the training object, and a third foreign object image in which a foreign object not expected to be a detection target is added to the third normal image; A model generation step of generating the inference model by training using training images acquired in the training acquisition step, The model generation method includes training in which information based on the first normal image is used as input to the inference model and information based on the first normal image is used as output from the inference model, training in which information based on the second foreign object image is used as input to the inference model and information based on the second normal image is used as output from the inference model, and training in which information based on the third foreign object image is used as input to the inference model and information based on the third normal image is used as output from the inference model. [2] The model generation method according to [1], wherein the foreign object not anticipated as a detection target is a natural image. [3] the second normal image and the third normal image are the first normal image; The model generation method according to [1] or [2], wherein in the training acquisition step, a foreign object expected to be a detection target is added to the first normal image to generate and acquire the second foreign object image, and a foreign object not expected to be a detection target is added to the first normal image to generate and acquire the third foreign object image. [4] The object is a specific type of object, The model generation method according to any one of [1] to [3], wherein the first normal image, the second normal image, the second foreign object image, the third normal image and the third foreign object image acquired in the training acquisition step are images in which the specific type of object is captured as the training object. [5] A model generation method according to any one of [1] to [4], wherein a ratio of the number of first normal images acquired in the training acquisition step, the number of combinations of the second normal images and the second foreign object images, and the number of combinations of the third normal images and the third foreign object images is a preset ratio. [6] The model generation method according to any one of [1] to [5], wherein the foreign object not anticipated as a detection target is an image drawn based on a formula. [7] A model generation method according to any one of [1] to [6], wherein the third foreign object image is an image in which a foreign object not expected to be detected is added to the third normal image by at least one of transparent addition and replacement addition. [8] A model generation method according to any one of [1] to [7], wherein the inference model includes a neural network having multiple layers, has a structure for connecting layers, and adds together the image after the connections with the input image. [9] A model generation method described in any of [1] to [8], in which in the model generation step, a new second inference model is generated by performing new training, in which a part that outputs information indicating the degree of foreign matter for each position of the image input to the inference model is added to the output side of the generated inference model.

[10] A model generation system that generates an inference model used to detect foreign objects contained in an image of an object, comprising: an acquisition means for acquiring, as training images, a first normal image showing a training object, a second normal image showing the training object, a second foreign object image in which a foreign object assumed to be a detection target is added to the second normal image, a third normal image showing the training object, and a third foreign object image in which a foreign object not assumed to be a detection target is added to the third normal image; A model generation means for generating the inference model by performing training using training images acquired by the acquisition means, The training includes training in which information based on the first normal image is used as input to the inference model and information based on the first normal image is used as output from the inference model, training in which information based on the second foreign object image is used as input to the inference model and information based on the second normal image is used as output from the inference model, and training in which information based on the third foreign object image is used as input to the inference model and information based on the third normal image is used as output from the inference model.

[11] A model generation program that causes a computer to operate as a model generation system that generates an inference model used to detect foreign objects contained in an image of an object, comprising: The computer, an acquisition means for acquiring, as training images, a first normal image showing a training object, a second normal image showing the training object, a second foreign object image in which a foreign object assumed to be a detection target is added to the second normal image, a third normal image showing the training object, and a third foreign object image in which a foreign object not assumed to be a detection target is added to the third normal image; and a model generation means for generating the inference model by performing training using training images acquired by the acquisition means. The training includes training in which information based on the first normal image is used as input to the inference model and information based on the first normal image is used as output from the inference model, training in which information based on the second foreign object image is used as input to the inference model and information based on the second normal image is used as output from the inference model, and training in which information based on the third foreign object image is used as input to the inference model and information based on the third normal image is used as output from the inference model.

[12] A foreign object detection method for detecting a foreign object contained in an image showing an object by using the inference model generated by the model generation method according to any one of [1] to [8], a detection acquisition step for acquiring a target image that is a target for detecting a foreign object; a calculation step of inputting information based on the target image acquired in the detection acquisition step into the inference model, performing a calculation, and obtaining an output from the inference model; a detection step of calculating a difference between information related to an input to the inference model in the calculation step and information related to an output from the inference model, and detecting a foreign object contained in the target image from the calculated difference; A foreign object detection method comprising:

[13] A foreign object detection method for detecting a foreign object contained in an image showing an object by using the second inference model generated by the model generation method described in [9], a detection acquisition step for acquiring a target image that is a target for detecting a foreign object; a calculation step of inputting information based on the target image acquired in the detection acquisition step into the second inference model, performing a calculation, and obtaining an output from the second inference model; a detection step of detecting a foreign object contained in the target image from the output from the second inference model obtained in the calculation step; A foreign object detection method comprising:

[14] A foreign object detection system that detects foreign objects contained in an image of an object by using the inference model generated by the model generation method according to any one of [1] to [8], A detection acquisition means for acquiring an object image that is a target for detecting a foreign object; a calculation means for inputting information based on the target image acquired by the detection acquisition means into the inference model, performing a calculation, and obtaining an output from the inference model; a detection means for calculating a difference between information related to an input to the inference model by the calculation means and information related to an output from the inference model, and detecting a foreign object contained in the target image from the calculated difference; A foreign object detection system comprising:

[15] A foreign object detection system that detects foreign objects contained in an image of an object by using the second inference model generated by the model generation method described in [9], A detection acquisition means for acquiring an object image that is a target for detecting a foreign object; a calculation means for inputting information based on the target image acquired by the detection acquisition means into the second inference model, performing a calculation, and obtaining an output from the second inference model; a detection means for detecting a foreign object contained in the target image from an output from the second inference model obtained by the calculation means; A foreign object detection system comprising:

[16] A foreign object detection program that causes a computer to operate as a foreign object detection system that detects foreign objects contained in an image showing an object by using the inference model generated by the model generation method according to any one of [1] to [8], The computer, A detection acquisition means for acquiring an object image that is a target for detecting a foreign object; a calculation means for inputting information based on the target image acquired by the detection acquisition means into the inference model, performing a calculation, and obtaining an output from the inference model; a detection means for calculating a difference between information related to an input to the inference model by the calculation means and information related to an output from the inference model, and detecting a foreign object contained in the target image from the calculated difference; A foreign object detection program that acts as a

[17] A foreign object detection program that causes a computer to operate as a foreign object detection system that detects foreign objects contained in an image showing an object by using the inference model generated by the model generation method described in [9], The computer, A detection acquisition means for acquiring an object image that is a target for detecting a foreign object; a calculation means for inputting information based on the target image acquired by the detection acquisition means into the second inference model, performing a calculation, and obtaining an output from the second inference model; a detection means for detecting a foreign object contained in the target image from an output from the second inference model obtained by the calculation means; A foreign object detection program that acts as a

[18] An inference model for causing a computer to receive image-based information, perform calculations according to the input, and output information, comprising: An inference model generated by a model generation method described in any one of [1] to [8]. [Explanation of symbols]

[0153] 10...model generation system, 11...training acquisition unit, 12...model generation unit, 20...foreign object detection system, 21...detection acquisition unit, 22...calculation unit, 23...detection unit, 100...model generation program, 101...training acquisition module, 102...model generation module, 110...recording medium, 111...program storage area, 201...detection acquisition module, 202...calculation module, 203...detection module, 200...foreign object detection program, 210...recording medium, 211...program storage area.

Claims

1. A model generation method for generating an inference model used to detect foreign objects contained in an image of an object, A training image acquisition step involves acquiring a first normal image containing the training target, a second normal image containing the training target, a second foreign object image in which a foreign object assumed to be the target of detection is added to the second normal image, a third normal image containing the training target, and a third foreign object image in which a foreign object not assumed to be the target of detection is added to the third normal image, as training images. The model generation step includes generating the inference model by performing training using the training images acquired in the training acquisition step, The image showing the aforementioned object is an X-ray image. A model generation method in which the third foreign object image is an image in which a foreign object not assumed to be the target of detection is added to the third normal image by at least one of transparent addition and replacement.

2. The model generation method according to claim 1, wherein foreign objects not assumed to be the target of detection are images drawn based on a calculation formula.

3. The model generation method according to claim 1 or 2, wherein in the third foreign object image, the position of the portion to which the foreign object not intended as a detection target is attached is irregular, and a plurality of such foreign objects of different sizes are attached.

4. The model generation method according to claim 1 or 2, wherein foreign objects not assumed to be the target of detection are natural images.

5. The second normal image and the third normal image are the first normal image. The model generation method according to claim 1 or 2, wherein in the training acquisition step, a foreign object assumed to be the target of detection is added to the first normal image to generate and acquire the second foreign object image, and a foreign object not assumed to be the target of detection is added to the first normal image to generate and acquire the third foreign object image.

6. The aforementioned object is of a specific type, The model generation method according to claim 1 or 2, wherein the first normal image, second normal image, second foreign object image, third normal image, and third foreign object image acquired in the training acquisition step are images in which the specific type of object is depicted as the training target object.

7. The model generation method according to claim 1 or 2, wherein the ratio of the number of first normal images acquired in the training acquisition step, the number of combinations of a second normal image and a second foreign object image, and the number of combinations of a third normal image and a third foreign object image is a predetermined ratio.

8. The model generation method according to claim 1 or 2, wherein the inference model includes a neural network having multiple layers, has a structure that connects the layers, and adds the image after the connection to the input image.

9. A model generation method for generating an inference model used for detecting foreign objects contained in an image of an object, A training image acquisition step involves acquiring a first normal image containing the training target, a second normal image containing the training target, a second foreign object image in which a foreign object assumed to be the target of detection is added to the second normal image, a third normal image containing the training target, and a third foreign object image in which a foreign object not assumed to be the target of detection is added to the third normal image, as training images. The model generation step includes generating the inference model by performing training using the training images acquired in the training acquisition step, The image showing the aforementioned object is an X-ray image. A model generation method comprising the model generation step, wherein a new second inference model is generated by performing new training on the output side of the generated inference model, which has a part added to output information indicating the degree of foreign matter in the image input to the inference model.

10. The model generation method according to claim 9, wherein the second inference model outputs a probability value for each pixel of the input image as information indicating the degree of the foreign matter, indicating whether the pixel is related to a foreign matter.

11. A method for detecting foreign objects contained in an image of an object, using the inference model generated by the model generation method described in claim 1 or 2, wherein the method detects foreign objects contained in an image of an object. A detection acquisition step to acquire a target image to be used for foreign object detection, A calculation step in which information based on the target image acquired in the detection acquisition step is input to the inference model, calculations are performed, and an output from the model is obtained; A detection step which involves calculating the difference between the information related to the input to the inference model in the calculation step and the information related to the output from the inference model, and detecting foreign objects contained in the target image from the calculated difference, A method for detecting foreign objects, including [specific examples of foreign object detection methods].

12. A method for detecting foreign objects contained in an image of an object, using the second inference model generated by the model generation method described in claim 9, wherein the method detects foreign objects contained in an image of an object, A detection acquisition step to acquire a target image to be used for foreign object detection, A calculation step in which information based on the target image acquired in the detection acquisition step is input to the second inference model, calculations are performed, and an output from the second inference model is obtained, A detection step in which foreign objects contained in the target image are detected from the output of the second inference model obtained in the calculation step, A method for detecting foreign objects, including [specific examples of foreign object detection methods].

13. A foreign object detection system that detects foreign objects contained in an image of an object using the inference model generated by the model generation method described in claim 1 or 2, A detection acquisition means for acquiring an image of a target to be detected as a foreign object, A calculation means that inputs information based on the target image acquired by the detection acquisition means into the inference model, performs calculations, and obtains an output from the model. A detection means that calculates the difference between the information input to the inference model by the calculation means and the information output from the inference model, and detects foreign objects contained in the target image from the calculated difference, A foreign object detection system equipped with the following features.

14. A foreign object detection system that detects foreign objects contained in an image of an object using the second inference model generated by the model generation method described in claim 9, A detection acquisition means for acquiring an image of a target to be detected as a foreign object, A calculation means inputs information based on the target image acquired by the detection acquisition means into the second inference model, performs calculations, and obtains an output from the second inference model. A detection means for detecting foreign objects contained in the target image from the output of the second inference model obtained by the calculation means, A foreign object detection system equipped with the following features.

15. A foreign object detection program that causes a computer to operate as a foreign object detection system that detects foreign objects contained in an image of an object using the inference model generated by the model generation method described in claim 1 or 2, The computer in question, A detection acquisition means for acquiring an image of a target to be detected as a foreign object, A calculation means that inputs information based on the target image acquired by the detection acquisition means into the inference model, performs calculations, and obtains an output from the model. A detection means that calculates the difference between the information input to the inference model by the calculation means and the information output from the inference model, and detects foreign objects contained in the target image from the calculated difference, A foreign object detection program that functions as such.

16. A foreign object detection program that causes a computer to operate as a foreign object detection system that detects foreign objects contained in an image of an object using the second inference model generated by the model generation method described in claim 9, The computer in question, A detection acquisition means for acquiring an image of a target to be detected as a foreign object, A calculation means inputs information based on the target image acquired by the detection acquisition means into the second inference model, performs calculations, and obtains an output from the second inference model. A detection means for detecting foreign objects contained in the target image from the output of the second inference model obtained by the calculation means, A foreign object detection program that functions as such.

Images