Control method, information processing system, information processing device, and program

Abstract

To facilitate acquisition of a sample image without any broken parts. [Solution] When a specific sample block image is corrupted, the computer controls the specific learned model to input specific captured block image information and generate updated specific sample block image information indicating a new specific sample block image based on the specific normal block image, and generates updated sample image information indicating a new sample image in which the corrupted specific sample block image has been modified based on the new specific sample block image indicated by the updated specific sample block image information.

Description

[Technical Field]

[0001] The present disclosure relates to a control method, an information processing system, an information processing device, and a program. [Background technology]

[0002] Inspection devices that use image processing to inspect captured objects, such as products, for abnormalities are known. Inspection devices sometimes compare captured images with sample images, which are images based on a group of non-defective product images generated by inputting information indicating the captured images into an image generation artificial intelligence (AI) that has learned from a group of non-defective product images. In this case, a phenomenon known as mode collapse may occur, in which a portion of the sample image is corrupted. When an inspection is performed using a sample image with a corrupted portion, the corrupted portion may cause overdetection or false detection. Therefore, in order for an inspection device to accurately perform an inspection using a sample image, if a corrupted portion is found in the sample image, it is necessary to identify and correct the corrupted portion, i.e., to perform so-called retouching.

[0003] Although it does not identify or correct defects in a sample image, Patent Document 1 discloses an image inspection device that adopts a normal image of an inspection object that has been determined to be normal in advance as a reference image, creates an upper limit threshold image by adding a preset upper threshold to the brightness value of each pixel of the reference image, and creates a lower limit threshold image by adding a preset lower threshold to the brightness value of each pixel of the reference image, and compares the inspection image obtained by photographing the inspection object with the upper limit threshold image and the lower limit threshold image on a pixel-by-pixel basis to determine whether the inspection object is good or bad. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-091360 Summary of the Invention [Problem to be solved by the invention]

[0005] The image inspection device described in Patent Document 1 may be able to identify defects in the sample image if the inspection image is replaced from the captured image with a sample image and the locations where defects have been over-detected or erroneously detected are identified. However, the image inspection device described in Patent Document 1 does not disclose or suggest a configuration for correcting defects in the sample image, and since the inspection image is compared with each threshold image pixel by pixel and defects in the sample image are also identified pixel by pixel, there is a problem in that it is difficult to properly correct all of the defects.

[0006] The present disclosure has been made in consideration of the above-described circumstances, and aims to make it easier to obtain a sample image without any defects. [Means for solving the problem]

[0007] In order to achieve the above object, a control method according to a first aspect of the present disclosure includes an information acquisition step in which a computer acquires captured image information that is information indicating a captured image that is an image of an image capturing object; a first information control step in which the computer inputs the captured image information acquired in the information acquisition step into a trained model to generate sample image information that is information indicating a sample image based on a normal image that is an image capturing image of a normal image capturing object; an image division step in which the computer divides the image into block images each composed of a predetermined number of pixels; and a failure judgment step in which the computer determines whether a specific sample block image is corrupted based on specific sample block image information that is information indicating a specific sample block image that is a block image of a specific location of the sample image. The method includes a determination step, a second information control step in which, if the specific sample block image is corrupted, the computer controls the specific learned model to input specific captured block image information, which is information indicating a block image of a specific location in the captured image indicated by the captured image information acquired in the information acquisition step, to generate updated specific sample block image information, which is information indicating a new specific sample block image based on a specific normal block image, which is a block image of a specific location in a normal image; and an information generation step in which, if the specific sample block image is corrupted, the computer generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the update specific sample block image information.

[0008] Furthermore, in order to achieve the above object, a control method according to a second aspect of the present disclosure includes an information acquiring step in which a computer acquires captured image information that is information indicating a captured image that is an image of an imaging object; an image dividing step in which the computer divides the image into a first block image consisting of a predetermined number of pixels and a second block image that is different from the first block image; and a first normal block image in which the computer inputs first captured block image information that is information indicating the first captured block image that is the first block image of the captured image indicated by the captured image information acquired in the information acquiring step to a first trained model, based on a first normal block image that is the first block image of a normal image that is an imaging object. and an information control step of inputting second captured block image information, which is information indicating a second captured block image that is a second block image of the captured image indicated by the captured image information acquired in the information acquisition step, to the second trained model to generate second sample block image information, which is information indicating a second sample block image based on a second normal block image that is a second block image of a normal image; and an information generation step of generating sample image information, which is information indicating a sample image based on a normal image, based on the first sample block image information and the second sample block image information. [Effects of the Invention]

[0009] According to the present disclosure, the computer generates updated sample image information indicating a new sample image obtained by modifying a corrupted specific sample block image based on a new specific sample block image indicated by the updated specific sample block image information, or generates sample image information indicating a sample image based on a normal image based on the first sample block image information and the second sample block image information generated by each trained model. As a result, the control method according to the present disclosure makes it easier to obtain a sample image without any defects than a control method that does not perform these preprocessing steps. [Brief explanation of the drawings]

[0010] [Figure 1]FIG. 1 is an explanatory diagram of an information processing system according to a first embodiment; [Figure 2] An explanatory diagram of an inspection device according to the first embodiment [Figure 3] FIG. 1 is a diagram showing a functional configuration of an information processing system according to a first embodiment; [Figure 4] FIG. 1 is a block diagram showing the hardware configuration of an information processing device, a learning device, and a storage device according to a first embodiment. [Figure 5] 1 is an explanatory diagram of an imaging block image, a sample block image, and a normal block image according to the first embodiment; [Figure 6A] FIG. 10 is an explanatory diagram of a case where a first sample block image is determined to be corrupted before inspection according to the first embodiment. [Figure 6B] FIG. 10 is an explanatory diagram of a case where a first sample block image is determined to be corrupted during inspection according to the first embodiment. [Figure 7A] FIG. 10 is an explanatory diagram of a case where it is determined that an abnormal part of a workpiece is included in a first imaging block image during inspection according to the first embodiment. [Figure 7B] FIG. 10 is an explanatory diagram of a case where it is determined that the first imaging block image does not include an abnormal part of the workpiece during inspection according to the first embodiment. [Figure 7C] FIG. 10 is an explanatory diagram of a case where it is determined that an error has occurred in a first imaging block image during an inspection according to the first embodiment; [Figure 8] FIG. 1 is an explanatory diagram of a neural network according to a first embodiment; [Figure 9] 1 is a flowchart showing a flow of a learning process according to the first embodiment. [Figure 10] 1 is a flowchart showing the flow of pre-examination determination processing according to the first embodiment; [Figure 11A] 1 is a flowchart showing the flow of an in-examination determination process according to the first embodiment. [Figure 11B] Continuation of the flowchart of FIG. 11A [Figure 12] 1 is a flowchart showing a flow of normal image information output processing according to the first embodiment. [Figure 13] An explanatory diagram of a trained model according to the first embodiment [Figure 14]FIG. 10 is a diagram showing a functional configuration of an information processing system according to a second embodiment. [Figure 15] 10 is a flowchart showing the flow of pre-examination determination processing according to the second embodiment. [Figure 16] 10 is a flowchart showing the flow of an in-examination determination process according to the second embodiment. [Figure 17] FIG. 10 is an explanatory diagram of an information processing system according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0011] A control method, an information processing system, an information processing device, and a program according to embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the same or corresponding parts in the drawings are denoted by the same reference numerals.

[0012] [Embodiment 1] (Regarding Information Processing System 1 According to Embodiment 1) 1 and 2, an information processing system 1 according to an embodiment of the present disclosure includes an inspection device 100 that uses image processing to inspect whether or not there is an abnormality in a workpiece 2 as an example of an inspection object or an imaged object, a learning device 200 that generates a trained model (described later) through machine learning, and a storage device 300 that stores information. The inspection device 100, the learning device 200, and the storage device 300 are capable of transmitting and receiving information via, for example, a wireless local area network (LAN), which is an example of a communication network.

[0013] The workpiece 2 is an electronic circuit board, such as a computer motherboard or mainboard. As shown in Fig. 2, electronic components such as a connection terminal 21, a CPU (Central Processing Unit) 22, a memory 23, a connector 24, a port 25, an electrolytic capacitor 26, and a film capacitor 27 are attached to the workpiece 2. In this embodiment, for example, the electrolytic capacitor 26 is a glossy component.

[0014] In the information processing system 1 according to this embodiment, the inspection device 100 stores in advance in the storage device 300 normal image information, which is information indicating normal images, which are images of good workpieces, that is, workpieces 2 determined to be normal by inspection, among captured images, which are images of workpieces 2. The learning device 200 acquires all normal image information stored in the storage device 300, i.e., information on a group of good workpiece images, as learning information, generates a trained model by machine learning using the training information, and stores the trained model in the storage device 300. The inspection device 100 acquires the trained model stored in the storage device 300, inputs captured image information indicating a captured image of the workpiece 2 that has become a new inspection target into the trained model, and generates sample image information indicating a sample image based on the normal image. The inspection device 100 then uses the sample image information to inspect the workpiece 2 for abnormalities.

[0015] Since there is a possibility that some parts of the sample image indicated by the sample image information generated by the trained model may be corrupted, the inspection device 100 identifies the corrupted parts and corrects them, i.e., retouches them, to prevent the corrupted parts from causing overdetection or erroneous detection. Furthermore, in order to reduce the variation in the sample images indicated by the sample image information generated by the captured images input to the trained model, the inspection device 100 preprocesses the learning information used by the learning device 200 for machine learning before storing it in the storage device 300.

[0016] (Regarding the inspection device 100 according to the first embodiment) Inspection device 100 includes a flat base 101 arranged on an XY plane, which is a horizontal plane. Inspection device 100 also includes a Y-axis actuator 102 as an example of a Y-axis moving member fixed to the upper surface of base 101 and extending in the Y-axis direction, which is the front-to-back direction on the XY plane, and a Z-axis actuator 103 as an example of a Z-axis moving member extending in the Z-axis direction, which is the up-down direction perpendicular to the XY plane. Inspection device 100 also includes an X-axis actuator 104 as an example of an X-axis moving member supported by Z-axis actuator 103 so as to be movable in the Z-axis direction and extending in the X-axis direction, which is the left-to-right direction on the XY plane.

[0017] The Y-axis actuator 102 includes a flat conveyance table 105 that is supported so as to be movable in the Y-axis direction and has the workpiece 2 fixed on its upper surface. The inspection device 100 also includes an observation camera 106 and a laser displacement meter 107 that are supported so as to be movable in the X-axis direction by the X-axis actuator 104. The observation camera 106 includes a camera 108 as an example of an imaging member, a lens 109 as an example of an optical system, and an illumination member 110.

[0018] The inspection device 100 also includes a displacement meter controller 111 as an example of a displacement measurement control unit that is connected to the laser displacement meter 107 and controls the measurement of the displacement between the workpiece 2 and the observation camera 106 by the laser displacement meter 107. The inspection device 100 also includes a lighting controller 112 as an example of an illumination light output control unit that is connected to the illumination member 110 and controls the output of illumination light by the illumination member 110. The inspection device 100 also includes an information processing device 113 as an example of an inference device that is connected to the camera 108 and performs information processing based on an image captured by the camera 108.

[0019] The inspection device 100 also includes a programmable logic controller (hereinafter referred to as "PLC") 114 as an example of an inspection device main body that is connected to a displacement meter controller 111, a lighting controller 112, and an information processing device 113 and performs overall control of the inspection device 100. The inspection device 100 also includes a GOT (Graphic Operation Terminal) 115 as an example of a display that is connected to the PLC 114 and displays information output from the PLC 114.

[0020] (Regarding information processing device 113 according to embodiment 1) The information processing device 113 is, for example, a computer device such as a personal computer (PC) used by a user of the inspection device. As shown in Fig. 3, the information processing device 113 includes an information acquisition unit 121 that acquires information and an information control unit 122 that controls information processing. The information control unit 122 includes a first information control unit 123 and a second information control unit 124. The information processing device 113 also includes an image division unit 125 that divides an image, a feature amount calculation unit 126 that calculates a feature amount of the image, a failure determination unit 127 that determines whether the image is failure or not, an abnormality determination unit 128 that determines an abnormality in the image, an information generation unit 129 that generates information, a gloss determination unit 130 that determines whether the image is glossy or not, and an information output unit 131 that outputs information.

[0021] (Regarding the learning device 200 according to the first embodiment) The learning device 200 is, for example, a computer equipped with a GPU (Graphics Processing Unit), a so-called GPU machine. Note that the learning device 200 is not limited to a GPU machine and may be any other computer capable of performing calculations for generating a trained model through machine learning using training information. For example, the learning device 200 may be a personal computer equipped with a CPU, a supercomputer, a workstation, or the like. The learning device 200 includes a training information acquisition unit 210 that acquires training information, a trained model generation unit 220 that generates a trained model, and a trained model output unit 230 that outputs the trained model.

[0022] (Regarding the storage device 300 according to the first embodiment) The storage device 300 is, for example, a computer equipped with database management software, i.e., a so-called database server. The storage device 300 includes a training information storage unit 310 that stores training information, a trained model storage unit 320 that stores trained models, and an information transmission / reception unit 330 that transmits and receives information.

[0023] (Hardware Configuration of Information Processing Device 113 According to Embodiment 1) 4, the information processing device 113 includes a control unit 51 that executes processing in accordance with a control program 59. The control unit 51 includes an arithmetic processing unit such as a CPU or a GPU. In accordance with the control program 59, the control unit 51 functions as an information control unit 122, a first information control unit 123, a second information control unit 124, an image dividing unit 125, a feature amount calculation unit 126, a failure determination unit 127, an abnormality determination unit 128, an information generation unit 129, and a gloss determination unit 130 shown in FIG.

[0024] 4, information processing device 113 includes main memory 52 into which control program 59 is loaded and which is used as a work area for control unit 51. Main memory 52 includes a volatile memory such as RAM (Random Access Memory).

[0025] Information processing device 113 also includes external storage unit 53 that stores control program 59 in advance. External storage unit 53 supplies information stored by this program to control unit 51 in accordance with instructions from control unit 51, and stores data supplied from control unit 51. External storage unit 53 includes non-volatile memory such as a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0026] Information processing device 113 also includes an operation unit 54 that is operated by a user. Information input via operation unit 54 is supplied to control unit 51. Operation unit 54 includes information input components such as a keyboard, a mouse, and a touch panel.

[0027] Furthermore, information processing device 113 includes display unit 55 that displays information input via operation unit 54 and information output by control unit 51. Display unit 55 includes a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0028] Information processing device 113 also includes a transmitting / receiving unit 56 that transmits and receives information. Transmitting / receiving unit 56 includes information communication components such as a communication network terminal device that connects to a network and a wireless communication device. Transmitting / receiving unit 56 functions as information acquisition unit 121 and information output unit 131 shown in FIG. 3.

[0029] Returning to FIG. 4, in the information processing device 113, the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55 and the transmission / reception unit 56 are all connected to the control unit 51 via the internal bus 50.

[0030] 3 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the information processing device 113 executes an information acquisition step performed by the information acquisition unit 121, an information control step performed by the information control unit 122, a first information control step performed by the first information control unit 123, and a second information control step performed by the second information control unit 124. Furthermore, for example, the information processing device 113 executes an image division step performed by the image division unit 125, a feature amount calculation step performed by the feature amount calculation unit 126, a failure determination step performed by the failure determination unit 127, and an abnormality determination step performed by the abnormality determination unit 128. Furthermore, for example, the information processing device 113 executes an information generation step performed by the information generation unit 129, a gloss determination step performed by the gloss determination unit 130, and an information output step performed by the information output unit 131.

[0031] (Hardware Configuration of Learning Device 200 According to Embodiment 1) 4, like the information processing device 113, the learning device 200 also includes a control unit 51, a main memory unit 52, an external memory unit 53, an operation unit 54, a display unit 55, and a transmission / reception unit 56. The control unit 51 functions as the trained model generation unit 220 shown in FIG. 3 in accordance with a control program 59, and the transmission / reception unit 56 functions as the training information acquisition unit 210 and the trained model output unit 230. Returning to FIG. 4, the learning device 200 realizes the functions of the above-mentioned units 210 to 230 shown in FIG. 3 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the learning device 200 executes a training information acquisition unit step performed by the training information acquisition unit 210, a trained model generation step performed by the trained model generation unit 220, and a trained model output step performed by the trained model output unit 230.

[0032] (Hardware Configuration of Storage Device 300 According to First Embodiment) 4, like the information processing device 113 and the learning device 200, the storage device 300 also includes a control unit 51, a main memory unit 52, an external memory unit 53, an operation unit 54, a display unit 55, and a transmission / reception unit 56. The external memory unit 53 functions as the learning information storage unit 310 and the trained model storage unit 320 shown in FIG. 3, and the transmission / reception unit 56 functions as the information transmission / reception unit 330. Returning to FIG. 4, the storage device 300 realizes the functions of the above-mentioned units 310 to 330 shown in FIG. 3 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the storage device 300 executes a learning information storage step performed by the learning information storage unit 310, a trained model storage step performed by the trained model storage unit 320, and an information transmission / reception step performed by the information transmission / reception unit 330.

[0033] (Details of Functional Configuration of Information Processing Device 113 According to Embodiment 1) 3, the information acquiring unit 121 acquires captured image information indicating a captured image of the workpiece 2 from the camera 108. Note that the captured image information acquired by the information acquiring unit 121 becomes normal image information if it is determined that there is no abnormality as a result of an inspection described below, whereas it becomes abnormal image information, which is information indicating an abnormal image that is an image of a defective workpiece, if it is determined that there is an abnormality.

[0034] The information acquisition unit 121 also acquires the trained model by transmitting predetermined trained model request information to the storage device 300 and receiving the trained model transmitted from the storage device 300. Here, in the present embodiment, the trained model includes a normal image trained model generated by the learning device 200 through machine learning using normal image information, and a normal block image trained model generated by machine learning using normal block image information indicating normal block images into which a normal image is divided by the image segmentation unit 125 (described later). Specifically, the normal block image trained models include a first normal block image trained model as an example of a specific trained model and a first normal block image trained model generated by machine learning using first normal block image information indicating a first normal block image (described later), a second normal block image trained model as an example of a specific trained model and a second normal block image trained model generated by machine learning using second normal block image information indicating a second normal block image, ..., and a ninth normal block image trained model as an example of a specific trained model and a ninth trained model generated by machine learning using ninth normal block image information indicating a ninth normal block image.

[0035] Furthermore, when checking the normal image trained model before inspecting the work 2, the information acquisition unit 121 acquires normal image information by transmitting predetermined normal image information request information to the storage device 300 and receiving the normal image information transmitted from the storage device 300. The information acquisition unit 121 acquires any of the normal image information stored in the storage device 300, as well as template image information indicating a template image, which will be described later.

[0036] The information control unit 122 performs control to input captured image information into the learned model to generate sample image information. Specifically, the first information control unit 123 performs control to input captured image information into the normal image learned model to generate sample image information. When checking the normal image learned model before inspecting the work 2, the first information control unit 123 performs control to input normal image information acquired in advance into the normal image learned model as captured image information to generate sample image information.

[0037] The image dividing unit 125 divides an image into block images each having a predetermined number of pixels. In this embodiment, for example, if n and m are integers equal to or greater than 2 and the number of pixels of the image is (3×n)×(3×m) pixels, the image dividing unit 125 divides the image into nine equal parts, dividing the n×m pixel image into a first block image, a second block image, ..., and a ninth block image. Specifically, the image dividing unit 125 divides the captured image indicated by the captured image information into imaging block images that are block images of the captured image, and divides the sample image indicated by the sample image information into sample block images that are block images of the sample image. Thus, as shown in FIG. 5 , the captured image is divided into a first imaging block image, a second imaging block image, ..., and a ninth imaging block image as imaging block images of n×m pixels, and the sample image is divided into a first sample block image, a second sample block image, ..., and a ninth sample block image as sample block images of n×m pixels.

[0038] When checking the normal image trained model before inspecting the work 2, the image dividing unit 125 divides the sample image indicated by the sample image information into sample block images, that is, a first sample block image, a second sample block image, ..., a ninth sample block image. In this case, the image dividing unit 125 divides a template image (described later) indicated by the template image information into a first template block image, a second template block image, ..., a ninth template block image, which are block images of the template image.

[0039] Returning to Fig. 3, the feature amount calculation unit 126 calculates a feature amount of a specific imaging block image, which is a block image of a specific portion of the captured image, and a feature amount of a specific sample block image, which is a block image of a specific portion of the sample image. Note that the feature amount of the specific imaging block image is a value of the mean squared error (MSE) (hereinafter referred to as "MSE value") between the specific imaging block image and a specific template block image, which is a block image of a specific portion of a template image predetermined from a plurality of types of normal images. Also, the feature amount of the specific sample block image is the MSE value between the specific template block image and the specific sample block image.

[0040] Therefore, for example, if the block image of a specific location is the first block image, the feature calculation unit 126 calculates the MSE value between the first template block image and the first captured block image as the feature of the first captured block image, and calculates the MSE value between the first template block image and the first sample block image as the feature of the first sample block image. Here, the template image is, for example, the most average normal image selected from multiple types of normal images. Therefore, the normal image trained model generates the sample image by performing a process to bring the captured image closer to the template image. Therefore, if the captured work 2 is normal, the MSE value of the first sample block image is likely to be smaller than the MSE value of the first captured block image.

[0041] In this embodiment, the feature calculation unit 126 calculates all of the feature amounts of the first imaging block image, the feature amounts of the second imaging block image, ..., the feature amounts of the ninth imaging block image, and the feature amounts of the first sample block image, the feature amounts of the second sample block image, ..., the feature amounts of the ninth sample block image. Furthermore, when checking the normal image trained model before inspecting the work 2, the feature calculation unit 126 calculates all of the feature amounts of the first sample block image, the feature amounts of the second sample block image, ..., the feature amounts of the ninth sample block image.

[0042] The corruption determination unit 127 determines whether a specific sample block image is corrupted based on the feature amount of the specific sample block image. As shown in FIG. 6B , if the feature amount of the specific imaging block image is below a predetermined threshold while the feature amount of the specific sample block image exceeds the threshold, the corruption determination unit 127 determines that the specific sample block image is corrupted. For example, if the MSE value of the first imaging block image is below a threshold while the MSE value of the first sample block image exceeds the threshold, the corruption determination unit 127 determines that the first sample block image is corrupted. Note that in this embodiment, the first image blocks, the second image blocks, ..., the ninth image blocks are all compared between the captured image and the sample image, and it is determined whether the first sample block image, the second sample block image, ..., the ninth sample block image are all corrupted.

[0043] 6A, when checking the normal image trained model before inspecting the work 2, if the feature amount of the specific sample block image exceeds a threshold, the failure determination unit 127 determines that the specific sample block image is failed. For example, if the MSE value of the first sample block image exceeds a threshold, the failure determination unit 127 determines that the first sample block image is failed.

[0044] Returning to FIG. 3 , the anomaly determination unit 128 determines whether or not an abnormal portion of the workpiece 2 is included in the specific imaging block image based on the comparison result between the feature amount of the specific imaging block image and the feature amount of the specific sample block image. When the feature amount of the specific imaging block image exceeds a threshold value while the feature amount of the specific sample block image is equal to or less than the threshold value, the anomaly determination unit 128 determines that an abnormal portion of the workpiece 2 is included in the specific imaging block image. For example, as shown in FIG. 7A , when the MSE value of the first imaging block image exceeds a threshold value while the MSE value of the first sample block image is equal to or less than the threshold value, the anomaly determination unit 128 determines that an abnormal portion of the workpiece 2 is included in the first imaging block image. Note that in this embodiment, the first image blocks, the second image blocks, ..., the ninth image blocks are all compared between the captured image and the sample image, and it is determined whether or not an abnormal portion of the workpiece 2 is included in each of the first imaging block image, the second imaging block image, ..., and the ninth imaging block image.

[0045] Furthermore, when the feature amount of the specific imaging block image and the feature amount of the specific sample block image are both equal to or less than the threshold, the abnormality determination unit 128 determines that the specific imaging block image does not contain an abnormal portion of the workpiece 2. For example, as shown in Fig. 7B, when the MSE value of the first imaging block image and the MSE value of the first sample block image are both equal to or less than the threshold, the abnormality determination unit 128 determines that the first imaging block image does not contain an abnormal portion of the workpiece 2.

[0046] On the other hand, when the feature amount of the specific imaging block image and the feature amount of the specific sample block image both exceed the threshold, the abnormality determination unit 128 determines that an error has occurred because there is a possibility that the specific sample block image is corrupted, that the specific imaging block image contains an abnormal portion of the work 2, or both have occurred. For example, as shown in Fig. 7C, when the MSE value of the first imaging block image and the MSE value of the first sample block image both exceed the threshold, the abnormality determination unit 128 determines that an error has occurred in the first imaging block image.

[0047] 3, when the specific sample block image is corrupted, the second information control unit 124 performs control to input specific imaging block image information into the specific normal block image trained model to generate updated specific sample block image information, which is information indicating a new specific sample block image based on the specific normal block image. For example, when the first sample block image is corrupted, the second information control unit 124 performs control to input the first imaging block image information into the first normal block image trained model to generate updated first sample block image information.

[0048] If the specific sample block image is corrupted, the information generation unit 129 generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updating specific sample block image information. After replacing the corrupted specific sample block image with the new specific sample block image indicated by the updating specific sample block image information, the information generation unit 129 performs image processing to smooth the brightness gradient of the edge portion, which is the boundary portion between the new specific sample block image and other sample block images. Note that, as image processing to smooth the brightness gradient of the edge portion, for example, a noise removal process such as a median filter that matches the edge portion to a neighboring image, or an image synthesis process such as Poisson blending that combines the edge image with a mask image can be used.

[0049] For example, if the first sample block image is corrupted, the information generating unit 129 first replaces the corrupted first sample block image with a new first sample block image indicated by the updated first sample block image information. Then, the information generating unit 129 performs image processing on the new replaced first sample block image to smooth the brightness gradient at the edge portion with the second sample block image and the edge portion with the fourth sample block image, and generates updated sample image information indicating the new sample image.

[0050] When the abnormality determination unit 128 determines that none of the captured block images contains an abnormal portion of the workpiece 2 during inspection, i.e., when the captured image information is determined to be normal image information as the inspection result, the gloss determination unit 130 determines whether the normal image indicated by the normal image information is a glossy normal image or a non-glossy normal image. The gloss determination unit 130 determines whether the normal image is a glossy normal image or a non-glossy normal image by determining whether the brightness of the glossy part of the normal workpiece 2 contained in the normal image exceeds a predetermined brightness threshold. Note that, as a result of the gloss determination unit 130 determining multiple types of normal images, the information processing device 113 counts the number of glossy normal images using a glossy normal image counter and counts the number of non-glossy normal images using a non-glossy normal image counter.

[0051] The information output unit 131 outputs determination result information, which is information indicating the determination result of the failure determination unit 127. For example, when the failure determination unit 127 determines that a specific sample block image is failed regardless of whether it is before or during inspection, the information output unit 131 outputs failure detection information, which is information indicating that the normal image trained model has generated a failed specific sample block image, to the PLC 114. Furthermore, when the information generation unit 129 changes the failed specific sample block image during inspection to generate updated sample image information, the information output unit 131 outputs retouching information, which is information indicating that updated sample image information has been generated, to the PLC 114.

[0052] Furthermore, when the abnormality determination unit 128 determines that an abnormal portion of the work 2 is included in a specific imaging block image during inspection, the information output unit 131 outputs, to the PLC 114, inspection abnormality information indicating that an abnormal portion of the work 2 is included in the specific imaging block image. Furthermore, when the abnormality determination unit 128 determines that no abnormal portion of the work 2 is included in any imaging block image during inspection, the information output unit 131 outputs, to the PLC 114, inspection normality information indicating that the work 2 is normal. Furthermore, when the abnormality determination unit 128 determines that an error has occurred during inspection, the information output unit 131 outputs, to the PLC 114, error information indicating that an error has occurred. As a result, the PLC 114 can, for example, display the acquired information on the GOT 115 or output a warning sound.

[0053] The information output unit 131 also outputs the normal image information and normal block image information obtained as the inspection results, i.e., the first normal block image information, the second normal block image information, ..., the ninth normal block image information, by transmitting the normal image information and normal block image information to the storage device 300 and storing them. Here, the information output unit 131 outputs the same number of glossy normal image information, which is normal image information indicating a glossy normal image, and the same number of non-glossy normal image information, which is normal image information indicating a non-glossy normal image, to the storage device 300 based on the value of the glossy normal image counter and the value of the non-glossy normal image counter, which are the determination results by the gloss determination unit 130. The information output unit 131 also outputs the same number of normal block image information indicating each normal block image of the glossy normal image and each normal block image information indicating each normal block image of the non-glossy normal image to the storage device 300.

[0054] (Details of the Functional Configuration of Learning Device 200 According to Embodiment 1) The learning information acquisition unit 210 acquires learning information by transmitting predetermined learning information request information to the storage device 300 and receiving the learning information transmitted from the storage device 300. The learning information acquisition unit 210 acquires all normal image information and normal block image information stored in the storage device 300, i.e., the first normal block image information, second normal block image information, ..., ninth normal block image information, as learning information.

[0055] The trained model generation unit 220 generates a trained model through machine learning using the training information acquired by the trained model generation unit 220. The trained model generation unit 220 generates a normal image trained model through machine learning using normal image information as training information. The trained model generation unit 220 also generates a first normal block image trained model through machine learning using first normal block image information as training information. The trained model generation unit 220 also generates a second normal block image trained model through machine learning using second normal block image information as training information, ..., and generates a ninth normal block image trained model through machine learning using ninth normal block image information as training information.

[0056] The machine learning learning algorithm used by the trained model generation unit 220 to generate the trained model described above is, for example, a neural network, which is an example of supervised learning. Here, supervised learning refers to a technique in which a learning device is provided with data pairs of inputs and resulting labels, and the learning device learns the features of the training information and infers the results from the inputs. The neural network is composed of an input layer containing multiple neurons, an intermediate layer also called a hidden layer containing multiple neurons, and an output layer containing multiple neurons. Note that although the intermediate layer is described as one layer, it may be two or more layers.

[0057] For example, in a three-layer neural network as shown in Figure 8, when multiple inputs are input to the input layer (X1, X2, X3), the values ​​are multiplied by the first weights W1 (w11, w12, ..., w16) and input to the middle layer (Y1, Y2), and the result is further multiplied by the second weights W2 (w21, w22, ..., w26) and output from the output layer (Z1, Z2, Z3). Therefore, the output result changes depending on the values ​​of the weights W1 and W2.

[0058] In this embodiment, the neural network learns multiple types of normal images, i.e., a group of good-quality images and normal block images, i.e., each block image of the group of good-quality images, through so-called supervised learning in accordance with learning information, which is training data acquired in advance by the learning device 200. That is, the neural network adjusts the weights W1 and W2 to input normal image information to the input layer and bring the results output from the output layer closer to the normal images and normal block images included in the training data. By performing the above-described learning, the learning device 200 generates a normal-image trained model, a first normal-block-image trained model, a second normal-block-image trained model, ..., a ninth normal-block-image trained model.

[0059] The trained model output unit 230 outputs the trained models generated by the trained model generation unit 220 by transmitting them to and storing them in the storage device 300. The trained model output unit 230 outputs a normal image trained model, a first normal block image trained model, a second normal block image trained model, ..., a ninth normal block image trained model.

[0060] (Details of the Functional Configuration of the Storage Device 300 According to the First Embodiment) The information transmitting / receiving unit 330 receives the learning information transmitted from the information processing device 113, and the learning information storage unit 310 stores the received learning information. The information transmitting / receiving unit 330 receives normal image information and normal block image information, i.e., first normal block image information, second normal block image information, ..., ninth normal block image information, as learning information, and the learning information storage unit 310 stores the received normal image information and normal block image information as learning information.

[0061] Furthermore, when the information transmitting / receiving unit 330 receives request information for learning information transmitted from the learning device 200, it transmits the learning information stored in the learning information storage unit 310 to the learning device 200. The information transmitting / receiving unit 330 transmits the normal image information and normal block image information stored in the learning information storage unit 310 to the learning device 200 as learning information.

[0062] Furthermore, when the information transmitting / receiving unit 330 receives request information for normal image information transmitted from the information processing device 113, it transmits the normal image information stored in the learning information storage unit 310 to the information processing device 113. The information transmitting / receiving unit 330 transmits any of the normal image information stored in the learning information storage unit 310 and the above-mentioned template image information to the information processing device 113.

[0063] Furthermore, the information transmitting and receiving unit 330 receives the trained models transmitted from the learning device 200, and the trained model storage unit 320 stores the received trained models. The information transmitting and receiving unit 330 receives the normal image trained model and the normal block image trained model, i.e., the first normal block image trained model, the second normal block image trained model, ..., the ninth normal block image trained model, and the trained model storage unit 320 stores the received normal image trained model and normal block image trained model as trained models.

[0064] Furthermore, when the information transmitting / receiving unit 330 receives request information for a trained model transmitted from the information processing device 113, it transmits the trained model stored in the trained model storage unit 320 to the information processing device 113. The information transmitting / receiving unit 330 transmits the normal image trained model and the normal block image trained model stored in the trained model storage unit 320 to the information processing device 113 as trained models.

[0065] (Regarding the learning process according to the first embodiment) Next, the operation of the learning device 200 to generate and output a trained model will be described using a flowchart. When the learning device 200 is powered on, it starts executing the learning process shown in Fig. 9. First, the learning information acquisition unit 210 transmits request information for learning information to the storage device 300 and acquires the learning information transmitted from the storage device 300 (step S201). For example, the learning information acquisition unit 210 acquires all of the normal image information, first normal block image information, second normal block image information, ..., ninth normal block image information stored in the storage device 300 as learning information.

[0066] After acquiring the information, the trained model generation unit 220 generates a trained model through machine learning using the acquired training information (step S202). For example, the trained model generation unit 220 generates a normal image trained model through machine learning using normal image information, and generates a first normal block image trained model through machine learning using first normal block image information. Also, for example, the trained model generation unit 220 generates a second normal block image trained model through machine learning using second normal block image information, ..., and generates a ninth normal block image trained model through machine learning using ninth normal block image information.

[0067] After generating the trained models, the trained model output unit 230 outputs the trained models by transmitting the trained models generated by the trained model generation unit 220 to the storage device 300 for storage (step S203), and ends the processing. For example, the trained model output unit 230 outputs a normal image trained model, a first normal block image trained model, a second normal block image trained model, ..., a ninth normal block image trained model.

[0068] (Regarding pre-test determination processing according to embodiment 1) Next, the operation of the information processing device 113 to determine a failure in a sample image before inspection will be described using a flowchart. When the information processing device 113 is powered on, it starts executing the pre-inspection determination process shown in FIG. 10. First, the information acquisition unit 121 transmits request information for trained models to the storage device 300 and acquires the trained models from the storage device 300 (step S101). For example, the information acquisition unit 121 acquires a normal image trained model and a normal block image trained model, i.e., a first normal block image trained model, a second normal block image trained model, ..., a ninth normal block image trained model. After acquiring the trained models, the information acquisition unit 121 transmits request information for normal image information to the storage device 300 and acquires normal image information and template image information from the storage device 300 (step S102).

[0069] After acquiring the information, the first information control unit 123 inputs the normal image information as captured image information into the normal image trained model and controls the generation of sample image information (step S103). After generating the sample image information, the image division unit 125 divides the sample image indicated by the sample image information into sample block images, and divides the template image indicated by the template image information into template block images (step S104). After image division, the feature calculation unit 126 calculates the feature of the sample block image (step S105).

[0070] For example, the image dividing unit 125 divides the sample image into a first sample block image, a second sample block image, ..., a ninth sample block image, and divides the template image into a first template block image, a second template block image, ..., a ninth template block image. Then, the feature calculation unit 126 calculates the MSE value between the first template block image and the first sample block image, the MSE value between the second template block image and the second sample block image, ..., the MSE value between the ninth template block image and the ninth sample block image, thereby calculating the feature values ​​of the first sample block image, the feature values ​​of the second sample block image, ..., the ninth sample block image.

[0071] After calculating the feature amounts, the failure determination unit 127 determines whether the specific sample block image is corrupted based on the feature amounts of the specific sample block image (step S106). For example, if the MSE value of the first sample block image exceeds a threshold, the failure determination unit 127 determines that the first sample block image is corrupted. If none of the sample block images is corrupted (step S106; N), the information processing device 113 terminates the processing. On the other hand, if the specific sample block image is corrupted (step S106; Y), the information output unit 131 outputs to the PLC 114 failure detection information indicating that the normal image trained model has generated a corrupted specific sample block image (step S107), and terminates the processing.

[0072] (Regarding the In-Test Determination Process According to the First Embodiment) Next, a flowchart will be used to explain the operation of the information processing device 113 to determine a failure in a sample image during inspection and to determine an abnormality in the captured image of the work 2. After completing the pre-inspection determination process described above, the information processing device 113 starts executing the in-inspection determination process shown in Figs. 11A and 11B. As shown in Fig. 11A, first, the information acquisition unit 121 acquires captured image information indicating the captured image of the work 2 from the camera 108 (step S111), and the first information control unit 123 inputs the captured image information into the normal image trained model to generate sample image information (step S112).

[0073] After generating the sample image information, the image dividing unit 125 divides the captured image indicated by the captured image information into imaging block images, and divides the sample image indicated by the sample image information into sample block images (step S113). After image division, the feature amount calculation unit 126 calculates the feature amounts of the imaging block images and the feature amounts of the sample block images (step S114).

[0074] For example, the image dividing unit 125 divides the captured image into a first imaging block image, a second imaging block image, ..., and a ninth imaging block image. Then, the feature amount calculation unit 126 calculates the MSE value between the first template block image and the first imaging block image, the MSE value between the second template block image and the second imaging block image, ..., and the MSE value between the ninth template block image and the ninth imaging block image, thereby calculating the feature amount of the first imaging block image, the feature amount of the second imaging block image, ..., and the feature amount of the ninth imaging block image. Note that the division of the sample image and the calculation of the feature amount of the sample block image are similar to the above-mentioned pre-inspection determination processing, and therefore will not be described in order to reduce redundant explanation.

[0075] After calculating the feature amounts, the failure determination unit 127 determines whether the specific sample block image is corrupted based on the result of comparing the feature amounts of the specific imaging block image with the feature amounts of the specific sample block image (step S115). For example, if the MSE value of the first imaging block image is equal to or less than a threshold value while the MSE value of the first sample block image exceeds the threshold value, the failure determination unit 127 determines that the first sample block image is corrupted. If the specific sample block image is corrupted (step S115; Y), the second information control unit 124 controls the specific normal block image trained model to input the specific imaging block image information and generate updated specific sample block image information (step S116). For example, if the first sample block image is corrupted, the second information control unit 124 controls the first normal block image trained model to input the first imaging block image information and generate updated first sample block image information.

[0076] After generating the specific sample block image information for update, the information generating unit 129 generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the specific sample block image information for update (step S117). For example, if the first sample block image is corrupted, the information generating unit 129 generates updated sample image information indicating a new sample image obtained by replacing the corrupted first sample block image with the new first sample block image indicated by the first sample block image information for update and then performing image processing to smooth the brightness gradient of the edge portion. Hereinafter, the updated sample image information will be described as sample image information.

[0077] After generating the information, the information output unit 131 outputs to the PLC 114 failure detection information indicating that the normal image trained model has generated a specific sample block image that has failed, and retouching information indicating that updated sample image information has been generated (step S118).

[0078] 11B, after the information output, or if none of the sample block images are damaged (step S115; Y), the abnormality determination unit 128 determines whether or not the specific imaging block image contains an abnormal portion of the workpiece 2 based on a result of comparing the feature amount of the specific imaging block image with the feature amount of the specific sample block image (step S119). For example, if the MSE value of the first imaging block image exceeds a threshold while the MSE value of the first sample block image is equal to or less than the threshold, the abnormality determination unit 128 determines that the first imaging block image contains an abnormal portion of the workpiece 2. If the specific imaging block image contains an abnormal portion of the workpiece 2 (step S119; Y), the information output unit 131 outputs inspection abnormality information indicating that the specific imaging block image contains an abnormal portion of the workpiece 2 to the PLC 114 (step S120), and ends the process.

[0079] On the other hand, if none of the imaging block images contains an abnormal portion of the workpiece 2 (step S119; N), the abnormality determination unit 128 determines whether or not an error that makes it impossible to determine an abnormality has occurred (step S121). If the feature amount of the specific imaging block image and the feature amount of the specific sample block image both exceed the threshold, the abnormality determination unit 128 determines that an error has occurred. If an error has occurred (step S121; Y), the information output unit 131 outputs error information indicating that an error has occurred to the PLC 114 (step S122), and ends the processing. On the other hand, if no error has occurred (step S121; N), the information output unit 131 outputs inspection normality information indicating that the workpiece 2 is normal to the PLC 114 (step S123), and ends the processing.

[0080] (Regarding normal image information output processing according to embodiment 1) Next, an operation of the information processing device 113 to output normal image information after inspection will be described using a flowchart. When the information processing device 113 determines that multiple types of captured image information are normal image information as the inspection result as a result of the above-described during-inspection determination process, it starts executing the normal image information output process shown in FIG. 12. First, the information processing device 113 resets the values ​​of the glossy normal image counter and the non-glossy normal image counter to 0 (step S151), and the gloss determination unit 130 determines whether the normal image indicated by the normal image information is a glossy normal image (step S152). For example, the gloss determination unit 130 determines whether the normal image is a glossy normal image or a non-glossy normal image by determining whether the brightness of a glossy part of a normal workpiece 2 included in the normal image exceeds a predetermined brightness threshold.

[0081] If the image is a glossy normal image (step S152; Y), +1 is added to the value of the glossy normal image counter (step S153). If the image is a non-glossy normal image (step S152; N), +1 is added to the value of the non-glossy normal image counter (step S154). After each counter increment, it is determined whether or not all normal images indicated by the normal image information have been determined (step S155). If all normal images have not been determined (step S155; N), the process returns to step S152 and the next normal image is determined. If all normal images have been determined (step S155; Y), the information output unit 131 outputs the same number of glossy normal image information and non-glossy normal image information to the storage device 300 based on the values ​​of the glossy normal image counter and the non-glossy normal image counter, and also outputs the same number of normal block image information based on the glossy normal image information and the normal block image information based on the non-glossy normal image information to the storage device 300 (step S156), and the process ends.

[0082] As described above, according to the information processing system 1 of this embodiment, in the information processing device 113, the information acquisition unit 121 acquires captured image information indicating a captured image of the workpiece 2. Furthermore, the information control unit 122 controls the learned model to input the captured image information acquired by the information acquisition unit 121 to generate sample image information indicating a sample image based on a normal image. Furthermore, the image division unit 125 divides the image into block images, and the feature calculation unit 126 calculates the feature amounts of a specific sample block image of the sample image. Then, the failure determination unit 127 determines whether the specific sample block image is failed based on the feature amounts of the specific sample block image, and the information output unit 131 outputs determination result information indicating the determination result.

[0083] In the conventionally known information processing system 1, abnormalities in the captured image are determined for each pixel, and if this system is used to determine defects in the sample image, it is necessary to determine defects in the sample image for each pixel, which may result in the sample image not being able to be identified. Furthermore, in the conventionally known information processing system 1, it is necessary to determine defects in the sample image for each pixel, so that the positioning of the object to be imaged during image capture must be performed with high precision in order to identify defects in the sample image.

[0084] In response to this, the information processing device 113 determines whether a specific sample block image is corrupted based on the feature values ​​of the specific sample block image calculated by dividing the sample image into sample block images with a predetermined number of pixels, and outputs the result. Therefore, the information processing device 113 determines whether a sample image is corrupted for each block image, including potentially corrupted areas of the sample image and their surrounding information. As a result, the information processing system 1 according to this embodiment can more easily identify corruption in a sample image generated by a trained model than an information processing system that does not determine whether a specific sample block image is corrupted based on the feature values ​​of the specific sample block image. Furthermore, by doing so, the information processing system 1 according to this embodiment can identify the presence or absence of a corruption in a sample image and the location of the corruption, allowing for positional misalignment without requiring highly accurate positioning of the object to be imaged during imaging, thereby reducing the manufacturing cost of the inspection device 100 required for positioning.

[0085] Furthermore, according to the information processing system 1 of this embodiment, the trained model that generates the sample image information is a normal image trained model that is generated by machine learning using normal image information. By doing this, the information processing system 1 according to this embodiment is even less susceptible to sample image corruption than an information processing system that does not generate sample image information using a normal image trained model.

[0086] Furthermore, according to the information processing system 1 of this embodiment, in the information processing device 113, the information acquisition unit 121 acquires captured image information indicating a captured image of a normal workpiece 2. Furthermore, the feature calculation unit 126 calculates the MSE value between a specific template block image of a predetermined template image included in the non-defective image group and the specific sample block image as the feature of the specific sample block image. Then, the failure determination unit 127 determines that the specific sample block image is failed if the feature of the specific sample block image exceeds a threshold.

[0087] In this way, for example, when the information processing device 113 acquires normal image information before an inspection, it inputs the acquired normal image information as captured image information into the trained model to generate sample image information, and calculates the MSE value of a specific sample block image based on the sample image information, thereby determining whether the specific sample block image is corrupted. As a result, the information processing system 1 according to this embodiment can check the vulnerability of the trained model based on the MSE value of a specific sample block image before actually performing an inspection.

[0088] Furthermore, according to the information processing system 1 of this embodiment, in the information processing device 113, the feature amount calculation unit 126 calculates the feature amount of a specific imaging block image of a captured image. Furthermore, the feature amount calculation unit 126 calculates the MSE value between the specific template block image and the specific imaging block image as the feature amount of the specific imaging block image, and also calculates the MSE value between the specific template block image and the specific sample block image as the feature amount of the specific sample block image. Then, the failure determination unit 127 determines that the specific sample block image is corrupted when the feature amount of the specific imaging block image is equal to or less than a threshold value while the feature amount of the specific sample block image exceeds the threshold value.

[0089] In this way, for example, when the information processing device 113 acquires captured image information during an inspection, it inputs the acquired captured image information into the trained model to generate sample image information, and calculates the MSE value of the specific captured block image and the MSE value of the specific sample block image based on the captured image information and the sample image information, thereby determining whether the specific sample block image is corrupted. As a result, even when no vulnerability was found in the preliminary check and the work 2 is ready for mass production, the information processing system 1 according to this embodiment can check the vulnerability of the trained model during actual inspection based on the MSE value of the specific captured block image and the MSE value of the specific sample block image.

[0090] In particular, according to the information processing system 1 of this embodiment, in the information processing device 113, the abnormality determination unit 128 determines whether or not the specific imaging block image contains an abnormal portion of the work 2 based on a result of comparing the feature amount of the specific imaging block image with the feature amount of the specific sample block image. Specifically, when the feature amount of the specific imaging block image exceeds a threshold value while the feature amount of the specific sample block image is equal to or less than the threshold value, the abnormality determination unit 128 determines that the specific imaging block image contains an abnormal portion of the work 2.

[0091] By doing this, the information processing device 113 can calculate the MSE value of a specific imaging block image and the MSE value of a specific sample block image based on the captured image information and the sample image information, and can not only determine whether the specific sample block image is corrupted or not, but also determine whether the specific imaging block image contains an abnormal part of the work 2.

[0092] Here, the phenomenon of the sample image breaking down may occur due to a lack of generation capability of the trained model in areas where there is high diversity in the sample image, such as areas where multiple parts of the workpiece 2 are arranged close together and the shape varies greatly depending on how each part is attached, or areas where the amount of reflected light received by the camera 108 varies greatly depending on how glossy parts of the workpiece 2 are attached.

[0093] In contrast, in the information processing system 1 according to the present embodiment, in the information processing device 113, the first information control unit 123 controls the generation of sample image information by inputting captured image information acquired by the information acquisition unit 121 into a normal image trained model, which is an example of a trained model generated by machine learning using normal image information. Furthermore, if the specific sample block image is corrupted, the second information control unit 124 inputs the specific captured block image information into the specific normal block image trained model generated by machine learning using the specific normal block image information to generate updated specific sample block image information indicating a new specific sample block image based on the specific normal block image. If the specific sample block image is corrupted, the information generation unit 129 generates updated sample image information indicating a new sample image obtained by updating the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information.

[0094] In this way, when a specific sample block image is corrupted, the information processing device 113 generates updated sample image information by changing the corrupted specific sample block image based on updated specific sample block image information generated by inputting specific imaging block image information into a specific normal block image learned model as an example of a learned model. Therefore, when a specific sample block image is corrupted due to insufficient generation capability of the normal image learned model, the information processing device 113 generates a new specific sample block image using a specific normal block image learned model that has a higher generation capability of specific sample block images than the normal image learned model, and generates new sample image information using the new specific sample block image.

[0095] As a result, the information processing system 1 according to this embodiment can more easily acquire a sample image without any defects than an information processing system that does not generate updated sample image information by changing a corrupted specific sample block image based on updated specific sample block image information generated by inputting specific imaging block image information into a specific normal block image trained model. Therefore, the information processing system 1 according to this embodiment can reduce overdetection and erroneous detection when the inspection device 100 performs an inspection using a sample image.

[0096] Furthermore, according to the information processing system 1 of this embodiment, the trained model that generates the sample image information is a normal image trained model generated by machine learning using normal image information, and the specific trained model that generates the update specific sample block image information is a specific normal block image trained model generated by machine learning using specific normal block image information. By doing this, the information processing system 1 according to this embodiment can more easily obtain a sample image without any defects than an information processing system that does not generate sample image information and updated specific sample block image information using a normal image trained model and a specific normal block image trained model.

[0097] Furthermore, according to the information processing system 1 of this embodiment, in the information processing device 113, the information generation unit 129 replaces the corrupted specific sample block image with a new specific sample block image indicated by the updated specific sample block image information, and then performs image processing to smooth the brightness gradient of the edge portion between the new specific sample block image and other sample block images.

[0098] By doing this, the information processing system 1 of the embodiment is less likely to make the above-mentioned edge portions of the new sample image indicated by the updated sample image information look unnatural than an information processing system 1 that does not perform image processing to smooth the brightness gradient of the edge portions after replacing a corrupted specific sample block image with a new specific sample block image, making it even easier to obtain a sample image without any corrupted portions.

[0099] Furthermore, in the information processing system 1 according to the present embodiment, when the information processing device 113 determines that multiple types of captured image information are normal image information as inspection results, the gloss determination unit 130 determines whether the normal image indicated by each type of normal image information is a glossy normal image or a non-glossy normal image, and the information processing device 113 counts the number of glossy normal images and the number of non-glossy normal images using a glossy normal image counter and a non-glossy normal image counter. Then, the information output unit 131 outputs the same number of glossy normal image information and non-glossy normal image information to the storage device 300 based on the values ​​of the glossy normal image counter and the non-glossy normal image counter.

[0100] As a result, in the storage device 300, the learning information storage unit 310 stores the same number of glossy normal image information and non-glossy normal image information. Also, in the learning device 200, the learning information acquisition unit 210 acquires the same number of glossy normal image information and non-glossy normal image information from the storage device 300 as learning information. Then, as shown in FIG. 13 , the trained model generation unit 220 generates a trained model by machine learning using the same number of acquired glossy normal image information and non-glossy normal image information, and the trained model output unit 230 outputs the generated trained model to the storage device 300.

[0101] In this way, the information processing device 113 generates sample image information by inputting the acquired captured image information into a trained model generated by machine learning using normal image information including glossy normal image information indicating a predetermined number of glossy normal images and non-glossy normal image information indicating the same number of non-glossy normal images. As a result, the information processing system 1 according to the present embodiment is more likely to generate sample images that allow for the presence or absence of gloss in glossy parts while reducing the degree of gloss to a level that is not too strong than an information processing system that does not generate a trained model by machine learning using the same number of glossy normal images and non-glossy normal images, thereby reducing the variation in sample images generated by the trained model.

[0102] [Embodiment 2] In the first embodiment, the information processing device 113 generates new specific sample block image information using a specific normal block image trained model when a specific sample block image is corrupted, but this is not limited to this. For example, other sample block images that are not corrupted may also be generated using a normal block image trained model. Hereinafter, the information processing system 1 according to the second embodiment will be described in detail with reference to FIGS. 14 to 16. Note that in the second embodiment, configurations different from those in the first embodiment will be described, and descriptions of the same configurations as those in the first embodiment will be omitted to avoid redundancy.

[0103] (Details of Functional Configuration of Information Processing Device 113 According to Embodiment 2) 14, the information processing device 113 according to this embodiment does not include the first information control unit 123. Returning to FIG. 3, the information acquisition unit 121 does not acquire a normal image trained model as the trained model, but acquires only normal block image trained models, that is, the first normal block image trained model, the second normal block image trained model, ..., the ninth normal block image trained model.

[0104] The second information control unit 124 controls the first normal block image trained model to input the first imaging block image information and generate first sample block image information that is information indicating a first sample block image based on the first normal block image. The second information control unit 124 also controls the second normal block image trained model to input the second imaging block image information and generate second sample block image information that is information indicating a second sample block image based on the second normal block image, ... and controls the ninth normal block image trained model to input the ninth imaging block image information and generate ninth sample block image information that is information indicating a ninth sample block image based on the ninth normal block image.

[0105] When checking the normal image trained model before inspecting the workpiece 2, the second information control unit 124 controls the first normal block image trained model to input first normal block image information as first captured block image information to generate first sample block image information. The second information control unit 124 also controls the second normal block image trained model to input second normal block image information as first captured block image information to generate second sample block image information, ... and controls the ninth normal block image trained model to input ninth normal block image information as ninth captured block image information to generate ninth sample block image information.

[0106] The information generating unit 129 generates sample image information indicating a sample image obtained by concatenating the first sample block image indicated by the first sample block image information, the second sample block image indicated by the second sample block image information, ..., and the ninth sample block image indicated by the ninth sample block image information. Note that the information generating unit 129 may perform image processing to smooth the luminance gradient at the edge portions between each sample block image.

[0107] (Regarding Pre-Test Determination Processing According to Embodiment 2) Next, the pre-test determination process according to this embodiment will be described with reference to a flowchart. As shown in Fig. 15, first, the information acquisition unit 121 executes the processes of steps S101 and S102 to acquire a first normal block image trained model, a second normal block image trained model, ..., a ninth normal block image trained model, normal image information, and template image information.

[0108] After acquiring the information, the image division unit 125 divides the normal image into a first normal block image, a second normal block image, ..., a ninth normal block image (step S108), and the second information control unit 124 inputs each normal block image information as each imaging block image information into each normal block image trained model, and controls the generation of each sample block image information (step S109). After generating the sample block image information, the image division unit 125 divides the template image indicated by the template image information into each template block image (step S110). After dividing the image, the information processing device 113 executes the processes of steps S105 to S107 and then ends the process.

[0109] (Regarding the In-Test Determination Process According to the Second Embodiment) Next, the in-inspection determination process according to this embodiment will be described using a flowchart. As shown in FIG. 16, first, the information acquisition unit 121 executes the process of step S111 to acquire captured image information, and the image division unit 125 divides the captured image indicated by the captured image information into a first imaging block image, a second imaging block image, ..., and a ninth imaging block image (step S131). After image division, the second information control unit 124 inputs each imaging block image information into each normal block image trained model and controls the generation of each sample block image information (step S132). After generating the sample block image information, the information generation unit 129 generates sample image information indicating a sample image obtained by concatenating each sample block image indicated by each sample block image information (step S133). After generating the sample image, the information processing device 113 executes the processes of steps S114 and S119 to S123, and then ends the process.

[0110] As described above, according to the information processing system 1 of this embodiment, in the information processing device 113, the image divider 125 divides the captured image of the workpiece 2 indicated by the captured image information acquired by the information acquisition unit 121 into a first normal block image, a second normal block image, ..., and a ninth normal block image. Furthermore, the second information control unit 124 of the information control unit 122 controls the first trained model to input the first captured block image information to generate first sample block image information. Furthermore, the second information control unit 124 controls the second trained model to input the second captured block image information to generate second sample block image information, ..., and controls the ninth trained model to generate ninth sample block image information. The information generation unit 129 generates sample image information based on the first sample block image information, the second sample block image information, ..., and the ninth sample block image information.

[0111] In this way, the information processing device 113 generates sample image information based on each sample block image information generated by inputting each imaging block image information into each normal block image trained model, making it less likely that a failure will occur than when sample image information is not generated on a sample block image information basis.As a result, the information processing system 1 according to this embodiment can more easily acquire a sample image without a failure than an information processing system that does not generate sample image information based on each sample block image information generated by inputting each imaging block image information into each normal block image trained model.

[0112] Furthermore, according to the information processing system 1 of this embodiment, the first trained model that generates the first sample block image information is a first normal block image trained model generated by machine learning using the first normal block image information, the second trained model that generates the second sample block image information is a second normal block image trained model generated by machine learning using the second normal block image information, ..., and the ninth trained model that generates the ninth sample block image information is a ninth normal block image trained model generated by machine learning using the ninth normal block image information. By doing this, the information processing system 1 according to this embodiment can more easily obtain sample images without defects than an information processing system that does not generate sample block image information using each normal block image trained model. In addition, the information processing system 1 according to this embodiment has the same effects as the information processing system 1 according to the first embodiment.

[0113] (Example of change) In the above-described first and second embodiments, the normal image trained model used by the first information control unit 123 and the normal block image trained models used by the second information control unit 124 are separate trained models, but this is not limiting. For example, the normal image trained model and the normal block image trained models may be integrated into a single trained model. In this way, the information processing system 1 can transmit and receive the normal image trained model and the normal block image trained models as a single image generation AI, and use them for image generation.

[0114] In the first and second embodiments, the information processing device 113, the learning device 200, and the storage device 300 are separate computer devices, but this is not limiting, and for example, the information processing device 113 and the storage device 300 may be integrated into a computer device, or the learning device 200 and the storage device 300 may be integrated into a computer device. Furthermore, for example, the information processing device 113, the learning device 200, and the storage device 300 may be integrated into a computer device.

[0115] In the first and second embodiments, the feature calculation unit 126 calculates the MSE value between the specific template block image and the specific sample block image as the feature of the specific sample block image, but this is not limiting. For example, the feature calculation unit 126 may calculate the average value and standard deviation of the luminance of the specific sample block image as the feature of the specific imaging block image. Furthermore, the feature calculation unit 126 may calculate the average value and standard deviation of the luminance of the specific normal block image as the feature of the specific normal block image, similar to the feature of the specific sample block image, or may calculate the average value and standard deviation of the luminance of the specific imaging block image as the feature of the specific imaging block image.

[0116] As in the first and second embodiments, it is preferable to check the vulnerability of the trained model by performing pre-test judgment processing, but it is not necessary to perform pre-test judgment processing.

[0117] In the above-described first and second embodiments, the failure determination unit 127 determines whether a specific sample block image is corrupted based on whether the feature amount of the specific sample block image exceeds a threshold value. However, this is not limited to this. For example, the failure determination unit 127 may input the specific sample block image into an artificial intelligence to generate determination result information on whether the specific sample block image is corrupted, and determine whether the specific sample block image is corrupted based on the determination result information. In this case, for example, it is necessary to newly generate an artificial intelligence that has learned the relationship between the specific sample block image and image corruption through machine learning.

[0118] In the above-described first and second embodiments, the abnormality determination unit 128 determines whether or not an abnormal portion of the work 2 is included in the specific imaging block image based on whether or not the feature amount of the specific imaging block image and the feature amount of the specific sample block image exceed a threshold value, but this is not limited to this. For example, the abnormality determination unit 128 may input the specific imaging block image and the specific sample block image into an artificial intelligence to generate determination result information on whether or not the specific imaging block image includes an abnormal portion of the work 2, and determine whether or not the specific imaging block image includes an abnormal portion of the work 2 based on the determination result information. In this case, for example, it is necessary to newly generate an artificial intelligence that has performed machine learning on the relationship between the specific imaging block image, the specific sample block image, and the abnormal portion of the work 2.

[0119] In the above-mentioned first and second embodiments, the learning device 200 and the storage device 300 are devices inside the information processing system 1, but this is not limited to this, and the learning device 200 and the storage device 300 may also be devices outside the information processing system 1.

[0120] Furthermore, the information processing device 113 does not need to include the functionality of an inference device, and for example, as shown in Fig. 17, an external cloud server system may include a learning device 200, a storage device 300, and an inference device 400. In other words, the trained model may be an external image generation AI, as long as it is generated by machine learning using a group of non-defective product images.

[0121] In this case, the information control unit 122 performs control to transmit the captured image information acquired by the information acquisition unit 121 to the inference device 400, and to receive from the inference device 400 sample image information that the inference device 400 has generated by inputting the captured image information into the learned model. Specifically, the first information control unit 123 performs control to transmit the captured image information to the inference device 400, and to receive from the inference device 400 sample image information that the inference device 400 has generated by inputting the captured image information into the normal image learned model. Furthermore, the second information control unit 124 performs control to transmit the specific captured block image to the inference device 400, and to receive from the inference device 400 specific sample block image information that the inference device 400 has generated by inputting the specific captured block image information into the specific normal block image learned model.

[0122] Furthermore, the inference device 400 may use the above-mentioned artificial intelligence to generate determination result information as to whether or not a specific sample block image is damaged, or generate determination result information as to whether or not a specific captured block image contains an abnormal part of the work 2. In other words, the above-mentioned artificial intelligence may be an external artificial intelligence.

[0123] In this case, the failure determination unit 127 and the abnormality determination unit 128, for example, transmit specific imaging block image information and specific sample block image information to the inference device 400, and after receiving from the inference device 400 determination result information generated by the inference device 400 inputting the specific imaging block image information and the specific sample block image into an artificial intelligence, determine whether the specific sample block image is failed or whether the specific imaging block image includes an abnormal portion of the work 2 based on the determination result information. In this way, the information processing system according to the modified example can omit a so-called learning phase such as the learning process shown in Fig. 9, and only need to perform so-called inference phases such as the pre-inspection determination process shown in Fig. 10 and the during-inspection determination process shown in Figs. 11A and 11B.

[0124] The external image generation AI and external artificial intelligence may be configured using known algorithms such as Transformer, BERT (Bidirectional Encoder Representations from Transformers), and GPT (Generative Pre-Training), or may be configured by combining a plurality of algorithms including these. Furthermore, it is preferable that the external image generation AI and external artificial intelligence perform so-called fine tuning, in which normal image information and each normal block image information are machine-learned as in the first and second embodiments, but fine tuning is not necessarily required.

[0125] In the first and second embodiments, the image dividing unit 125 divides the image of n×m pixels into nine equal parts, creating a first block image, a second block image, ..., and a ninth block image. However, the present invention is not limited to this, and the image does not need to be divided equally. In other words, the predetermined number of pixels by which the image dividing unit 125 divides the image into block images is not limited to one type, and may be two or more types. By doing so, it is possible to adjust the number of pixels in a block image at a specific location where, for example, a glossy part of the workpiece 2 may be present and the block image may be damaged, or where an abnormal part of the workpiece 2 may be included.

[0126] In the above-described first and second embodiments, the feature amount calculation unit 126 calculates the feature amounts of all block images divided by the image division unit 125, the failure determination unit 127 determines whether all sample block images divided by the image division unit 125 are failed, and the abnormality determination unit 128 determines whether all captured block images divided by the image division unit 125 contain an abnormal portion of the workpiece 2. However, the present invention is not limited to this. For example, if there is a block image of a specific location that may be failed due to the presence of a glossy part of the workpiece 2 or that may contain an abnormal portion of the workpiece 2, the feature amount calculation unit 126 may calculate only the feature amount of the block image of the specific location, the failure determination unit 127 may determine only whether the sample block image of the specific location is failed, and the abnormality determination unit 128 may determine only whether the captured block image of the specific location contains an abnormal portion of the workpiece 2.

[0127] In this case, in the learning device 200, the learning information acquisition unit 210 may acquire only normal image information and specific normal block image information as learning information, the learned model generation unit 220 may generate only a normal image trained model and a specific normal block image as trained models, and the trained model output unit 230 may output only the normal image trained model and the specific normal block image trained model. Furthermore, in the information processing device 113, the information acquisition unit 121 may acquire only a normal image trained model and a specific normal block image trained model as trained models. In this way, if the sample block image of a specific location is corrupted, the second information control unit 124 inputs specific imaging block image information into the specific normal block image trained model to generate updated specific sample block image information, and the information generation unit 129 can generate updated sample image information using the updated specific sample block image information.

[0128] As in the first embodiment, it is preferable to generate a trained model by machine learning using an equal number of glossy normal image information and non-glossy normal image information, but this is not limiting. For example, the difference between the numerical value of the type of glossy normal image information and the numerical value of the type of non-glossy normal image information included in the normal image information used in the machine learning of the trained model may be equal to or less than a predetermined type threshold. Even in this case, since the trained model can be generated by machine learning using an equal number of glossy normal image information and non-glossy normal image information, it is easier to generate sample images in which the degree of glossiness is reduced to a level that is not too strong while allowing for the presence or absence of gloss in glossy parts of the workpiece 2 in the captured image, thereby reducing the variation in the sample images generated by the trained model. Furthermore, although variation in the sample images generated by the trained model may occur, for example, the difference between the numerical value of the type of glossy normal image information and the numerical value of the type of non-glossy normal image information included in the normal image information used in the machine learning of the trained model may exceed the type threshold.

[0129] In the first and second embodiments, the machine learning learning algorithm by which the trained model generation unit 220 generates the trained model is a neural network, but the present invention is not limited to this and may be, for example, another supervised learning learning algorithm such as a support vector machine. Also, the learning algorithm is not limited to supervised learning as long as the trained model is generated by machine learning using a group of good product images and sample image information can be generated when captured image information is input, and may be, for example, another learning algorithm such as reinforcement learning.

[0130] In the first embodiment, the information processing device 113 includes the gloss determination unit 130, and the information processing device 113 outputs the same number of glossy normal image information and non-glossy normal image information to the storage device 300, thereby ensuring that the learning device 200 uses the same number of glossy normal image information and non-glossy normal image information for machine learning. However, this is not limiting. For example, the storage device 300 may include the gloss determination unit 130, and control may be performed such that the storage device 300 stores the same number of glossy normal image information and non-glossy normal image information, or that the storage device 300 transmits the same number of glossy normal image information and non-glossy normal image information to the learning device 200. Furthermore, for example, the learning device 200 may include the gloss determination unit 130, and control may be performed such that the learning device 200 acquires the same number of glossy normal image information and non-glossy normal image information from the storage device 300, or that the learning device 200 uses the same number of glossy normal image information and non-glossy normal image information for machine learning.

[0131] The core processing components of information processing device 113, learning device 200, and storage device 300, including control unit 51, main memory unit 52, external memory unit 53, operation unit 54, display unit 55, transmission / reception unit 56, and internal bus 50, can be realized using a standard computer system rather than a dedicated system. For example, a computer program for executing the above operations may be stored and distributed on a recording medium readable by information processing device 113, learning device 200, and storage device 300, such as a DVD-ROM (Read-Only Memory), and the computer program may be installed on a computer to configure information processing device 113, learning device 200, and storage device 300 to execute the above processes. Alternatively, the computer program may be stored in a storage device of a server device on a communication network such as the Internet, and downloaded by a standard computer system to configure information processing device 113, learning device 200, and storage device 300.

[0132] In addition, when the functions of the information processing device 113, learning device 200, and storage device 300 are realized by sharing the functions between an OS (operating system) and an application program, or by collaboration between the OS and an application program, only the application program portion may be stored on a recording medium or storage device.

[0133] It is also possible to superimpose a computer program on a carrier wave and provide it via a communication network. For example, the computer program may be posted on a bulletin board system (BBS) on the communication network and provided via the network. The computer program may then be started and executed under the control of the OS in the same way as other application programs, thereby performing the above-mentioned processing.

[0134] In addition, the control methods, information processing system 1, information processing device 113, and program configurations according to the above-mentioned first and second embodiments are merely examples, and can be changed and modified as desired as long as they solve the problems that the present disclosure aims to solve.

[0135] The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are considered to be within the scope of the present disclosure.

[0136] Various aspects of the present disclosure are summarized below as appendices.

[0137] (Appendix 1) an information acquisition step in which the computer acquires captured image information that is information indicating a captured image that is an image of an imaging target; A first information control step in which the computer inputs the captured image information acquired in the information acquisition step into a learned model to generate sample image information that is information indicating a sample image based on a normal image that is a normal captured image of the imaged object; an image dividing step in which the computer divides the image into block images each consisting of a predetermined number of pixels; a failure determination step in which the computer determines whether the specific sample block image is broken or not based on specific sample block image information, which is information indicating the specific sample block image, which is the block image of a specific location of the sample image; a second information control step in which the computer, when the specific sample block image is corrupted, inputs specific captured block image information, which is information indicating the block image of the specific location of the captured image indicated by the captured image information acquired in the information acquisition step, into a specific trained model to generate updated specific sample block image information, which is information indicating a new specific sample block image based on a specific normal block image, which is the block image of the specific location of the normal image; an information generating step in which, when the specific sample block image is corrupted, the computer generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information; A control method comprising: (Appendix 2) In the information generating step, the computer replaces the corrupted specific sample block image with the new specific sample block image indicated by the update specific sample block image information, and then performs image processing to smooth the brightness gradient of the edge portion, which is the boundary portion between the new specific sample block image and another sample block image. 10. The control method according to claim 1. (Appendix 3) The trained model is a normal image trained model generated by machine learning using normal image information, which is information indicating the normal image, The specific trained model is a specific normal block image trained model generated by machine learning using specific normal block image information, which is information indicating the specific normal block image. 3. The control method according to claim 1 or 2. (Appendix 4) The normal image trained model and the specific normal block image trained model are the same artificial intelligence. 3. The control method according to claim 3. (Appendix 5) an information acquisition step in which the computer acquires captured image information that is information indicating a captured image that is an image of an imaging target; an image dividing step in which the computer divides the image into a first block image composed of a predetermined number of pixels and a second block image different from the first block image; an information control step in which the computer inputs first image capture block image information, which is information indicating a first image capture block image that is the first block image of the captured image indicated by the captured image information acquired in the information acquisition step, into a first trained model to control generation of first sample block image information, which is information indicating a first sample block image based on a first normal block image that is the first block image of a normal image that is a normal captured image of the captured object, and inputs second image capture block image information, which is information indicating a second image capture block image that is the second block image of the captured image indicated by the captured image information acquired in the information acquisition step, into a second trained model to control generation of second sample block image information, which is information indicating a second sample block image based on a second normal block image that is the second block image of the normal image; an information generating step in which the computer generates sample image information, which is information indicating a sample image based on the normal image, based on the first sample block image information and the second sample block image information; A control method comprising: (Appendix 6) The first trained model is a first normal block image trained model generated by machine learning using first normal block image information, which is information indicating the first normal block image, The second trained model is a second normal block image trained model generated by machine learning using second normal block image information, which is information indicating a second normal block image that is the second block image of the normal image. 6. The control method according to claim 5. (Appendix 7) The first normal block image trained model and the second normal block image trained model are the same artificial intelligence. 6. The control method according to claim 6. (Appendix 8) an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; a first information control unit that controls inputting the captured image information acquired by the information acquisition unit into a normal image trained model generated by machine learning using normal image information that is information indicating a normal image, which is the captured image of a normal imaging object, to generate sample image information that is information indicating a sample image based on the normal image; an image dividing unit that divides an image into block images each consisting of a predetermined number of pixels; a failure determination unit that determines whether the specific sample block image is broken or not based on specific sample block image information, which is information indicating the specific sample block image, which is the block image of a specific location of the sample image; a second information control unit that, when the specific sample block image is corrupted, inputs specific captured block image information, which is information indicating the block image of the specific location of the captured image indicated by the captured image information acquired by the information acquisition unit, into a specific normal block image trained model generated by machine learning using specific normal block image information, which is information indicating the block image of the specific location of the normal image, to generate updating specific sample block image information, which is information indicating a new specific sample block image based on the specific normal block image; an information generating unit that generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information, when the specific sample block image is corrupted; An information processing system comprising: (Appendix 9) an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an image dividing unit that divides an image into a first block image composed of a predetermined number of pixels and a second block image different from the first block image; an information control unit that controls inputting first imaged block image information, which is information indicating the first imaged block image that is the first block image of the captured image indicated by the captured image information acquired by the information acquisition unit, into a first normal block image trained model generated by machine learning using first normal block image information, which is information indicating the first normal block image that is the first block image of a normal image that is the captured image of a normal image of the captured object, to generate first sample block image information, which is information indicating a first sample block image based on the first normal block image; and an information control unit that controls inputting second imaged block image information, which is information indicating the second imaged block image that is the second block image of the captured image indicated by the captured image information acquired by the information acquisition unit, into a second normal block image trained model generated by machine learning using second normal block image information, which is information indicating a second normal block image that is the second block image of the normal image, to generate second sample block image information, which is information indicating a second sample block image based on the second normal block image; an information generating unit that generates sample image information, which is information indicating a sample image based on the normal image, based on the first sample block image information and the second sample block image information; An information processing system comprising: (Appendix 10) an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; a first information control unit that controls inputting the captured image information acquired by the information acquisition unit into a normal image trained model generated by machine learning using normal image information that is information indicating a normal image, which is the captured image of a normal imaging object, to generate sample image information that is information indicating a sample image based on the normal image; an image dividing unit that divides an image into block images each consisting of a predetermined number of pixels; a failure determination unit that determines whether the specific sample block image is broken or not based on specific sample block image information, which is information indicating the specific sample block image, which is the block image of a specific location of the sample image; a second information control unit that, when the specific sample block image is corrupted, inputs specific captured block image information, which is information indicating the block image of the specific location of the captured image indicated by the captured image information acquired by the information acquisition unit, into a specific normal block image trained model generated by machine learning using specific normal block image information, which is information indicating the block image of the specific location of the normal image, to generate updating specific sample block image information, which is information indicating a new specific sample block image based on the specific normal block image; an information generating unit that generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information, when the specific sample block image is corrupted; An information processing device comprising: (Appendix 11) an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an image dividing unit that divides an image into a first block image composed of a predetermined number of pixels and a second block image different from the first block image; an information control unit that controls inputting first imaged block image information, which is information indicating the first imaged block image that is the first block image of the captured image indicated by the captured image information acquired by the information acquisition unit, into a first normal block image trained model generated by machine learning using first normal block image information, which is information indicating the first normal block image that is the first block image of a normal image that is the captured image of a normal image of the captured object, to generate first sample block image information, which is information indicating a first sample block image based on the first normal block image; and an information control unit that controls inputting second imaged block image information, which is information indicating the second imaged block image that is the second block image of the captured image indicated by the captured image information acquired by the information acquisition unit, into a second normal block image trained model generated by machine learning using second normal block image information, which is information indicating a second normal block image that is the second block image of the normal image, to generate second sample block image information, which is information indicating a second sample block image based on the second normal block image; an information generating unit that generates sample image information, which is information indicating a sample image based on the normal image, based on the first sample block image information and the second sample block image information; An information processing device comprising: (Appendix 12) Computer, an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; a first information control unit that controls inputting the captured image information acquired by the information acquisition unit into a normal image trained model generated by machine learning using normal image information that is information indicating a normal image, which is the captured image of a normal image object, to generate sample image information that is information indicating a sample image based on the normal image; an image dividing unit that divides an image into block images each consisting of a predetermined number of pixels; a failure determination unit that determines whether the specific sample block image is failed based on specific sample block image information, which is information indicating the specific sample block image, which is the block image of a specific location of the sample image; a second information control unit that controls, when the specific sample block image is corrupted, to input specific captured block image information, which is information indicating the block image of the specific location of the captured image indicated by the captured image information acquired by the information acquisition unit, into a specific normal block image trained model generated by machine learning using specific normal block image information, which is information indicating the block image of the specific location of the normal image, to generate updating specific sample block image information, which is information indicating a new specific sample block image based on the specific normal block image; an information generating unit that generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information, when the specific sample block image is corrupted; A program that functions as a (Appendix 13) Computer, an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an image dividing unit that divides an image into a first block image composed of a predetermined number of pixels and a second block image different from the first block image; an information control unit that controls inputting first imaged block image information, which is information indicating the first imaged block image that is the first block image of the captured image indicated by the captured image information acquired by the information acquisition unit, into a first normal block image trained model generated by machine learning using first normal block image information, which is information indicating the first normal block image that is the first block image of a normal image that is the captured image of a normal image of the captured object, to generate first sample block image information, which is information indicating a first sample block image based on the first normal block image; and also controls inputting second imaged block image information, which is information indicating the second imaged block image that is the second block image of the captured image indicated by the captured image information acquired by the information acquisition unit, into a second normal block image trained model generated by machine learning using second normal block image information, which is information indicating a second normal block image that is the second block image of the normal image, to generate second sample block image information, which is information indicating a second sample block image based on the second normal block image; an information generating unit that generates sample image information, which is information indicating a sample image based on the normal image, based on the first sample block image information and the second sample block image information; A program that functions as a [Explanation of symbols]

[0138] 1...information processing system, 2...work, 21...connection terminal, 22...CPU, 23...memory, 24...connector, 25...port, 26...electrolytic capacitor, 27...film capacitor, 50...internal bus, 51...control unit, 52...main memory unit, 53...external memory unit, 54...operation unit, 55...display unit, 56...transmitting / receiving unit, 59...control program, 100...inspection device, 101...base, 102...Y-axis actuator, 103...Z-axis actuator, 104...X-axis actuator, 105...transport table, 106...observation camera, 107...laser displacement meter, 108...camera, 109...lens, 110...illumination member, 111...displacement meter controller Roller, 112...lighting controller, 113...information processing device, 114...PLC, 115...GOT, 121...information acquisition unit, 122...information control unit, 123...first information control unit, 124...second information control unit, 125...image segmentation unit, 126...feature calculation unit, 127...failure determination unit, 128...abnormality determination unit, 129...information generation unit, 130...gloss determination unit, 131...information output unit, 200...learning device, 210...learning information acquisition unit, 220...learned model generation unit, 230...learned model output unit, 300...storage device, 310...learning information storage unit, 320...learned model storage unit, 330...information transmission and reception unit, 400...inference device.

Claims

1. an information acquisition step in which the computer acquires captured image information that is information indicating a captured image that is an image of an imaging target; A first information control step in which the computer inputs the captured image information acquired in the information acquisition step into a trained model to generate sample image information that is information indicating a sample image based on a normal image that is a normal captured image of the image capture object; an image dividing step in which the computer divides the image into block images each consisting of a predetermined number of pixels; a failure determination step in which the computer determines whether the specific sample block image is broken or not based on specific sample block image information, which is information indicating a specific sample block image that is the block image of a specific location of the sample image; a second information control step in which the computer, when the specific sample block image is corrupted, inputs specific captured block image information, which is information indicating the block image of the specific location of the captured image indicated by the captured image information acquired in the information acquisition step, into a specific trained model to generate updated specific sample block image information, which is information indicating a new specific sample block image based on a specific normal block image, which is the block image of the specific location of the normal image; an information generating step in which, when the specific sample block image is corrupted, the computer generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information; A control method comprising:

2. In the information generating step, the computer replaces the corrupted specific sample block image with the new specific sample block image indicated by the update specific sample block image information, and then performs image processing to smooth the brightness gradient of the edge portion, which is the boundary portion between the new specific sample block image and another sample block image. The control method according to claim 1 .

3. The trained model is a normal image trained model generated by machine learning using normal image information, which is information indicating the normal image, The specific trained model is a specific normal block image trained model generated by machine learning using specific normal block image information, which is information indicating the specific normal block image. The control method according to claim 1 or 2.

4. The normal image trained model and the specific normal block image trained model are the same artificial intelligence. The control method according to claim 3 .

5. an information acquisition step in which the computer acquires captured image information that is information indicating a captured image that is an image of an imaging target; an image dividing step in which the computer divides the image into a first block image composed of a predetermined number of pixels and a second block image different from the first block image; an information control step in which the computer inputs first image capture block image information, which is information indicating a first image capture block image that is the first block image of the captured image indicated by the captured image information acquired in the information acquisition step, into a first trained model to control generation of first sample block image information, which is information indicating a first sample block image based on a first normal block image that is the first block image of a normal image that is a normal captured image of the captured object, and inputs second image capture block image information, which is information indicating a second image capture block image that is the second block image of the captured image indicated by the captured image information acquired in the information acquisition step, into a second trained model to control generation of second sample block image information, which is information indicating a second sample block image based on a second normal block image that is the second block image of the normal image; an information generating step in which the computer generates sample image information, which is information indicating a sample image based on the normal image, based on the first sample block image information and the second sample block image information; A control method comprising:

6. The first trained model is a first normal block image trained model generated by machine learning using first normal block image information, which is information indicating the first normal block image, The second trained model is a second normal block image trained model generated by machine learning using second normal block image information, which is information indicating a second normal block image that is the second block image of the normal image. The control method according to claim 5.

7. The first normal block image trained model and the second normal block image trained model are the same artificial intelligence. The control method according to claim 6.

8. an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; A first information control unit that inputs the captured image information acquired by the information acquisition unit into a trained model and controls the generation of sample image information that is information indicating a sample image based on a normal image that is a normal captured image of the imaging object; an image dividing unit that divides an image into block images each consisting of a predetermined number of pixels; a failure determination unit that determines whether the specific sample block image is broken or not based on specific sample block image information, which is information indicating the specific sample block image, which is the block image of a specific location of the sample image; a second information control unit that, when the specific sample block image is corrupted, controls the specific trained model to input specific captured block image information, which is information indicating the block image of the specific location of the captured image indicated by the captured image information acquired by the information acquisition unit, to generate updated specific sample block image information, which is information indicating a new specific sample block image based on a specific normal block image, which is the block image of the specific location of the normal image; an information generating unit that generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information, when the specific sample block image is corrupted; An information processing system comprising:

9. an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an image dividing unit that divides an image into a first block image composed of a predetermined number of pixels and a second block image different from the first block image; an information control unit that controls a first trained model to generate first sample block image information that is information indicating a first captured block image, which is the first block image of the captured image indicated by the captured image information acquired by the information acquisition unit, and that controls a first trained model to generate first sample block image information that is information indicating a first sample block image based on a first normal block image, which is the first block image of a normal image, which is the captured image of a normal image of the captured object, and that controls a second trained model to generate second sample block image information that is information indicating a second captured block image, which is information indicating a second normal block image, which is the second block image of the captured image indicated by the captured image information acquired by the information acquisition unit; an information generating unit that generates sample image information, which is information indicating a sample image based on the normal image, based on the first sample block image information and the second sample block image information; An information processing system comprising:

10. an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; A first information control unit that inputs the captured image information acquired by the information acquisition unit into a trained model and controls the generation of sample image information that is information indicating a sample image based on a normal image that is a normal captured image of the imaging object; an image dividing unit that divides an image into block images each consisting of a predetermined number of pixels; a failure determination unit that determines whether the specific sample block image is broken or not based on specific sample block image information, which is information indicating the specific sample block image, which is the block image of a specific location of the sample image; a second information control unit that, when the specific sample block image is corrupted, controls the specific trained model to input specific captured block image information, which is information indicating the block image of the specific location of the captured image indicated by the captured image information acquired by the information acquisition unit, to generate updated specific sample block image information, which is information indicating a new specific sample block image based on a specific normal block image, which is the block image of the specific location of the normal image; an information generating unit that generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information, when the specific sample block image is corrupted; An information processing device comprising:

11. an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an image dividing unit that divides an image into a first block image composed of a predetermined number of pixels and a second block image different from the first block image; an information control unit that controls a first normal block image trained model to generate first sample block image information that is information indicating a first captured block image, which is the first block image of the captured image indicated by the captured image information acquired by the information acquisition unit, and that controls a second trained model to generate second sample block image information that is information indicating a second captured block image, which is the second block image of the captured image indicated by the captured image information acquired by the information acquisition unit, and that controls a second trained model to generate second sample block image information that is information indicating a second sample block image, which is information indicating a second normal block image, which is the second block image of the normal image; an information generating unit that generates sample image information, which is information indicating a sample image based on the normal image, based on the first sample block image information and the second sample block image information; An information processing device comprising:

12. Computer, an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; a first information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model to generate sample image information that is information indicating a sample image based on a normal image that is a normal captured image of the imaging object; an image dividing unit that divides an image into block images each consisting of a predetermined number of pixels; a failure determination unit that determines whether the specific sample block image is broken or not based on specific sample block image information, which is information indicating the specific sample block image, which is the block image of a specific location of the sample image; a second information control unit that controls, when the specific sample block image is corrupted, to input specific captured block image information, which is information indicating the block image of the specific location of the captured image indicated by the captured image information acquired by the information acquisition unit, into a specific trained model, and generates updated specific sample block image information, which is information indicating a new specific sample block image based on a specific normal block image, which is the block image of the specific location of the normal image; an information generating unit that generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information, when the specific sample block image is corrupted; A program that functions as a

13. Computer, an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an image dividing unit that divides an image into a first block image composed of a predetermined number of pixels and a second block image different from the first block image; an information control unit that controls a first trained model to input first image capture block image information, which is information indicating a first image capture block image that is the first block image of the captured image indicated by the captured image information acquired by the information acquisition unit, to generate first sample block image information, which is information indicating a first sample block image based on a first normal block image that is the first block image of a normal image that is a normal captured image of the captured object, and that controls a second trained model to input second image capture block image information, which is information indicating a second image capture block image that is the second block image of the captured image indicated by the captured image information acquired by the information acquisition unit, to generate second sample block image information, which is information indicating a second sample block image based on a second normal block image that is the second block image of the normal image; an information generating unit that generates sample image information, which is information indicating a sample image based on the normal image, based on the first sample block image information and the second sample block image information; A program that functions as a

Images