Control method, information processing system, information processing device, and program

Abstract

Reduce the variation in sample images generated by the trained model. [Solution] A computer acquires captured image information, which is information indicating a captured image of a workpiece, and controls the computer to input the acquired captured image information into a trained model generated by machine learning using normal image information to generate sample image information, which is information indicating a sample image based on the normal image. The normal images include a first normal image and a second normal image different from the first normal image, and brightness normalization processing, which is image processing that adjusts the average brightness and deviation value to predetermined values, has been performed in advance as preprocessing on the first normal image and the second normal image.

Description

[Technical Field]

[0001] The present disclosure relates to a control method, an information processing system, an information processing device, and a program. [Background technology]

[0002] Inspection devices that use image processing to inspect captured objects, such as products, for abnormalities are known. In some cases, instead of capturing captured images of the object, the inspection device uses sample images, which are images based on a group of non-defective product images generated by inputting the captured images into an image generation AI (artificial intelligence) that has learned from a group of non-defective product images, as the inspection image. In this case, for example, variations in the shape and brightness of glossy portions of the captured object in the captured images may occur due to factors such as the object, lighting equipment, and the position and condition of the imaging equipment at the time of capture. This may result in variations in the sample images generated by the image generation AI, which may lead to overdetection or false detection. Therefore, in order for the inspection device to perform accurate inspections using sample images, it is necessary to reduce the variations in the sample images generated by the image generation AI.

[0003] Patent Document 1 discloses an inspection device that makes it possible to set optimal image processing conditions by evaluating brightness variations caused by the object being imaged or the image detection system. The inspection device in Patent Document 1 finds the noise characteristics of the secondary electron image caused by the image detection system, determines optimal image processing parameters according to the object being imaged based on these characteristics, and evaluates variations caused by mismatches in the object being imaged by performing a comparison process using these noise characteristics and the image of the object being imaged. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-304842 Summary of the Invention [Problem to be solved by the invention]

[0005] The inspection device described in Patent Document 1 can perform inspections by tolerating brightness variations caused by the object being imaged or the image detection system, and variations due to mismatches in the object being imaged. However, if unacceptable variations occur, there is a risk that this may cause overdetection or erroneous detection.

[0006] The present disclosure has been made in consideration of the above-mentioned circumstances, and aims to reduce the variation in sample images generated by a trained model. [Means for solving the problem]

[0007] In order to achieve the above object, a control method according to a first aspect of the present disclosure includes an information acquisition step in which a computer acquires captured image information, which is information indicating a captured image, which is an image of an object to be imaged; and an information control step in which the computer inputs the captured image information acquired in the information acquisition step into a trained model generated by machine learning using normal image information, which is information indicating a normal image, which is an image based on a captured image of a normal object to be imaged, to generate sample image information, which is information indicating a sample image based on the normal image, wherein the normal images include a first normal image and a second normal image different from the first normal image, and brightness normalization processing, which is image processing that adjusts the average brightness and deviation value to predetermined values, has been performed in advance as preprocessing on the first normal image and the second normal image.

[0008] Furthermore, in order to achieve the above object, a control method according to a second aspect of the present disclosure includes an information acquisition step in which a computer acquires captured image information, which is information indicating a captured image, which is an image of an image capture target, and an information control step in which the computer inputs the captured image information acquired in the information acquisition step into a trained model generated by machine learning using normal image information, which is information indicating a normal image, which is an image based on an image capture target that is a normal image capture target, to generate sample image information, which is information indicating a sample image based on the normal image, wherein the normal image includes a first normal image, which is an image capture target of the normal image capture target, and a second normal image on which chromaticity variation processing, which is image processing that adjusts the chromaticity of any of red light, green light, and blue light corresponding to the three primary colors of light that represent the first normal image, has been performed in advance as preprocessing. [Effects of the Invention]

[0009] According to the present disclosure, the normal image information used in machine learning is pre-processed by brightness normalization and chromaticity variation processing. As a result, the control method according to the present disclosure can reduce the variation in sample images generated by the trained model compared to a control method that does not perform these pre-processing steps. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram of an information processing system according to a first embodiment; [Figure 2] An explanatory diagram of an inspection device according to the first embodiment [Figure 3] FIG. 1 is a diagram showing a functional configuration of an information processing system according to a first embodiment; [Figure 4] FIG. 1 is a block diagram showing the hardware configuration of an information processing device, a learning device, and a storage device according to a first embodiment. [Figure 5] 1 is an explanatory diagram of an imaging block image, a sample block image, and a normal block image according to the first embodiment; [Figure 6A] FIG. 10 is an explanatory diagram of a case where a first sample block image is determined to be corrupted before inspection according to the first embodiment. [Figure 6B]FIG. 10 is an explanatory diagram of a case where a first sample block image is determined to be corrupted during inspection according to the first embodiment. [Figure 7A] FIG. 10 is an explanatory diagram of a case where it is determined that an abnormal part of a workpiece is included in a first imaging block image during inspection according to the first embodiment. [Figure 7B] FIG. 10 is an explanatory diagram of a case where it is determined that the first imaging block image does not include an abnormal part of the workpiece during inspection according to the first embodiment. [Figure 7C] FIG. 10 is an explanatory diagram of a case where it is determined that an error has occurred in a first imaging block image during an inspection according to the first embodiment; [Figure 8] FIG. 1 is an explanatory diagram of a neural network according to a first embodiment; [Figure 9] 1 is a flowchart showing a flow of a learning process according to the first embodiment. [Figure 10] 1 is a flowchart showing the flow of pre-examination determination processing according to the first embodiment; [Figure 11A] 1 is a flowchart showing the flow of an in-examination determination process according to the first embodiment. [Figure 11B] Continuation of the flowchart of FIG. 11A [Figure 12] 1 is a flowchart showing a flow of normal image information output processing according to the first embodiment. [Figure 13] An explanatory diagram of a trained model according to the first embodiment [Figure 14] FIG. 1 is a diagram showing a functional configuration of an information processing system according to second and third embodiments. [Figure 15] An explanatory diagram of a trained model according to the second embodiment [Figure 16] 10 is a flowchart showing the flow of pre-examination determination processing according to the third embodiment. [Figure 17] 10 is a flowchart showing the flow of an in-examination determination process according to the third embodiment. [Figure 18] An explanatory diagram of a trained model according to the third embodiment [Figure 19] FIG. 10 is an explanatory diagram of an information processing system according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0011] A control method, an information processing system, an information processing device, and a program according to embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the same or corresponding parts in the drawings are denoted by the same reference numerals.

[0012] [Embodiment 1] (Regarding Information Processing System 1 According to Embodiment 1) 1 and 2, an information processing system 1 according to an embodiment of the present disclosure includes an inspection device 100 that uses image processing to inspect whether or not there is an abnormality in a workpiece 2 as an example of an inspection object or an imaged object, a learning device 200 that generates a trained model (described later) through machine learning, and a storage device 300 that stores information. The inspection device 100, the learning device 200, and the storage device 300 are capable of transmitting and receiving information via, for example, a wireless local area network (LAN), which is an example of a communication network.

[0013] The workpiece 2 is an electronic circuit board, such as a computer motherboard or mainboard. As shown in Fig. 2, electronic components such as a connection terminal 21, a CPU (Central Processing Unit) 22, a memory 23, a connector 24, a port 25, an electrolytic capacitor 26, and a film capacitor 27 are attached to the workpiece 2. In this embodiment, for example, the electrolytic capacitor 26 is a glossy component.

[0014] In the information processing system 1 according to this embodiment, the inspection device 100 stores in advance in the storage device 300 normal image information, which is information indicating normal images, which are images of good workpieces, that is, workpieces 2 determined to be normal by inspection, among captured images, which are images of workpieces 2. The learning device 200 acquires all normal image information stored in the storage device 300, i.e., information on a group of good workpiece images, as learning information, generates a trained model by machine learning using the training information, and stores the trained model in the storage device 300. The inspection device 100 acquires the trained model stored in the storage device 300, inputs captured image information indicating a captured image of the workpiece 2 that has become a new inspection target into the trained model, and generates sample image information indicating a sample image based on the normal image. The inspection device 100 then uses the sample image information to inspect the workpiece 2 for abnormalities.

[0015] Since there is a possibility that some parts of the sample image indicated by the sample image information generated by the trained model may be corrupted, the inspection device 100 identifies the corrupted parts and corrects them, i.e., retouches them, to prevent the corrupted parts from causing overdetection or erroneous detection. Furthermore, in order to reduce the variation in the sample images indicated by the sample image information generated by the captured images input to the trained model, the inspection device 100 preprocesses the learning information used by the learning device 200 for machine learning before storing it in the storage device 300.

[0016] (Regarding the inspection device 100 according to the first embodiment) Inspection device 100 includes a flat base 101 arranged on an XY plane, which is a horizontal plane. Inspection device 100 also includes a Y-axis actuator 102 as an example of a Y-axis moving member fixed to the upper surface of base 101 and extending in the Y-axis direction, which is the front-to-back direction on the XY plane, and a Z-axis actuator 103 as an example of a Z-axis moving member extending in the Z-axis direction, which is the up-down direction perpendicular to the XY plane. Inspection device 100 also includes an X-axis actuator 104 as an example of an X-axis moving member supported by Z-axis actuator 103 so as to be movable in the Z-axis direction and extending in the X-axis direction, which is the left-to-right direction on the XY plane.

[0017] The Y-axis actuator 102 includes a flat conveyance table 105 that is supported so as to be movable in the Y-axis direction and has the workpiece 2 fixed on its upper surface. The inspection device 100 also includes an observation camera 106 and a laser displacement meter 107 that are supported so as to be movable in the X-axis direction by the X-axis actuator 104. The observation camera 106 includes a camera 108 as an example of an imaging member, a lens 109 as an example of an optical system, and an illumination member 110.

[0018] The inspection device 100 also includes a displacement meter controller 111 as an example of a displacement measurement control unit that is connected to the laser displacement meter 107 and controls the measurement of the displacement between the workpiece 2 and the observation camera 106 by the laser displacement meter 107. The inspection device 100 also includes a lighting controller 112 as an example of an illumination light output control unit that is connected to the illumination member 110 and controls the output of illumination light by the illumination member 110. The inspection device 100 also includes an information processing device 113 as an example of an inference device that is connected to the camera 108 and performs information processing based on an image captured by the camera 108.

[0019] The inspection device 100 also includes a programmable logic controller (hereinafter referred to as "PLC") 114 as an example of an inspection device main body that is connected to a displacement meter controller 111, a lighting controller 112, and an information processing device 113 and performs overall control of the inspection device 100. The inspection device 100 also includes a GOT (Graphic Operation Terminal) 115 as an example of a display that is connected to the PLC 114 and displays information output from the PLC 114.

[0020] (Regarding information processing device 113 according to embodiment 1) The information processing device 113 is, for example, a computer device such as a personal computer (PC) used by a user of the inspection device. As shown in Fig. 3, the information processing device 113 includes an information acquisition unit 121 that acquires information and an information control unit 122 that controls information processing. The information control unit 122 includes a first information control unit 123 and a second information control unit 124. The information processing device 113 also includes an image division unit 125 that divides an image, a feature amount calculation unit 126 that calculates a feature amount of the image, a failure determination unit 127 that determines whether the image is failure or not, an abnormality determination unit 128 that determines an abnormality in the image, an information generation unit 129 that generates information, a gloss determination unit 130 that determines whether the image is glossy or not, and an information output unit 131 that outputs information.

[0021] (Regarding the learning device 200 according to the first embodiment) The learning device 200 is, for example, a computer equipped with a GPU (Graphics Processing Unit), a so-called GPU machine. Note that the learning device 200 is not limited to a GPU machine and may be any other computer capable of performing calculations for generating a trained model through machine learning using training information. For example, the learning device 200 may be a personal computer equipped with a CPU, a supercomputer, a workstation, or the like. The learning device 200 includes a training information acquisition unit 210 that acquires training information, a trained model generation unit 220 that generates a trained model, and a trained model output unit 230 that outputs the trained model.

[0022] (Regarding the storage device 300 according to the first embodiment) The storage device 300 is, for example, a computer equipped with database management software, i.e., a so-called database server. The storage device 300 includes a training information storage unit 310 that stores training information, a trained model storage unit 320 that stores trained models, and an information transmission / reception unit 330 that transmits and receives information.

[0023] (Hardware Configuration of Information Processing Device 113 According to Embodiment 1) 4, the information processing device 113 includes a control unit 51 that executes processing in accordance with a control program 59. The control unit 51 includes an arithmetic processing unit such as a CPU or a GPU. In accordance with the control program 59, the control unit 51 functions as an information control unit 122, a first information control unit 123, a second information control unit 124, an image dividing unit 125, a feature amount calculation unit 126, a failure determination unit 127, an abnormality determination unit 128, an information generation unit 129, and a gloss determination unit 130 shown in FIG.

[0024] 4, information processing device 113 includes main memory 52 into which control program 59 is loaded and which is used as a work area for control unit 51. Main memory 52 includes a volatile memory such as RAM (Random Access Memory).

[0025] Information processing device 113 also includes external storage unit 53 that stores control program 59 in advance. External storage unit 53 supplies information stored by this program to control unit 51 in accordance with instructions from control unit 51, and stores data supplied from control unit 51. External storage unit 53 includes non-volatile memory such as a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0026] Information processing device 113 also includes an operation unit 54 that is operated by a user. Information input via operation unit 54 is supplied to control unit 51. Operation unit 54 includes information input components such as a keyboard, a mouse, and a touch panel.

[0027] Furthermore, information processing device 113 includes display unit 55 that displays information input via operation unit 54 and information output by control unit 51. Display unit 55 includes a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0028] Information processing device 113 also includes a transmitting / receiving unit 56 that transmits and receives information. Transmitting / receiving unit 56 includes information communication components such as a communication network terminal device that connects to a network and a wireless communication device. Transmitting / receiving unit 56 functions as information acquisition unit 121 and information output unit 131 shown in FIG. 3.

[0029] Returning to FIG. 4, in the information processing device 113, the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55 and the transmission / reception unit 56 are all connected to the control unit 51 via the internal bus 50.

[0030] 3 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the information processing device 113 executes an information acquisition step performed by the information acquisition unit 121, an information control step performed by the information control unit 122, a first information control step performed by the first information control unit 123, and a second information control step performed by the second information control unit 124. Furthermore, for example, the information processing device 113 executes an image division step performed by the image division unit 125, a feature amount calculation step performed by the feature amount calculation unit 126, a failure determination step performed by the failure determination unit 127, and an abnormality determination step performed by the abnormality determination unit 128. Furthermore, for example, the information processing device 113 executes an information generation step performed by the information generation unit 129, a gloss determination step performed by the gloss determination unit 130, and an information output step performed by the information output unit 131.

[0031] (Hardware Configuration of Learning Device 200 According to Embodiment 1) 4, like the information processing device 113, the learning device 200 also includes a control unit 51, a main memory unit 52, an external memory unit 53, an operation unit 54, a display unit 55, and a transmission / reception unit 56. The control unit 51 functions as the trained model generation unit 220 shown in FIG. 3 in accordance with a control program 59, and the transmission / reception unit 56 functions as the training information acquisition unit 210 and the trained model output unit 230. Returning to FIG. 4, the learning device 200 realizes the functions of the above-mentioned units 210 to 230 shown in FIG. 3 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the learning device 200 executes a training information acquisition unit step performed by the training information acquisition unit 210, a trained model generation step performed by the trained model generation unit 220, and a trained model output step performed by the trained model output unit 230.

[0032] (Hardware Configuration of Storage Device 300 According to First Embodiment) 4, like the information processing device 113 and the learning device 200, the storage device 300 also includes a control unit 51, a main memory unit 52, an external memory unit 53, an operation unit 54, a display unit 55, and a transmission / reception unit 56. The external memory unit 53 functions as the learning information storage unit 310 and the trained model storage unit 320 shown in FIG. 3, and the transmission / reception unit 56 functions as the information transmission / reception unit 330. Returning to FIG. 4, the storage device 300 realizes the functions of the above-mentioned units 310 to 330 shown in FIG. 3 by the control unit 51 using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources. For example, the storage device 300 executes a learning information storage step performed by the learning information storage unit 310, a trained model storage step performed by the trained model storage unit 320, and an information transmission / reception step performed by the information transmission / reception unit 330.

[0033] (Details of Functional Configuration of Information Processing Device 113 According to Embodiment 1) 3, the information acquiring unit 121 acquires captured image information indicating a captured image of the workpiece 2 from the camera 108. Note that the captured image information acquired by the information acquiring unit 121 becomes normal image information if it is determined that there is no abnormality as a result of an inspection described below, whereas it becomes abnormal image information, which is information indicating an abnormal image that is an image of a defective workpiece, if it is determined that there is an abnormality.

[0034] The information acquisition unit 121 also acquires the trained model by transmitting predetermined trained model request information to the storage device 300 and receiving the trained model transmitted from the storage device 300. Here, in the present embodiment, the trained model includes a normal image trained model generated by the learning device 200 through machine learning using normal image information, and a normal block image trained model generated by machine learning using normal block image information indicating normal block images into which a normal image is divided by the image segmentation unit 125 (described later). Specifically, the normal block image trained models include a first normal block image trained model as an example of a specific trained model and a first normal block image trained model generated by machine learning using first normal block image information indicating a first normal block image (described later), a second normal block image trained model as an example of a specific trained model and a second normal block image trained model generated by machine learning using second normal block image information indicating a second normal block image, ..., and a ninth normal block image trained model as an example of a specific trained model and a ninth trained model generated by machine learning using ninth normal block image information indicating a ninth normal block image.

[0035] Furthermore, when checking the normal image trained model before inspecting the work 2, the information acquisition unit 121 acquires normal image information by transmitting predetermined normal image information request information to the storage device 300 and receiving the normal image information transmitted from the storage device 300. The information acquisition unit 121 acquires any of the normal image information stored in the storage device 300, as well as template image information indicating a template image, which will be described later.

[0036] The information control unit 122 performs control to input captured image information into the learned model to generate sample image information. Specifically, the first information control unit 123 performs control to input captured image information into the normal image learned model to generate sample image information. When checking the normal image learned model before inspecting the work 2, the first information control unit 123 performs control to input normal image information acquired in advance into the normal image learned model as captured image information to generate sample image information.

[0037] The image dividing unit 125 divides an image into block images each having a predetermined number of pixels. In this embodiment, for example, if n and m are integers equal to or greater than 2 and the number of pixels of the image is (3×n)×(3×m) pixels, the image dividing unit 125 divides the image into nine equal parts, dividing the n×m pixel image into a first block image, a second block image, ..., and a ninth block image. Specifically, the image dividing unit 125 divides the captured image indicated by the captured image information into imaging block images that are block images of the captured image, and divides the sample image indicated by the sample image information into sample block images that are block images of the sample image. Thus, as shown in FIG. 5 , the captured image is divided into a first imaging block image, a second imaging block image, ..., and a ninth imaging block image as imaging block images of n×m pixels, and the sample image is divided into a first sample block image, a second sample block image, ..., and a ninth sample block image as sample block images of n×m pixels.

[0038] When checking the normal image trained model before inspecting the work 2, the image dividing unit 125 divides the sample image indicated by the sample image information into sample block images, that is, a first sample block image, a second sample block image, ..., a ninth sample block image. In this case, the image dividing unit 125 divides a template image (described later) indicated by the template image information into a first template block image, a second template block image, ..., a ninth template block image, which are block images of the template image.

[0039] Returning to Fig. 3, the feature amount calculation unit 126 calculates a feature amount of a specific imaging block image, which is a block image of a specific portion of the captured image, and a feature amount of a specific sample block image, which is a block image of a specific portion of the sample image. Note that the feature amount of the specific imaging block image is a value of the mean squared error (MSE) (hereinafter referred to as "MSE value") between the specific imaging block image and a specific template block image, which is a block image of a specific portion of a template image predetermined from a plurality of types of normal images. Also, the feature amount of the specific sample block image is the MSE value between the specific template block image and the specific sample block image.

[0040] Therefore, for example, if the block image of a specific location is the first block image, the feature calculation unit 126 calculates the MSE value between the first template block image and the first captured block image as the feature of the first captured block image, and calculates the MSE value between the first template block image and the first sample block image as the feature of the first sample block image. Here, the template image is, for example, the most average normal image selected from multiple types of normal images. Therefore, the normal image trained model generates the sample image by performing a process to bring the captured image closer to the template image. Therefore, if the captured work 2 is normal, the MSE value of the first sample block image is likely to be smaller than the MSE value of the first captured block image.

[0041] In this embodiment, the feature calculation unit 126 calculates all of the feature amounts of the first imaging block image, the feature amounts of the second imaging block image, ..., the feature amounts of the ninth imaging block image, and the feature amounts of the first sample block image, the feature amounts of the second sample block image, ..., the feature amounts of the ninth sample block image. Furthermore, when checking the normal image trained model before inspecting the work 2, the feature calculation unit 126 calculates all of the feature amounts of the first sample block image, the feature amounts of the second sample block image, ..., the feature amounts of the ninth sample block image.

[0042] The corruption determination unit 127 determines whether a specific sample block image is corrupted based on the feature amount of the specific sample block image. As shown in FIG. 6B , if the feature amount of the specific imaging block image is below a predetermined threshold while the feature amount of the specific sample block image exceeds the threshold, the corruption determination unit 127 determines that the specific sample block image is corrupted. For example, if the MSE value of the first imaging block image is below a threshold while the MSE value of the first sample block image exceeds the threshold, the corruption determination unit 127 determines that the first sample block image is corrupted. Note that in this embodiment, the first image blocks, the second image blocks, ..., the ninth image blocks are all compared between the captured image and the sample image, and it is determined whether the first sample block image, the second sample block image, ..., the ninth sample block image are all corrupted.

[0043] 6A, when checking the normal image trained model before inspecting the work 2, if the feature amount of the specific sample block image exceeds a threshold, the failure determination unit 127 determines that the specific sample block image is failed. For example, if the MSE value of the first sample block image exceeds a threshold, the failure determination unit 127 determines that the first sample block image is failed.

[0044] Returning to FIG. 3 , the anomaly determination unit 128 determines whether or not an abnormal portion of the workpiece 2 is included in the specific imaging block image based on the comparison result between the feature amount of the specific imaging block image and the feature amount of the specific sample block image. When the feature amount of the specific imaging block image exceeds a threshold value while the feature amount of the specific sample block image is equal to or less than the threshold value, the anomaly determination unit 128 determines that an abnormal portion of the workpiece 2 is included in the specific imaging block image. For example, as shown in FIG. 7A , when the MSE value of the first imaging block image exceeds a threshold value while the MSE value of the first sample block image is equal to or less than the threshold value, the anomaly determination unit 128 determines that an abnormal portion of the workpiece 2 is included in the first imaging block image. Note that in this embodiment, the first image blocks, the second image blocks, ..., the ninth image blocks are all compared between the captured image and the sample image, and it is determined whether or not an abnormal portion of the workpiece 2 is included in each of the first imaging block image, the second imaging block image, ..., and the ninth imaging block image.

[0045] Furthermore, when the feature amount of the specific imaging block image and the feature amount of the specific sample block image are both equal to or less than the threshold, the abnormality determination unit 128 determines that the specific imaging block image does not contain an abnormal portion of the workpiece 2. For example, as shown in Fig. 7B, when the MSE value of the first imaging block image and the MSE value of the first sample block image are both equal to or less than the threshold, the abnormality determination unit 128 determines that the first imaging block image does not contain an abnormal portion of the workpiece 2.

[0046] On the other hand, when the feature amount of the specific imaging block image and the feature amount of the specific sample block image both exceed the threshold, the abnormality determination unit 128 determines that an error has occurred because there is a possibility that the specific sample block image is corrupted, that the specific imaging block image contains an abnormal portion of the work 2, or both have occurred. For example, as shown in Fig. 7C, when the MSE value of the first imaging block image and the MSE value of the first sample block image both exceed the threshold, the abnormality determination unit 128 determines that an error has occurred in the first imaging block image.

[0047] 3, when the specific sample block image is corrupted, the second information control unit 124 performs control to input specific imaging block image information into the specific normal block image trained model to generate updated specific sample block image information, which is information indicating a new specific sample block image based on the specific normal block image. For example, when the first sample block image is corrupted, the second information control unit 124 performs control to input the first imaging block image information into the first normal block image trained model to generate updated first sample block image information.

[0048] If the specific sample block image is corrupted, the information generation unit 129 generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the updating specific sample block image information. After replacing the corrupted specific sample block image with the new specific sample block image indicated by the updating specific sample block image information, the information generation unit 129 performs image processing to smooth the brightness gradient of the edge portion, which is the boundary portion between the new specific sample block image and other sample block images. Note that, as image processing to smooth the brightness gradient of the edge portion, for example, a noise removal process such as a median filter that matches the edge portion to a neighboring image, or an image synthesis process such as Poisson blending that combines the edge image with a mask image can be used.

[0049] For example, if the first sample block image is corrupted, the information generating unit 129 first replaces the corrupted first sample block image with a new first sample block image indicated by the updated first sample block image information. Then, the information generating unit 129 performs image processing on the new replaced first sample block image to smooth the brightness gradient at the edge portion with the second sample block image and the edge portion with the fourth sample block image, and generates updated sample image information indicating the new sample image.

[0050] When the abnormality determination unit 128 determines that none of the captured block images contains an abnormal portion of the workpiece 2 during inspection, i.e., when the captured image information is determined to be normal image information as the inspection result, the gloss determination unit 130 determines whether the normal image indicated by the normal image information is a glossy normal image or a non-glossy normal image. The gloss determination unit 130 determines whether the normal image is a glossy normal image or a non-glossy normal image by determining whether the brightness of the glossy part of the normal workpiece 2 contained in the normal image exceeds a predetermined brightness threshold. Note that, as a result of the gloss determination unit 130 determining multiple types of normal images, the information processing device 113 counts the number of glossy normal images using a glossy normal image counter and counts the number of non-glossy normal images using a non-glossy normal image counter.

[0051] The information output unit 131 outputs determination result information, which is information indicating the determination result of the failure determination unit 127. For example, when the failure determination unit 127 determines that a specific sample block image is failed regardless of whether it is before or during inspection, the information output unit 131 outputs failure detection information, which is information indicating that the normal image trained model has generated a failed specific sample block image, to the PLC 114. Furthermore, when the information generation unit 129 changes the failed specific sample block image during inspection to generate updated sample image information, the information output unit 131 outputs retouching information, which is information indicating that updated sample image information has been generated, to the PLC 114.

[0052] Furthermore, when the abnormality determination unit 128 determines that an abnormal portion of the work 2 is included in a specific imaging block image during inspection, the information output unit 131 outputs, to the PLC 114, inspection abnormality information indicating that an abnormal portion of the work 2 is included in the specific imaging block image. Furthermore, when the abnormality determination unit 128 determines that no abnormal portion of the work 2 is included in any imaging block image during inspection, the information output unit 131 outputs, to the PLC 114, inspection normality information indicating that the work 2 is normal. Furthermore, when the abnormality determination unit 128 determines that an error has occurred during inspection, the information output unit 131 outputs, to the PLC 114, error information indicating that an error has occurred. As a result, the PLC 114 can, for example, display the acquired information on the GOT 115 or output a warning sound.

[0053] The information output unit 131 also outputs the normal image information and normal block image information obtained as the inspection results, i.e., the first normal block image information, the second normal block image information, ..., the ninth normal block image information, by transmitting the normal image information and normal block image information to the storage device 300 and storing them. Here, the information output unit 131 outputs the same number of glossy normal image information, which is normal image information indicating a glossy normal image, and the same number of non-glossy normal image information, which is normal image information indicating a non-glossy normal image, to the storage device 300 based on the value of the glossy normal image counter and the value of the non-glossy normal image counter, which are the determination results by the gloss determination unit 130. The information output unit 131 also outputs the same number of normal block image information indicating each normal block image of the glossy normal image and each normal block image information indicating each normal block image of the non-glossy normal image to the storage device 300.

[0054] (Details of the Functional Configuration of Learning Device 200 According to Embodiment 1) The learning information acquisition unit 210 acquires learning information by transmitting predetermined learning information request information to the storage device 300 and receiving the learning information transmitted from the storage device 300. The learning information acquisition unit 210 acquires all normal image information and normal block image information stored in the storage device 300, i.e., the first normal block image information, second normal block image information, ..., ninth normal block image information, as learning information.

[0055] The trained model generation unit 220 generates a trained model through machine learning using the training information acquired by the trained model generation unit 220. The trained model generation unit 220 generates a normal image trained model through machine learning using normal image information as training information. The trained model generation unit 220 also generates a first normal block image trained model through machine learning using first normal block image information as training information. The trained model generation unit 220 also generates a second normal block image trained model through machine learning using second normal block image information as training information, ..., and generates a ninth normal block image trained model through machine learning using ninth normal block image information as training information.

[0056] The machine learning learning algorithm used by the trained model generation unit 220 to generate the trained model described above is, for example, a neural network, which is an example of supervised learning. Here, supervised learning refers to a technique in which a learning device is provided with data pairs of inputs and resulting labels, and the learning device learns the features of the training information and infers the results from the inputs. The neural network is composed of an input layer containing multiple neurons, an intermediate layer also called a hidden layer containing multiple neurons, and an output layer containing multiple neurons. Note that although the intermediate layer is described as one layer, it may be two or more layers.

[0057] For example, in a three-layer neural network as shown in Figure 8, when multiple inputs are input to the input layer (X1, X2, X3), the values ​​are multiplied by the first weights W1 (w11, w12, ..., w16) and input to the middle layer (Y1, Y2), and the result is further multiplied by the second weights W2 (w21, w22, ..., w26) and output from the output layer (Z1, Z2, Z3). Therefore, the output result changes depending on the values ​​of the weights W1 and W2.

[0058] In this embodiment, the neural network learns multiple types of normal images, i.e., a group of good-quality images and normal block images, i.e., each block image of the group of good-quality images, through so-called supervised learning in accordance with learning information, which is training data acquired in advance by the learning device 200. That is, the neural network adjusts the weights W1 and W2 to input normal image information to the input layer and bring the results output from the output layer closer to the normal images and normal block images included in the training data. By performing the above-described learning, the learning device 200 generates a normal-image trained model, a first normal-block-image trained model, a second normal-block-image trained model, ..., a ninth normal-block-image trained model.

[0059] The trained model output unit 230 outputs the trained models generated by the trained model generation unit 220 by transmitting them to and storing them in the storage device 300. The trained model output unit 230 outputs a normal image trained model, a first normal block image trained model, a second normal block image trained model, ..., a ninth normal block image trained model.

[0060] (Details of the Functional Configuration of the Storage Device 300 According to the First Embodiment) The information transmitting / receiving unit 330 receives the learning information transmitted from the information processing device 113, and the learning information storage unit 310 stores the received learning information. The information transmitting / receiving unit 330 receives normal image information and normal block image information, i.e., first normal block image information, second normal block image information, ..., ninth normal block image information, as learning information, and the learning information storage unit 310 stores the received normal image information and normal block image information as learning information.

[0061] Furthermore, when the information transmitting / receiving unit 330 receives request information for learning information transmitted from the learning device 200, it transmits the learning information stored in the learning information storage unit 310 to the learning device 200. The information transmitting / receiving unit 330 transmits the normal image information and normal block image information stored in the learning information storage unit 310 to the learning device 200 as learning information.

[0062] Furthermore, when the information transmitting / receiving unit 330 receives request information for normal image information transmitted from the information processing device 113, it transmits the normal image information stored in the learning information storage unit 310 to the information processing device 113. The information transmitting / receiving unit 330 transmits any of the normal image information stored in the learning information storage unit 310 and the above-mentioned template image information to the information processing device 113.

[0063] Furthermore, the information transmitting and receiving unit 330 receives the trained models transmitted from the learning device 200, and the trained model storage unit 320 stores the received trained models. The information transmitting and receiving unit 330 receives the normal image trained model and the normal block image trained model, i.e., the first normal block image trained model, the second normal block image trained model, ..., the ninth normal block image trained model, and the trained model storage unit 320 stores the received normal image trained model and normal block image trained model as trained models.

[0064] Furthermore, when the information transmitting / receiving unit 330 receives request information for a trained model transmitted from the information processing device 113, it transmits the trained model stored in the trained model storage unit 320 to the information processing device 113. The information transmitting / receiving unit 330 transmits the normal image trained model and the normal block image trained model stored in the trained model storage unit 320 to the information processing device 113 as trained models.

[0065] (Regarding the learning process according to the first embodiment) Next, the operation of the learning device 200 to generate and output a trained model will be described using a flowchart. When the learning device 200 is powered on, it starts executing the learning process shown in Fig. 9. First, the learning information acquisition unit 210 transmits request information for learning information to the storage device 300 and acquires the learning information transmitted from the storage device 300 (step S201). For example, the learning information acquisition unit 210 acquires all of the normal image information, first normal block image information, second normal block image information, ..., ninth normal block image information stored in the storage device 300 as learning information.

[0066] After acquiring the information, the trained model generation unit 220 generates a trained model through machine learning using the acquired training information (step S202). For example, the trained model generation unit 220 generates a normal image trained model through machine learning using normal image information, and generates a first normal block image trained model through machine learning using first normal block image information. Also, for example, the trained model generation unit 220 generates a second normal block image trained model through machine learning using second normal block image information, ..., and generates a ninth normal block image trained model through machine learning using ninth normal block image information.

[0067] After generating the trained models, the trained model output unit 230 outputs the trained models by transmitting the trained models generated by the trained model generation unit 220 to the storage device 300 for storage (step S203), and ends the processing. For example, the trained model output unit 230 outputs a normal image trained model, a first normal block image trained model, a second normal block image trained model, ..., a ninth normal block image trained model.

[0068] (Regarding pre-test determination processing according to embodiment 1) Next, the operation of the information processing device 113 to determine a failure in a sample image before inspection will be described using a flowchart. When the information processing device 113 is powered on, it starts executing the pre-inspection determination process shown in FIG. 10. First, the information acquisition unit 121 transmits request information for trained models to the storage device 300 and acquires the trained models from the storage device 300 (step S101). For example, the information acquisition unit 121 acquires a normal image trained model and a normal block image trained model, i.e., a first normal block image trained model, a second normal block image trained model, ..., a ninth normal block image trained model. After acquiring the trained models, the information acquisition unit 121 transmits request information for normal image information to the storage device 300 and acquires normal image information and template image information from the storage device 300 (step S102).

[0069] After acquiring the information, the first information control unit 123 inputs the normal image information as captured image information into the normal image trained model and controls the generation of sample image information (step S103). After generating the sample image information, the image division unit 125 divides the sample image indicated by the sample image information into sample block images, and divides the template image indicated by the template image information into template block images (step S104). After image division, the feature calculation unit 126 calculates the feature of the sample block image (step S105).

[0070] For example, the image dividing unit 125 divides the sample image into a first sample block image, a second sample block image, ..., a ninth sample block image, and divides the template image into a first template block image, a second template block image, ..., a ninth template block image. Then, the feature calculation unit 126 calculates the MSE value between the first template block image and the first sample block image, the MSE value between the second template block image and the second sample block image, ..., the MSE value between the ninth template block image and the ninth sample block image, thereby calculating the feature values ​​of the first sample block image, the feature values ​​of the second sample block image, ..., the ninth sample block image.

[0071] After calculating the feature amounts, the failure determination unit 127 determines whether the specific sample block image is corrupted based on the feature amounts of the specific sample block image (step S106). For example, if the MSE value of the first sample block image exceeds a threshold, the failure determination unit 127 determines that the first sample block image is corrupted. If none of the sample block images is corrupted (step S106; N), the information processing device 113 terminates the processing. On the other hand, if the specific sample block image is corrupted (step S106; Y), the information output unit 131 outputs to the PLC 114 failure detection information indicating that the normal image trained model has generated a corrupted specific sample block image (step S107), and terminates the processing.

[0072] (Regarding the In-Test Determination Process According to the First Embodiment) Next, a flowchart will be used to explain the operation of the information processing device 113 to determine a failure in a sample image during inspection and to determine an abnormality in the captured image of the work 2. After completing the pre-inspection determination process described above, the information processing device 113 starts executing the in-inspection determination process shown in Figs. 11A and 11B. As shown in Fig. 11A, first, the information acquisition unit 121 acquires captured image information indicating the captured image of the work 2 from the camera 108 (step S111), and the first information control unit 123 inputs the captured image information into the normal image trained model to generate sample image information (step S112).

[0073] After generating the sample image information, the image dividing unit 125 divides the captured image indicated by the captured image information into imaging block images, and divides the sample image indicated by the sample image information into sample block images (step S113). After image division, the feature amount calculation unit 126 calculates the feature amounts of the imaging block images and the feature amounts of the sample block images (step S114).

[0074] For example, the image dividing unit 125 divides the captured image into a first imaging block image, a second imaging block image, ..., and a ninth imaging block image. Then, the feature amount calculation unit 126 calculates the MSE value between the first template block image and the first imaging block image, the MSE value between the second template block image and the second imaging block image, ..., and the MSE value between the ninth template block image and the ninth imaging block image, thereby calculating the feature amount of the first imaging block image, the feature amount of the second imaging block image, ..., and the feature amount of the ninth imaging block image. Note that the division of the sample image and the calculation of the feature amount of the sample block image are similar to the above-mentioned pre-inspection determination processing, and therefore will not be described in order to reduce redundant explanation.

[0075] After calculating the feature amounts, the failure determination unit 127 determines whether the specific sample block image is corrupted based on the result of comparing the feature amounts of the specific imaging block image with the feature amounts of the specific sample block image (step S115). For example, if the MSE value of the first imaging block image is equal to or less than a threshold value while the MSE value of the first sample block image exceeds the threshold value, the failure determination unit 127 determines that the first sample block image is corrupted. If the specific sample block image is corrupted (step S115; Y), the second information control unit 124 controls the specific normal block image trained model to input the specific imaging block image information and generate updated specific sample block image information (step S116). For example, if the first sample block image is corrupted, the second information control unit 124 controls the first normal block image trained model to input the first imaging block image information and generate updated first sample block image information.

[0076] After generating the specific sample block image information for update, the information generating unit 129 generates updated sample image information, which is information indicating a new sample image obtained by changing the corrupted specific sample block image based on the new specific sample block image indicated by the specific sample block image information for update (step S117). For example, if the first sample block image is corrupted, the information generating unit 129 generates updated sample image information indicating a new sample image obtained by replacing the corrupted first sample block image with the new first sample block image indicated by the first sample block image information for update and then performing image processing to smooth the brightness gradient of the edge portion. Hereinafter, the updated sample image information will be described as sample image information.

[0077] After generating the information, the information output unit 131 outputs to the PLC 114 failure detection information indicating that the normal image trained model has generated a specific sample block image that has failed, and retouching information indicating that updated sample image information has been generated (step S118).

[0078] 11B, after the information output, or if none of the sample block images are damaged (step S115; Y), the abnormality determination unit 128 determines whether or not the specific imaging block image contains an abnormal portion of the workpiece 2 based on a result of comparing the feature amount of the specific imaging block image with the feature amount of the specific sample block image (step S119). For example, if the MSE value of the first imaging block image exceeds a threshold while the MSE value of the first sample block image is equal to or less than the threshold, the abnormality determination unit 128 determines that the first imaging block image contains an abnormal portion of the workpiece 2. If the specific imaging block image contains an abnormal portion of the workpiece 2 (step S119; Y), the information output unit 131 outputs inspection abnormality information indicating that the specific imaging block image contains an abnormal portion of the workpiece 2 to the PLC 114 (step S120), and ends the process.

[0079] On the other hand, if none of the imaging block images contains an abnormal portion of the workpiece 2 (step S119; N), the abnormality determination unit 128 determines whether or not an error that makes it impossible to determine an abnormality has occurred (step S121). If the feature amount of the specific imaging block image and the feature amount of the specific sample block image both exceed the threshold, the abnormality determination unit 128 determines that an error has occurred. If an error has occurred (step S121; Y), the information output unit 131 outputs error information indicating that an error has occurred to the PLC 114 (step S122), and ends the processing. On the other hand, if no error has occurred (step S121; N), the information output unit 131 outputs inspection normality information indicating that the workpiece 2 is normal to the PLC 114 (step S123), and ends the processing.

[0080] (Regarding normal image information output processing according to embodiment 1) Next, an operation of the information processing device 113 to output normal image information after inspection will be described using a flowchart. When the information processing device 113 determines that multiple types of captured image information are normal image information as the inspection result as a result of the above-described during-inspection determination process, it starts executing the normal image information output process shown in FIG. 12. First, the information processing device 113 resets the values ​​of the glossy normal image counter and the non-glossy normal image counter to 0 (step S151), and the gloss determination unit 130 determines whether the normal image indicated by the normal image information is a glossy normal image (step S152). For example, the gloss determination unit 130 determines whether the normal image is a glossy normal image or a non-glossy normal image by determining whether the brightness of a glossy part of a normal workpiece 2 included in the normal image exceeds a predetermined brightness threshold.

[0081] If the image is a glossy normal image (step S152; Y), +1 is added to the value of the glossy normal image counter (step S153). If the image is a non-glossy normal image (step S152; N), +1 is added to the value of the non-glossy normal image counter (step S154). After each counter increment, it is determined whether or not all normal images indicated by the normal image information have been determined (step S155). If all normal images have not been determined (step S155; N), the process returns to step S152 and the next normal image is determined. If all normal images have been determined (step S155; Y), the information output unit 131 outputs the same number of glossy normal image information and non-glossy normal image information to the storage device 300 based on the values ​​of the glossy normal image counter and the non-glossy normal image counter, and also outputs the same number of normal block image information based on the glossy normal image information and the normal block image information based on the non-glossy normal image information to the storage device 300 (step S156), and the process ends.

[0082] As described above, according to the information processing system 1 of this embodiment, in the information processing device 113, the information acquisition unit 121 acquires captured image information indicating a captured image of the workpiece 2. Furthermore, the information control unit 122 controls the input of the captured image information acquired by the information acquisition unit 121 into a trained model generated by machine learning using normal image information to generate sample image information indicating a sample image based on the normal image. Furthermore, the image division unit 125 divides the image into block images, and the feature calculation unit 126 calculates the feature amounts of a specific sample block image of the sample image. Then, the failure determination unit 127 determines whether the specific sample block image is failed based on the feature amounts of the specific sample block image, and the information output unit 131 outputs determination result information indicating the determination result.

[0083] In the conventionally known information processing system 1, abnormalities in the captured image are determined for each pixel, and if this system is used to determine defects in the sample image, it is necessary to determine defects in the sample image for each pixel, which may result in the sample image not being able to be identified. Furthermore, in the conventionally known information processing system 1, it is necessary to determine defects in the sample image for each pixel, so that the positioning of the object to be imaged during image capture must be performed with high precision in order to identify defects in the sample image.

[0084] In response to this, the information processing device 113 determines whether a specific sample block image is corrupted based on the feature values ​​of the specific sample block image calculated by dividing the sample image into sample block images with a predetermined number of pixels, and outputs the result. Therefore, the information processing device 113 determines whether a sample image is corrupted for each block image, including potentially corrupted areas of the sample image and their surrounding information. As a result, the information processing system 1 according to this embodiment can more easily identify corruption in a sample image generated by a trained model than an information processing system that does not determine whether a specific sample block image is corrupted based on the feature values ​​of the specific sample block image. Furthermore, by doing so, the information processing system 1 according to this embodiment can identify the presence or absence of a corruption in a sample image and the location of the corruption, allowing for positional misalignment without requiring highly accurate positioning of the object to be imaged during imaging, thereby reducing the manufacturing cost of the inspection device 100 required for positioning.

[0085] Furthermore, according to the information processing system 1 of this embodiment, the trained model that generates the sample image information is a normal image trained model that is generated by machine learning using normal image information. By doing this, the information processing system 1 according to this embodiment is even less susceptible to sample image corruption than an information processing system that does not generate sample image information using a normal image trained model.

[0086] Furthermore, according to the information processing system 1 of this embodiment, in the information processing device 113, the information acquisition unit 121 acquires captured image information indicating a captured image of a normal workpiece 2. Furthermore, the feature calculation unit 126 calculates the MSE value between a specific template block image of a predetermined template image included in the non-defective image group and the specific sample block image as the feature of the specific sample block image. Then, the failure determination unit 127 determines that the specific sample block image is failed if the feature of the specific sample block image exceeds a threshold.

[0087] In this way, for example, when the information processing device 113 acquires normal image information before an inspection, it inputs the acquired normal image information as captured image information into the trained model to generate sample image information, and calculates the MSE value of a specific sample block image based on the sample image information, thereby determining whether the specific sample block image is corrupted. As a result, the information processing system 1 according to this embodiment can check the vulnerability of the trained model based on the MSE value of a specific sample block image before actually performing an inspection.

[0088] Furthermore, according to the information processing system 1 of this embodiment, in the information processing device 113, the feature amount calculation unit 126 calculates the feature amount of a specific imaging block image of a captured image. Furthermore, the feature amount calculation unit 126 calculates the MSE value between the specific template block image and the specific imaging block image as the feature amount of the specific imaging block image, and also calculates the MSE value between the specific template block image and the specific sample block image as the feature amount of the specific sample block image. Then, the failure determination unit 127 determines that the specific sample block image is corrupted when the feature amount of the specific imaging block image is equal to or less than a threshold value while the feature amount of the specific sample block image exceeds the threshold value.

[0089] In this way, for example, when the information processing device 113 acquires captured image information during an inspection, it inputs the acquired captured image information into the trained model to generate sample image information, and calculates the MSE value of the specific captured block image and the MSE value of the specific sample block image based on the captured image information and the sample image information, thereby determining whether the specific sample block image is corrupted. As a result, even when no vulnerability was found in the preliminary check and the work 2 is ready for mass production, the information processing system 1 according to this embodiment can check the vulnerability of the trained model during actual inspection based on the MSE value of the specific captured block image and the MSE value of the specific sample block image.

[0090] In particular, according to the information processing system 1 of this embodiment, in the information processing device 113, the abnormality determination unit 128 determines whether or not the specific imaging block image contains an abnormal portion of the work 2 based on a result of comparing the feature amount of the specific imaging block image with the feature amount of the specific sample block image. Specifically, when the feature amount of the specific imaging block image exceeds a threshold value while the feature amount of the specific sample block image is equal to or less than the threshold value, the abnormality determination unit 128 determines that the specific imaging block image contains an abnormal portion of the work 2.

[0091] By doing this, the information processing device 113 can calculate the MSE value of a specific imaging block image and the MSE value of a specific sample block image based on the captured image information and the sample image information, and can not only determine whether the specific sample block image is corrupted or not, but also determine whether the specific imaging block image contains an abnormal part of the work 2.

[0092] Here, the phenomenon of the sample image breaking down may occur due to a lack of generation capability of the trained model in areas where there is high diversity in the sample image, such as areas where multiple parts of the workpiece 2 are arranged close together and the shape varies greatly depending on how each part is attached, or areas where the amount of reflected light received by the camera 108 varies greatly depending on how glossy parts of the workpiece 2 are attached.

[0093] In contrast, in the information processing system 1 according to the present embodiment, in the information processing device 113, the first information control unit 123 controls the generation of sample image information by inputting captured image information acquired by the information acquisition unit 121 into a normal image trained model, which is an example of a trained model generated by machine learning using normal image information. Furthermore, if the specific sample block image is corrupted, the second information control unit 124 inputs the specific captured block image information into the specific normal block image trained model generated by machine learning using the specific normal block image information to generate updated specific sample block image information indicating a new specific sample block image based on the specific normal block image. If the specific sample block image is corrupted, the information generation unit 129 generates updated sample image information indicating a new sample image obtained by updating the corrupted specific sample block image based on the new specific sample block image indicated by the updated specific sample block image information.

[0094] In this way, when a specific sample block image is corrupted, the information processing device 113 generates updated sample image information by changing the corrupted specific sample block image based on updated specific sample block image information generated by inputting specific imaging block image information into a specific normal block image learned model as an example of a learned model. Therefore, when a specific sample block image is corrupted due to insufficient generation capability of the normal image learned model, the information processing device 113 generates a new specific sample block image using a specific normal block image learned model that has a higher generation capability of specific sample block images than the normal image learned model, and generates new sample image information using the new specific sample block image.

[0095] As a result, the information processing system 1 according to this embodiment can more easily acquire a sample image without any defects than an information processing system that does not generate updated sample image information by changing a corrupted specific sample block image based on updated specific sample block image information generated by inputting specific imaging block image information into a specific normal block image trained model. Therefore, the information processing system 1 according to this embodiment can reduce overdetection and erroneous detection when the inspection device 100 performs an inspection using a sample image.

[0096] Furthermore, according to the information processing system 1 of this embodiment, the trained model that generates the sample image information is a normal image trained model generated by machine learning using normal image information, and the specific trained model that generates the update specific sample block image information is a specific normal block image trained model generated by machine learning using specific normal block image information. By doing this, the information processing system 1 according to this embodiment can more easily obtain a sample image without any defects than an information processing system that does not generate sample image information and updated specific sample block image information using a normal image trained model and a specific normal block image trained model.

[0097] Furthermore, according to the information processing system 1 of this embodiment, in the information processing device 113, the information generation unit 129 replaces the corrupted specific sample block image with a new specific sample block image indicated by the updated specific sample block image information, and then performs image processing to smooth the brightness gradient of the edge portion between the new specific sample block image and other sample block images.

[0098] By doing this, the information processing system 1 of the embodiment is less likely to make the above-mentioned edge portions of the new sample image indicated by the updated sample image information look unnatural than an information processing system 1 that does not perform image processing to smooth the brightness gradient of the edge portions after replacing a corrupted specific sample block image with a new specific sample block image, making it even easier to obtain a sample image without any corrupted portions.

[0099] Furthermore, in the information processing system 1 according to the present embodiment, when the information processing device 113 determines that multiple types of captured image information are normal image information as inspection results, the gloss determination unit 130 determines whether the normal image indicated by each type of normal image information is a glossy normal image or a non-glossy normal image, and the information processing device 113 counts the number of glossy normal images and the number of non-glossy normal images using a glossy normal image counter and a non-glossy normal image counter. Then, the information output unit 131 outputs the same number of glossy normal image information and non-glossy normal image information to the storage device 300 based on the values ​​of the glossy normal image counter and the non-glossy normal image counter.

[0100] As a result, in the storage device 300, the learning information storage unit 310 stores the same number of glossy normal image information and non-glossy normal image information. Also, in the learning device 200, the learning information acquisition unit 210 acquires the same number of glossy normal image information and non-glossy normal image information from the storage device 300 as learning information. Then, as shown in FIG. 13 , the trained model generation unit 220 generates a trained model by machine learning using the same number of acquired glossy normal image information and non-glossy normal image information, and the trained model output unit 230 outputs the generated trained model to the storage device 300.

[0101] In this way, the information processing device 113 generates sample image information by inputting the acquired captured image information into a trained model generated by machine learning using normal image information including glossy normal image information indicating a predetermined number of glossy normal images and non-glossy normal image information indicating the same number of non-glossy normal images. As a result, the information processing system 1 according to the present embodiment is more likely to generate sample images that allow for the presence or absence of gloss in glossy parts while reducing the degree of gloss to a level that is not too strong than an information processing system that does not generate a trained model by machine learning using the same number of glossy normal images and non-glossy normal images, thereby reducing the variation in sample images generated by the trained model.

[0102] [Embodiment 2] In the first embodiment, the learning device 200 performs preprocessing of the learning process by making the number of glossy normal image information and non-glossy normal image information included in the normal image information serving as learning information equal, but the preprocessing of the learning process is not limited to this. For example, preprocessing of the learning process may involve image processing of the normal image indicated by the normal image information serving as learning information. The information processing system 1 according to the second embodiment will be described in detail below with reference to FIGS. 14 and 15. Note that in the second embodiment, only the configurations different from those in the first embodiment will be described, and the same configurations as those in the first embodiment will not be described to avoid redundancy.

[0103] (Details of Functional Configuration of Information Processing Device 113 According to Embodiment 2) As shown in FIG. 14 , the information processing device 113 according to the present embodiment further includes an image processing unit 132 that performs image processing, but does not include the gloss determination unit 130. The control unit 51 shown in FIG. 4 functions as the image processing unit 132 shown in FIG. 14 in accordance with a control program 59. In the information processing device 113, the control unit 51 realizes the functions of the image processing unit 132 by using the main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, and the transmission / reception unit 56 as resources, and executes, for example, an image processing step, a first image processing step, and a second image processing step performed by the image processing unit 132. When the captured image information is determined to be normal image information as the inspection result, the image processing unit 132 performs chromaticity variation processing, which is image processing that adjusts the chromaticity of any one of red light, green light, and blue light corresponding to the three primary colors of light that represent the normal image, for the normal image indicated by the normal image information.

[0104] The image processing unit 132 generates, for example, first normal image information which is normal image information indicating a first normal image which is the original normal image, and second normal image information which is normal image information indicating multiple types of second normal images which have a stronger chromaticity of red light than the first normal image. Also, the image processing unit 132 generates, for example, third normal image information which is normal image information indicating multiple types of third normal images which have a stronger chromaticity of green light than the first normal image, and fourth normal image information which is normal image information indicating multiple types of fourth normal images which have a stronger chromaticity of blue light than the first normal image.

[0105] The image processing unit 132 also performs brightness variation processing, which is image processing for adjusting the brightness of each normal image indicated by each piece of normal image information. The image processing unit 132 generates, for example, fifth normal image information, which is normal image information indicating multiple types of fifth normal images obtained by adjusting the brightness of the first normal image indicated by the first normal image information, and sixth normal image information, which is normal image information indicating multiple types of sixth normal images obtained by adjusting the brightness of each second normal image indicated by the second normal image information. The image processing unit 132 also generates, for example, seventh normal image information, which is normal image information indicating multiple types of seventh normal images obtained by adjusting the brightness of each third normal image indicated by the third normal image information, and eighth normal image information, which is normal image information indicating multiple types of eighth normal images obtained by adjusting the brightness of each third normal image indicated by the fourth normal image information.

[0106] In this embodiment, the information output unit 131 outputs all of the normal image information to the storage device 300. The information output unit 131 also outputs all of the normal block image information indicating each normal block image of each normal image to the storage device 300. As a result, in the storage device 300, the learning information storage unit 310 stores, as learning information, each normal image information and each normal block image information corresponding to each normal image information.

[0107] In the learning device 200, the learning information acquisition unit 210 acquires, as learning information, each piece of normal image information and each piece of normal block image information corresponding to each piece of normal image information. The trained model generation unit 220 generates a normal image trained model through machine learning using each piece of normal image information as learning information, and generates each normal block image trained model through machine learning using each piece of normal block image information corresponding to each piece of normal image information as learning information. The trained model output unit 230 then outputs these generated trained models to the storage device 300 for storage. Therefore, the trained models acquired by the information acquisition unit 121 from the storage device 300 are the normal image trained model and each normal block image trained model generated through machine learning using each piece of normal image information, the normal block image information corresponding to each piece of normal image information, and each normal block image trained model.

[0108] As described above, in the information processing system 1 according to the present embodiment, when the inspection result of the information processing device 113 determines that multiple types of captured image information are normal image information, the image processing unit 132 performs chromaticity variation processing to generate first, second, third, and fourth normal image information based on the normal image information. Then, the information output unit 131 outputs all of the normal image information to the storage device 300.

[0109] As a result, in the storage device 300, the training information storage unit 310 stores not only first normal image information indicating the first normal image, which is the original normal image, but also second normal image information indicating a second normal image in which the chromaticity of red light has been adjusted, third normal image information indicating a third normal image in which the chromaticity of green light has been adjusted, and fourth normal image information indicating a fourth normal image in which the chromaticity of blue light has been adjusted, for the three primary colors of light representing the first normal image. Furthermore, in the learning device 200, the training information acquisition unit 210 acquires the first normal image information, second normal image information, third normal image information, and fourth normal image information from the storage device 300 as training information. Then, as shown in FIG. 15 , the trained model generation unit 220 generates a trained model by machine learning using the acquired first normal image information, second normal image information, third normal image information, and fourth normal image information, and the trained model output unit 230 outputs the generated trained model to the storage device 300.

[0110] In this way, the information processing device 113 generates sample image information by inputting the acquired captured image information into a trained model generated by machine learning using normal image information including the first normal image information, the second normal image information, the third normal image information, and the fourth normal image information. As a result, the information processing system 1 according to the present embodiment is more likely to generate sample images with average colors while allowing for changes in chromaticity and color tone of the captured image than an information processing system that does not generate a trained model by machine learning using the first normal image information, the second normal image information, the third normal image information, and the fourth normal image information, and can reduce variation in sample images generated by the trained model.

[0111] Therefore, for example, the information processing system 1 according to this embodiment can tolerate changes in the chromaticity and color tone of the substrate serving as the work 2 due to external factors such as individual differences in the camera 108, lens 109, and lighting member 110 included in the observation camera 106, and differences in the lot of the work 2, and can stably generate sample images with little variation to inspect the work 2 for abnormalities. In addition, the information processing system 1 according to this embodiment has the same effects as the information processing system 1 according to the first embodiment.

[0112] [Embodiment 3] In the second embodiment, a chromaticity variation process is performed to vary the chromaticity of the three primary colors of light that express a normal image as preprocessing for the normal image indicated by the normal image information machine-learned by the trained model, but the preprocessing is not limited to this, and other processes may be performed as preprocessing. Below, an information processing system 1 according to a third embodiment will be described in detail with reference to Fig. 14 and Figs. 16 to 18. Note that in the third embodiment, configurations different from those in the second embodiment will be described, and a redundant description of the same configuration as in the second embodiment will be omitted.

[0113] (Details of Functional Configuration of Information Processing Device 113 According to Embodiment 3) As shown in Fig. 14, in this embodiment, when the captured image information is determined to be normal image information as the inspection result, the image processing unit 132 performs a luminance normalization process on the normal image indicated by the normal image information, which is an image process for adjusting the average value and deviation value of the luminance of the normal image to predetermined values. Specifically, as the luminance normalization process, the image processing unit 132 performs a process for changing the average value and deviation value of the luminance of the normal image to the average value and deviation value of the luminance of the template image. As a result, as shown in Fig. 18, the histogram of the normal image after the luminance normalization process is more similar to the histogram of the template image in terms of luminance and chromaticity distribution of red light, green light, and blue light than the histogram of the original normal image before the luminance normalization process.

[0114] Here, for example, consider a case where there are three types of normal images: a first normal image, a second normal image, and a third normal image. In this case, the image processing unit 132 performs brightness normalization processing on all normal images, and generates first normal image information indicating the first normal image after the brightness normalization processing has been performed, second normal image information indicating the second normal image after the brightness normalization processing has been performed, and third normal image information indicating the second normal image after the brightness normalization processing has been performed.

[0115] Furthermore, when the information acquisition unit 121 acquires captured image information from the storage device 300, the image processing unit 132 performs a luminance normalization process before inputting the captured image information into the trained model, where the average value and deviation value of the luminance of the captured image indicated by the captured image information are set to the average value and deviation value of the luminance of the template image. As a result, the histogram of the captured image after the luminance normalization process is more similar to the histogram of the template image in terms of luminance and chromaticity distribution of red light, green light, and blue light than the histogram of the original captured image before the luminance normalization process.

[0116] Furthermore, when the information control unit 122 causes the trained model to generate sample image information, the image processing unit 132 performs a luminance normalization process in which the average value and deviation value of the luminance of the sample image indicated by the sample image information are set to the average value and deviation value of the luminance of the captured image or the template image after the luminance normalization process has been performed. As a result, the histogram of the sample image after the luminance normalization process is more similar to the histograms of the captured image and template image after the luminance normalization process in terms of luminance and chromaticity distribution of red light, green light, and blue light than the histogram of the original sample image before the luminance normalization process.

[0117] (Regarding Pre-Test Determination Processing According to Embodiment 3) Next, the pre-examination determination process according to this embodiment will be described using a flowchart. As shown in FIG. 16, first, the information acquisition unit 121 executes the processes of steps S101 to S103. Therefore, the first information control unit 123 controls the normal image learning model to input normal image information indicating a normal image after brightness normalization processing as captured image information to generate sample image information. After generating the sample image information, the image processing unit 132 executes brightness normalization processing, in which the average value and standard deviation of the brightness of the sample image indicated by the sample image information are set to the average value and standard deviation of the brightness of the template image (step S141). After image processing, the information processing device 113 executes the processes of steps S104 to S107 and ends the process.

[0118] (Regarding the In-Test Determination Process According to the Third Embodiment) Next, the in-examination determination process according to this embodiment will be described with reference to a flowchart. As shown in Fig. 17, first, the information acquisition unit 121 executes the process of step S111 to acquire captured image information, and the image processing unit 132 performs a process of normalizing the brightness of the captured image indicated by the captured image information to the average brightness and deviation of the brightness of the template image (step S142). After the image processing, the first information control unit 123 inputs the captured image information indicating the captured image after the brightness normalization process into the normal image trained model, and controls the generation of sample image information (step S143).

[0119] After generating the sample image information, the image processing unit 132 performs a brightness normalization process in which the average value and deviation value of the brightness of the sample image indicated by the sample image information are set to the average value and deviation value of the brightness of the captured image after the brightness normalization process (step S144). After the image processing, the information processing device 113 executes the processes of steps S113 to S123 and ends the process.

[0120] As described above, according to the information processing system 1 of this embodiment, in the information processing device 113, when it is determined that multiple types of captured image information are normal image information as the inspection result, the image processing unit 132 performs brightness normalization processing, and adjusts, for example, the average value and deviation value of the brightness of the first normal image indicated by the first normal image information, the second normal image indicated by the second normal image information, and the third normal image indicated by the third normal image information to predetermined values.

[0121] As a result, in the storage device 300, the learning information storage unit 310 stores first normal image information indicating a first normal image whose average luminance value and deviation value have been adjusted to predetermined values, second normal image information indicating a second normal image, and third normal image information indicating a third normal image. Also, as shown in FIG. 18 , in the learning device 200, the learning information acquisition unit 210 acquires, as learning information, the first normal image information, second normal image information, and third normal image information indicating normal images whose chromaticity and luminance distributions of the three primary colors of light representing the image are similar to each other from the storage device 300. Then, the trained model generation unit 220 generates a trained model by machine learning using the acquired first normal image information, second normal image information, and third normal image information, and the trained model output unit 230 outputs the generated trained model to the storage device 300.

[0122] In this way, the information processing device 113 generates sample image information by inputting the acquired captured image information into a trained model generated by machine learning using normal image information including the first normal image information, the second normal image information, and the third normal image information. As a result, the information processing system 1 according to the present embodiment is more likely to generate sample images in which the average value and deviation values ​​of luminance are closer to predetermined values ​​regardless of changes in chromaticity and luminance of the captured image than an information processing system that does not generate a trained model by machine learning using the first normal image information, the second normal image information, and the third normal image information, and can reduce variation in sample images generated by trained models.

[0123] In particular, according to the information processing system 1 of this embodiment, the image processing unit 132 performs the luminance normalization process by converting the average luminance value and deviation value of the normal image into the average luminance value and deviation value of the template image. By doing this, the information processing system 1 according to this embodiment is more likely to generate sample images whose average luminance and standard deviation are closer to the average luminance and standard deviation of the template image than an information processing system that does not use the average luminance and standard deviation of the normal image as the average luminance and standard deviation of the template image, thereby further reducing the variation in sample images generated by the trained model.

[0124] Furthermore, according to the information processing system 1 of this embodiment, the image processing unit 132 performs brightness normalization processing on the captured image indicated by the captured image information acquired by the information acquisition unit 121 from the storage device 300 as preprocessing before the captured image is input into the trained model, in order to convert the average brightness value and standard deviation value to the average brightness value and standard deviation value of the template image.

[0125] In this way, the first information control unit 123 performs control to input captured image information indicating the captured image after the luminance normalization process has been performed into the normal image trained model to generate sample image information. As a result, the information processing system 1 according to the present embodiment is more likely to generate sample images whose average luminance and deviation values ​​are closer to the average luminance and deviation values ​​of the template image than an information processing system that does not input captured image information indicating the captured image after the luminance normalization process into the normal image trained model to generate sample image information, and can further reduce the variation in sample images generated by the trained model.

[0126] Furthermore, according to the information processing system 1 of this embodiment, after the information control unit 122 causes the trained model to generate sample image information, the image processing unit 132 performs a brightness normalization process as post-processing, in which the average value and standard deviation of the brightness of the sample image indicated by the sample image information are set to the average value and standard deviation of the brightness of the captured image after the brightness normalization process has been performed.

[0127] By doing this, the information processing system 1 according to this embodiment is more likely to generate sample images whose average brightness value and standard deviation value are closer to the average brightness value and standard deviation value of the template image than an information processing system that does not use the average brightness value and standard deviation value of the sample image indicated by the sample image information generated by the normal image trained model as the average brightness value and standard deviation value of the captured image after the brightness normalization process has been performed, thereby further reducing the variation in sample images generated by the trained model. In addition, the information processing system 1 according to this embodiment has the same effects as the information processing system 1 according to the second embodiment.

[0128] (Example of change) In the above-described first to third embodiments, the normal image trained model used by the first information control unit 123 and the normal block image trained models used by the second information control unit 124 are separate trained models, but this is not limiting. For example, the normal image trained model and each normal block image trained model may be integrated into a trained model. In this way, the information processing system 1 can transmit and receive the normal image trained model and each normal block image trained model as a single image generation AI, and use them for image generation.

[0129] In the above-described first to third embodiments, the information processing device 113, the learning device 200, and the storage device 300 are separate computer devices, but this is not limiting, and for example, the information processing device 113 and the storage device 300 may be integrated into a computer device, or the learning device 200 and the storage device 300 may be integrated into a computer device. Furthermore, for example, the information processing device 113, the learning device 200, and the storage device 300 may be integrated into a computer device.

[0130] In the above-described first to third embodiments, the feature calculation unit 126 calculates the MSE value between the specific template block image and the specific sample block image as the feature of the specific sample block image, but this is not limiting. For example, the feature calculation unit 126 may calculate the average value and standard deviation of the luminance of the specific sample block image as the feature of the specific imaging block image. Furthermore, the feature calculation unit 126 may calculate the average value and standard deviation of the luminance of the specific normal block image as the feature of the specific normal block image, similar to the feature of the specific sample block image, and may calculate the average value and standard deviation of the luminance of the specific imaging block image as the feature of the specific imaging block image.

[0131] As in the first to third embodiments, it is preferable to check the vulnerability of the trained model by performing pre-test determination processing, but the pre-test determination processing does not have to be performed.

[0132] In the above-described first to third embodiments, the failure determination unit 127 determines whether a specific sample block image is corrupted based on whether the feature amount of the specific sample block image exceeds a threshold value, but this is not limited to this. For example, the failure determination unit 127 may input the specific sample block image into an artificial intelligence to generate determination result information on whether the specific sample block image is corrupted, and determine whether the specific sample block image is corrupted based on the determination result information. In this case, for example, it is necessary to newly generate an artificial intelligence that has learned the relationship between the specific sample block image and image corruption through machine learning.

[0133] In the above-described first to third embodiments, the abnormality determination unit 128 determines whether or not an abnormal portion of the work 2 is included in the specific imaging block image based on whether or not the feature amount of the specific imaging block image and the feature amount of the specific sample block image exceed a threshold value, but this is not limited to this. For example, the abnormality determination unit 128 may input the specific imaging block image and the specific sample block image into an artificial intelligence to generate determination result information on whether or not the specific imaging block image includes an abnormal portion of the work 2, and determine whether or not the specific imaging block image includes an abnormal portion of the work 2 based on the determination result information. In this case, for example, it is necessary to newly generate an artificial intelligence that has learned by machine learning the relationship between the specific imaging block image, the specific sample block image, and the abnormal portion of the work 2.

[0134] In the above embodiments 1 to 3, the learning device 200 and the storage device 300 are devices inside the information processing system 1, but this is not limited to this, and the learning device 200 and the storage device 300 may also be devices outside the information processing system 1.

[0135] Furthermore, the information processing device 113 does not need to include the functionality of an inference device. For example, as shown in Fig. 19, an external cloud server system may include a learning device 200, a storage device 300, and an inference device 400. In other words, the trained model may be an external image generation AI, as long as it is generated by machine learning using a group of non-defective product images.

[0136] In this case, the information control unit 122 performs control to transmit the captured image information acquired by the information acquisition unit 121 to the inference device 400, and to receive from the inference device 400 sample image information that the inference device 400 has generated by inputting the captured image information into the learned model. Specifically, the first information control unit 123 performs control to transmit the captured image information to the inference device 400, and to receive from the inference device 400 sample image information that the inference device 400 has generated by inputting the captured image information into the normal image learned model. Furthermore, the second information control unit 124 performs control to transmit the specific captured block image to the inference device 400, and to receive from the inference device 400 specific sample block image information that the inference device 400 has generated by inputting the specific captured block image information into the specific normal block image learned model.

[0137] Furthermore, the inference device 400 may use the above-mentioned artificial intelligence to generate determination result information as to whether or not a specific sample block image is damaged, or generate determination result information as to whether or not a specific captured block image contains an abnormal part of the work 2. In other words, the above-mentioned artificial intelligence may be an external artificial intelligence.

[0138] In this case, the failure determination unit 127 and the abnormality determination unit 128, for example, transmit specific imaging block image information and specific sample block image information to the inference device 400, and after receiving from the inference device 400 determination result information generated by the inference device 400 inputting the specific imaging block image information and the specific sample block image into an artificial intelligence, determine whether the specific sample block image is failed or whether the specific imaging block image includes an abnormal portion of the work 2 based on the determination result information. In this way, the information processing system according to the modified example can omit a so-called learning phase such as the learning process shown in Fig. 9, and only need to perform so-called inference phases such as the pre-inspection determination process shown in Fig. 10 and the during-inspection determination process shown in Figs. 11A and 11B.

[0139] The external image generation AI and external artificial intelligence may be configured using known algorithms such as Transformer, BERT (Bidirectional Encoder Representations from Transformers), and GPT (Generative Pre-Training), or may be configured by combining a plurality of algorithms including these. Furthermore, it is preferable that the external image generation AI and external artificial intelligence perform so-called fine tuning, in which normal image information and each normal block image information are machine-learned as in the above-described first to third embodiments, but fine tuning is not necessarily required.

[0140] In the above-described first to third embodiments, the image dividing unit 125 divides the image of n×m pixels into nine equal parts, creating a first block image, a second block image, ..., a ninth block image, but this is not limited to this, and the image does not have to be divided equally. In other words, the predetermined number of pixels by which the image dividing unit 125 divides the image into block images is not limited to one type, but may be two or more types. By doing so, it is possible to adjust the number of pixels of a block image at a specific location where, for example, a glossy part of the workpiece 2 may be present and the block image may be damaged, or where an abnormal part of the workpiece 2 may be included.

[0141] In the above-described first to third embodiments, the feature amount calculation unit 126 calculates the feature amounts of all block images divided by the image division unit 125, the failure determination unit 127 determines whether all sample block images divided by the image division unit 125 are failed, and the abnormality determination unit 128 determines whether all captured block images divided by the image division unit 125 contain an abnormal portion of the workpiece 2. However, the present invention is not limited to this. For example, if there is a block image of a specific location that may be failed due to the presence of a glossy part of the workpiece 2 or that may contain an abnormal portion of the workpiece 2, the feature amount calculation unit 126 may calculate only the feature amount of the block image of the specific location, the failure determination unit 127 may determine only whether the sample block image of the specific location is failed, and the abnormality determination unit 128 may determine only whether an abnormal portion of the workpiece 2 is contained in the captured block image of the specific location.

[0142] In this case, in the learning device 200, the learning information acquisition unit 210 may acquire only normal image information and specific normal block image information as learning information, the learned model generation unit 220 may generate only a normal image trained model and a specific normal block image as trained models, and the trained model output unit 230 may output only the normal image trained model and the specific normal block image trained model. Furthermore, in the information processing device 113, the information acquisition unit 121 may acquire only a normal image trained model and a specific normal block image trained model as trained models. In this way, if the sample block image of a specific location is corrupted, the second information control unit 124 inputs specific imaging block image information into the specific normal block image trained model to generate updated specific sample block image information, and the information generation unit 129 can generate updated sample image information using the updated specific sample block image information.

[0143] As in the first embodiment, it is preferable to generate a trained model by machine learning using an equal number of glossy normal image information and non-glossy normal image information, but this is not limiting. For example, the difference between the numerical value of the type of glossy normal image information and the numerical value of the type of non-glossy normal image information included in the normal image information used in the machine learning of the trained model may be equal to or less than a predetermined type threshold. Even in this case, since the trained model can be generated by machine learning using an equal number of glossy normal image information and non-glossy normal image information, it is easier to generate sample images in which the degree of glossiness is reduced to a level that is not too strong while allowing for the presence or absence of gloss in glossy parts of the workpiece 2 in the captured image, thereby reducing the variation in the sample images generated by the trained model. Furthermore, although variation in the sample images generated by the trained model may occur, for example, the difference between the numerical value of the type of glossy normal image information and the numerical value of the type of non-glossy normal image information included in the normal image information used in the machine learning of the trained model may exceed the type threshold.

[0144] In the above-described first to third embodiments, the machine learning learning algorithm by which the trained model generation unit 220 generates the trained model is a neural network, but the present invention is not limited to this and may be, for example, another supervised learning learning algorithm such as a support vector machine. Also, the learning algorithm is not limited to supervised learning as long as the trained model is generated by machine learning using a group of good product images and sample image information can be generated when captured image information is input, and may be, for example, another learning algorithm such as reinforcement learning.

[0145] In the first embodiment, the information processing device 113 includes the gloss determination unit 130, and the information processing device 113 outputs the same number of glossy normal image information and non-glossy normal image information to the storage device 300, thereby ensuring that the learning device 200 uses the same number of glossy normal image information and non-glossy normal image information for machine learning. However, this is not limiting. For example, the storage device 300 may include the gloss determination unit 130, and control may be performed such that the storage device 300 stores the same number of glossy normal image information and non-glossy normal image information, or that the storage device 300 transmits the same number of glossy normal image information and non-glossy normal image information to the learning device 200. Furthermore, for example, the learning device 200 may include the gloss determination unit 130, and control may be performed such that the learning device 200 acquires the same number of glossy normal image information and non-glossy normal image information from the storage device 300, or that the learning device 200 uses the same number of glossy normal image information and non-glossy normal image information for machine learning.

[0146] As in the second embodiment, the image processing unit 132 preferably generates second normal image information indicating a second normal image having a strong chromaticity of red light, third normal image information indicating a third normal image having a strong chromaticity of green light, and fourth normal image information indicating a fourth normal image having a strong chromaticity of blue light, but is not limited to this, and generation of any of the normal image information may be omitted. For example, if the chromaticity of blue light is likely to vary due to external factors such as individual differences in the camera 108, lens 109, and illumination member 110 included in the observation camera 106, or differences in the lot of the workpiece 2, the image processing unit 132 may generate only the fourth normal image information.

[0147] As in the above-mentioned second embodiment, it is preferable that the image processing unit 132 generates normal image information indicating multiple types of normal images each having a strong chromaticity of each color light of red, green, and blue, but this is not limited to this, and each normal image information may indicate only one type of normal image.

[0148] In the above-mentioned second embodiment, it is preferable that the image processing unit 132 generates each normal image information indicating each normal image in which the chromaticity of any one of the red, green, and blue color lights is strong, but this is not limited thereto, and each normal image indicated by each normal image information may have the chromaticity of two or more of the red, green, and blue color lights strong.

[0149] In the second embodiment, the information processing device 113 includes the image processing unit 132, and the information processing device 113 performs chromaticity variation processing to generate the first, second, third, and fourth normal image information based on the normal image information. However, this is not limiting. For example, the storage device 300 may include the image processing unit 132, and the storage device 300 may perform chromaticity variation processing to generate the first, second, third, and fourth normal image information. Alternatively, for example, the learning device 200 may include the image processing unit 132, and the learning device 200 may perform chromaticity variation processing to generate the first, second, third, and fourth normal image information.

[0150] In the third embodiment, the information processing device 113 includes the image processing unit 132, and the information processing device 113 performs luminance normalization processing to generate the first normal image information, the second normal image information, and the third normal image information based on the normal image information. However, this is not limiting. For example, the storage device 300 may include the image processing unit 132, and the storage device 300 may perform luminance normalization processing to generate the first normal image information, the second normal image information, and the third normal image information. Furthermore, for example, the learning device 200 may include the image processing unit 132, and the learning device 200 may perform luminance normalization processing to generate the first normal image information, the second normal image information, and the third normal image information.

[0151] The core processing components of information processing device 113, learning device 200, and storage device 300, including control unit 51, main memory unit 52, external memory unit 53, operation unit 54, display unit 55, transmission / reception unit 56, and internal bus 50, can be realized using a standard computer system rather than a dedicated system. For example, a computer program for executing the above operations may be stored and distributed on a recording medium readable by information processing device 113, learning device 200, and storage device 300, such as a DVD-ROM (Read-Only Memory), and the computer program may be installed on a computer to configure information processing device 113, learning device 200, and storage device 300 to execute the above processes. Alternatively, the computer program may be stored in a storage device of a server device on a communication network such as the Internet, and downloaded by a standard computer system to configure information processing device 113, learning device 200, and storage device 300.

[0152] In addition, when the functions of the information processing device 113, learning device 200, and storage device 300 are realized by sharing the functions between an OS (operating system) and an application program, or by collaboration between the OS and an application program, only the application program portion may be stored on a recording medium or storage device.

[0153] It is also possible to superimpose a computer program on a carrier wave and provide it via a communication network. For example, the computer program may be posted on a bulletin board system (BBS) on the communication network and provided via the network. The computer program may then be started and executed under the control of the OS in the same way as other application programs, thereby performing the above-mentioned processing.

[0154] In addition, the control methods, information processing system 1, information processing device 113, and program configurations according to the above-mentioned embodiments 1 to 3 are merely examples, and can be changed and modified as desired as long as they solve the problems that the present disclosure aims to solve.

[0155] The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are considered to be within the scope of the present disclosure.

[0156] Various aspects of the present disclosure are summarized below as appendices.

[0157] (Appendix 1) an information acquisition step in which the computer acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control step in which the computer inputs the captured image information acquired in the information acquisition step into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of the normal image object, to generate sample image information, which is information indicating a sample image based on the normal image; Including, the normal images include a first normal image and a second normal image different from the first normal image, The first normal image and the second normal image have been subjected to a brightness normalization process as a pre-processing, which is an image process for adjusting the average value and deviation value of brightness to predetermined values. Control method. (Appendix 2) The first normal image and the second normal image are subjected to the brightness normalization process, in which the average value and deviation value of brightness are set to the average value and deviation value of brightness of a predetermined template image included in the normal images. 10. The control method according to claim 1. (Appendix 3) a first image processing step in which the computer performs the brightness normalization process on the captured image indicated by the captured image information acquired in the information acquisition step before inputting the captured image to a trained model, thereby normalizing the brightness of the captured image to the average brightness and deviation values ​​of the brightness of the template image included in the normal image; Further comprising: 3. The control method according to claim 1 or 2. (Appendix 4) a second image processing step in which the computer performs, as the luminance normalization processing, a process of setting an average value and a deviation value of luminance of the sample image indicated by the sample image information generated in the information control step to the average value and the deviation value of luminance of the captured image after the luminance normalization processing has been performed; Further comprising: 3. The control method according to claim 3. (Appendix 5) an information acquisition step in which the computer acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control step in which the computer inputs the captured image information acquired in the information acquisition step into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal imaging target that is a normal imaging target, to generate sample image information, which is information indicating a sample image based on the normal image; Including, The normal image includes a first normal image which is the captured image of the normal image capture object, and a second normal image on which a chromaticity change process, which is an image process for adjusting the chromaticity of any one of red light, green light, and blue light corresponding to the three primary colors of light expressing the first normal image, has been performed as a preprocessing. Control method. (Appendix 6) an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of the normal image object, to generate sample image information, which is information indicating a sample image based on the normal image; Equipped with the normal images include a first normal image and a second normal image different from the first normal image, The first normal image and the second normal image have been subjected to a brightness normalization process as a pre-processing, which is an image process for adjusting the average value and deviation value of brightness to predetermined values. Information processing system. (Appendix 7) an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal image capture target that is a normal image capture target, to generate sample image information, which is information indicating a sample image based on the normal image; Equipped with The normal image includes a first normal image which is the captured image of the normal image capture object, and a second normal image on which a chromaticity change process, which is an image process for adjusting the chromaticity of any one of red light, green light, and blue light corresponding to the three primary colors of light expressing the first normal image, has been performed as a preprocessing. Information processing system. (Appendix 8) an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of the normal image object, to generate sample image information, which is information indicating a sample image based on the normal image; Equipped with the normal images include a first normal image and a second normal image different from the first normal image, The first normal image and the second normal image have been subjected to a brightness normalization process as a pre-processing, which is an image process for adjusting the average value and deviation value of brightness to predetermined values. Information processing device. (Appendix 9) an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal image capture target that is a normal image capture target, to generate sample image information, which is information indicating a sample image based on the normal image; Equipped with The normal image includes a first normal image which is the captured image of the normal image capture object, and a second normal image on which a chromaticity change process, which is an image process for adjusting the chromaticity of any one of red light, green light, and blue light corresponding to the three primary colors of light expressing the first normal image, has been performed as a preprocessing. Information processing device. (Appendix 10) Computer, an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal image object, to generate sample image information, which is information indicating a sample image based on the normal image; It functions as the normal images include a first normal image and a second normal image different from the first normal image, The first normal image and the second normal image have been subjected to a brightness normalization process as a pre-processing, which is an image process for adjusting the average value and deviation value of brightness to predetermined values. program. (Appendix 11) Computer, an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal imaging target that is a normal imaging target, to generate sample image information, which is information indicating a sample image based on the normal image; It functions as The normal image includes a first normal image which is the captured image of the normal image capture object, and a second normal image on which a chromaticity change process, which is an image process for adjusting the chromaticity of any one of red light, green light, and blue light corresponding to the three primary colors of light expressing the first normal image, has been performed as a preprocessing. program. [Explanation of symbols]

[0158] 1...information processing system, 2...work, 21...connection terminal, 22...CPU, 23...memory, 24...connector, 25...port, 26...electrolytic capacitor, 27...film capacitor, 50...internal bus, 51...control unit, 52...main memory unit, 53...external memory unit, 54...operation unit, 55...display unit, 56...transmitting / receiving unit, 59...control program, 100...inspection device, 101...base, 102...Y-axis actuator, 103...Z-axis actuator, 104...X-axis actuator, 105...transport table, 106...observation camera, 107...laser displacement meter, 108...camera, 109...lens, 110...illumination member, 111...displacement meter controller, 1 12...lighting controller, 113...information processing device, 114...PLC, 115...GOT, 121...information acquisition unit, 122...information control unit, 123...first information control unit, 124...second information control unit, 125...image segmentation unit, 126...feature calculation unit, 127...failure determination unit, 128...abnormality determination unit, 129...information generation unit, 130...gloss determination unit, 131...information output unit, 132...image processing unit, 200...learning device, 210...learning information acquisition unit, 220...learned model generation unit, 230...learned model output unit, 300...storage device, 310...learning information storage unit, 320...learned model storage unit, 330...information transmission and reception unit, 400...inference device.

Claims

1. an information acquisition step in which the computer acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control step in which the computer inputs the captured image information acquired in the information acquisition step into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of the normal image object, to generate sample image information, which is information indicating a sample image based on the normal image; Including, the normal image includes a first normal image and a second normal image different from the first normal image, a brightness normalization process, which is image processing for adjusting an average value and a deviation value of brightness to predetermined values, is performed in advance as preprocessing on the first normal image and the second normal image; Control method.

2. The first normal image and the second normal image are subjected to the luminance normalization process, in which the average luminance value and the deviation value are set to the average luminance value and the deviation value of the luminance value of a predetermined template image included in the normal images. The control method according to claim 1 .

3. a first image processing step in which the computer performs the brightness normalization process on the captured image indicated by the captured image information acquired in the information acquisition step before inputting the captured image to a trained model, thereby normalizing the brightness of the captured image to the average brightness and deviation values ​​of the brightness of the template image included in the normal image; Further comprising: The control method according to claim 2 .

4. a second image processing step in which the computer performs, as the luminance normalization processing, a process of setting an average value and a deviation value of luminance of the sample image indicated by the sample image information generated in the information control step to the average value and the deviation value of luminance of the captured image after the luminance normalization processing has been performed; Further comprising: The control method according to claim 3 .

5. an information acquisition step in which the computer acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control step in which the computer inputs the captured image information acquired in the information acquisition step into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal imaging target that is a normal imaging target, to generate sample image information, which is information indicating a sample image based on the normal image; Including, The normal image includes a first normal image which is the captured image of the normal image capture object, and a second normal image on which a chromaticity variation process, which is an image process for adjusting the chromaticity of any one of red light, green light, and blue light corresponding to the three primary colors of light expressing the first normal image, has been performed as a preprocessing. Control method.

6. an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of the normal image object, to generate sample image information, which is information indicating a sample image based on the normal image; Equipped with the normal image includes a first normal image and a second normal image different from the first normal image, a brightness normalization process, which is image processing for adjusting an average value and a deviation value of brightness to predetermined values, is performed in advance as preprocessing on the first normal image and the second normal image; Information processing system.

7. an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal image capture target that is a normal image capture target, to generate sample image information, which is information indicating a sample image based on the normal image; Equipped with The normal image includes a first normal image which is the captured image of the normal image capture object, and a second normal image on which a chromaticity variation process, which is an image process for adjusting the chromaticity of any one of red light, green light, and blue light corresponding to the three primary colors of light expressing the first normal image, has been performed as a preprocessing. Information processing system.

8. an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of the normal image object, to generate sample image information, which is information indicating a sample image based on the normal image; Equipped with the normal image includes a first normal image and a second normal image different from the first normal image, a brightness normalization process, which is image processing for adjusting an average value and a deviation value of brightness to predetermined values, is performed in advance as preprocessing on the first normal image and the second normal image; Information processing device.

9. an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal image capture target that is a normal image capture target, to generate sample image information, which is information indicating a sample image based on the normal image; Equipped with The normal image includes a first normal image which is the captured image of the normal image capture object, and a second normal image on which a chromaticity variation process, which is an image process for adjusting the chromaticity of any one of red light, green light, and blue light corresponding to the three primary colors of light expressing the first normal image, has been performed as a preprocessing. Information processing device.

10. Computer, an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal image object, to generate sample image information, which is information indicating a sample image based on the normal image; It functions as the normal image includes a first normal image and a second normal image different from the first normal image, a brightness normalization process, which is image processing for adjusting an average value and a deviation value of brightness to predetermined values, is performed in advance as preprocessing on the first normal image and the second normal image; program.

11. Computer, an information acquisition unit that acquires captured image information that is information indicating a captured image that is an image of an imaging target; an information control unit that controls inputting the captured image information acquired by the information acquisition unit into a trained model generated by machine learning using normal image information, which is information indicating a normal image that is an image based on the captured image of a normal imaging target that is a normal imaging target, to generate sample image information, which is information indicating a sample image based on the normal image; It functions as The normal image includes a first normal image which is the captured image of the normal image capture object, and a second normal image on which a chromaticity variation process, which is an image process for adjusting the chromaticity of any one of red light, green light, and blue light corresponding to the three primary colors of light expressing the first normal image, has been performed as a preprocessing. program.

Images