Control device, inspection device, image generation method, inspection method, and program

Abstract

The present invention provides a control device, an inspection device, an image generation method, an inspection method, and a program that can generate and utilize training images that apply shading according to the shape of the subject and minimize the domain gap with the real image. [Solution] The learning image generation device comprises a first generation unit 381 that generates a 3D model, a second generation unit 382 that generates a pseudo-damaged image based on 3D shape data of a subject, which is obtained by synthesizing a pre-created scratch shape including distance information in the depth direction of the scratch at a predetermined position on the surface of the subject of the 3D model, a third generation unit 383 that generates an image of the damaged area based on the pseudo-damaged image and the image without a scratch, and a fourth generation unit 384 that synthesizes the image of the damaged area with the real image to generate a learning image for machine learning training.

Description

Technical Field

[0001] The present disclosure relates to a control device, an inspection device, an image generation method, an inspection method, and a program.

Background Art

[0002] In Patent Document 1, a technique is disclosed in which a two-dimensional image of a defect shape, created based on a defect model formed three-dimensionally in advance, is synthesized with a surface image of a piston, which is an inspection object imaged by a camera, to generate a plurality of defective product sample images. In this technique, the surface of the piston is inspected based on the learning result obtained by machine learning using the plurality of defective product sample images.

[0003] Also, in Patent Document 2, a technique is disclosed in which a virtual body representing the three-dimensional shape of an inspection object is generated in a virtual space, an image with a defect added to this virtual body is generated, and a plurality of teacher data associating this image with defect information including the depth information of the defect are generated by changing at least the parameters of the defect. In this technique, an inspection model is generated using the plurality of generated teacher data, an imaging image obtained by imaging the inspection object is input to this inspection model, and the depth of the defect of the inspection object included in the imaging image is estimated.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the aforementioned Patent Document 1 generates a sample image of a defective product by rendering a pseudo-defective image, which is created by cutting out only the damaged area and converting it into a two-dimensional image, and then compositing it with a normal surface image. This has the problem that shading caused by the shape of the subject is not applied.

[0006] Furthermore, in the aforementioned Patent Document 2, since pseudo-abnormal images containing defects are generated using computer graphics (hereinafter simply referred to as "CG") in a virtual space, a problem arises in that as the area of ​​CG used in the pseudo-abnormal image increases, the pseudo-abnormal image becomes more artificial, and a domain gap occurs between the real image and the CG image, which reduces the accuracy of inferences made by the inspection model regarding the real image.

[0007] This disclosure is made in view of the above, and aims to provide a control device, inspection device, image generation method, inspection method, and program that can generate and utilize training images that apply shading according to the shape of the subject and minimize the domain gap with the real image. [Means for solving the problem]

[0008] The control device according to this disclosure includes: a first generation unit that generates a 3D model that reproduces the positional relationship and shape of the optical system, including an imaging device and an illumination device used in the actual inspection process, and the subject to be inspected in the actual inspection process; a second generation unit that generates a pseudo-damaged image based on 3D shape data of the subject, in which a scratch shape including distance information in the depth direction of the scratch, which has been created in advance, is synthesized at a predetermined position on the surface of the subject in the 3D model; a third generation unit that generates a scratch abnormality area image by extracting the scratch abnormality area, including the scratch, based on the pseudo-damaged image and a non-damaged image of the subject generated based on 3D shape data of the subject in which the scratch shape has not been synthesized; and a fourth generation unit that generates a training image for machine learning by synthesizing the scratch abnormality area image at the same position as the extraction position from which the scratch abnormality area image was extracted from the pseudo-damaged image, onto an actual image of the subject.

[0009] Furthermore, the inspection apparatus according to this disclosure includes a trained model that outputs whether or not there is a surface abnormality on the object to be inspected as an output parameter, an estimation unit that inputs data of an actual image of the object to be inspected captured by an imaging device whose positional relationship with the object to be inspected and the lighting device used in the actual inspection process is predetermined, and estimates the surface abnormality of the object to be inspected using the output result output from the trained model, and a display control unit that displays the information of the estimation result of the estimation unit on a display unit.

[0010] Furthermore, the image generation method according to this disclosure involves a control device generating a 3D model that reproduces the positional relationship and shape of the optical system, including an imaging device and an illumination device used in the actual inspection process, and the subject to be inspected in the actual inspection process; generating a pseudo-damaged image based on 3D shape data of the subject, which is obtained by synthesizing a pre-created scratch shape, including distance information in the depth direction of the scratch, at a predetermined position on the surface of the subject in the 3D model; generating a scratch abnormality area image by extracting the scratch abnormality area, which includes the scratch, based on the pseudo-damaged image and a scratch-free image generated based on the 3D shape data of the subject, which does not have the scratch shape synthesized; reading out the actual image of the subject from the recording unit; and generating a training image for machine learning by synthesizing the scratch abnormality area image on the actual image at the same position as the extraction position from which the scratch abnormality area image was extracted from the pseudo-damaged image.

[0011] Furthermore, the inspection method relating to this disclosure involves an inspection device that reads data from a recording unit of an actual image of the object to be inspected, captured by an imaging device whose positional relationship with the object to be inspected and the lighting device used in the actual inspection process is predetermined, into a trained model that outputs whether or not there is an abnormality on the surface of the object to be inspected as an output parameter, estimates the abnormality on the surface of the object to be inspected using the output result output from the trained model, and displays the information of the estimation result on a display unit.

[0012] Furthermore, the program relating to this disclosure causes the control device to generate a 3D model that reproduces the positional relationship and shape of the optical system, including an imaging device and an illumination device used in the actual inspection process, and the subject to be inspected in the actual inspection process; generate a pseudo-damaged image based on 3D shape data of the subject, which is obtained by synthesizing a scratch shape, including distance information in the depth direction of the scratch, that was previously created, at a predetermined position on the surface of the subject in the 3D model; generate a scratch abnormality area image by extracting the scratched abnormality area based on the pseudo-damaged image and a scratch-free image generated based on the 3D shape data of the subject in which the scratch shape has not been synthesized; read out the actual image of the subject captured by the recording unit; and generate a training image for machine learning by synthesizing the scratch abnormality area image on the actual image at the same position as the extraction position from which the scratch abnormality area image was extracted from the pseudo-damaged image.

[0013] Furthermore, the program relating to this disclosure causes the inspection device to input data of an actual image of the object to be inspected, captured by an imaging device whose positional relationship with the object to be inspected and the lighting device used in the actual inspection process is predetermined, into a trained model that outputs whether or not there is an abnormality on the surface of the object to be inspected as an output parameter; estimate the abnormality on the surface of the object to be inspected using the output result output from the trained model; and display the information of the estimation result on the display unit. [Effects of the Invention]

[0014] According to this disclosure, it is possible to generate and use training images that have shading applied according to the shape of the subject, minimizing the domain gap with the real image. [Brief explanation of the drawing]

[0015] [Figure 1] Figure 1 is a block diagram showing the functional configuration of an inspection system according to one embodiment. [Figure 2] Figure 2 is a flowchart showing an overview of the generation process performed by a control device according to one embodiment. [Figure 3] FIG. 3 is a diagram showing an example of a 3D model generated by a first generation unit included in a control device according to an embodiment. [Figure 4] FIG. 4 is a diagram schematically showing an imaging range and a subject of an imaging device of a 3D model according to an embodiment. [Figure 5] FIG. 5 is a diagram showing an example of scratch shape data read by a second generation unit included in a control device according to an embodiment from a scratch shape image data recording unit 35. [Figure 6] FIG. 6 is a diagram showing an example of scratch shape data after rendering processing by a second generation unit 382 with respect to the scratch shape data by a second generation unit included in a control device according to an embodiment. [Figure 7] FIG. 7 is an enlarged view of region A1 in FIG. 6. [Figure 8] FIG. 8 is a diagram showing an example of a scratch abnormal site image generated by a third generation unit included in a control device according to an embodiment. [Figure 9] FIG. 9 is an enlarged view of region A2 in FIG. 8. [Figure 10] FIG. 10 is a diagram showing an example of a learning image generated by a fourth generation unit included in a control device according to an embodiment. [Figure 11] FIG. 11 is an enlarged view of region A3 in FIG. 10. [Figure 12] FIG. 12 is a diagram showing another example of a learning image generated by a fourth generation unit included in a control device according to an embodiment. [Figure 13] FIG. 13 is an enlarged view of region A4 in FIG. 12. [Figure 14] FIG. 14 is a flowchart showing an outline of an inspection process executed by a control device according to an embodiment.

Embodiments for Carrying Out the Invention

[0016] The control device and inspection device according to embodiments of this disclosure will be described below with reference to the drawings. Note that the components in the embodiments described below include those that are easily substituted or substantially identical to those that are easily substituted by those skilled in the art. Furthermore, the figures referenced in the following description only schematically show the shape, size, and positional relationships to the extent that the contents of this disclosure can be understood. In other words, this disclosure is not limited to the shapes, sizes, and positional relationships exemplified in the figures.

[0017] [Inspection System] Figure 1 is a block diagram showing the functional configuration of an inspection system according to one embodiment. The inspection system 1 shown in Figure 1 is a system that captures an image of a subject 100, which is the object to be inspected, generates image data, and inspects for abnormalities on the surface of the subject 100 based on this image data. The inspection system 1 shown in Figure 1 comprises an observation device 2 that captures an image of the subject 100 and generates image data, and a control device 3 that generates training images for inspecting abnormalities on the surface of the subject 100 and for learning machine learning based on the image data generated by the observation device 2.

[0018] [Configuration of the observation device] First, let's explain the functional configuration of observation device 2. The observation device 2 comprises an actual inspection process optical system 20 used in the actual inspection process, and a stage 21. The actual inspection process optical system 20 comprises an illumination device 201 that irradiates light toward the subject 100, and an imaging device 202 that captures images of the subject 100 and generates image data.

[0019] The illumination device 201 is configured, for example, using a ring illumination that forms an annular shape. The illumination device 201 is positioned so that the center of the annular shape coincides with the optical axis of the imaging device 202. Under the control of the control device 3, the illumination device 201 irradiates the subject 100 with illumination light at a predetermined illuminance.

[0020] The imaging device 202 is composed of one or more lenses and an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) that receives the image of the subject 100 formed by the lenses and generates image data. Under the control of the control device 3, the imaging device 202 captures the subject 100 and generates image data, and outputs this image data to the control device 3.

[0021] Stage 21 is on which the subject 100 is placed. Stage 21 is configured to be movable in the left-right, front-back, and up-down directions by a drive mechanism (not shown).

[0022] [Control device configuration] Next, the functional configuration of the control device 3 will be described. The control device 3 comprises a communication unit 30, an input unit 31, a display unit 32, an output unit 33, a 3D image data recording unit 34, a scratch shape image data recording unit 35, a learning image data recording unit 36, a recording unit 37, and a control unit 38. In one embodiment, the control device 3 also functions as an inspection device.

[0023] The communication unit 30 is connected to the actual inspection process optical system 20 via a communication cable so as to be able to communicate with it. The communication unit 30 outputs control signals from the control unit 38 to the actual inspection process optical system 20, and also outputs image data from the actual inspection process optical system 20 to the control unit 38. The communication unit 30 is configured using a communication module.

[0024] The input unit 31 receives various operation inputs in response to external operations and outputs signals corresponding to the received operations to the control unit 38. The input unit 31 is configured using input interfaces such as a keyboard, mouse, buttons, switches, and touch panel.

[0025] The display unit 32 displays various information related to the control device 3 under the control of the control unit 38. The display unit 32 is configured using a liquid crystal display or an organic electroluminescent display (OLED display), etc.

[0026] The output unit 33 outputs various types of information under the control of the control unit 38. The output unit 33 is configured using, for example, a printer, a speaker, etc.

[0027] The 3D image data recording unit 34 records a 3D model (3D model data) for each inspection that can reproduce the positional relationship and shape between the actual inspection process optical system 20, which includes the imaging device 202 and illumination device 201 used in the actual inspection process, and the subject 100 to be inspected in the actual inspection process. The 3D image data recording unit 34 is configured using an HDD (Hard Disk Drive) and an SSD (Solid State Drive), etc.

[0028] The scratch shape image data recording unit 35 records scratch shape image data of the 3D shape data of the subject, which is obtained by synthesizing multiple scratch shapes with different shapes for rendering when generating the training image data described later. The scratch shape image data recording unit 35 is configured using an HDD and an SSD, etc.

[0029] The training image data recording unit 36 ​​records multiple training image data for machine learning. The training image data recording unit 36 ​​is configured using an HDD and an SSD, etc.

[0030] The recording unit 37 is configured using volatile memory and non-volatile memory, etc., and records various information related to the control device 3. The recording unit 37 includes a program recording unit 371 that records various programs executed by the control device 3, and a trained model recording unit 372 that records a trained model that has learned from training images (training image data). Here, the trained model is trained using machine learning, such as deep learning, with multiple training images described later, and takes image data as an input parameter and outputs whether or not there is an abnormality on the surface of the object to be inspected as an output parameter. For example, support vector machines and K-nearest neighbors are used as the training model.

[0031] The control unit 38 performs various processes related to the control device 3 and comprehensively controls each part that constitutes the control device 3. The control unit 38 is implemented using a processor having hardware such as a CPU (Central Processing Unit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), and memory, which is a temporary storage area used by the processor. The control unit 38 includes a first generation unit 381, a second generation unit 382, ​​a third generation unit 383, a fourth generation unit 384, a determination unit 385, a fifth generation unit 386, an estimation unit 387, and a display control unit 388.

[0032] The first generation unit 381 reads 3D image data specified by the user via the input unit 31 from the 3D image data recording unit 34 and generates a 3D model in virtual space (memory) that reproduces the positional relationship and shape of the illumination device 201, imaging device 202, and subject 100 used in the actual inspection process optical system 20.

[0033] The second generation unit 382 reads scratch shape data specified by the user via the input unit 31 from the scratch shape image data recording unit 35 and generates a pseudo-scratched image based on the 3D shape data of the subject, which is obtained by synthesizing the scratch shape, including distance information in the depth direction of the scratch, which was previously created, at a predetermined position on the surface of the subject 100 in the 3D model.

[0034] The third generation unit 383 generates an image of a damaged area, which includes the damaged area, based on the pseudo-damaged image generated by the second generation unit 382 and the image without damage, which is generated based on the 3D shape data of the subject 100 that has not had the damage shape data of the 3D model generated by the first generation unit 381 synthesized.

[0035] The fourth generation unit 384 generates a training image by combining the image of the damaged area with the actual image (actual image data) of the subject 100 captured by the observation device 2, at the same position where the third generation unit 383 extracted the damaged area image from the pseudo-damaged image.

[0036] The determination unit 385 determines whether the number of training images generated by the fourth generation unit 384 has reached a predetermined number. Here, the predetermined number is, for example, 1000 images. The determination unit 385 also determines whether there is an abnormality on the surface of the subject 100 based on the estimation result of the estimation unit 387.

[0037] The fifth generation unit 386 takes multiple training images recorded in the training image data recording unit 36 ​​as training data, inputs them into the training model for machine learning, generates a trained model that uses real image data (imaging image data) as input parameters and outputs whether or not there are scratches on the surface of the subject 100 as output parameters, and records it in the trained model recording unit 372.

[0038] The estimation unit 387 inputs the data of the actual image captured of the object to be inspected into a trained model recorded by the trained model recording unit 372, and estimates whether or not there is an abnormality in the object 100 based on the presence or absence of scratches on the surface of the object 100, which is an output parameter output by the trained model.

[0039] If the determination unit 385 determines that there is an abnormality on the surface of the subject 100, the display control unit 388 displays the estimation result information from the estimation unit 387, for example, that there is an abnormality on the surface of the subject 100, on the display unit 32.

[0040] [Overview of the generation process] Next, we will describe the overview of the training image generation process performed by the control device 3. Figure 2 is a flowchart showing the overview of the training image generation process performed by the control device 3.

[0041] As shown in Figure 2, first, the first generation unit 381 reads 3D image data specified by the user from the 3D image data recording unit 34 via the input unit 31 and generates a 3D model in virtual space (in memory) that reproduces the positional relationship and shape of the illumination device 201, imaging device 202, and subject 100 used in the actual inspection process optical system 20 (step S101).

[0042] Figure 3 shows an example of a 3D model generated by the first generation unit 381. Figure 4 is a schematic diagram showing the imaging range of the imaging device 202 and the subject 100 shown in Figure 3. As shown in Figure 3, the first generation unit 381 reads a 3D model (3D image data) specified by the user via the input unit 31 from the 3D image data recording unit 34 and generates a 3D model P1 in virtual space (memory) that reproduces the positional relationships and shapes of the illumination device 201, imaging device 202, and subject 100 used in the actual inspection process optical system 20. Specifically, the first generation unit 381 generates a 3D model P1 in virtual space that reproduces the positional relationships and shapes of the illumination device 201, imaging device 202, and subject 100 based on 3D measurement data (3D measurement data) obtained by the user in advance using a well-known CG rendering tool or CAD tool to measure the positional relationships and shapes of the illumination device 201, imaging device 202, and subject 100. Here, measurement data refers to point cloud data measured by laser scanning, etc. Furthermore, in the actual inspection process, the relative positions and shapes of the illumination device 201, imaging device 202, and subject 100 are fixed in advance, so the 3D model P1 (3D modeling) can be easily and accurately reproduced in the virtual space. Note that multiple 3D models P1 (3D image data) are recorded in the 3D image data recording unit 34 for each type of optical system 20 and subject 100 in the actual inspection process. Moreover, as shown in Figure 4, the imaging device 202 of the 3D model P1 has an imaging range W1 (field of view) that includes the subject 100.

[0043] Returning to Figure 2, we will continue the explanation from step S102 onwards. In step S102, the second generation unit 382 reads scratch shape data specified by the user via the input unit 31 from the scratch shape image data recording unit 35 and generates a pseudo-scratched image of the subject 100 based on the 3D shape data of the subject 100, which is obtained by synthesizing a scratch shape including distance information in the depth direction of the scratch, which has been previously created, at a predetermined position on the surface of the subject 100 of the 3D model P1.

[0044] Figure 5 shows an example of scratch shape data read from the scratch shape image data recording unit 35 by the second generation unit 382. Figure 6 shows an example of scratch shape data after rendering processing by the second generation unit 382. Figure 7 is an enlarged view of region A1 in Figure 6.

[0045] As shown in Figure 5, the second generation unit 382 acquires three-dimensional scratch shape data (scratch shape model P2) from the scratch shape image data recording unit 35. Here, the three-dimensional scratch shape model P2 is a depth map that includes distance information obtained by measuring the shape of the scratch, the width of the scratch, and the distance in the depth direction of the scratch at each position (each pixel) using a stereomicroscope or scale, for the scratch shape area 200 of a sample that has been previously created by the user by scratching an actual subject 100. The depth direction of the scratch is the distance (depth) from the surface of the subject 100 to the bottom of the scratch or the distance from the imaging surface of the image sensor of the imaging device 202 to the bottom of the scratch on the subject 100. Subsequently, the second generation unit 382 generates two-dimensional scratch shape data having the scratch shape area 200 by performing a rendering process on the scratch shape model P2 that converts well-known three-dimensional image data into two-dimensional data. Subsequently, as shown in Figures 6 and 7, the second generation unit 382 generates a pseudo-damaged image P3 by compositing the pseudo-damaged image data having the damaged area 200 with the damaged area model P2 at a predetermined position on the subject 100 of the 3D model P1. Here, the predetermined position may be a position specified by the user via the input unit 31, or it may be a position randomly selected by the second generation unit 382. In this case, the second generation unit 382 generates the pseudo-damaged image P3 by associating the position information (pixel address) of the subject 100 of the 3D model P1 into which the damaged area 200 has been composited with the pseudo-damaged image P3.

[0046] Returning to Figure 2, we will continue the explanation from step S103 onwards. In step S103, the third generation unit 383 generates a damaged area image by extracting the damaged area containing the damage, based on the pseudo-damaged image P3 generated by the second generation unit 382 and the undamaged image generated based on the 3D model P1 of the subject 100 in which the damage shape model P2 has not been synthesized.

[0047] Figure 8 shows an example of an image of a wound abnormality generated by the third generation unit 383. Figure 9 is a magnified view of region A2 in Figure 8.

[0048] As shown in Figures 8 and 9, the third generation unit 383 generates a damaged area image P4 by extracting the damaged area containing the damage, based on the pseudo-damaged image P3 generated by the second generation unit 382 and the undamaged image generated based on the 3D model P1 of the subject 100 in which the damage shape model P2 has not been synthesized. In this case, the third generation unit 383 generates the damaged area image P4 by extracting pixels with differences in brightness values ​​at the same position in the pseudo-damaged image P3 and the 3D model P1 of the subject 100 in which the damage shape model P2 has not been synthesized, as damaged areas 300. Specifically, the third generation unit 383 compares the luminance values ​​of pixels at the same positions in the 3D model P1 of the subject 100, in which the pseudo-damaged image P3 and the dam shape model P2 have not been combined. Pixels with a luminance value of, for example, 128 in the 3D model P1 of the subject 100 are treated as pixels with the same luminance as the reference plane of the 3D model P1, and pixels with other luminance values ​​are extracted from the pseudo-damaged image P3 as damage areas 300 to generate the damage area image P4. In this case, the third generation unit 383 generates the damage area image P4 by associating the positional information (pixel addresses) of the damage areas 300 extracted from the pseudo-damaged image P3 with the damage area image P4.

[0049] Returning to Figure 2, we will continue the explanation from step S104 onwards. In step S104, the fourth generation unit 384 generates a training image by combining the damaged area image P4 with the actual image corresponding to the imaging data of the subject 100 generated by the observation device 2, at the same position in the actual image where the damaged area 300 was extracted from the pseudo-damaged image P3 by the third generation unit 383. In this case, the fourth generation unit 384 generates a training image by combining the damaged area image P4 with the actual image at the same position in the actual image as the position information of the damaged area 300 associated with the damaged area image P4.

[0050] Figure 10 shows an example of a training image generated by the fourth generation unit 384. Figure 11 is a magnified view of area A3 in Figure 10. Figure 12 shows another example of a training image generated by the fourth generation unit 384. Figure 13 is a magnified view of area A4 in Figure 12. Note that in Figures 10 and 11, the image of the abnormal area is superimposed on the edge portion (raised portion) where shadows occur in the real image, while in Figures 12 and 13, the image of the abnormal area is superimposed on the flat portion (flat portion) where no shadows occur in the real image and where illumination light from the illumination device 201 reaches.

[0051] As shown in Figures 10 to 13, the fourth generation unit 384 generates training images P5 and P6 by combining the image of the abnormal damage area P4 with the actual image corresponding to the imaging data of the subject 100 generated by the observation device 2, at the same position where the third generation unit 383 extracted the abnormal damage area 300 from the pseudo-damaged image P3. In this case, in training image P5 shown in Figures 10 and 11, the abnormal damage area 300 is darkened with a shadow (represented by hatching). In contrast, in training image P6 shown in Figures 12 and 13, the abnormal damage area 300 is bright white.

[0052] In this way, the fourth generation unit 384 can generate a training image P5 or training image P6 that matches the shape of the subject 100 in the real image with the reflection characteristics at a position corresponding to that shape.

[0053] Returning to Figure 2, we will continue the explanation from step S105 onwards. In step S105, the determination unit 385 determines whether the number of training images generated by the fourth generation unit 384 has reached a predetermined number. Here, the predetermined number is, for example, 1000 images. Note that the number of training images is not limited to this and can be changed as appropriate, depending on the input operation by the input unit 31 or the performance of the learning model described later. If the determination unit 385 determines that the number of training images generated by the fourth generation unit 384 has reached a predetermined number (step S105: Yes), the control device 3 proceeds to step S107. On the other hand, if the determination unit 385 determines that the number of training images generated by the fourth generation unit 384 has not reached a predetermined number (step S105: No), the control device 3 proceeds to step S106.

[0054] In step S106, the second generation unit 382 reads the scratch shape data specified by the user via the input unit 31 from the scratch shape image data recording unit 35 and changes the composite position of the scratch shape image on the surface of the subject 100 in the 3D model P1. After step S106, the control device 3 returns to step S102.

[0055] In step S107, the fifth generation unit 386 inputs the multiple training images recorded in the training image data recording unit 36 ​​as training data into the training model to perform machine learning, generates a trained model that uses image data as input parameters and outputs whether or not there are scratches on the surface of the subject 100 as output parameters, and records it in the trained model recording unit 372.

[0056] Next, the determination unit 385 determines whether the user has operated the input unit 31 to select another subject (step S108). If the determination unit 385 determines that the user has operated the input unit 31 to select another subject (step S108: Yes), the control device 3 returns to step S101. On the other hand, if the determination unit 385 determines that the user has not operated the input unit 31 to select another subject (step S108: No), the control device 3 terminates this process.

[0057] [Overview of the inspection process] Next, we will describe the overview of the inspection process performed by the control device 3. Figure 14 is a flowchart showing the overview of the inspection process performed by the control device 3.

[0058] As shown in Figure 14, first, the estimation unit 387 acquires image data of the actual image captured by the observation device 2 of the subject 100 to be inspected via the communication unit 30 (step S201).

[0059] Next, the estimation unit 387 inputs the real image data (image data) acquired from the observation device 2 into the trained model recorded by the trained model recording unit 372, and estimates the abnormality of the surface of the object to be inspected using the output result of the trained model, which indicates whether or not there are scratches on the surface of the object 100 (step S202). In this case, if the trained model outputs that there are scratches on the surface (external surface) of the object 100, the estimation unit 387 estimates that there is an abnormality in the object 100 to be inspected, while if the trained model outputs that there are no scratches on the surface of the object 100, the estimation unit 387 estimates that there is no abnormality in the object 100 to be inspected.

[0060] Subsequently, the determination unit 385 determines whether or not there is an abnormality on the surface of the subject 100 based on the estimation result of the estimation unit 387 (step S203). If the determination unit 385 determines that there is an abnormality on the surface of the subject 100 (step S203: Yes), the control device 3 proceeds to step S204. On the other hand, if the determination unit 385 determines that there is no abnormality on the surface of the subject 100 (step S203: No), the control device 3 proceeds to step S205.

[0061] In step S204, the display control unit 388 displays information from the estimation result of the estimation unit 387 on the display unit 32, for example, that an abnormality has occurred on the surface of the subject 100. This allows the user to understand that an abnormality has occurred on the surface of the subject 100. After step S204, the control device 3 proceeds to step S205.

[0062] Next, the determination unit 385 determines whether or not an instruction signal to terminate the inspection has been input from the input unit 31 (step S205). If the determination unit 385 determines that an instruction signal to terminate the inspection has been input from the input unit 31 (step S205: Yes), the control device 3 terminates this process. On the other hand, if the determination unit 385 determines that no instruction signal to terminate the inspection has been input from the input unit 31 (step S205: No), the control device 3 returns to step S201.

[0063] According to the embodiment described above, the fourth generation unit 384 synthesizes a damaged area image P4 with the actual image of the subject 100 captured by the observation device 2, at the same position where the damaged area 300 was extracted from the pseudo-damaged image P3 by the third generation unit 383. This allows for shading according to the shape of the subject 100 and generates a training image with a minimized domain gap with the actual image.

[0064] Furthermore, according to one embodiment, the third generation unit 383 extracts pixels from the pseudo-damaged image P3 that have a difference in brightness value at the same position in the pseudo-damaged image P3 and the 3D model P1 as the damaged area 300, thereby enabling the generation of an accurate damaged area image P4.

[0065] Furthermore, according to one embodiment, the fifth generation unit 386 inputs multiple training images recorded in the training image data recording unit 36, which have physically consistent reflection characteristics, as training data into the training model for machine learning, and generates a trained model that uses image data as input parameters and outputs whether or not there are scratches on the surface of the subject 100 as output parameters. This reduces the rate of undetected and overdetected images and improves inference performance for real images.

[0066] Furthermore, according to one embodiment, the estimation unit 387 inputs image data to a trained model recorded by the trained model recording unit 372, and estimates whether or not there are any abnormalities in the subject 100 based on the presence or absence of scratches on the surface of the subject 100, which is output as an output parameter output by the trained model. This eliminates the need for the user to perform a visual inspection of the subject's appearance.

[0067] In one embodiment, the second generation unit 382 synthesizes scratch shape image data having the scratch abnormality area 300 at a predetermined position on the subject 100 of the 3D model P1 to generate a pseudo-scratched image P3. Then, the third generation unit 383 generates a scratch abnormality area image P4 based on the pseudo-scratched image P3 generated by the second generation unit 382 and the 3D model P1. However, these generation steps may be omitted. In this case, the fourth generation unit 384 synthesizes the scratch shape image data recorded by the scratch shape image data recording unit 35 with the actual image to generate a training image. This reduces the amount of work in the memory of the control unit 38.

[0068] Furthermore, in one embodiment, the fifth generation unit 386 of the control device 3 generated a trained model using multiple training images recorded in the training image data recording unit 36, but the invention is not limited to this. For example, the function of the fifth generation unit 386 may be provided in a device different from the control device 3, and a model generation device that generates a trained model may be provided in this device. This model generation device may acquire multiple training images recorded by the training image data recording unit 36 ​​from the control device 3 via a network, and generate a trained model using these acquired multiple training images.

[0069] Furthermore, in one embodiment, the control device 3 estimated the surface abnormality of the subject 100, which is the object to be inspected, but this is not the only possible outcome. For example, the inspection device could be configured to use a separate device from the control device 3, which contains the functions of the estimation unit 387 and the trained model recording unit 372, to estimate the surface abnormality of the subject 100. In this inspection device, the imaging device 202 of the observation device 2 inputs the data of the actual image captured by the object to be inspected into the trained model, and estimates the surface abnormality of the subject 100 based on the presence or absence of scratches on the surface of the subject 100, which is an output parameter output by the trained model. Of course, the inspection device does not necessarily need to have the trained model recording unit 372; the trained model could be recorded on an external server via a network, and the trained model could be read from this server to estimate the presence or absence of abnormalities in the subject 100.

[0070] (Other embodiments) Furthermore, the program to be executed by the inspection system according to one embodiment of this disclosure is provided as installable or executable file data recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disk), USB medium, or flash memory.

[0071] Furthermore, the program to be executed by the inspection system according to one embodiment of this disclosure may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0072] In this specification, while the flowcharts have used expressions such as "first," "then," and "followed by" to indicate the sequence of processes between steps, the order of processes necessary to carry out the present invention is not uniquely determined by these expressions. That is, the order of processes in the flowcharts described herein can be changed within a reasonable range.

[0073] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of Symbols]

[0074] 1. Inspection System 2. Observation device 3. Control device 20 Actual Inspection Process Optical System 32 Display section 34 3D Image Data Recording Unit 35 Damage shape image data recording unit 36. Learning Image Data Recording Unit 38 Control Unit 100 subjects 200 Scar shape part 201 Lighting equipment 202 Imaging device 300 Abnormal wound area 371 Program Recording Section 372 Trained Model Recording Unit 381 First generation unit 382 Second generation unit 383 Third generation unit 384 Fourth generation unit 385 Judgment section 386 Fifth generation unit 387 Estimation Department 388 Display Control Unit P1 3D Model P2 Scratch Shape Model P3 Image with simulated damage P4 Images of the injured or abnormal area P5, P6 Training Images

Claims

1. A first generation unit generates a three-dimensional model that reproduces the positional relationship and shape of the optical system, including the imaging and illumination devices used in the actual inspection process, and the object to be inspected in the actual inspection process. A second generation unit generates a pseudo-damaged image based on three-dimensional shape data of the subject, which is obtained by synthesizing a scratch shape including distance information in the depth direction of the scratch, which was created in advance, at a predetermined position on the surface of the subject in the three-dimensional model of the subject. A third generation unit generates an image of a damaged area, which extracts the damaged area including the damage, based on the aforementioned pseudo-damaged image and an image without damage generated based on the three-dimensional shape data of the subject in which the damage shape has not been synthesized. A fourth generation unit generates a training image for machine learning by synthesizing the image of the damaged area with the actual image of the subject, at the same position as the extraction position from which the image of the damaged area was extracted from the pseudo-damaged image, Equipped with, Control device.

2. A control device according to claim 1, The third generation unit is, Pixels with a difference in brightness value at the same position in the pseudo-damaged image and the undamaged image are extracted as the damaged area to generate the damaged area image. Control device.

3. A control device according to claim 1, The system further includes a fifth generation unit that inputs the aforementioned training images as input parameters into a learning model to perform machine learning and generates a trained model that outputs whether or not there is an abnormality on the surface of the object being inspected as an output parameter. The fourth generation unit is, Multiple training images are generated by combining the images of the damaged or abnormal areas with the actual image at the same position and at different positions. The fifth generation unit is, The trained model is generated by inputting multiple training images into the training model and performing machine learning. Control device.

4. An estimation unit inputs data of an actual image of the object being inspected, captured by an imaging device whose positional relationship with the object being inspected and the lighting device used in the actual inspection process is predetermined, into a trained model that outputs whether or not there is an abnormality on the surface of the object being inspected as an output parameter, and estimates the abnormality on the surface of the object being inspected using the output result output from the trained model. A display control unit that displays the information of the estimation result from the estimation unit on the display unit, Equipped with, Inspection device.

5. The control device A 3D model is generated that reproduces the positional relationship and shape of the optical system, including the imaging and illumination devices used in the actual inspection process, and the object being inspected in the actual inspection process. Based on the three-dimensional shape data of the subject, a pseudo-image with a scratch is generated by synthesizing a scratch shape, which includes distance information in the depth direction of the scratch, with a predetermined position on the surface of the subject in the three-dimensional model. Based on the aforementioned pseudo-damaged image and the undamaged image generated from the three-dimensional shape data of the subject in which the damage shape is not synthesized, an image of the damaged area containing the damage is generated. The recording unit reads out the actual image of the subject, and the unit synthesizes the image of the damaged area with the actual image at the same position as the extracted position from the pseudo-damaged image to generate a training image for machine learning. Image generation method.

6. The inspection device, A trained model outputs whether or not there is an abnormality on the surface of the object to be inspected as an output parameter. The trained model reads from the recording unit and inputs actual image data of the object to be inspected, captured by an imaging device whose positional relationship with the object to be inspected and the lighting device used in the actual inspection process is predetermined. The trained model then uses the output result to estimate the abnormality on the surface of the object to be inspected. The information of the estimation result is displayed on the display unit. Testing method.

7. In the control device, A 3D model is generated that reproduces the positional relationship and shape of the optical system, including the imaging and illumination devices used in the actual inspection process, and the object being inspected in the actual inspection process. Based on the three-dimensional shape data of the subject, a pseudo-image with a scratch is generated by synthesizing a scratch shape, which includes distance information in the depth direction of the scratch, with a predetermined position on the surface of the subject in the three-dimensional model. Based on the aforementioned pseudo-damaged image and the undamaged image generated from the three-dimensional shape data of the subject in which the damage shape is not synthesized, an image of the damaged area containing the damage is generated. The actual image captured of the subject is read from the recording unit, and the image of the damaged area is synthesized onto the actual image at the same position as the image of the damaged area extracted from the pseudo-damaged image to generate a training image for machine learning. To execute program.

8. In the inspection device, A trained model outputs whether or not there is an abnormality on the surface of the object to be inspected as an output parameter. The trained model receives data of an actual image of the object to be inspected, captured by an imaging device whose positional relationship with the object to be inspected and the lighting device used in the actual inspection process is predetermined. The trained model then uses the output results to estimate the abnormality on the surface of the object to be inspected. To display the information of the estimation result on the display unit. To execute program.

Images