Image processing device and image processing method

Abstract

This image processing device comprises: a virtual model generation unit that uses three-dimensional shape data regarding an object to be inspected as a basis to generate a virtual model (600) in which the three-dimensional shape of the object to be inspected is represented in a virtual space; a defect model acquisition unit for acquiring a defect model (620) that is represented in the virtual space and includes defect data; and a defect application unit that applies a defect to the virtual model (600) on the basis of the defect model (620).

Description

Image processing device and image processing method

[0001] This application claims the benefit of priority from Japanese Patent Application No. 2024-108024, filed on July 4, 2024, the contents of which are incorporated herein by reference.

[0002] Conventionally, visual inspection of an object has been performed using an imaging device. For example, Patent Literature 1 discloses a technique for determining whether or not an object has a defect based on an image captured using the principle of photometric stereo.

[0003] Japanese Patent Application Laid-Open No. 2020-38215

[0004] In visual inspection of objects using deep learning models in image analysis, a wide variety of training data is required to create the deep learning model. However, when the object is, for example, a high-quality, low-volume product, it is difficult to obtain captured images as training data. Therefore, it has been difficult to generate the wide variety of training data required to create a deep learning model.

[0005] In consideration of the above-described problems, the present disclosure aims to provide an image processing device and an image processing method that can easily generate training data.

[0006] In order to solve the above problem, an image processing device according to one aspect of the present disclosure includes a virtual model generation unit that generates a virtual model that represents the three-dimensional shape of an object to be inspected in a virtual space, a defect model acquisition unit that acquires a defect model that is represented in the virtual space and includes defect data, and a defect assignment unit that assigns defects to the virtual model based on the defect model.

[0007] The defect model may be configured with a plane in which the shape of the defect to be applied to the virtual model is specified by a grayscale image.

[0008] The defect imparting unit may select at least one point from a predetermined group of points included in a replica model that replicates a specific surface that is part of the surface of the virtual model, and set the position of the defect model so that the surface of the defect model intersects with a normal line of the replica model that passes through the selected point.

[0009] The defect imparting unit may move each point of a predetermined group of points included in the surface of the defect model in the normal direction of the defect model, and identify a contact point where each point comes into contact with the surface of the virtual model.

[0010] The contact point may be identified based on the dot product of the normal vector of each point in the point cloud included in the surface of the defect model and the inward vector of the virtual model.

[0011] The system may include a 3DCG image generation unit that generates at least three 3DCG images with different virtual light sources based on a virtual model with defects, at least three virtual light sources that illuminate the virtual model with defects, and a virtual imaging device that images the illuminated virtual model, and an image generation unit that generates an image as training data based on a direction vector of the surface of the virtual model derived based on the at least three 3DCG images.

[0012] In order to solve the above problem, an image processing method according to one aspect of the present disclosure includes the steps of generating a virtual model that represents the three-dimensional shape of an object to be inspected in a virtual space, obtaining a defect model that is represented in the virtual space and includes defect data, and adding defects to the virtual model based on the defect model.

[0013] According to the present disclosure, it is possible to easily generate training data.

[0014] FIG. 1 is a schematic configuration diagram of an image processing system according to this embodiment. FIG. 2 is a schematic block diagram of the image processing system according to this embodiment. FIG. 3 is a block diagram showing an example of the functional configuration of an image processing device according to this embodiment. FIG. 4 is a schematic configuration diagram showing an example of a 3D graphic model generated in a virtual space. FIG. 5 is a diagram showing an example of a defect model according to this embodiment. FIG. 6 is a first diagram for explaining the positions of a virtual model, a duplicate model, and a defect model in a virtual space. FIG. 7 is a second diagram for explaining the positions of a virtual model, a duplicate model, and a defect model in a virtual space. FIG. 8 is a diagram showing how a point cloud included in a defect model is moved to a virtual model. FIG. 9 is a diagram for explaining the relationship between each point of the defect model and a contact point on the surface of the virtual model. FIG. 10 is a diagram showing an example of a first 3DCG image. FIG. 11 is a diagram showing an example of a second 3DCG image. FIG. 12 is a diagram showing an example of a third 3DCG image. FIG. 13 is a diagram showing an example of a first image. FIG. 14 is an explanatory diagram for explaining a direction vector of a surface of an object to be inspected illuminated by light from a first virtual light source. Fig. 15 is an explanatory diagram for explaining a direction vector of a surface of an inspection object illuminated by light from a second virtual light source. Fig. 16 is an explanatory diagram for explaining a direction vector of a surface of an inspection object illuminated by light from a third virtual light source. Fig. 17 is a diagram showing an example of an inspection model. Fig. 18 is a flowchart showing an example of an image processing method according to this embodiment.

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Specific dimensions, materials, numerical values, etc. shown in the embodiments are merely examples for ease of understanding and do not limit the present disclosure unless otherwise specified. In this specification and drawings, elements having substantially the same functions and configurations are designated by the same reference numerals to avoid redundant explanation, and elements not directly related to the present disclosure are not shown.

[0016] FIG. 1 is a schematic diagram of an image processing system 100 according to this embodiment. The image processing system 100 is an inspection system for performing a visual inspection of an inspection target (object) 200. In FIG. 1, the X direction, Y direction, and Z direction are mutually orthogonal directions. The X direction and Y direction are horizontal directions, and the Z direction is a vertical direction. As shown in FIG. 1, the image processing system 100 includes one imaging device 300, multiple light sources 400, and an image processing device 500.

[0017] The inspection object 200 is an actual product that is placed in real space and undergoes visual inspection by the image processing system 100. In this embodiment, the inspection object 200 is, for example, a general industrial product. However, the inspection object 200 is not limited to an industrial product as long as it can be imaged by the imaging device 300.

[0018] In this embodiment, a defect 210, which is an abnormal portion, may be included in a portion of the inspection object 200. The defect 210 is formed in a portion of the surface of the inspection object 200, and is, for example, a protrusion or depression formed on the surface of the inspection object 200. The defect 210 in the inspection object 200 is generated, for example, by a reduction in thickness of the inspection object 200 due to corrosion, a foreign object being caught during press working of the inspection object 200, or a tool colliding with the surface of the inspection object 200.

[0019] The imaging device 300 is a real camera placed in real space. The imaging device 300 is placed, for example, on an extension line in the +Z direction of the inspection object 200. In other words, the camera viewpoint of the imaging device 300 is located on an extension line in the +Z direction of the inspection object 200, and the imaging direction of the imaging device 300 is the -Z direction toward the inspection object 200. In this way, in this embodiment, the imaging device 300 images the inspection object 200 from above. However, without being limited to this, the imaging device 300 may image the inspection object 200 from below or from the side.

[0020] The imaging device 300 captures an image of the inspection object 200 and generates a captured image of the inspection object 200. The imaging device 300 outputs the generated captured image of the inspection object 200 to the image processing device 500. In this embodiment, one imaging device 300 is arranged for one inspection object 200. One imaging device 300 is arranged facing the inspection object 200 so that the inspection object 200 is included in the imaging range. With the position of the imaging device 300 fixed, multiple captured images of the inspection object 200 are captured.

[0021] At least three light sources 400 are prepared. The reason for using at least three light sources is to capture images of the inspection object 200 using a photometric stereo method. The photometric stereo method is a three-dimensional measurement technique in which images of the inspection object 200 illuminated from a plurality of different illumination directions are captured and a normal vector, which is a directional vector of the surface of the inspection object 200, is obtained from the shading information. Here, the normal vector is a vector perpendicular to the surface of the inspection object 200. The photometric stereo method is also called photometric stereo.

[0022] The multiple light sources 400 include a first light source 400A, a second light source 400B, and a third light source 400C. The first light source 400A, the second light source 400B, and the third light source 400C are actual light sources arranged in real space. The first light source 400A is arranged on the +Y direction side of the inspection object 200. In this case, when the position of the first light source 400A is taken as the reference position, the first direction L1 in which light from the first light source 400A is irradiated is the -Y direction and a direction tilted at -45° with respect to the horizontal direction. In other words, the first direction L1 is the -Y direction with respect to the imaging direction of the imaging device 300 and a direction tilted at -45° with respect to the horizontal direction. The first light source 400A illuminates the inspection object 200 from the first direction L1.

[0023] The second light source 400B is disposed on the +X direction side of the inspection object 200. In this case, the second direction L2 in which the light from the second light source 400B is irradiated is the -X direction and a direction tilted at -45 degrees with respect to the horizontal direction when the position of the second light source 400B is taken as the reference position. In other words, the second direction L2 is the -X direction with respect to the imaging direction of the imaging device 300 and a direction tilted at -45 degrees with respect to the horizontal direction. The second light source 400B illuminates the inspection object 200 from the second direction L2, which is different from the first direction L1.

[0024] The third light source 400C is disposed on the -Y direction side of the inspection object 200. In this case, the third direction L3 in which the light from the third light source 400C is emitted is the +Y direction and a direction tilted at -45 degrees with respect to the horizontal direction when the position of the third light source 400C is taken as the reference position. In other words, the third direction L3 is the +Y direction with respect to the imaging direction of the imaging device 300 and a direction tilted at -45 degrees with respect to the horizontal direction. The third light source 400C illuminates the inspection object 200 from the third direction L3, which is different from the first direction L1 and the second direction L2.

[0025] In this manner, the inspection object 200 of this embodiment is disposed so that it can be imaged by the imaging device 300 from directly above, and can be illuminated by the first light source 400A, the second light source 400B, and the third light source 400C from an angle of 45° above at 90° intervals in the circumferential direction. With the positions of the first light source 400A, the second light source 400B, and the third light source 400C fixed, the inspection object 200 is illuminated by light emitted from any one of the first light source 400A, the second light source 400B, and the third light source 400C.

[0026] In the present embodiment, an example will be described in which the first light source 400A, the second light source 400B, and the third light source 400C are arranged at 90° intervals, such as 0°, 90°, and 180°, in the circumferential direction of the inspection object 200. However, without being limited to this, the first light source 400A, the second light source 400B, and the third light source 400C may be arranged at equal 120° intervals, such as 0°, 120°, and 240°, in the circumferential direction of the inspection object 200. Furthermore, when the plurality of light sources 400 is composed of four light sources, the four light sources may be arranged at equal 90° intervals, such as 0°, 90°, 180°, and 270°, in the circumferential direction of the inspection object 200.

[0027] In this way, the first light source 400A, the second light source 400B, and the third light source 400C illuminate the inspection object 200 from different directions. In this embodiment, the number of the plurality of light sources 400 is three. However, the number of the plurality of light sources 400 is not limited to three as long as it is at least three or more.

[0028] The image processing device 500 is electrically connected to the image capturing device 300 and the plurality of light sources 400, and controls the image capturing device 300 and the plurality of light sources 400. The image processing device 500 controls the image capturing device 300 and the plurality of light sources 400 so as to illuminate the inspection object 200 from different illumination directions at least three different times in order to capture an image of the inspection object 200 by the photometric stereo method.

[0029] Specifically, the image processing device 500 controls the first light source 400A to irradiate the inspection object 200 with light in a first direction L1 at a first timing, thereby acquiring a first captured image of the illuminated inspection object 200. The image processing device 500 also controls the second light source 400B to irradiate the inspection object 200 with light in a second direction L2 at a second timing different from the first timing, thereby acquiring a second captured image of the illuminated inspection object 200. The second timing is, for example, a timing after the first timing. The image processing device 500 also controls the third light source 400C to irradiate the inspection object 200 with light in a third direction L3 at a third timing different from the first and second timings, thereby acquiring a third captured image of the illuminated inspection object 200. The third timing is, for example, a timing after the second timing.

[0030] 2 is a schematic block diagram of the image processing system 100 according to this embodiment. As shown in FIG. 2, the image processing device 500 includes an I / F 510, a storage device 520, a system bus 530, one or more processors 540, and one or more memories 550. The I / F 510 is an interface for communicating with the imaging device 300, the first light source 400A, the second light source 400B, and the third light source 400C. Note that the image processing device 500 according to this embodiment also functions as a generation device that generates training data, as will be described in detail below.

[0031] The storage device 520 is composed of RAM, flash memory, HDD, etc., and holds various information necessary for the processing of the processor 540 described below. Specifically, the storage device 520 stores three-dimensional shape data. The three-dimensional shape data is data expressed as a collection of line information and surface information each including two points, and is data that can identify, for example, the three-dimensional shape of a virtual model 600 (described later). The three-dimensional shape data includes, for example, point cloud data, which is a collection of points configured with three-dimensional coordinates represented by X, Y, and Z. A 3D graphics model is obtained by converting this point cloud data into surface data. The surface data is data that forms the surface of the 3D graphics model, and is, for example, mesh data or surface data. The 3D graphics model is generated by converting the point cloud data into mesh data or surface data. In this embodiment, the storage device 520 stores three-dimensional shape data that represents the three-dimensional shape of the inspection object 200. The three-dimensional shape data in this embodiment is, for example, 3D CAD data. However, without being limited to this, the three-dimensional shape data may also be 3D data obtained by measuring the inspection object 200 and the defect 210 using a three-dimensional optical measuring instrument such as a laser displacement meter or a stereo imaging device, or an X-ray CT device.

[0032] The system bus 530 is a transmission path that electrically connects the I / F 510, the storage device 520, the processor 540, and the memory 550 and transmits data among them.

[0033] The processor 540 includes, for example, a CPU (Central Processing Unit). The memory 550 includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM is a storage element that stores programs and calculation parameters used by the CPU. The RAM is a storage element that temporarily stores data such as variables and parameters used in processing executed by the CPU.

[0034] 3 is a block diagram showing an example of the functional configuration of an image processing device 500 according to this embodiment. For example, as shown in FIG. 3, the image processing device 500 includes a 3D graphic model generation unit (virtual model generation unit) 500a, a defect model acquisition unit 500b, a defect assignment unit 500c, a 3DCG image generation unit 500d, a first image generation unit (image generation unit) 500e, a captured image generation unit 500f, a second image generation unit 500g, a teacher data generation unit 500h, a model learning unit 500i, and an estimation unit 500j.

[0035] 2 cooperates with a program stored in memory 550 and executes the program stored in memory 550. This realizes various processes including the processes described below that are performed by the 3D graphic model generation unit 500a, defect model acquisition unit 500b, defect assignment unit 500c, 3DCG image generation unit 500d, first image generation unit 500e, captured image generation unit 500f, second image generation unit 500g, teacher data generation unit 500h, model learning unit 500i, and estimation unit 500j.

[0036] The 3D graphic model generating unit 500a generates a virtual model 600, which is a 3D graphic model that represents the three-dimensional shape of the inspection object 200 in a virtual space S, based on the three-dimensional shape data of the inspection object 200 stored in the storage device 520. The position and orientation of the virtual model 600 that imitates the inspection object 200 provided in the virtual space S are set to be the same as the position and orientation of the actual inspection object 200 shown in FIG.

[0037] Furthermore, the 3D graphic model generation unit 500a sets the position of the camera viewpoint and the imaging direction of the virtual imaging device 700 in the virtual space S. Here, the position of the camera viewpoint and the imaging direction of the virtual imaging device 700 are set to be the same as the camera viewpoint and the imaging direction of the actual imaging device 300 shown in FIG.

[0038] Furthermore, 3D graphic model generation unit 500a sets the position and illumination direction of first virtual light source 800A in virtual space S. Here, the position and illumination direction of first virtual light source 800A are set to be the same as the position and illumination direction of real first light source 400A shown in FIG. 1 . 3D graphic model generation unit 500a sets the position and illumination direction of second virtual light source 800B in virtual space S. Here, the position and illumination direction of second virtual light source 800B are set to be the same as the position and illumination direction of real second light source 400B shown in FIG. 1 . 3D graphic model generation unit 500a sets the position and illumination direction of third virtual light source 800C in virtual space S. Here, the position and illumination direction of third virtual light source 800C are set to be the same as the position and illumination direction of real third light source 400C shown in FIG. 1 . First virtual light source 800A, second virtual light source 800B, and third virtual light source 800C are also collectively referred to as multiple virtual light sources 800.

[0039] 4 is a schematic configuration diagram showing an example of a 3D graphic model generated in virtual space S. As shown in Fig. 4, in virtual space S, a virtual model 600 simulating inspection object 200, a virtual imaging device 700 simulating imaging device 300, and a plurality of virtual light sources 800 simulating a plurality of light sources 400 are arranged. Note that the plurality of virtual light sources 800 includes at least three virtual light sources in order to capture an image of virtual model 600 simulating inspection object 200 by a photometric stereo method.

[0040] In the XYZ coordinate system within virtual space S, virtual model 600 is disposed at a position corresponding to inspection target object 200 shown in Fig. 1 . Furthermore, in the XYZ coordinate system within virtual space S, virtual imaging device 700 is disposed at a position corresponding to imaging device 300 shown in Fig. 1 . In the XYZ coordinate system within virtual space S, first virtual light source 800A is disposed at a position corresponding to first light source 400A shown in Fig. 1 . In the XYZ coordinate system within virtual space S, second virtual light source 800B is disposed at a position corresponding to second light source 400B shown in Fig. 1 . In the XYZ coordinate system within virtual space S, third virtual light source 800C is disposed at a position corresponding to third light source 400C shown in Fig. 1 .

[0041] In this embodiment, virtual imaging device 700 is disposed on an extension line in the +Z direction with respect to virtual model 600. Furthermore, first virtual light source 800A is disposed on the +Y direction side with respect to virtual model 600. In this case, first direction L1 in which light from first virtual light source 800A is emitted is the −Y direction and a direction tilted at −45° with respect to the horizontal direction when the position of first virtual light source 800A is taken as the reference position. In other words, first direction L1 is the −Y direction with respect to the imaging direction of virtual imaging device 700 and a direction tilted at −45° with respect to the horizontal direction.

[0042] Second virtual light source 800B is disposed on the +X direction side with respect to virtual model 600. In this case, second direction L2, in which light from second virtual light source 800B is emitted, is the −X direction and a direction tilted at −45 degrees with respect to the horizontal when the position of second virtual light source 800B is taken as the reference position. In other words, second direction L2 is the −X direction with respect to the imaging direction of virtual imaging device 700 and a direction tilted at −45 degrees with respect to the horizontal. Third virtual light source 800C is disposed on the −Y direction side with respect to virtual model 600. In this case, third direction L3, in which light from third virtual light source 800C is emitted, is the +Y direction and a direction tilted at −45 degrees with respect to the horizontal when the position of third virtual light source 800C is taken as the reference position. In other words, third direction L3 is the +Y direction with respect to the imaging direction of virtual imaging device 700 and a direction tilted at −45 degrees with respect to the horizontal.

[0043] In this way, the virtual model 600 of this embodiment is positioned so that it can be imaged from directly above by the virtual imaging device 700, and so that it can be illuminated by the first virtual light source 800A, the second virtual light source 800B, and the third virtual light source 800C at 90° intervals around the circumference from a diagonal angle of 45° above.

[0044] Note that the imaging direction of imaging device 300 and the first direction L1, second direction L2, and third direction L3 of multiple light sources 400 shown in Figure 1 are the same as the imaging direction of virtual imaging device 700 and the first direction L1, second direction L2, and third direction L3 of multiple virtual light sources 800 shown in Figure 4.

[0045] Incidentally, in visual inspection of an inspection object 200 using a deep learning model in image analysis, in order to create a deep learning model, a large number of captured images, for example, hundreds to tens of thousands, are required as training data in which defects 210 in the inspection object 200 are identified.

[0046] However, when the inspection object 200 is a high-quality, low-volume product, such as an aircraft part, it is difficult to obtain captured images as training data. Furthermore, when captured images as training data are obtained during a repair process after the aircraft part is manufactured, it is impossible to obtain training data before the aircraft part is manufactured. Therefore, it has been difficult to generate the wide variety of training data required to create a deep learning model.

[0047] Therefore, this embodiment includes a defect model acquisition unit 500b that acquires a wide variety of defect models for adding a wide variety of defects to a virtual model 600, which is a 3D graphic model of the inspection target 200. This embodiment also includes a defect assignment unit 500c that assigns defects to the virtual model 600 based on the wide variety of defect models acquired by the defect model acquisition unit 500b. By acquiring a wide variety of defect models using the defect model acquisition unit 500b and assigning a wide variety of defects based on the wide variety of defect models to the virtual model 600 using the defect assignment unit 500c, a wide variety of images can be obtained as training data. As a result, a deep learning model can be easily created.

[0048] FIG. 5 is a diagram showing an example of a defect model 620 according to this embodiment. The defect model 620 is, for example, a planar 3D graphic model, and is generated based on three-dimensional shape data that represents the three-dimensional shape of the defect model 620 in a virtual space. Specifically, the defect model 620 is generated by converting point cloud data, which is the three-dimensional shape data of the defect model 620, into surface data such as mesh data or surface data. The image of the defect model 620 shown in FIG. 5 is a grayscale image, as will be described in detail later, and the pixel value of each pixel in the image of FIG. 5 is expressed in 256 gradations ranging from "0" to "255." The closer the pixel value is to "0," the closer it is to black shown in FIG. 5, and the closer the pixel value is to "255," the closer it is to white shown in FIG. 5. In addition to the point cloud data, the defect model 620 includes image information, maximum value information, rotation angle information, and magnification information as defect data. In this embodiment, an example will be described in which the defect model 620 is a planar 3D graphic model. However, the present invention is not limited to this, and the defect model 620 may be a three-dimensional 3D graphic model. Furthermore, the number of point clouds per unit area of ​​the defect model 620 is the same as the number of point clouds per unit area of ​​the virtual model 600. However, the present invention is not limited to this, and the number of point clouds per unit area of ​​the defect model 620 may be greater than the number of point clouds per unit area of ​​the virtual model 600. This allows the shape of the defect 610 assigned to the virtual model 600 to be more detailed. Furthermore, the number of point clouds per unit area of ​​the defect model 620 may be less than the number of point clouds per unit area of ​​the virtual model 600. This allows the shape of the defect 610 assigned to the virtual model 600 to be coarse.

[0049] The image information is, for example, the grayscale image shown in FIG. 5 . The grayscale image includes a plurality of pixels. The total number of pixels included in the grayscale image is the same as the total number of points in the predetermined point cloud included in the defect model 620. Furthermore, a pixel value of each pixel in the grayscale image is associated with each point in the point cloud of the defect model 620. The pixel value of each pixel in the grayscale image is expressed in 256 gradations ranging from "0" to "255." A value corresponding to the depth of the defect 610 assigned to the virtual model 600 is set for each pixel in the grayscale image. For example, the deeper the defect 610, the closer the pixel value is to "255" (white in FIG. 5 ), and the shallower the defect 610, the closer the pixel value is to "0" (black in FIG. 5 ). In the example shown in FIG. 5 , each pixel in the grayscale image corresponding to each point in the point cloud of the defect model 620 is shown. In this embodiment, multiple types of grayscale images are prepared in advance. Therefore, it is possible to generate a plurality of types of defect models 620 in the virtual space according to the type of grayscale image.

[0050] The maximum value information is, for example, the maximum depth value or the maximum height value of the defect 610 to be assigned to the virtual model 600. If the defect 610 is a depression, the maximum depth value is set as the maximum value information, and if the defect 610 is a protrusion, the maximum height value is set as the maximum value information. Specifically, for example, a value of "+5" mm is set as the maximum depth value of the defect 610. Also, for example, a value of "-5" mm is set as the maximum height value of the defect 610. In other words, whether the defect 610 is a depression or a protrusion is determined by the "+" or "-" sign of the maximum value information.

[0051] The rotation angle information is a value that represents the orientation of the defect model 620, and is, for example, an angle value in the range of "0" degrees to "360" degrees. Fig. 5 shows an example in which the rotation angle information is "0" degrees, which is the reference angle.

[0052] The magnification information is a value that represents the magnification of the defect model 620, and for example, the magnification information is a value of "1", "0.5", "2", etc. Fig. 5 shows an example in which the magnification information is "1", which is the reference magnification.

[0053] The defect model acquisition unit 500b acquires at least one defect model 620 represented in a virtual space. For example, the defect model acquisition unit 500b acquires the defect model 620 determined in response to an operator's selection of image information, maximum value information, rotation angle information, and magnification information. Alternatively, the defect model acquisition unit 500b acquires the defect model 620 determined in response to an automatic random selection of image information, maximum value information, rotation angle information, and magnification information. The defect model acquisition unit 500b may regularly vary and combine the image information, maximum value information, rotation angle information, and magnification information. For example, the defect model acquisition unit 500b may fix at least one of the image information, maximum value information, rotation angle information, and magnification information, and randomly vary the information other than the fixed information.

[0054] The defect adding unit 500c adds a defect 610 to the virtual model 600 based on the defect model 620 acquired by the defect model acquiring unit 500b. The process of adding the defect 610 to the virtual model 600 based on the defect model 620 will be described below.

[0055] Fig. 6 is a first diagram for explaining the positions of the virtual model 600, the duplicate model 630, and the defect model 620 in the virtual space. Fig. 7 is a second diagram for explaining the positions of the virtual model 600, the duplicate model 630, and the defect model 620 in the virtual space.

[0056] 6 and 7 , the defect imparting unit 500c generates a duplicate model 630 by duplicating a specific surface SA1, which is a part of the surface of the virtual model 600. The duplicate model 630 is a 3D graphic model, and is generated based on three-dimensional shape data that represents the three-dimensional shape of the specific surface SA1 of the virtual model 600 in virtual space. Specifically, the duplicate model 630 is generated by converting point cloud data, which is the three-dimensional shape data of the specific surface SA1 of the virtual model 600, into surface data.

[0057] The defect adding unit 500c translates the replica model 630 outward from the virtual model 600 along the normal direction of the geometric center point CE0 of the specific surface SA1. The defect adding unit 500c also selects at least one point from a predetermined group of points included in the replica model 630. Here, the group of points included in the replica model 630 is a group of points constituting the surface data of the replica model 630. The at least one point is a point automatically selected at random by the defect adding unit 500c. However, this is not limiting, and the at least one point may be a point manually selected by an operator. In the example shown in FIGS. 6 and 7 , the defect adding unit 500c selects the geometric center point CE1 of the replica model 630.

[0058] The defect adding unit 500c controls the orientation of the surface of the defect model 620 so that the normal direction of the surface of the defect model 620 coincides with the normal direction passing through the geometric center point CE1 of the replica model 630. For example, the defect adding unit 500c sets the orientation of the surface of the defect model 620 so that the normal direction of the geometric center point CE2 of the surface of the defect model 620 coincides with the normal direction of the geometric center point CE1 of the replica model 630.

[0059] Furthermore, the defect adding unit 500c sets the position of the defect model 620 so that an extension line L11 that passes through the selected geometric center point CE1 and extends in the normal direction of the replica model 630 intersects with a surface of the defect model 620. For example, as shown in FIG. 6 , the defect adding unit 500c sets the position of the defect model 620 so that the extension line L11 intersects with a geometric center point CE2 of the surface of the defect model 620.

[0060] 8 is a diagram showing how the point cloud included in the defect model 620 is moved to the virtual model 600. Here, the point cloud included in the defect model 620 is a point cloud that constitutes the surface data of the defect model 620. As shown in FIG. 8, the defect adding unit 500c moves the point cloud included in the defect model 620 in a direction R2 opposite to the normal direction R1 of the surface of the defect model 620.

[0061] The defect imparting unit 500c stops the movement at a position where the point cloud of the defect model 620 contacts the surface of the virtual model 600. In the example shown in Fig. 8 , a first point P1, a second point P2, and a third point P3, which are point clouds included in the defect model 620, move to the positions of a first contact point P11, a second contact point P12, and a third contact point P13 on the surface of the virtual model 600. Here, there are two contact points on the surface of the virtual model 600 with which each point of the defect model 620 can come into contact.

[0062] 9 is a diagram for explaining the relationship between each point of the defect model 620 and the contact point on the surface of the virtual model 600. In FIG. 9, the solid arrows indicate inward vectors of the virtual model 600, and the hollow arrows indicate normal vectors of the defect model 620.

[0063] 9, on the surface of the virtual model 600, a fourth contact point P21 is located on an extension of the normal line of the first contact point P11. A fifth contact point P22 is located on an extension of the normal line of the second contact point P12. A sixth contact point P23 is located on an extension of the normal line of the third contact point P13.

[0064] The defect assignment unit 500c determines whether the dot product of the normal vector at each point of the defect model 620 and the inward vector at each contact point of the virtual model 600 is positive or negative. If the dot product is positive, the defect assignment unit 500c deselects the point of the defect model 620 as a contact point to be moved, and if the dot product is negative, the defect assignment unit 500c selects the point of the defect model 620 as a contact point to be moved.

[0065] 9 , the dot products of the first contact point P11, the second contact point P12, and the third contact point P13 on the surface of the virtual model 600 with the normal vectors of the respective points of the defect model 620 are negative. Furthermore, the dot products of the fourth contact point P21, the fifth contact point P22, and the sixth contact point P23 on the surface of the virtual model 600 with the normal vectors of the respective points of the defect model 620 are positive. Therefore, the defect imparting unit 500c selects the first contact point P11, the second contact point P12, and the third contact point P13 on the surface of the virtual model 600 as the contact points to which the first point P1, the second point P2, and the third point P3 of the defect model 620 will be moved. In this way, the contact points to which the respective points constituting the defect model 620 will be moved are identified on the surface of the virtual model 600.

[0066] 9 , the defect assigning unit 500c forms a first displacement point P31 of the defect 610 in the virtual model 600 by displacing the selected first contact point P11 inward along a normal direction starting from the first contact point P11. The displacement amount from the first contact point P11 to the first displacement point P31 is determined based on the maximum value information and the image information. For example, if the maximum value information is "+5" mm and the pixel value of the grayscale image corresponding to the first point P1 of the defect model 620 is "50," the defect assigning unit 500c determines the displacement amount to be "0.98" mm.

[0067] Similarly, the defect imparting unit 500c forms a second displacement point P32 of the defect 610 in the virtual model 600 by displacing the selected second contact point P12 inward along the normal direction with the second contact point P12 as the base point. The defect imparting unit 500c also forms a third displacement point P33 of the defect 610 in the virtual model 600 by displacing the selected third contact point P13 inward along the normal direction with the third contact point P13 as the base point. In this manner, the defect imparting unit 500c imparts the defect 610 to the virtual model 600 by displacing points on the surface of the virtual model 600 based on the defect model 620. Hereinafter, the virtual model 600 to which the defect 610 has been imparted may be referred to as a defect-added virtual model or simply as a virtual model. Furthermore, the portion of the defect-added virtual model that includes only the defect 610 may be referred to as a defect shape model.

[0068] Since multiple types of image information of the defect model 620 are prepared in advance, the defect imparting unit 500c can impart multiple types of defects 610 to the virtual model 600. As a result, multiple types of virtual models 600 having a wide variety of defects 610 are generated.

[0069] The 3DCG image generation unit 500d emits light from one of the multiple virtual light sources 800, virtually captures an image of the virtual model 600 illuminated by the light from the one virtual light source using the virtual imaging device 700, and generates one 3DCG image. Specifically, the 3DCG image generation unit 500d generates a 3DCG image by performing physically based rendering on the virtual model 600. Physically based rendering is a method of calculating an image captured by the virtual imaging device 700 based on real-world optical laws such as reflection, transmission, and diffusion. In this way, the 3DCG image generation unit 500d generates a 3DCG image based on the virtual model 600 to which the defect 610 has been added.

[0070] 3DCG image generation unit 500d virtually captures an image of virtual model 600 using virtual imaging device 700 while changing the irradiation direction of light from multiple virtual light sources 800, and generates multiple 3DCG images. Specifically, 3DCG image generation unit 500d first emits light from first virtual light source 800A of multiple virtual light sources 800, and virtually captures an image of virtual model 600 illuminated by the light from first virtual light source 800A using virtual imaging device 700, thereby generating first 3DCG image 710. Next, 3DCG image generation unit 500d emits light from second virtual light source 800B of multiple virtual light sources 800, and virtually captures an image of virtual model 600 illuminated by the light from second virtual light source 800B using virtual imaging device 700, thereby generating second 3DCG image 720. Finally, the 3DCG image generation unit 500d emits light from a third virtual light source 800C among the multiple virtual light sources 800, virtually captures an image of the virtual model 600 illuminated by the light from the third virtual light source 800C using the virtual imaging device 700, and generates a third 3DCG image 730.

[0071] The first 3DCG image 710, the second 3DCG image 720, and the third 3DCG image 730 are images virtually captured at different times. Note that the first 3DCG image 710, the second 3DCG image 720, and the third 3DCG image 730 are images virtually captured using a photometric stereo method under the same or similar capturing conditions as the first captured image, the second captured image, and the third captured image.

[0072] Fig. 10 is a diagram showing an example of a first 3DCG image 710. Fig. 11 is a diagram showing an example of a second 3DCG image 720. Fig. 12 is a diagram showing an example of a third 3DCG image 730. As shown in Figs. 10 to 12, the first 3DCG image 710, the second 3DCG image 720, and the third 3DCG image 730 include images of the virtual model 600 and the defect 610 imparted to the surface of the virtual model 600.

[0073] 10 , virtual model 600 is illuminated by light emitted from first virtual light source 800A in first direction L1. As a result, as shown in FIG. 10 , a shadow region SH, indicated by hatching, is formed at the end of virtual model 600 on the first direction L1 side. Furthermore, a shadow region SH, indicated by hatching, is formed inside the depression on the first direction L1 side of defect 610. Furthermore, no shadow region SH is formed inside the depression on the opposite side of defect 610 from first direction L1 due to the illumination of light.

[0074] 11 , virtual model 600 is illuminated by light emitted from second virtual light source 800B in second direction L2. As a result, as shown in FIG. 11 , a shadow region SH, indicated by hatching, is formed at the end of virtual model 600 on the second direction L2 side. Furthermore, a shadow region SH, indicated by hatching, is formed inside the depression on the second direction L2 side of defect 610. Furthermore, no shadow region SH is formed inside the depression on the opposite side of defect 610 from second direction L2 due to the illumination of light.

[0075] 12 , virtual model 600 is illuminated by light emitted from third virtual light source 800C in third direction L3. As a result, as shown in FIG. 12 , a shadow region SH, indicated by hatching, is formed at the end of virtual model 600 on the third direction L3 side. Furthermore, a shadow region SH, indicated by hatching, is formed inside the depression on the third direction L3 side of defect 610. Furthermore, no shadow region SH is formed inside the depression on the opposite side of defect 610 from third direction L3 due to the illumination of light.

[0076] The first image generation unit 500e generates the first image 740 based on the direction vector of the surface of the virtual model 600 derived based on the first 3DCG image 710, the second 3DCG image 720, and the third 3DCG image 730. A method for generating the first image 740 will be described in detail below.

[0077] FIG. 13 is a diagram showing an example of a first image 740. The first image 740 is a so-called normal map image. The normal map image is an image in which a direction vector n of the surface of the inspection object 200 or the virtual model 600 is derived for each pixel of a captured image of the inspection object 200 or a 3DCG image of the virtual model 600, and visualized as RGB pixel values. The direction vector n of the surface of the inspection object 200 or the virtual model 600 is derived using the following equation (1). Note that the first method of deriving the direction vector n of the surface of the virtual model 600 and generating the first image 740, which is a normal map image, is similar to the second method of deriving the direction vector n of the surface of the inspection object 200 and generating the second image, which is a normal map image. Therefore, the first method will be described in detail below, and a detailed description of the second method will be omitted.

[0078] Fig. 14 is an explanatory diagram for explaining direction vector n of the surface of virtual model 600 illuminated by light from first virtual light source 800A. Fig. 15 is an explanatory diagram for explaining direction vector n of the surface of virtual model 600 illuminated by light from second virtual light source 800B. Fig. 16 is an explanatory diagram for explaining direction vector n of the surface of virtual model 600 illuminated by light from third virtual light source 800C.

[0079] 14 , l = (a1, a2, a3) indicates the direction vector of first virtual light source 800A. i = A indicates a luminance value, which is the pixel value of each pixel of first 3DCG image 710. n = (x, y, z) indicates the direction vector of the surface. The direction vector l of first virtual light source 800A, luminance value i, and direction vector n of the surface have a relationship of i = n · l. In other words, luminance value i is expressed as the dot product of the direction vector l of first virtual light source 800A and the direction vector n of the surface.

[0080] 15 , l = (b1, b2, b3) indicates the direction vector of second virtual light source 800B. i = B indicates the luminance value, which is the pixel value of each pixel of second 3DCG image 720. n = (x, y, z) indicates the direction vector of the surface. The relationship between direction vector l, luminance value i, and direction vector n of the surface of second virtual light source 800B is i = n · l. In other words, the luminance value i is expressed as the dot product of direction vector l of second virtual light source 800B and direction vector n of the surface.

[0081] 16 , l = (c1, c2, c3) indicates the direction vector of third virtual light source 800C. i = C indicates a luminance value, which is the pixel value of each pixel of third 3DCG image 730. n = (x, y, z) indicates the direction vector of the surface. The relationship between direction vector l, luminance value i, and direction vector n of the surface of third virtual light source 800C is i = n · l. In other words, the luminance value i is expressed as the dot product of direction vector l of third virtual light source 800C and direction vector n of the surface.

[0082] Since the direction vector l and brightness value i of the first virtual light source 800A, the second virtual light source 800B, and the third virtual light source 800C are known, the direction vector n of the surface can be derived by solving the simultaneous equations of the above formula (1).

[0083] To generate a normal map image, it is necessary to visualize the direction vector n of the surface derived for each pixel as RGB pixel values. In this embodiment, the normal map image is derived using the following formulas (2) to (5).

[0084] The following formula (2) expresses the pixels of the first 3DCG image 710, the second 3DCG image 720, and the third 3DCG image 730 arranged in a row as a matrix I. In formula (2), img1 receives the luminance values ​​of pixels 1 to p of the first 3DCG image 710. img2 receives the luminance values ​​of pixels 1 to p of the second 3DCG image 720. img3 receives the luminance values ​​of pixels 1 to p of the third 3DCG image 730. The luminance values ​​input to pixels 1 to p are values ​​in the range of 0 to 255 normalized to values ​​in the range of 0.0 to 1.0.

[0085] The following formula (3) is a pseudo-inverse matrix L obtained by converting a matrix L representing the direction vectors of first virtual light source 800A, second virtual light source 800B, and third virtual light source 800C. －１ Here, a case will be described in which first direction L1 of first virtual light source 800A, second direction L2 of second virtual light source 800B, and third direction L3 of third virtual light source 800C are all inclined at +45° with respect to the horizontal direction with virtual model 600 as the reference. However, this is not limiting, and first direction L1, second direction L2, and third direction L3 may be inclined at -45° with respect to the horizontal direction, or at an angle other than 45°. Furthermore, in formula (3), matrix L is converted into a pseudo-inverse matrix L －１ However, the matrix L may be converted into an inverse matrix.

[0086] The following formula (4) is the matrix I of formula (2) and the pseudo-inverse matrix L of formula (3). －１ The matrix product L －１ I. Furthermore, the following formula (5) represents the magnitude of the vector at each pixel in the matrix product of formula (4) converted to 1.

[0087] The normal map image is obtained by normalizing the values ​​in the range from the minimum value to the maximum value of the matrix in the above formula (5) to values ​​in the range from 0 to 255, so that (R, G, B) = (x, y, z). In this way, the first image generation unit 500e can generate the first image 740, which is the normal map image shown in Fig. 13, based on the first 3DCG image 710, second 3DCG image 720, and third 3DCG image 730 shown in Figs. 10 to 12.

[0088] The captured image generating unit 500f generates at least three captured images using different light sources based on at least a first light source 400A, a second light source 400B, and a third light source 400C that illuminate the inspection object 200 and an imaging device 300 that captures an image of the illuminated inspection object 200. The at least three captured images include a first captured image, a second captured image, and a third captured image. The first captured image is an image captured by the imaging device 300 of the inspection object 200 illuminated by light irradiated from the first light source 400A in a first direction L1. The second captured image is an image captured by the imaging device 300 of the inspection object 200 illuminated by light irradiated from the second light source 400B in a second direction L2. The third captured image is an image captured by the imaging device 300 of the inspection object 200 illuminated by light irradiated from the third light source 400C in a third direction L3.

[0089] The second image generation unit 500g generates a second image based on a direction vector n of the surface of the inspection object 200 derived based on at least three captured images generated by the captured image generation unit 500f. Specifically, the second image generation unit 500g generates a second image, which is a normal map image, based on the direction vector n of the surface of the inspection object 200 derived based on the first captured image, the second captured image, and the third captured image. The normal map image, which is the second image, is also derived using the above formulas (2) to (5), similar to the normal map image, which is the first image 740. In generating the second image, the terms used in generating the first image 740 are replaced as follows: "virtual model 600" is replaced with "inspection object 200." "first 3DCG image 710" is replaced with "first captured image." "second 3DCG image 720" is replaced with "second captured image." "Third 3DCG image 730" is to be read as "third captured image." "First virtual light source 800A" is to be read as "first light source 400A." "Second virtual light source 800B" is to be read as "second light source 400B." "Third virtual light source 800C" is to be read as "third light source 400C." The method for generating the second image is the same as the method for generating first image 740, and therefore detailed description thereof will be omitted.

[0090] The teacher data generation unit 500h generates correct answer data indicating the position of the defect 610 in the first image 740 based on the defect information of the defect 610 assigned by the defect assignment unit 500c. Here, the point cloud data, which is three-dimensional shape data of the defect 610, includes coordinate information constituting the defect 610 in three-dimensional space, as well as position information indicating the position of the defect 610 and depth information indicating its depth. Information about the defect 610, including the position information indicating the position of the defect 610 and the depth information indicating its depth, is also referred to as defect information. The defect information can be derived, for example, based on a defect shape model. For example, photometric stereo photography may be performed on the defect shape model as described above, and the resulting normal map image of only the defect 610 may be used as the defect information. The teacher data generation unit 500h can generate correct answer data based on the defect information of the defect 610.

[0091] The correct answer data is associated with a pixel corresponding to the position of the defect 610 in the first image 740 as an abnormal portion, and for example, a numerical value "1" is associated with it. Also, the correct answer data is associated with a pixel corresponding to a position other than the defect 610 in the first image 740 as a normal portion, and for example, a numerical value "0" is associated with it.

[0092] Furthermore, the correct answer data is associated with each pixel by a numerical value between "0" and "1" other than "0" based on the depth distribution at the position of the defect 610 in the first image 740. For example, the correct answer data is such that, for a pixel corresponding to the position of the defect 610, the deeper the defect 610, the closer to "1" the correct answer data is associated, and the shallower the defect 610, the closer to "0" the correct answer data is associated.

[0093] In the first image 740, pixel values ​​change depending on the distance from the imaging surface of the virtual imaging device 700 to the virtual model 600. For example, the pixel values ​​of the first image 740 are expressed in 256 gradations ranging from "0" to "255." Here, for example, the closer the distance from the imaging surface of the virtual imaging device 700, the closer the pixel value is to "0," and the farther the distance from the imaging surface, the closer the pixel value is to "255." Because the pixel values ​​of the first image 740 include distance information related to the distance from the imaging surface of the virtual imaging device 700, the depth distribution of the defect 610 can be expressed by converting the pixel values.

[0094] For example, for a pixel corresponding to the position of the defect 610, the deeper the defect 610, i.e., the closer the pixel value is to "255," the closer to "1" the correct data is associated. Furthermore, for a pixel corresponding to the position of the defect 610, the shallower the defect 610, i.e., the closer the pixel value is to "0," the closer to "0" the correct data is associated. However, without being limited to this, the correct data may be set to "0" or "1," for example, depending on whether the pixel value of the pixel corresponding to the position of the defect 610 is equal to or greater than a predetermined threshold value within the range of "0" to "255." For example, if the pixel value of the pixel corresponding to the position of the defect 610 is less than the threshold value, "0" may be set as the correct data, and if the pixel value is equal to or greater than the threshold value, "1" may be set as the correct data. Furthermore, without being limited to this, for example, if the pixel value of the pixel corresponding to the position of the defect 610 is less than the threshold value, "0.5" may be set as the correct data, and if the pixel value is equal to or greater than the threshold value, "1" may be set as the correct data.

[0095] The teacher data generation unit 500h associates the first image 740 with the correct answer data, collectively defines the teacher data, and stores the teacher data in the storage device 520. In this embodiment, the teacher data generation unit 500h generates teacher data that associates defect information about the defect 610, including at least the position information and depth information of the defect 610, with the first image 740. The teacher data generation unit 500h generates a plurality of such teacher data.

[0096] Specifically, the 3DCG image generation unit 500d generates a plurality of types of first 3DCG images 710, second 3DCG images 720, and third 3DCG images 730 based on a plurality of types of virtual models 600 having a wide variety of defects 610. At this time, the 3DCG image generation unit 500d randomly changes parameters of the imaging conditions of the virtual imaging device 700 to generate a wide variety of first 3DCG images 710, second 3DCG images 720, and third 3DCG images 730. The imaging conditions include the surface optical characteristics of the inspection object 200 in the virtual model 600, the orientation of the imaging surface of the virtual imaging device 700, the orientation of the virtual light source 800, the light intensity of the virtual light source 800, etc.

[0097] The surface optical characteristics of the inspection object 200 include, for example, surface reflectance, diffuse reflectance, surface roughness, etc. However, changing the parameters of the imaging conditions is not a necessary condition, and the 3DCG image generation unit 500d may generate multiple types of first 3DCG images 710, second 3DCG images 720, and third 3DCG images 730 based on multiple types of virtual models 600 in which only the parameters of the defects 610 have been changed.

[0098] A plurality of pieces of teacher data can be generated by changing at least the parameters of the defect 610. The teacher data generating unit 500h stores the generated plurality of types of teacher data in the storage device 520.

[0099] The model learning unit 500i inputs multiple types of training data stored in the storage device 520 into the test model M, and trains the test model M so that output data that is close to the correct data contained in the training data is obtained.

[0100] The inspection model M of this embodiment includes, for example, a neural network (NN). The neural network is a convolutional neural network (CNN) trained by supervised learning. However, a neural network other than a convolutional neural network may also be used. Furthermore, a learning model other than a neural network may also be used.

[0101] The test model M is, for example, a deep learning model in image analysis. In this embodiment, the test model M is configured by, for example, a combination of a neural network structure and parameters that represent the strength of the connections between each neuron. Each connection between neurons is provided with a parameter that is a coefficient. Each parameter is configured to be adjustable. The internal state of the test model M is represented by a set of numerical values ​​that are a combination of the neural network structure, called internal variables, and the parameters between each neuron.

[0102] 17 is a diagram showing an example of the test model M. As shown in FIG. 17, training data is input to the test model M, and each neuron N １ , N ２ , N３ , N ４ , N ５ , N Ｍ－１ , N Ｍ By passing through the above, output data is output in which the output node value 0 to 1 is associated with each pixel as described above.

[0103] The inspection model M is trained based on training data including the first image 740 to which correct answer data is linked in advance. The training data is input to the inspection model M, and the values ​​of the internal variables of the inspection model M are set so as to reduce the error between the output data of the inspection model M and the correct answer data linked to the first image 740 of the training data. In this way, the inspection model M is trained based on multiple types of training data.

[0104] The model learning unit 500i may use the first image 740 to which no supervised data is linked as training data. That is, the model learning unit 500i may input the first image 740 to which no supervised data is linked to the inspection model M, and train the inspection model M so that the error between the information indicating the position of the defect 610 output from the inspection model M and the supervised data is reduced. In this case, the first image generation unit 500e functions as a training data generation unit.

[0105] Furthermore, the model learning unit 500i may use the second image as training data in addition to the first image 740. That is, the model learning unit 500i may input the first image 740 and the second image to the inspection model M, and train the inspection model M so that the error between the information indicating the position of the defect 610 output from the inspection model M and the ground truth data is reduced. In this case, the first image generation unit 500e and the second image generation unit 500g function as training data generation units.

[0106] The estimation unit 500j inputs the second image generated by the second image generation unit 500g based on the first captured image, the second captured image, and the third captured image into the inspection model M. At this time, the inspection model M classifies each pixel of the second image, and a value greater than "0" among the numerical values ​​"0" to "1" is associated as an output node with the defect 210 in the second image, i.e., the pixel corresponding to the defect 210. Furthermore, a numerical value "0" is associated as an output node with the pixel corresponding to a normal portion of the second image. In this way, in this embodiment, the second image input to the inspection model M is output in the output format of semantic segmentation.

[0107] The estimation unit 500j estimates the position of the defect 210 in the inspection object 200 included in the second image based on the output data of the inspection model M, i.e., the numerical values ​​"0" to "1" as the output nodes, and further estimates the depth of the defect 210. Specifically, the estimation unit 500j estimates the position of a pixel associated with a numerical value of an output node other than "0" as the position of the defect 210. Furthermore, the estimation unit 500j estimates the depth of the defect 210 according to the magnitude of the numerical value of the output node other than "0". In this way, the estimation unit 500j estimates the position and depth distribution of the defect 210 according to the position of a pixel associated with a numerical value of an output node other than "0" and the magnitude of the numerical value of the output node other than "0".

[0108] The estimation unit 500j displays information indicating the estimated position of the defect 210 in the inspection object 200 and information indicating the estimated depth of the defect 210 (hereinafter also referred to as estimation result data) on a display (not shown). For example, the estimation unit 500j adds a predetermined color to the position of the defect 210 and superimposes it on the second image displayed on the display. At this time, the estimation unit 500j may superimpose and display the second image displayed on the display by changing the color depending on the depth of the defect 210. For example, the estimation unit 500j may superimpose and display color information on the second image so that the deeper the defect 210, the darker the color, and the shallower the defect 210, the lighter the color.

[0109] 18 is a flowchart showing an example of an image processing method according to this embodiment. As shown in FIG. 18, the 3D graphic model generation unit 500a generates a virtual model 600 that represents the three-dimensional shape of the inspection object 200 in a virtual space S based on three-dimensional shape data of the inspection object 200 (step S100). The defect model acquisition unit 500b acquires a defect model 620 that is represented in the virtual space S and includes defect data (step S102).

[0110] The defect adding unit 500c adds a defect 610 to the virtual model 600 based on the defect model 620 (step S104). The 3DCG image generating unit 500d generates at least three 3DCG images with different virtual light sources (step S106). The first image generating unit 500e generates a first image 740, which is a normal map image, based on a direction vector n of the surface of the virtual model 600 derived based on the at least three 3DCG images (step S108).

[0111] The teacher data generation unit 500h generates teacher data in which the supervised answer data indicating the position of the defect 610 is associated with the first image 740 (step S110). The model learning unit 500i inputs only the teacher data generated by the teacher data generation unit 500h into the inspection model M, and trains the inspection model M (step S112).

[0112] The captured image generation unit 500f generates at least three captured images with different light sources (step S114). The second image generation unit 500g generates a second image, which is a normal map image, based on the direction vector of the surface of the inspection object 200 derived based on the at least three captured images (step S116).

[0113] The estimation unit 500j inputs the second image, which is a normal map image, into the trained inspection model M, and estimates the position of the defect 210 in the inspection object 200 included in the second image (step S118).

[0114] As described above, the image processing device 500 of this embodiment includes a 3D graphic model generation unit 500a, a defect model acquisition unit 500b, and a defect assignment unit 500c. The 3D graphic model generation unit 500a generates a virtual model 600 that represents the three-dimensional shape of the inspection target 200 in a virtual space. The defect model acquisition unit 500b acquires a defect model 620 that is represented in the virtual space and includes defect data. The defect assignment unit 500c assigns defects 610 to the virtual model 600 based on the defect model 620. This makes it possible to easily assign a wide variety of defects 610 to the virtual model 600 in accordance with the wide variety of defect models 620. As a result, it is possible to easily generate a wide variety of training data.

[0115] Furthermore, the defect model 620 is configured with a plane in which the shape of the defect 610 to be added to the virtual model 600 is specified by a grayscale image. A value corresponding to the depth of the defect 610 is set as the pixel value of each pixel in the grayscale image. This makes it possible to easily add a wide variety of defects 610 to the virtual model 600 by preparing a wide variety of grayscale images in advance.

[0116] Furthermore, the defect imparting unit 500c selects at least one point from a predetermined point group included in a duplicate model 630 obtained by duplicating a specific surface SA1 that is a part of the surface of the virtual model 600, and sets the position of the defect model 620 so that the surface of the defect model 620 intersects with a normal line of the duplicate model 630 that passes through the selected point. By using the duplicate model 630, it is possible to easily impart a defect 610 to a location such as a narrow portion or an edge of the virtual model 600 that is difficult to select and set manually. Therefore, it is possible to reduce the likelihood of bias in the position of the defect 610 imparted to the virtual model 600.

[0117] Furthermore, the defect adding unit 500c moves each point of the point cloud included in the surface of the defect model 620 in the normal direction of the defect model 620, and identifies the contact point where each point comes into contact with the surface of the virtual model 600. This makes it possible to easily reflect the point cloud included in the defect model 620 on the surface of the virtual model 600.

[0118] Furthermore, the contact point is identified based on the dot product of the normal vector of each point of the point cloud included in the surface of the defect model 620 and the inward vector of the virtual model 600. This makes it possible to prevent the point cloud included in the defect model 620 from being reflected in an unintended location on the surface of the virtual model 600.

[0119] The image processing device 500 of this embodiment also includes a 3DCG image generation unit 500d and a first image generation unit 500e. The 3DCG image generation unit 500d generates at least three 3DCG images with different virtual light sources 800 based on a virtual model 600 with a defect 610, at least three virtual light sources 800 that illuminate the virtual model 600 with the defect 610, and a virtual imaging device 700 that captures an image of the illuminated virtual model 600. The first image generation unit 500e generates a first image 740, which is a normal map serving as training data, based on a directional vector of the surface of the virtual model 600 derived from the at least three 3DCG images. This makes it possible to generate a wide variety of training data corresponding to a wide variety of defect models 620. As a result, a large amount of training data can be generated by simulation, enabling the inspection model M to be trained with high accuracy.

[0120] Furthermore, in this embodiment, the first image 740 and the second image input to the inspection model M are both normal map images of the same type. Therefore, the difference in appearance between the first image 740 and the second image can be reduced, and as a result, the accuracy of the position of the defect 210 in the inspection object 200 estimated by the trained inspection model M can be improved.

[0121] Furthermore, according to this embodiment, when training the inspection model M, training data in which ground truth data indicating the position of the defect 610 is associated with the first image 740 is used. This makes it possible to improve the accuracy of the position of the defect 610 estimated by the inspection model M.

[0122] Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to the above embodiments. It is clear to those skilled in the art that various modifications and alterations can be made within the scope of the claims, and it is understood that these modifications and alterations naturally fall within the technical scope of the present disclosure. For example, in the above embodiment, an example in which a depression is formed as the defect 610 has been described. However, this is not limited to this, and a protrusion may be formed as the defect 610. For example, by changing the sign of the maximum value information of the defect model 620 from "+" to "-", the defect 610 can be changed from a depression defect to a protrusion defect. Furthermore, the defect 610 added to the virtual model 600 may include both a depression and a protrusion.

[0123] The present disclosure can contribute, for example, to Goal 12 of the Sustainable Development Goals (SDGs), "Ensure sustainable consumption and production patterns."

[0124] REFERENCE SIGNS LIST 100 Image processing system 200 Inspection object 300 Imaging device 400 Multiple light sources 400A First light source 400B Second light source 400C Third light source 500 Image processing device 600 Virtual model 700 Virtual imaging device 800 Multiple virtual light sources 800A First virtual light source 800B Second virtual light source 800C Third virtual light source

Claims

1. An image processing device comprising: a virtual model generation unit that generates a virtual model that represents the three-dimensional shape of an object to be inspected in a virtual space; a defect model acquisition unit that acquires a defect model that is represented in the virtual space and includes defect data; and a defect assignment unit that assigns defects to the virtual model based on the defect model.

2. The image processing device according to claim 1, wherein the defect model is configured by a plane in which the shape of the defect to be assigned to the virtual model is specified by a grayscale image.

3. The image processing device according to claim 2, wherein the defect imparting unit selects at least one point from a predetermined group of points included in a duplicate model that duplicates a specific face that is part of the surface of the virtual model, and sets the position of the defect model so that the face of the defect model intersects with a normal line of the duplicate model that passes through the selected point.

4. The image processing device according to claim 3, wherein the defect imposing unit moves each point of a predetermined group of points included in the surface of the defect model in the normal direction of the defect model, and identifies contact points where each of the points comes into contact with the surface of the virtual model.

5. The image processing device according to claim 4, wherein the contact points are identified based on the dot product of the normal vector of each of the points in the point cloud included in the surface of the defect model and an inward vector of the virtual model.

6. An image processing device according to any one of claims 1 to 5, comprising: a 3DCG image generation unit that generates at least three 3DCG images with different virtual light sources based on the virtual model with the defect added, at least three virtual light sources that illuminate the virtual model with the defect added, and a virtual imaging device that images the illuminated virtual model; and an image generation unit that generates an image as training data based on a direction vector of the surface of the virtual model derived based on the at least three 3DCG images.

7. An image processing method comprising: a step of generating a virtual model that represents the three-dimensional shape of an object to be inspected in a virtual space; a step of acquiring a defect model that is represented in the virtual space and includes defect data; and a step of adding defects to the virtual model based on the defect model.

Images