Learning device, image processing device, learning method, image processing method, learning program, and image processing program

Abstract

A learning device (100) includes a defective part shape data storage unit (107) for storing defective part shape data, generated from a new defective product image by a trained model constructed by a learning model generation unit (105), in association with the new defective product image. The image processing device (200) includes: a defective part shape addition unit (207) for extracting defective part shape data, corresponding to a defective product image approximate to input product shape data, from the defective part shape data storage unit (107) and adding the defective part shape data to the product shape data; and a defective product image generation unit (210) for generating a defective product image by optical simulation based on inspection illumination information and the product shape data including the added defective part shape data.

Description

Learning device, image processing device, learning method, image processing method, learning program, and image processing program

[0001] The present disclosure relates to a learning device, an image processing device, a learning method, an image processing method, a learning program, and an image processing program.

[0002] 2. Description of the Related Art Conventionally, appearance inspection is performed to inspect the appearance state of an object based on image data of an object image, which is an image of the object.

[0003] A known method for visual inspection involves inputting image data of a target image into a discriminant model to discriminate the state of the target object. The discriminant model is generated by machine learning the parameters of the target image based on previously prepared training data (image data of the target image). In this machine learning, the more training data there is, the more accurate the discriminant model can be generated. However, creating the training data is cumbersome and places a considerable burden on the person involved.

[0004] Patent Document 1 discloses inventions relating to a learning device, an inspection device, a learning method, and an inspection method that generate a trained model through machine learning using training data while reducing the burden of creating training data.

[0005] Patent Document 2 discloses a learning device, an image processing device, and a program that enable easy generation of learning data.

[0006] International Publication No. 2019 / 230356 Japanese Patent Application Laid-Open No. 2020-27424

[0007] However, the inventions described in Patent Documents 1 and 2 vary depending on the environment in which the device is placed, for example, due to different lighting conditions, such as backlighting or frontlighting, on the target product, and therefore the appearance of the product image changes. Therefore, capturing product images in the actual inspection environment is desirable for improving inspection accuracy. However, there is a problem in that it is cumbersome and time-consuming to adjust the lighting conditions of the product to certain conditions, start up the inspection device, and then acquire images of the product with abnormalities. Another problem is that it is difficult to conveniently obtain product samples containing abnormalities.

[0008] The present disclosure aims to provide a learning device, image processing device, learning method, image processing method, learning program, and image processing program that are capable of generating images of defective products by optical simulation, taking into account the lighting conditions in the surrounding environment in which the product is located.

[0009] The learning device of the present disclosure includes a defect data storage unit that stores pairs of defective product images and defective part shape data corresponding to the defective product images; a learning model generation unit that constructs a trained model that generates defective part shape data corresponding to a newly input defective product image by machine learning the correspondence between the defective product image and the defective part shape data; a defective product image input unit that receives the new defective product image; and a defective part shape data storage unit that stores the defective part shape data generated by the trained model from the new defective product image in association with the new defective product image.

[0010] The image processing device of the present disclosure includes a product shape data input unit to which product shape data indicating the shape of a product is input, a shape data editing unit having a defective part shape adding unit that extracts defective part shape data corresponding to a defective product image that approximates the product shape data input to the product shape data input unit from the defective part shape data storage unit described in claim 1 and adds the data to the product shape data, an inspection illumination information setting unit that sets inspection illumination information including the light distribution characteristics and radiation intensity of inspection illumination, and a defective product image generation unit that generates a defective product image by optical simulation based on the inspection illumination information and the product shape data to which the defective part shape data has been added.

[0011] The learning method disclosed herein is a learning method executed by a computer, and includes the steps of: storing a pair of defective product images and defective part shape data corresponding to the defective product images; constructing a trained model that generates defective part shape data corresponding to a newly input defective product image by machine learning the correspondence between the defective product images and the defective part shape data; inputting the new defective product image; and storing the defective part shape data generated by the trained model from the new defective product image in a storage device in association with the new defective product image.

[0012] The image processing method disclosed herein is an image processing method executed by a computer, and includes the steps of: inputting product shape data indicating the shape of a product; extracting defective part shape data corresponding to a defective product image that approximates the input product shape data from the storage device described in claim 7 and adding it to the product shape data; setting inspection illumination information including the light distribution characteristics and radiant intensity of inspection illumination; and generating a defective product image by optical simulation based on the inspection illumination information and the product shape data to which the defective part shape data has been added.

[0013] The learning program disclosed herein causes a computer to execute the following steps: storing a pair of defective product images and defective part shape data corresponding to the defective product images; constructing a trained model that generates defective part shape data corresponding to a newly input defective product image by machine learning the correspondence between the defective product image and the defective part shape data; inputting the new defective product image; and storing the defective part shape data generated by the trained model from the new defective product image in a storage device in association with the new defective product image.

[0014] The image processing program of the present disclosure causes a computer to execute the following steps: inputting product shape data indicating the shape of a product; extracting defective part shape data corresponding to a defective product image that approximates the input product shape data from the storage device described in claim 7 and adding it to the product shape data; setting inspection lighting information including the light distribution characteristics and radiation intensity of inspection lighting; and generating a defective product image by optical simulation based on the inspection lighting information and the product shape data to which the defective part shape data has been added.

[0015] According to the present disclosure, it is possible to provide a learning device, image processing device, learning method, image processing method, learning program, and image processing program that are capable of generating images of defective products by optical simulation, taking into account the lighting conditions in the surrounding environment in which the product exists.

[0016] FIG. 1 is a schematic diagram showing an example of a defective product image. FIG. 2 is a schematic diagram showing an example of a product image without defects. FIG. 3 is a schematic diagram showing an example of a defective part shape. FIG. 4 is a schematic diagram showing an example of the configuration of an information processing device according to embodiment 1. FIG. 5 is a block diagram showing an example of the hardware configuration of a learning device and an image processing device according to embodiment 1. FIG. 6 is a flowchart showing an example of a method for generating a trained model and creating defective part shape data according to embodiment 1. FIG. 7 is a flowchart showing an example of a method for generating a defective product image by optical simulation according to embodiment 1. FIG. 8 is a schematic diagram showing an example of the configuration of an information processing device according to embodiment 2. FIG. 9 is a flowchart showing an example of a method for generating a defective product image by optical simulation according to embodiment 2.

[0017] The learning device and image processing device according to the embodiments will be described below with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and modified.

[0018] First Embodiment First, a defective product image generated in the first embodiment will be described. FIG. 1 shows an example of a defective product image, and FIG. 2 shows an example of a defect-free product image, which is an image of a product without defects. FIG. 1 illustrates a part with a line scratch 12. In addition to the line scratch 12, target defects include, for example, dents such as dents, cracks, chips, stains, sink marks, and blisters, which are determined by visual inspection. When the defective product image in FIG. 1 is collected and stored as a learning dataset, it may be either a photographed image of the actual product or a computer-generated (CG) image generated by simulation. FIG. 3 illustrates an example of the shape of a defective part. The shape of the defective part is unevenness data representing the shape of the defective part (data including only concave parts, only convex parts, or both concave and convex parts). The degree of unevenness due to the scratch is expressed, for example, by a normal map that represents the normal direction of the XYZ coordinate system using pixel values ​​of an RGB image. Alternatively, the degree of unevenness due to the scratch may be represented by polygon data recording coordinate values ​​in three-dimensional space.

[0019] 4 is a schematic diagram showing an example of the configuration of an information processing device 10 according to Embodiment 1. As shown in FIG. 4, the information processing device 10 includes a learning device 100 and an image processing device 200.

[0020] The learning device 100 generates a trained model by learning the correspondence between defective product images and defective part shapes from the defective product images and the shapes of defective parts in the defective product images, and outputs defective part shape data from new defective product images input to the trained model. The image processing device 200 adds the defective part shape data output by the learning device 100 to the product shape data, and generates defective product images by optical simulation based on material information, camera information, inspection lighting information, and ambient light setting information.

[0021] The learning device 100 includes a defect data storage unit 101 , a defective product image 102 , defective part shape data 103 , a learning model 104 , a learning model generation unit 105 , a defective product image input unit 106 , and a defect part shape data storage unit 107 .

[0022] The image processing device 200 includes a setting value optimization unit 201, a material information setting unit 202, a camera information setting unit 203, an inspection lighting information setting unit 204, an ambient light setting unit 205, a product shape data editing unit 206, a defective part shape adding unit 207, a product shape data input unit 208, a defective product image generation unit 210, and an optical simulation unit 211.

[0023] The defect data storage unit 101 stores, as learning data, a plurality of pairs of defective product images 102 and defective part shape data 103 corresponding to the defective product images 102. As shown in FIG. 1, the defective product images are images of a product containing a defect such as a line scratch 12. The images stored as the defective product images 102 may be either actual photographs of the actual product or CG images obtained by simulation or the like, as long as they contain a defect. As shown in FIG. 3, the defect part shape data is shape data representing the shape of the defect. While FIG. 3 shows an example of defect part shape data in a normal map, shape data represented by polygon information or mesh information in a three-dimensional space may also be used. However, when the defect data storage unit 101 collects the defect part shape data 103 using a normal map, the data formats must be consistent. Furthermore, the defective product images 102 and the defect part shape data 103 must be aligned in a one-to-one correspondence.

[0024] The learning model 104 includes a learning model generation unit 105. The learning model 104 constructs a surrogate model so as to output defective part shape data of a defective product image input from a defective product image input unit 106. The learning model 104 constructs a trained model in the learning model generation unit 105.

[0025] The learning model generation unit 105 constructs a learned model that generates defect part shape data 103 corresponding to a newly input defective product image 102 by machine learning the correspondence between the defective product image 102 and the defect part shape data 103 stored in the defect data storage unit 101. Machine learning techniques include neural networks, deep learning, AutoML, support vector machines, decision trees, random forests, and k-nearest neighbor methods.

[0026] A new defective product image to be input to the trained model is input to the defective product image input unit 106. The image to be input to the defective product image input unit 106 may be a plurality of defective product images as test data, or a single defective product image.

[0027] The defect part shape data storage unit 107 stores defect part shape data generated from a new defective part product image input by the defective product image input unit 106 based on a learned model learned by machine learning in the learning model generation unit 105, in association with the new defective part product image. The defect part shape data storage unit 107 stores a plurality of pairs of new defective part product images and defect part shape data generated from the new defective part product images. The defect part shape data has the same data format as the shape data stored in the defect part shape data 103 in the defect data storage unit 101. If the defect part shape data 103 is mesh data, the shape data output to the defect part shape data storage unit 107 is also mesh data. If the defect part shape data 103 is a normal map, the defect part shape data storage unit 107 is also a normal map. Alternatively, the defect part shape data 103 and the shape data output to the defect part shape data storage unit 107 may be polygon data.

[0028] The image processing device 200 adds the defective part shape data stored in the defective part shape data storage unit 107 to the product shape data input by the product shape data input unit 208, and performs optical simulation in the optical simulation unit 211 in the defective product image generation unit 210 based on the outputs of the setting value optimization unit 201, the material information setting unit 202, the camera information setting unit 203, the inspection lighting information setting unit 204, and the ambient light setting unit 205, to generate a defective product image.

[0029] The product shape data input unit 208 inputs shape data of products and parts. The data may be CAD data, or may be three-dimensional volume data such as polygon data or mesh data.

[0030] The product shape data editing unit 206 adds, in the defect shape adding unit 207, the defect part shape data stored in the defect part shape data storage unit 107 to the product shape data input by the product shape data input unit 208. For example, the defect part shape adding unit 207 extracts a defect part shape corresponding to a defective product image that is similar to the input product shape data from the defect part shape data storage unit 107 of the learning device 100 and adds it to the input product shape data. The defect part shape adding unit 207's function for identifying defective product images that are similar to the input product shape data is constructed by machine learning using product shape data and defective product images that are similar to each other and prepared as learning data. The position and size of the defect part shape may be set randomly, or may be added at a position or size specified by the user. Furthermore, one scratch may be added, or multiple scratches and multiple types of scratches may be added.

[0031] The processing in the product shape data editing unit 206 is not limited to adding the shape of the defective part, but may also add shape information such as a hairline-finished surface texture or a pattern in the case of metal in order to make the input product shape data look more realistic. In this case, the surface texture may be defined by normal data, rather than being limited to adding three-dimensional volume data.

[0032] The material information setting unit 202 sets material information for the product shape data, including the defect shape data, edited by the product shape data editing unit 206. Since the materials used for products are generally aluminum, iron, titanium, or synthetic resin, multiple pieces of material information are stored in advance, and the user selects a material. A different material may be set, not limited to the specific material stored. In this case, the material information to be set may be a texture image including object color information and specular reflection intensity information for the material, or BRDF data based on a bidirectional reflectance distribution function.

[0033] The camera information setting unit 203 sets the angle of view when imaging the product shape data edited by the product shape data editing unit 206. Parameters of the camera information setting unit for setting the angle of view include the camera position, rotation angle, sensor size, focal length, and aperture. Other camera parameters may include information on ISO (International Organization for Standardization) sensitivity, shutter speed, and camera sensitivity of a color sensor or monochrome sensor. One or more cameras may be installed.

[0034] The inspection illumination information setting unit 204 sets parameter information related to the inspection illumination device that illuminates the product shape data. The parameter information related to the inspection illumination device is set by selecting an appropriate one from multiple types of inspection illumination devices stored in advance, and setting the illumination position, rotation angle, light distribution characteristics of the illumination, radiation intensity (the intensity of the illumination), etc. The inspection illumination devices stored in advance include bar illumination, ring illumination, coaxial epi-illumination, and pattern illumination, which are capable of uniformly illuminating a long area, as actually used in appearance inspection.

[0035] The setting value optimization unit 201 optimizes the values ​​to be set by the camera information setting unit 203 and the inspection lighting information setting unit 204, based on the defect part shape data referenced from the defect part shape data storage unit 107 and the product shape data input by the product shape data input unit 208. For example, it sets parameters such as the position and angle of the camera and lighting, and the type of lighting, that make it easy to recognize the defect part shape.

[0036] The ambient light setting unit 205 inputs a panoramic image as ambient light information to reproduce the lighting environment around the product shape and the reflection of the surrounding environment. The ambient light information is a two-dimensional image of the entire surrounding scenery, and may be in a format called a cubic skybox, a spherical shape, or a rectangular shape. Furthermore, an HDR (High Dynamic Range) format is preferable, but it does not have to be an HDR format. The ambient light setting unit 205 may select from multiple pre-registered ambient light information or may set a specified image. The ambient light setting unit 205 may also set a different lighting source and create a new ambient environment based on that lighting information. For example, to simulate external light that changes depending on the time of day, the angle of incidence of parallel light simulating sunlight may be changed, and parallel light with a different color may be added to simulate a sunset or sunrise.

[0037] The defective product image generation unit 210 generates a display image using an optical simulation unit 211 based on the material information set by the material information setting unit 202, the camera information set by the camera information setting unit 203, the inspection illumination information set by the inspection illumination information setting unit 204, the ambient light information set by the ambient light setting unit 205, and the product shape data to which the defective part shape data has been added by the product shape data editing unit 206, all of which are specified in the virtual optical environment. The simulation by the optical simulation unit 211 includes ray tracing or path tracing. Alternatively, the optical simulation unit 211 may generate a defective product image by optical simulation based on the inspection illumination information and the product shape data to which the defective part shape data has been added.

[0038] The output unit 212 displays the display image generated by the defective product image generation unit 210. The output unit 212 is, for example, a monitor such as a liquid crystal display, an organic electroluminescence (EL) display, a micro light-emitting diode (LED) display, or a cathode ray tube (CRT). The output unit 212 may also be an aerial display, a head-mounted display (HMD), a virtual reality (VR) device, an augmented reality (AR) device, a tablet terminal, a smartphone, or a television.

[0039] 5 is a block diagram showing an example of the hardware configuration of the learning device 100 and image processing device 200 according to embodiment 1. As shown in FIG. 5, each of the learning device 100 and image processing device 200 is configured by a computer having CPUs (Central Processing Units) 21 and 31, which are arithmetic elements (processors), main memories 22 and 32, input / output interfaces (I / O interfaces) 23 and 33, and storage units 24 and 34, each connected to a system bus 25 and 35. The learning device 100 and image processing device 200 may be configured by multiple computers connected via a network, or may be configured by a processing circuit as dedicated hardware, such as a single circuit or a composite circuit.

[0040] The CPUs 21 and 31 are integrated circuits (ICs) that perform arithmetic processing. In addition to the CPUs 21 and 31, arithmetic elements such as a digital signal processor (DSP), a graphics processing unit (GPU), a network processor, or a field programmable gate array (FPGA) may also be used.

[0041] The CPU 21 of the learning device 100 executes the learning program according to the first embodiment to establish a learning method that functions as a learning model generation function for learning the correspondence between defective product images and the defect part shape data. As a result, the CPU executes the learning program to function as the learning model generation unit 105. The learning program is provided, for example, on a recording medium on which it is recorded.

[0042] By executing the image processing program according to the first embodiment, the CPU 31 of the image processing device 200 implements an image processing method that functions as a defect shape adding function that adds defect shape data to product shape data, a material information setting function that sets material information for the product shape data, a camera information setting function that sets the angle of view, etc., when imaging the product shape data, an inspection illumination information setting function that sets parameter information related to lighting for the product shape data, a setting value optimization function that sets parameters such as the position and angle of the camera and lighting that make it easy to recognize the defect shape, an ambient light setting function that inputs a panoramic image as ambient light information, and an optical simulation function that generates a display image by ray tracing or path tracing. As a result, by executing the image processing program, the CPU functions as a defect shape adding unit 207, a material information setting unit 202, a camera information setting unit 203, an inspection illumination information setting unit 204, a setting value optimization unit 201, an ambient light setting unit 205, and an optical simulation unit. The image processing program may be provided, for example, on a recording medium on which it is recorded.

[0043] The main memories 22 and 32 are configured with a volatile storage device such as a RAM (Random Access Memory) or a non-volatile storage device such as a ROM (Read Only Memory). The storage units 24 and 34 are configured with a non-volatile storage device such as a HDD (Hard Disk Drive) or a flash memory. For example, the storage unit 24 of the learning device 100 is provided with a defect data storage unit 101, a defect part shape data storage unit 107, etc.

[0044] The I / O interface 23 of the learning device 100 is a port to which the defective product image input unit 106 and the I / O interface 33 of the image processing device 200 are connected.

[0045] The I / O interface 33 of the image processing device 200 is a port to which the product shape data input unit 208, the output unit 212, the I / O interface 23 of the learning device 100, etc. are connected.

[0046] FIG. 6 is a flowchart showing an example of a method for generating a trained model and creating defect part shape data according to the first embodiment.

[0047] In step S101, defective product images serving as learning data are stored in the defective data storage unit 101.

[0048] In step S102, defective part shape data is stored in the defect data storage unit 101 as learning data corresponding to the defective product image stored in step S101.

[0049] In step S103, the learning model generation unit 105 generates a learned model by learning using the defective product image stored in step S101 and the defective part shape data stored in step S102.

[0050] In step S104, a defective product image is input to the trained model generated in step S103 from the defective product image input unit 106. The defective product image input in step S104 may be input as test data.

[0051] In step S105, the learning model 104 generates defective part shape data from the defective product image input in step S104.

[0052] In step S106, the defect part shape data generated in step S105 is stored in the defect part shape data storage unit 107, and the process ends.

[0053] FIG. 7 is a flowchart showing an example of a method for generating a defective product image by optical simulation.

[0054] In step S201, shape data such as CAD data of a product or part is input to the product shape data input unit 208.

[0055] In step S202, the defective part shape adding unit 207 in the product shape data editing unit 206 acquires the defective part shape data stored in the defective part shape data storage unit 107 of the learning device 100, and adds the acquired defective part shape data to the product shape data input in step S201.

[0056] In step S203, the material information setting unit 202 sets material information including material information, object color, reflection characteristics, and normal information of the material.

[0057] In step S204, the camera information setting unit 203 sets information about the camera to be installed in the virtual space. The camera information to be set includes information about the camera position, rotation angle, sensor size, focal length, aperture, ISO sensitivity, shutter speed, and camera sensitivity of the color sensor or monochrome sensor. The setting values ​​of each piece of information are optimized by the setting value optimization unit 201. The setting value optimization unit 201 optimizes the values ​​to be set by the camera information setting unit 203 based on the defective part shape data and product shape data referenced via the product shape data editing unit 206 and the camera information setting unit 203.

[0058] In step S205, the inspection illumination information setting unit 204 sets information about the inspection illumination. Examples of the inspection illumination include information about the arrangement of the illumination, such as ring illumination, bar illumination, coaxial illumination, line illumination, or dome illumination, information about the illumination shape, the illumination wavelength, the illumination light intensity, the illumination installation position, and the illumination installation angle. The setting value of each piece of information is optimized by the setting value optimization unit 201. The setting value optimization unit 201 optimizes the values ​​set by the inspection illumination information setting unit 204 based on the defect part shape data and product shape data referenced via the product shape data editing unit 206 and the inspection illumination information setting unit 204.

[0059] In step S206, ambient light information is set in the ambient light setting unit 205. The ambient light information is input as a spherical image for the purpose of reproducing the environment around the target product shape and calculating reflections on the product shape. The spherical image may be input as a panoramic image or as a development also called a cube map or a skybox.

[0060] The procedure from step S203 to step S206 is not limited to the order shown in FIG. 6, and may be performed in any order.

[0061] In step S207, based on the information input in the series of steps S201 to S206, the optical simulation unit 211 in the defective product image generation unit 210 generates a defective product image by optical simulation, and then the process ends. The optical simulation is performed by path tracing or ray tracing, for example.

[0062] As described above, according to the first embodiment, by learning the correlation between defective product images and defective part shape data and generating a trained model, it is possible to obtain defective part shape data from defective product images even if there are no actual defective products.

[0063] Furthermore, according to the first embodiment, by performing an optical simulation based on the shape of the defective part, it is possible to perform a simulation of taking an image of a defective product taking into account the inspection environment, lighting parameters, or camera parameters. Furthermore, by generating an image of a defective product taking into account the actual environment, it is possible to improve the detection accuracy and inspection accuracy during inspection.

[0064] Furthermore, according to embodiment 1, it is possible to perform a simulation of taking images of defective products taking into account the surrounding environment, making it possible to generate images of defective products that take into account the installation environment of the inspection device or the effects of external light such as sunsets, thereby improving detection accuracy and inspection accuracy during inspection.

[0065] Second Embodiment Next, a second embodiment will be described. Fig. 8 is a schematic diagram showing an example of the configuration of an information processing device 14 according to the second embodiment. The information processing device 14 according to the second embodiment differs from the first embodiment in that an inspection device shape data input unit 209 is added to the image processing device 300. However, the other configurations are the same as those of the first embodiment, and therefore the same components as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment, and detailed description thereof will be omitted.

[0066] In the second embodiment, shape data of the inspection device that affects reflections on the product shape is input from the inspection device shape data input unit 209, and optical simulation is performed on the appearance of the product's inspection device as well as the product or part, thereby enabling detailed simulation of detailed reflections on the product or part and the influence of shadows caused by the inspection device shape.

[0067] FIG. 9 is a flowchart showing an example of a method for generating a defective product image by optical simulation.

[0068] In step S301, shape data such as CAD data of a product or part is input to the product shape data input unit 208.

[0069] In step S302, the defect part shape adding unit 207 in the product shape data editing unit 206 acquires the defect part shape data stored in the defect part shape data storage unit 107 of the learning device 100, and adds the acquired defect part shape data to the product shape data input in step S201. Also in step S302, shape data of the inspection device that affects reflection on the product shape is input from the inspection device shape data input unit 209.

[0070] In step S303, the material information setting unit 202 sets material information including material information, object color, reflection characteristics, and normal information of the material.

[0071] In step S304, the camera information setting unit 203 sets optimal values ​​for the camera information to be installed in the virtual space. The camera information to be set includes information on the camera position, rotation angle, sensor size, focal length, aperture, ISO sensitivity, shutter speed, and camera sensitivity of the color sensor or monochrome sensor. The setting values ​​for each piece of information are optimized by the setting value optimization unit 201. The setting value optimization unit 201 optimizes the values ​​to be set by the camera information setting unit 203 based on the defect part shape data, product shape data, and inspection device shape data referenced via the product shape data editing unit 206 and the camera information setting unit 203.

[0072] In step S305, the inspection illumination information setting unit 204 sets optimal values ​​for the information on the inspection illumination. Examples of the inspection illumination include information on the arrangement of the illumination, such as ring illumination, bar illumination, coaxial illumination, line illumination, or dome illumination, information on the illumination shape, illumination wavelength, illumination light intensity, illumination installation position, and illumination installation angle. The setting value of each piece of information is optimized by the setting value optimization unit 201. The setting value optimization unit 201 optimizes the values ​​set by the inspection illumination information setting unit 204 based on the defect shape data, product shape data, and inspection device shape data referenced via the product shape data editing unit 206 and the inspection illumination information setting unit 204.

[0073] In step S306, information about ambient light is set in the ambient light setting unit 205. The information about ambient light is input as a spherical image for the purpose of reproducing the environment around the target product shape and calculating reflections on the product shape. The spherical image may be input as a panoramic image or as a development also called a cube map or a skybox.

[0074] The procedure from step S303 to step S306 is not limited to the order shown in FIG. 9, and may be performed in any order.

[0075] In step S307, based on the information input in the series of steps S201 to S206, the optical simulation unit 211 in the defective product image generation unit 210 generates a defective product image by optical simulation, and then the process ends. The optical simulation is performed by path tracing or ray tracing, for example.

[0076] As described above, according to the second embodiment, the setting value optimization unit 201 optimizes the values ​​set by the camera information setting unit 203 and the inspection illumination information setting unit 204 based on the defect part shape data, product shape data, and inspection device shape data. As a result, it is possible to simulate the capture of an image of a defective product taking into account shadows, reflections, or illumination caused by the structure of the inspection device. Furthermore, by generating an image of a defective product taking into account the actual environment, it is possible to improve the detection accuracy and inspection accuracy during inspection.

[0077] Furthermore, according to the second embodiment, the camera information and the inspection illumination information are each optimized based on the defective part shape data, the product shape data, and the inspection device shape data, thereby enabling a quick photography simulation to be performed.

[0078] 10 Information processing device, 12 Line scratch, 14 Information processing device, 21 CPU, 22 Main memory, 23 I / O interface, 24 Memory unit, 31 CPU, 32 Main memory, 33 I / O interface, 34 Memory unit, 100 Learning device, 101 Defect data memory unit, 102 Defective product image, 103 Defective part shape data, 104 Learning model, 105 Learning model generation unit, 106 Defective product image input unit, 107 Defective part shape data memory unit, 200 Image processing device, 201 Setting value optimization unit, 202 Material information setting unit, 203 Camera information setting unit, 204 Inspection lighting information setting unit, 205 Ambient light setting unit, 206 Product shape data editing unit, 207 Defective part shape addition unit, 208 Product shape data input unit, 209 Inspection device shape data input unit, 210 Defective product image generation unit, 211 Optical simulation unit, 300 image processing device.

Claims

1. A learning device comprising: a defect data storage unit that stores pairs of defective product images and defective part shape data corresponding to the defective product images; a learning model generation unit that constructs a trained model that generates defective part shape data corresponding to a newly input defective product image by machine learning the correspondence between the defective product images and the defective part shape data; a defective product image input unit that receives the new defective product image; and a defective part shape data storage unit that stores the defective part shape data generated by the trained model from the new defective product image in association with the new defective product image.

2. An image processing device comprising: a product shape data input unit to which product shape data indicating the shape of a product is input; a shape data editing unit having a defective part shape adding unit that extracts defective part shape data corresponding to a defective product image that approximates the product shape data input to the product shape data input unit from the defective part shape data storage unit described in claim 1 and adds it to the product shape data; an inspection lighting information setting unit that sets inspection lighting information including the light distribution characteristics and radiant intensity of inspection lighting; and a defective product image generation unit that generates a defective product image by optical simulation based on the inspection lighting information and the product shape data to which the defective part shape data has been added.

3. An image processing device as described in claim 2, further comprising: a material information setting unit that sets material information of the product shape data to which the defective part shape data has been added; a camera information setting unit that sets parameters including the position and angle of view of a camera that will image the product shape data to which the defective part shape data has been added; and an ambient light setting unit that inputs a panoramic image as ambient light information of the product, wherein the defective product image generation unit generates a defective product image by optical simulation based on the inspection illumination information, the product shape data to which the defective part shape data has been added, the material information, the parameters, and the ambient light information.

4. An image processing device as described in claim 3, further comprising a setting value optimization unit that optimizes the parameters set by the camera information setting unit and the values ​​of the inspection lighting information set by the inspection lighting information setting unit based on the defective part shape data and the product shape data.

5. An image processing device as described in claim 4, further comprising an inspection device shape data input unit for inputting shape data of an inspection device, wherein the defective product image generation unit generates a defective product image by optical simulation based on the inspection illumination information, the product shape data to which the defective part shape data has been added, the material information, the parameters, the ambient light information, and the shape data of the inspection device.

6. An image processing device as described in claim 5, wherein the setting value optimization unit optimizes the parameters set by the camera information setting unit and the values ​​of the inspection lighting information set by the inspection lighting information setting unit based on the defective part shape data, the product shape data, and the shape data of the inspection device.

7. A learning method executed by a computer, comprising: a step of storing a pair of defective product images and defective part shape data corresponding to the defective product images; a step of constructing a trained model that generates defective part shape data corresponding to a newly input defective product image by machine learning the correspondence between the defective product images and the defective part shape data; a step of inputting the new defective product image; and a step of storing the defective part shape data generated by the trained model from the new defective product image in a storage device in association with the new defective product image.

8. An image processing method executed by a computer, comprising the steps of: inputting product shape data indicating the shape of a product; extracting defective part shape data corresponding to a defective product image that approximates the input product shape data from the storage device described in claim 7 and adding the data to the product shape data; setting inspection lighting information including the light distribution characteristics and radiant intensity of inspection lighting; and generating a defective product image by optical simulation based on the inspection lighting information and the product shape data to which the defective part shape data has been added.

9. A learning program that causes a computer to execute the following steps: storing a pair of defective product images and defective part shape data corresponding to the defective product images; constructing a trained model that generates defective part shape data corresponding to a newly input defective product image by machine learning the correspondence between the defective product images and the defective part shape data; inputting the new defective product image; and storing the defective part shape data generated by the trained model from the new defective product image in a storage device in association with the new defective product image.

10. An image processing program that causes a computer to execute the following steps: inputting product shape data indicating the shape of a product; extracting from the storage device described in claim 9 defective part shape data corresponding to a defective product image that approximates the input product shape data and adding it to the product shape data; setting inspection lighting information including the light distribution characteristics and radiant intensity of inspection lighting; and generating a defective product image by optical simulation based on the inspection lighting information and the product shape data to which the defective part shape data has been added.

Images