Teacher data set generating device, flaw estimating device, and teacher data set generating method

Abstract

This teacher data set generating device comprises: an image acquiring unit that acquires a non-polarized image of an object to be learned; a flaw position information acquiring unit that acquires position information of a flaw in the object to be learned; and a teacher data set generating unit that generates a teacher data set by associating the non-polarized image with the flaw position information.

Description

Teacher data set generation device, flaw estimation device, and teacher data set generation method

[0001] The present invention relates to a teacher data set generation device, a flaw estimation device, and a teacher data set generation method.

[0002] In the case of visual inspections that inspect an object for color, scratches, foreign matter, etc., a camera suited to the inspection purpose is generally used. For example, an RGB camera that captures images in the unpolarized visible light range can be used to inspect the color of an object. There are also techniques that use polarized light to detect scratches on an object, for example. Patent Document 1 discloses a technique for inspecting whether an optical disc has scratches based on the light reflected by the optical disc. By utilizing the technique described in Patent Document 1 and using a polarization camera that captures polarized light components, it is possible to inspect an object for scratches. In this way, when inspecting color and scratches, for example, both an RGB camera and a polarization camera are used.

[0003] Patent No. 3154074

[0004] However, since two cameras are used, it is more expensive than using one camera as in normal photography.The purpose of this embodiment is to provide a teacher dataset generation device, a flaw estimation device, and a teacher dataset generation method for estimating the position of a flaw based on an unpolarized image.

[0005] One aspect of this embodiment is a teacher dataset generation device that includes an image acquisition unit that acquires an unpolarized image of a learning object, a flaw position information acquisition unit that acquires position information of a flaw in the learning object, and a teacher dataset generation unit that generates a teacher dataset by associating the unpolarized image with the flaw position information.

[0006] According to this embodiment, the position of a flaw can be estimated based on an unpolarized image.

[0007] 1 is a diagram showing the configuration of a teacher dataset generation system according to the present embodiment; FIG. 2 is a diagram showing the configuration of a teacher dataset generation device according to the present embodiment; FIG. 3 is a diagram showing an example of the configuration of a camera according to the present embodiment; FIG. 4 is a diagram showing an example of the configuration of a camera according to the present embodiment; FIG. 5 is an example of a polarized image captured by a camera; FIG. 6 is an example of an RGB image captured by a camera; FIG. 7 is a DOLP image converted from a polarized image; FIG. 8 is a diagram showing an example of teacher data generated by a teacher dataset generation unit; A flowchart showing the operation of a teacher dataset generation system according to the present embodiment; A diagram showing the configuration of an RGB image flaw estimation model generation device according to the present embodiment; A flowchart showing the operation of an RGB image flaw estimation model generation device according to the present embodiment; A diagram showing the configuration of a flaw estimation system according to the present embodiment; A diagram showing the configuration of an RGB image flaw estimation device according to the present embodiment; A flowchart showing the operation of the flaw estimation system according to the present embodiment.

[0008] [System Overview] Conventionally, there have been inspection systems that inspect the color and flaws of products, for example, on factory production lines. Polarization cameras, which can capture the polarization state of the surface of a product, are known to be able to detect the presence of flaws on the surface of a product with greater accuracy than RGB cameras, which capture images without using polarization. One example of a conventional inspection system using a polarization camera is one that inspects the color of a product using images captured by an RGB camera and inspects the flaws on the product using polarized images captured by a polarization camera, thereby enabling simultaneous inspection of both the color and flaws of the product. This conventional inspection system, configured in this manner, simultaneously inspects products for color defects and flaws, allowing for the removal of defective products from the production line. However, the above-described conventional inspection system uses two types of cameras, an RGB camera and a polarization camera, resulting in a complex system configuration and a problem in that the cost of the inspection system cannot be reduced. However, if flaws could be detected based on images captured by an RGB camera with the same accuracy as when captured by a polarization camera, the polarization camera could be omitted from the inspection system. In the following explanation, an unpolarized image will be described as an RGB image, which is a two-dimensional image containing brightness information and color information, but this also includes brightness images (black and white images), which are two-dimensional images containing only brightness information and no color information.

[0009] Therefore, in this embodiment, we propose to use machine learning or the like to learn the correspondence between a flaw in a product captured by a polarization camera and an image of the flawed area captured by an RGB camera, thereby detecting flaws based on the image captured by the RGB camera with the same accuracy as when the flaw is captured by a polarization camera. When performing machine learning, a common challenge is how to efficiently generate training data. In this embodiment, the training data is a polarization image of the learning target (e.g., a product) or position information of a flaw indicated by the polarization image. In this embodiment, the combination of the training data, which is the polarization image (or position information of the flaw indicated by the polarization image) and the RGB image captured by the RGB camera, is referred to as a training dataset.

[0010] Below, we will explain the functional configuration of each in the following order: the teacher dataset generation system 1 that generates this teacher dataset; the RGB image flaw estimation model generation device 20 that generates a flaw estimation model using RGB images based on the teacher dataset generated by the teacher dataset generation system 1; and the RGB image flaw estimation device 30 that estimates flaws based on the flaw estimation model generated by the RGB image flaw estimation model generation device 20.

[0011] <Teacher Dataset Generation> Fig. 1 is a diagram showing the configuration of a teacher dataset generation system 1 according to this embodiment. The teacher dataset generation system 1 is a system that generates a dataset in which RGB images of a learning object correspond to flaw position information. The teacher dataset generation system 1 includes a teacher dataset generation device 10, a camera 12, and a polarization image flaw estimation device 14. Here, the learning object may be, for example, a manufactured product flowing through a factory production line.

[0012] The teacher dataset generation system 1 according to this embodiment can be configured by adding a teacher dataset generation device 10 to a conventionally existing inspection system.

[0013] The teacher dataset generation device 10 generates a teacher dataset based on RGB images and flaw position information for the same learning object. Figure 2 is a diagram showing the configuration of the teacher dataset generation device 10 according to this embodiment. The teacher dataset generation device 10 includes an RGB image acquisition unit 100, a flaw position information acquisition unit 102, a teacher dataset generation unit 104, and a teacher dataset output unit 106.

[0014] The RGB image acquisition unit 100 acquires an RGB image of the learning object from the camera 12 .

[0015] The flaw position information acquisition unit 102 acquires position information of the flaw to be learned from the polarization image flaw estimation device 14. The flaw position information is, for example, the position of a pixel that indicates the flaw in the image. The flaw position information is, for example, coordinates in a predetermined reference system. The flaw position information may be one or more coordinates that identify the position of the flaw, or may be information such as a graphic that identifies the extent of the flaw.

[0016] The teacher dataset generation unit 104 generates a teacher dataset by associating the RGB image of the learning object acquired by the RGB image acquisition unit 100 with the position information of the learning object flaw acquired by the flaw position information acquisition unit 102.

[0017] The camera 12 captures RGB images and polarization images of the learning target. The camera 12 outputs the captured RGB images to the teacher dataset generation device 10. The camera 12 outputs the captured polarization images to the polarization image flaw estimation device 14. The camera 12 preferably captures the RGB images and polarization images on the same optical axis. FIG. 3 is a diagram showing an example of the configuration of the camera 12 according to this embodiment. The camera 12 includes a lens 121, a prism 122, an RGB sensor 123, a polarization sensor 124, an RGB processing unit 125, and a polarization processing unit 126. In the camera 12 shown in FIG. 3, light entering the lens 121 is split by the prism 122 and input to the RGB sensor 123 and the polarization sensor 124. The RGB sensor 123 detects red, green, and blue light of the input light, and the RGB processing unit 125 processes the light to generate an RGB image.

[0018] The polarization sensor 124 detects polarization images of the input light in multiple directions. The polarization processing unit 126 processes the polarization images in multiple directions, thereby generating a polarization image that includes all polarization direction components. The polarization sensor 124 detects polarization images in multiple directions, for example, by forming polarizers in multiple directions, such as four directions, on the photodiodes of the pixels. The polarization processing unit 126 performs signal processing on the polarization images in multiple directions for each pixel, thereby calculating all polarization direction components for each pixel and generating a polarization image that includes all polarization direction components. Therefore, the polarization sensor 124 can detect a flaw in the learning object even if the posture of the learning object or the position of the flaw are not specified and the polarization direction due to the flaw is various.

[0019] The camera 12 may capture the RGB image and the polarized image at a close angle such that the RGB image and the polarized image can be considered to have substantially the same optical axis. FIG. 4 is a diagram showing an example of the configuration of the camera 12 according to this embodiment. The camera 12 includes two lenses 121-1 and 121-2, an RGB sensor 123, a polarization sensor 124, an RGB processing unit 125, and a polarization processing unit 126. In FIG. 4, light entering the lens 121-1 is input to the RGB sensor 123. In FIG. 4, light entering the lens 121-2 is input to the polarization sensor 124. The operations of the RGB sensor 123, polarization sensor 124, RGB processing unit 125, and polarization processing unit 126 in FIG. 4 are the same as the operations of the RGB sensor 123, polarization sensor 124, RGB processing unit 125, and polarization processing unit 126 in FIG. 3.

[0020] Fig. 5A is an example of a polarized image captured by camera 12. Fig. 5B is an example of an RGB image captured by camera 12. The polarized image captured by camera 12 includes components in all polarization directions. Area 51 is a scratch, and area 52 is white paint simulating dirt. Comparing Fig. 5A and Fig. 5B, the polarized image shown in Fig. 5A includes all polarization components, making it difficult to distinguish from the RGB image shown in Fig. 5B. In the polarized image shown in Fig. 5A, it is difficult to determine whether the object is a scratch or something other than a scratch, such as a scratch in the paint.

[0021] However, if the image is processed to show only specific polarization information, the camera 12 can more easily estimate the location of the flaw from the polarized image. Figure 6 shows a degree of linear polarization (DOLP) image converted from the polarized image shown in Figure 5. Because the degree of polarization of light in the flaw is higher than in the paint, the difference between the flaw area 51 and the white paint area 52 is clear in the DOLP image.

[0022] The polarized image flaw estimation device 14 acquires a polarized image from the camera 12. The polarized image flaw estimation device 14 estimates the position of a flaw based on the polarized image. The polarized image flaw estimation device 14 outputs information on the estimated flaw position to the teacher data set generation device 10.

[0023] The polarized image flaw estimation device 14 estimates the flaw position based on the polarized image using a model (hereinafter referred to as the polarized image flaw estimation model) that receives a polarized image and outputs flaw position information. The polarized image flaw estimation model is a model generated by learning using a data set in which polarized images are associated with flaw position information captured in the polarized image. The learning method is not particularly limited, and the polarized image flaw estimation model is, for example, a neural network. The polarized image flaw estimation device 14 may be configured to estimate the flaw position from a polarized image that includes all polarization direction components, as shown in FIG. 5, or may be configured to estimate the flaw position from a DOLP image, as shown in FIG. 6.

[0024] 7A and 7B are diagrams illustrating an example of training data generated by the training data set generation unit. FIG. 7A is a diagram illustrating an example of training data (RGB image) generated by the training data set generation unit. FIG. 7B is a diagram illustrating an example of training data (flaw position information) generated by the training data set generation unit. The training data is data in which an RGB image of a training object is associated with position information of the training object's flaw, for example, data in which the RGB image shown in FIG. 7A is associated with an image showing the flaw position estimated from the polarization image shown in FIG. 7B. The image showing the flaw position estimated from the polarization image is, for example, a binary image in which the flaw position is white and non-flaw positions are black, as shown in FIG. 7B.

[0025] 8 is a flowchart showing the operation of the teacher dataset generation system 1 according to this embodiment. The camera 12 captures an RGB image and a polarized image of the learning target (step S121). The camera 12 outputs the RGB image to the teacher dataset generation device 10 (step S122). The camera 12 outputs the polarized image to the polarized image flaw estimation device 14 (step S123). The teacher dataset generation device 10 acquires the RGB image output by the camera 12 (step S101). The polarized image flaw estimation device 14 acquires the polarized image output by the camera 12 (step S141). The polarized image flaw estimation device 14 estimates flaw position information based on the polarized image (step S142). The polarized image flaw estimation device 14 outputs the flaw position information to the teacher dataset generation device 10 (step S143).

[0026] The teacher dataset generation device 10 acquires the flaw position information output by the polarization image flaw estimation device 14 (step S102). The teacher dataset generation device 10 generates a teacher dataset that associates the RGB image with the flaw position information (step S103). The teacher dataset generation device 10 outputs the teacher dataset (step S104).

[0027] As described above, the teacher dataset generation device 10 can generate a teacher dataset in which RGB images are associated with flaw position information. The teacher dataset generation system 1 can be created by adding a configuration for acquiring RGB images (an RGB sensor 123 and an RGB processing unit 125) to a conventional system that acquires polarized images and estimates flaw positions from the polarized images, and then adding the teacher dataset generation device 10. The teacher dataset generation system 1 can be installed in a factory, and a large amount of teacher data can be generated by using the camera 12 to capture images of products moving on the production line. Therefore, the teacher dataset generation system 1 can easily create large amounts of datasets.

[0028] Furthermore, by capturing RGB images and polarized images on the same optical axis using the camera 12, the flaw positions indicated by the RGB images in the teacher data and the flaw positions estimated from the polarized images appear at the same positions without any deviation. This allows the generated teacher dataset to be used to improve the accuracy of an estimation model that estimates flaw positions based on RGB images. Note that, without using the polarized image flaw estimation device 14, a person who observes the learning object or the RGB image of the learning object may input flaw position information into the teacher dataset generation device 10, allowing the flaw position information acquisition unit 102 to acquire flaw position information for the learning object. In this case, the camera 12 does not need to capture polarized images.

[0029] 9 is a diagram showing the configuration of an RGB image flaw estimation model generation device 20 according to this embodiment. The RGB image flaw estimation model generation device 20 includes a teacher data set acquisition unit 200, an estimation model generation unit 202, and an estimation model output unit 204.

[0030] The teacher dataset acquisition unit 200 acquires the teacher dataset from the teacher dataset generation device 10. The estimation model generation unit 202 generates an RGB image flaw estimation model by learning using the teacher dataset. The RGB image flaw estimation model is a model that estimates the position of a flaw by inputting an RGB image. The learning method is not particularly limited, and the RGB image flaw estimation model is, for example, a neural network.

[0031] The estimation model output unit 204 outputs the generated RGB image flaw estimation model.

[0032] 10 is a flowchart showing the operation of the RGB image flaw estimation model generation device 20 according to this embodiment. The teacher dataset acquisition unit 200 acquires a teacher dataset from the teacher dataset generation device 10 (step S201). The estimation model generation unit 202 generates an RGB image flaw estimation model by learning using the teacher dataset (step S202). The estimation model output unit 204 outputs the RGB image flaw estimation model (step S203).

[0033] <Flaw Estimation System> Fig. 11 is a diagram showing the configuration of a flaw estimation system 3 according to this embodiment. The flaw estimation system 3 includes an RGB image flaw estimation device 30 and an RGB camera 32. The RGB camera 32 photographs an object to be estimated and captures an RGB image. The RGB camera 32 outputs the captured RGB image to the RGB image flaw estimation device 30. The RGB image flaw estimation device 30 estimates and outputs the position of a flaw in the RGB image based on the RGB image. The flaw estimation system 3 is installed in, for example, a factory. The RGB camera 32 photographs products transported on a production line, and the RGB image flaw estimation device 30 detects the position of a flaw in the product and outputs the detection result.

[0034] 12 is a diagram showing the configuration of an RGB image flaw estimation device 30 according to this embodiment. The RGB image flaw estimation device 30 includes an RGB image acquisition unit 300, a flaw position estimation unit 302, a flaw position information output unit 304, and a storage unit 310. The storage unit 310 stores the RGB image flaw estimation model output by the RGB image flaw estimation model generation device 20.

[0035] The RGB image acquisition unit 300 acquires an RGB image from the RGB camera 32 .

[0036] The flaw position estimation unit 302 estimates the flaw position based on the RGB image using the RGB image flaw estimation model. The flaw position estimation unit 302 estimates the flaw position by inputting the RGB image into the RGB image flaw estimation model and outputting an estimation result of the flaw position.

[0037] The flaw position information output unit 304 outputs information on the estimated flaw position.

[0038] 13 is a flowchart showing the operation of the flaw estimation system 3 according to this embodiment. The RGB camera 32 captures an RGB image of an object to be estimated (step S321). The RGB camera 32 outputs the RGB image to the RGB image flaw estimation device 30 (step S322).

[0039] The RGB image acquisition unit 300 acquires an RGB image from the RGB camera 32 (step S301). The flaw position estimation unit 302 estimates the flaw position based on the RGB image using an RGB image flaw estimation model (step S302). The flaw position information output unit 304 outputs information on the estimated flaw position (step S303).

[0040] As described above, the flaw estimation system 3 can estimate the position of the flaw of the estimation target based on the RGB image. In addition to estimating the position of the flaw of the estimation target based on the RGB image, the flaw estimation system 3 can also inspect the color of the estimation target based on the RGB image. The flaw estimation system 3 can inspect not only the color of the estimation target but also the flaw by simply capturing the RGB image.

[0041] Other Embodiments One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like can be made within the scope that does not deviate from the gist of the present invention.

[0042] The processing of the teacher dataset generation device 10, the polarization image flaw estimation device 14, the RGB image flaw estimation model generation device 20, or the RGB image flaw estimation device 30 in the above-described embodiments may be implemented by a computer using software. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, and media that store programs for a fixed period of time, such as volatile memory within the computer systems that serve as the server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in a computer system, or may be one that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0043] 1 Teacher dataset generation system, 10 Teacher dataset generation device, 100 RGB image acquisition unit, 102 Scratch position information acquisition unit, 104 Teacher dataset generation unit, 106 Teacher dataset output unit, 12 Camera, 121 Lens, 122 Prism, 123 RGB sensor, 124 Polarization sensor, 125 RGB processing unit, 126 Polarization processing unit, 14 Polarized image scratch estimation device, 20 RGB image scratch estimation model generation device, 200 Teacher dataset acquisition unit, 202 Estimation model generation unit, 204 Estimation model output unit, 3 Scratch estimation system, 30 RGB image scratch estimation device, 300 RGB image acquisition unit, 302 Scratch position estimation unit, 304 Scratch position information output unit, 310 Memory unit, 32 RGB camera

Claims

1. A teacher dataset generation device comprising: an image acquisition unit that acquires an unpolarized image of a learning object; a flaw position information acquisition unit that acquires position information of a flaw in the learning object; and a teacher dataset generation unit that generates a teacher dataset by matching the unpolarized image with the flaw position information.

2. The teacher dataset generation device according to claim 1, wherein the scratch position information is information estimated based on the polarization image of the learning object.

3. The teacher dataset generation device according to claim 1, wherein the unpolarized image is a luminance image, which is a two-dimensional image that does not contain polarization information but contains luminance information, or an RGB image, which is a two-dimensional image that further contains color information.

4. The training dataset generating device according to claim 2, wherein the unpolarized image and the polarized image are images taken on substantially the same optical axis.

5. A flaw estimation device that estimates the position of a flaw to be estimated from an unpolarized image of the target using an image flaw estimation model trained using a teacher dataset generated by a teacher dataset generation device according to any one of claims 1 to 4.

6. A teacher dataset generating method comprising: an image acquisition step of acquiring a non-polarized image of a learning object; a scratch position information acquisition step of acquiring scratch position information of the learning object; and a teacher dataset generating step of generating a teacher dataset by matching the non-polarized image with the scratch position information.

Images