Appearance determination device and appearance determination method

Abstract

To enable quality determination of appearance of a determination target using color photometric stereo.SOLUTION: An appearance determination device (10) disclosed herein comprises one or more processors (11) configured to perform a determination step (S13) for determining the quality of a determination target based on color images showing color optical images of the determination target as generated by photometric stereo and shape images showing the shape of the determination target as generated by the photometric stereo.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an appearance determination device and an appearance determination method.

Background Art

[0002] Illuminance difference stereo is a technique for analyzing the appearance of an article using a plurality of light sources with different directions (see, for example, Non-Patent Documents 1 and 2).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, illuminance difference stereo is usually used for monochrome grayscale images and is not premised on being applied to color images. That is, illuminance difference stereo generally abstracts color information (wavelength dependence of light reflection from an article), which is important for determining the quality of the appearance of an article.

[0005] The present invention aims to realize an appearance determination device and an appearance determination method that determine the quality of the appearance of an object to be judged using colorized illuminance difference stereo. [Means for solving the problem]

[0006] To solve the above problems, an appearance determination device according to one aspect of the present invention comprises one or more processors, the processors performing a determination step of determining whether an object to be determined is good or bad based on at least one of a color image representing the optical image of the object to be determined in color, generated by illuminance difference stereo, and a shape image representing the shape of the object to be determined, also generated by illuminance difference stereo. [Effects of the Invention]

[0007] According to one aspect of the present invention, an appearance determination device and an appearance determination method can be realized that determine the quality of the appearance of an object to be judged using colorized illuminance difference stereo. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows the configuration of an appearance determination system according to one embodiment of the present invention. [Figure 2] This flowchart illustrates an example of an appearance determination method according to an embodiment of the present invention. [Figure 3] This flowchart illustrates an example of a method for generating color reflection images (color images) and normal images (shape images). [Figure 4] This flowchart illustrates other examples of methods for generating color reflection images (color images) and normal images (shape images). [Figure 5] This flowchart shows an example of a method for estimating the light source vector. [Figure 6] This diagram illustrates an example of a method for excluding abnormal light source vectors. [Figure 7] This is a diagram that schematically represents the model. [Figure 8] This diagram shows an example of the results of the visual inspection. [Figure 9]This diagram illustrates another example of the visual inspection result. [Modes for carrying out the invention]

[0009] The following describes in detail one embodiment of the present invention. Figure 1 is a diagram showing the configuration of the appearance determination system 1 according to one embodiment of the present invention. The appearance determination system 1 is a system for determining whether an object OB to be judged by appearance is a good product (whether it has defects or not), and includes an imaging unit MP and an appearance determination device 10.

[0010] The imaging unit MP is an imaging device for illuminance difference stereo imaging and includes a darkroom BX, multiple light sources LS(1) to LS(n), and a camera CA. Hereafter, light sources LS(1) to LS(n) will be collectively referred to as light source LS(i). For illuminance difference stereo imaging, the imaging unit MP uses light sources LS(i) with different orientations to image the object OB to be judged and generates multiple images IM of the object OB to be judged.

[0011] Darkroom BX is a space set up for placing and photographing the object to be judged OB. Darkroom BX has walls to block out external light, and the object to be judged OB, light source LS(i), and camera CA are installed inside Darkroom BX.

[0012] Light source LS(i) is the illumination used when camera CA photographs object OB. The light sources LS(i) are positioned and oriented so as to illuminate object OB from different directions. To colorize the illuminance difference stereo, light sources LS(i) emit light containing light from different first, second, and third wavelength ranges.

[0013] Here, as examples of the first, second, and third wavelength ranges, we can cite the wavelength ranges of R (red), G (green), and B (blue). For clarity, in the following, the first, second, and third wavelength ranges will be represented as R, G, and B, respectively.

[0014] The camera CA is an imaging device that captures the object OB to be determined. Here, for the sake of clarity, only one camera CA is shown, but a plurality of cameras CA may be installed at positions and orientations for capturing the object OB to be determined from different directions.

[0015] The appearance determination device 10 is, for example, a personal computer, and determines whether the object OB to be determined is a non-defective product (i.e., has no defects) based on a plurality of images IM of the object OB to be determined captured by the imaging unit MP.

[0016] The appearance determination device 10 includes a processor 11, a primary memory 12, a secondary memory 13, an input / output IF (interface) 14, a communication IF 15, and a bus 16. The processor 11, the primary memory 12, the secondary memory 13, the input / output IF 14, and the communication IF 15 are interconnected via the bus 16.

[0017] The secondary memory 13 stores (non-volatile memory) an appearance determination program P1 and a model M1. The processor 11 deploys the appearance determination program P1 and the model M1 stored in the secondary memory 13 onto the primary memory 12. Then, the processor 11 executes the appearance determination method according to the instructions included in the appearance determination program P1 deployed on the primary memory 12. The model M1 deployed on the primary memory 12 is used when the processor 11 executes the appearance determination method.

[0018] Examples of devices that can be used as the processor 11 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a microcontroller, or a combination thereof. The processor 11 may also be referred to as an "arithmetic unit".

[0019] Examples of devices that can be used as primary memory 12 include semiconductor RAM (Random Access Memory). Primary memory 12 is sometimes called "main memory". Examples of devices that can be used as secondary memory 13 include flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), ODD (Optical Disk Drive), FDD (Floppy® Disk Drive), or combinations thereof. Secondary memory 13 is sometimes called "auxiliary memory". Secondary memory 13 may be built into the appearance determination device 10, or it may be built into another computer (for example, a computer that makes up a cloud server) connected to the appearance determination device 10 via input / output IF 14 or communication IF 15. In this embodiment, storage in the appearance determination device 10 is realized by two memories (primary memory 12 and secondary memory 13), but it is not limited to this. That is, storage in the appearance determination device 10 may be realized by one memory. In this case, for example, one memory area of ​​that memory can be used as primary memory 12, and another memory area of ​​that memory can be used as secondary memory 13.

[0020] Input and / or output devices are connected to I / O IF14. Examples of I / O IF14 interfaces include USB (Universal Serial Bus), ATA (Advanced Technology Attachment), SCSI (Small Computer System Interface), and PCI (Peripheral Component Interconnect).

[0021] An input device connected to the input / output IF 14 is a camera CA. Data acquired from the camera CA in the appearance determination method is input to the appearance determination device 10 and stored in the primary memory 12. Other input devices connected to the input / output IF 14 include a keyboard, mouse, touchpad, microphone, or a combination thereof. Output devices connected to the input / output IF 14 include a display, projector, printer, speaker, headphones, or a combination thereof. Information provided to the user in the appearance determination method is output from the appearance determination device 10 via these output devices. The appearance determination device 10 may have a keyboard that functions as an input device and a display that functions as an output device, similar to a laptop computer. Alternatively, the appearance determination device 10 may have a touch panel that functions as both an input and output device, similar to a tablet computer.

[0022] Other computers are connected to the communication IF15 via a network, either by wired or wireless connection. Examples of communication IF15 interfaces include Ethernet® and Wi-Fi®. Available networks include PAN (Personal Area Network), LAN (Local Area Network), CAN (Campus Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), GAN (Global Area Network), or internetworks including these networks. The internetworks may be an intranet, an extranet, or the Internet. Data acquired by the appearance determination device 10 from other computers in the appearance determination method, and data provided by the appearance determination device 10 to other computers in the appearance determination method, are transmitted and received via these networks. The camera CA and the appearance determination device 10 may be connected via the input / output IF14, or via the communication IF15.

[0023] In this embodiment, the appearance determination method is performed using a single processor (processor 11), but the present invention is not limited thereto. That is, the appearance determination method may be performed using multiple processors. In this case, the multiple processors that work together to perform the appearance determination method may be provided in a single computer and communicate with each other via a bus, or they may be distributed across multiple computers and communicate with each other via a network. As an example, it is conceivable that a processor built into a computer constituting a cloud server and a processor built into a computer owned by a user of that cloud server may work together to perform the appearance determination method.

[0024] Furthermore, in this embodiment, Model M1 is stored in memory (secondary memory 13) built into the same computer as the processor (processor 11) that executes the appearance determination method, but the present invention is not limited to this. That is, Model M1 may be stored in memory built into a different computer from the one that executes the appearance determination method. In this case, the computer that houses the memory storing Model M1 can communicate with the computer that houses the processor that executes the appearance determination method via a network. As an example, it is conceivable that Model M1 be stored in memory built into a computer that constitutes a cloud server, and that a processor built into a computer owned by a user of that cloud server executes the appearance determination method.

[0025] Furthermore, although the Model M1 is stored in a single memory (secondary memory 13) in this embodiment, the present invention is not limited thereto. That is, the Model M1 may be distributed and stored in multiple memories. In this case, the multiple memories for storing the Model M1 may be provided in a single computer (which may or may not be a computer with a built-in processor for executing the appearance determination method), or they may be distributed and provided in multiple computers (which may or may not include a computer with a built-in processor for executing the appearance determination method). As an example, it is conceivable to distribute and store the Model M1 in the memory built into each of the multiple computers that constitute a cloud server.

[0026] (Overview of the visual inspection method) The appearance determination device 10 comprises one or more processors 11 and executes an appearance determination method. Figure 2 is a flowchart showing an example of an appearance determination method executed by the processor 11. The appearance determination method includes a preparation step (step S11), a generation step (step S12), and a determination step (step S13).

[0027] The preparation process is performed as needed, for example, to form or strengthen model M1. Details of this will be described later. The generation process is the process of generating a color image and a shape image of the object OB based on multiple image IMs of the object OB captured by the imaging unit MP. The multiple image IMs are images of the object OB illuminated by multiple light sources LS(i) with different orientations. The determination process is the process of determining whether the object OB is good or bad based on at least one of the color image and the shape image.

[0028] Object OB is an article subject to visual inspection. Examples of objects OB include cast iron or molds used in casting. However, objects OB are not limited to castings or molds; they may be other articles.

[0029] The color image is a color optical image of the object OB being judged, generated by illuminance difference stereo. As an example, the color image is a color reflection image representing the reflectance distribution on the surface of the object OB in three distinct wavelength ranges of light: the first wavelength range (R), the second wavelength range (G), and the third wavelength range (B).

[0030] In illuminance difference stereo, an albedo image representing the reflectance distribution on the object OB is typically generated based on a monochrome grayscale image. That is, the albedo image is a monochrome grayscale image. In contrast, in this embodiment, the illuminance difference stereo is colorized to generate a color reflection image that represents the reflectance distribution on the object OB in color. Details of the generation of the color reflection image will be described later.

[0031] The shape image is an image representing the shape of the object OB to be judged, generated by illuminance difference stereo. As an example, the shape image is a normal image representing the distribution of normals on the surface of the object to be judged. The normal image is generated along with the color reflection image based on multiple images IM of the object OB captured by the imaging unit MP.

[0032] A good product is an object (for example, a casting) that is free of defects. Examples of defects include chips, cracks, dents, protrusions, and the inclusion of foreign matter (screws, paper fragments, rust, sand, etc.). Minor inconsistencies such as surface irregularities, color variations, and the presence or absence of burrs are acceptable in a good product.

[0033] (Details of the production process) The details of the generation process (step S12) will be explained below. Figure 3 is a flowchart showing an example of a method for generating a color reflection image (color image) and a normal image (shape image). The generation process shown in Figure 3 can be broadly divided into (A) the estimation process of the light source vector L(i) (step S21), (B) the acquisition process of multiple images IM(i) of the object OB to be judged (step S22), and (C) the generation process of the color image and shape image of the object OB to be judged (steps S23, S24a~S24c, S25, S26). These will be explained in detail below.

[0034] (A) Estimation process of the light source vector L(i) (Step S21) The processor 11 estimates the light source vector L(i) (the direction of each of the multiple light sources LS(i)) (step S21).

[0035] The light source vector L(i) is a vector representing the direction and distance of the light source LS(i) relative to the object OB being judged, and is used to create a color reflection image (color image) and a normal image (shape image). This estimation means determining the direction of each of the multiple light sources LS(i).

[0036] In illuminance difference stereo, the light source vector L(i) can be determined by measuring the positional relationship between the light source LS(i) and the object to be judged OB. However, this measurement can sometimes be difficult. In such cases, estimation of the light source vector L(i) becomes necessary. The details of this will be described later.

[0037] (B) Step to acquire multiple images IM(i) of the object OB to be judged (shooting, switching of lighting) (Step S22) The processor 11 acquires multiple images IM(i) of the object OB to be judged (step S22). Prior to this acquisition, the object OB to be judged is placed inside the darkroom BX.

[0038] By switching the light source LS(i) and capturing images of the object OB with the camera CA, multiple images IM(i) of the object OB can be obtained. The multiple images IM(i) are multiple images of the object OB illuminated by light from each of the multiple light sources LS(i) with different orientations. Here, image IM(i) is captured using light containing R (first wavelength range), B (second wavelength range), and B (third wavelength range), and is a color image containing R, G, and B pixels.

[0039] (C) Process for generating color and shape images of the object OB to be judged (Steps S23, S24a~S24c, S25, S26) In this generation process, a color image and a shape image of the object OB to be determined are generated based on multiple images IM(i) and the determined orientation (light source vector L(i)). This generation process can be divided into (1) the extraction of multiple R, G, and B images (step S23), (2) the generation of R, G, and B reflection images and R, G, and B normal images (steps S24a to S24c), (3) the generation of a normal image (shape image) (step S25), and (4) the generation of a color reflection image (color image) (step S26). These will be explained in detail below.

[0040] (1) Extraction of multiple R images IMr(i), G images IMg(i), and B images IMb(i) from multiple images IM(i) (Step S23) The processor 11 extracts from multiple images IM(i) multiple first images in the first wavelength range (R) (multiple R images IMr(i)), multiple second images in the second wavelength range (G) (multiple G images IMg(i)), and multiple third images in the third wavelength range (B) (multiple B images IMb(i)) (step S23).

[0041] By extracting R, G, and B pixels from a color image IM(i), it is possible to generate an R image IMr(i) composed of R pixels, a G image IMg(i) composed of G pixels, and a B image IMb(i) composed of B pixels.

[0042] As previously described, in conventional illuminance difference stereo, color is not considered, and an albedo image (in this case, a reflection image) is generated from a monochrome grayscale image. In this embodiment, in order to colorize the illuminance difference stereo, the R image IMr(i), G image IMg(i), and B image IMb(i) are extracted from the image IM(i) and processed individually. As a result, it becomes possible to colorize the illuminance difference stereo.

[0043] (2) Generation of R, G, and B reflection images and R, G, and B normal images based on multiple R, G, and B images (steps S24a to S24c) The processor 11 generates an R reflection image (a first reflection image representing the distribution of reflectance on the surface of the object OB to be determined, as seen from the pixels of R) and an R normal image (a first normal image representing the first distribution in the normal direction on the surface of the object OB to be determined, as seen from the pixels of R (step S24a).

[0044] Similarly, based on multiple G images IMg(i) (multiple second images), a G reflection image (a second reflection image representing the distribution of reflectance on the surface of the object OB to be judged, as seen from the pixels of G) and a G normal image (a second normal image representing the second distribution in the normal direction on the surface of the object OB to be judged, as seen from the pixels of B) are generated (step S24b). Based on multiple B images IMb(i) (multiple third images), a B reflection image (a third reflection image representing the distribution of reflectance on the surface of the object OB to be judged, as seen from the pixels of B) and a B normal image (a third normal image representing the third distribution in the normal direction on the surface of the object OB to be judged, as seen from the pixels of B) are generated (step S24c).

[0045] Furthermore, the first, second, and third distributions in the normal direction do not necessarily need to be represented as images such as R-normal images, G-normal images, and B-normal images. It is sufficient to show the distribution in the normal direction.

[0046] In a typical illuminance difference stereo, the luminance I(i), reflectance ρ, light source vector L(i), and normal vector n of a specific pixel in the image IM(i) are related by the following equation (1). I(i) = ρ(L(i)·n) …Equation (1)

[0047] Here, the light source vector L(i) is estimated in step S21, and the luminance I(i) is obtained from the image IM(i), but the reflectance ρ and the normal vector n are unknown. Therefore, the reflectance ρ and the normal vector n are calculated as a system of simultaneous equations by solving multiple equations for different light source vectors L(i).

[0048] As a result, the reflectance ρ and normal vector n can be determined for all pixels of image IM(i), and a reflection image (so-called albedo image) representing the distribution of reflectance ρ on the object OB to be judged, and a shape image (so-called normal vector image) representing the distribution of normal vector n on the object OB to be judged, can be generated.

[0049] In this embodiment, instead of equation (1), the following equations (2a) to (2c) are used and applied to R, G, and B (light in the first wavelength range, second wavelength range, and third wavelength range). Ir(i)=ρr(L(i)·nr) …Equation (2a) Ig(i)=ρg(L(i)·ng) …Equation (2b) Ib(i)=ρb(L(i)·nb) …Equation (2c) Ir(i), Ig(i), Ib(i): Brightness I(i) at a single point on the object OB in R, G, and B, i.e., the brightness of a pixel in image IM(i) in R, G, and B. ρg, ρb, and ρg: Reflectance ρ at a single point on the object OB in R, G, and B ng, ng, and nb: Normal vectors at a point on the object OB in R, G, and B.

[0050] In other words, for R, G, and B respectively, reflectances ρr, ρg, and ρb, and normal vectors nr, ng, and nb can be calculated based on the light source vector L(i) (the direction of the calculated light source LS(i)). Based on the reflectances ρr, ρg, and ρb, R reflection images corresponding to R pixels, G reflection images corresponding to G pixels, and B reflection images corresponding to B pixels (which, as described later, are ultimately color reflection images) can be generated. Furthermore, based on the normal vectors nr, ng, and nb, R normal images corresponding to R pixels, G normal images corresponding to G pixels, and B normal images corresponding to B pixels (which, as described later, are ultimately normal images) can be generated.

[0051] (3) Generation of a normal image (shape image) by averaging R, G, and B normal images (Step S25) The processor 11 generates a normal image (shape image) by averaging the distribution of normals in the R normal image (first normal image) (first distribution), the distribution of normals in the G normal image (second normal image) (second distribution), and the distribution of normals in the B normal image (third normal image) (third distribution) (step S25). In other words, a normal image (shape image) can be generated by averaging the normal directions in the first, second, and third distributions.

[0052] The normal vector on the object OB is a quantity corresponding to its shape and is considered to be fundamentally independent of the wavelength of light. Therefore, it is reasonable to combine the R, G, and B normal vectors nr, ng, and nb into a single normal vector n. Specifically, the average value of the normal vectors nr, ng, and nb at adjacent locations is calculated and used as the normal vector n. Using this normal vector n, a normal image representing the distribution of normals on the object OB is generated.

[0053] Here, after removing outliers from the normal vectors nr, ng, and nb, the normal vectors nr, ng, and nb may be averaged to obtain the normal vector n and generate a normal image. Removing outliers improves the accuracy of the normal vector n.

[0054] To exclude outliers, the method for excluding abnormal light source vectors, as described later, can be applied. For example, at adjacent locations on the object OB, a normal vector n that is more than a predetermined value (distance) away from the average nav of multiple obtained normal vectors nr, ng, and nb can be excluded as an anomaly. Alternatively, for example, DBSCAN can be used to exclude outliers.

[0055] Here, the normal vectors nr, ng, and nb may not be averaged, and the normal image may be generated using any one of the normal vectors nr, ng, and nb.

[0056] (4) Generation of a color reflection image (color image) from the R, G, and B reflection images (step S26) The processor 11 synthesizes the R reflection image (first reflection image), the G reflection image (second reflection image), and the B reflection image (third reflection image) to generate a color reflection image (step S26). That is, by using the reflectances ρr, ρg, and ρb of R, G, and B as the R, G, and B luminances (ρr, ρg, ρb) of a single pixel, a color reflection image can be generated.

[0057] As described above, a color reflection image and a normal image are generated from multiple images IM(i) of the object OB to be judged.

[0058] In the above, in steps S24a to S24c, the reflectance ρ and normal vector n are calculated for each of the R, G, and B pixels. This process requires a significant amount of computation. The method described below can reduce the computational load and generate color reflection images and normal images.

[0059] Figure 4 is a flowchart illustrating another example of a method for generating a color reflection image (color image) and a normal image (shape image). Steps S21 and S22 are the same as in Figure 3, so their explanation is omitted.

[0060] Here, the processor 11 converts multiple images IM(i) to grayscale and obtains multiple grayscale images IM(i) (step S31). Subsequently, the processor 11 generates a normal image based on the multiple grayscale images IM(i) (step S32).

[0061] At this point, the normal vector n is calculated using equation (1) described above, and the normal image is generated by imaging the normal vector n. In this way, it is not necessary to calculate the normal vectors nr, ng, and nb for each of R, G, and B, thus reducing the computational load. Although the reflectance ρ is calculated simultaneously with the normal vector n, it is not necessary to use this reflectance ρ.

[0062] For the color reflection image, the R reflection image, G reflection image, and B reflection image are generated from the R image, G image, and B image extracted in the same manner as in Figure 3 (step S23) (steps S33a to S33c). At this time, equations (3a) to (3c) can be used instead of equations (2a) to (2c). Ir(i) = ρr(L(i)·n) …Equation (3a) Ig(i) = ρg(L(i)·n) …Equation (3b) Ib(i) = ρb(L(i)·n) …Equation (3c)

[0063] In other words, the reflectances ρr, ρg, and ρb can be calculated using the normal vector n calculated in step S32. In this case, since the normal vector n is known, only the reflectances ρr, ρg, and ρb are unknown, and the amount of computation can be greatly reduced.

[0064] As described above, by using multiple grayscale images IM(i), the computational complexity when generating color reflection images and normal images can be reduced.

[0065] (Details of light source vector estimation) The details of the light source vector estimation (step S21) are explained below. Figure 5 is a flowchart showing an example of the light source vector estimation process. As previously described, this estimation process functions to determine the orientation of each of the multiple light sources LS(i).

[0066] (1) Selection of light source and shooting (Steps S41, S42) The processor 11 selects a light source LS(i) (step S41) and takes multiple images of the reference object (for example, 30 times) (step S42). Multiple images of the reference object illuminated by light from one light source LS(i), which is selected sequentially from multiple light sources LS(1) to LS(n), are acquired. Prior to this imaging, the reference object is placed in the darkroom BX.

[0067] The reference object is, for example, a white board (paper, for instance), and is usually one whose normal direction (normal vector n) and reflectance ρ are known. As described later, in order to determine one light source vector L(i), a set of three images are taken, for example, with the orientation (normal vector n) of the reference object changed. In other words, the "30 times" here means that it is possible to calculate one light source vector L(i) at least 10 times.

[0068] (2) Calculation of multiple light source vectors (Step S43) The processor 11 calculates multiple light source vectors L(i) corresponding to one light source LS(i) from the multiple captured images (step S43). Multiple values ​​of the direction (light source vector L(i)) of one light source LS(i) are obtained.

[0069] If the normal vector n and reflectance ρ of the reference object are known, the light source vector L(i) can be easily calculated using equation (1) described above. However, since one light source vector L(i) has three variables (x, y, z), it is usually calculated from three images in which the orientation (direction of the normal) of the reference object is changed. As a result, for one light source L(i), for example, 10 light source vectors L(i) can be obtained from 30 images.

[0070] (3) Exclusion of abnormal light source vectors and averaging of light source vectors (Steps S44, S45) The processor 11 removes abnormal light source vectors L(i) from the obtained light source vectors L(i) (step S44) and averages the remaining light source vectors (step S45). Based on the light source vectors remaining after removing the abnormal light source vectors, an estimated value Lp(i) of the light source vector L(i) (direction of light source LS(i)) is obtained. From multiple values ​​(light source vectors L(i)), values ​​that deviate from the average value of these multiple values ​​(light source vectors L(i)) by a predetermined value or more are removed, and based on the multiple values ​​(multiple light source vectors L(i)) from which values ​​that deviate by a predetermined value or more have been removed, the direction of light source LS(i) for one (estimated value Lp(i) of light source vector L(i)) is determined.

[0071] The above process is repeated until all light sources LS(i) are selected and the light source vector L(i) is estimated (step S46).

[0072] Figure 6 illustrates an example of a method for excluding abnormal light source vectors. The light source vector L(i) is represented as a point (x, y, z) on the xyz coordinate system. The object to be judged, OB, is placed at the origin O.

[0073] Here, the light source vector L(i) is divided into the light source vector L1(i) within the normal range R and the (abnormal) light source vector L0(i) outside the normal range R. However, both are light source vectors L(i) of the same light source LS(i) and should ideally be the same. However, due to measurement errors and calculation errors, the calculated light source vector L(i) may deviate significantly from its original value. Therefore, by excluding the abnormal light source vector L0(i) in the process of estimating the light source vector L(i), the accuracy of the light source vector L(i) estimation can be improved.

[0074] Here, from the average Lav(i) of multiple obtained light source vectors L(i), light source vectors L0(i) that are more than a predetermined value (distance) D are excluded as abnormal, and the estimated value of the light source vector Lp(i) is calculated as the average of the remaining light source vectors L1(i).

[0075] The distance D may be defined as a numerical value such as "0.05", or it may be defined as, for example, 1σ based on the standard deviation σ of the multiple light source vectors L(i) obtained.

[0076] Here, to exclude outliers, other methods, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), may be used. DBSCAN crystallizes the data based on the density of coordinates and removes points in low-density areas as outliers (noise).

[0077] (Details of the judgment process) The details of the decision process will be explained below, but first, let's describe Model M1. Figure 7 is a schematic representation of Model M1. Model M1 has an input layer 21, an intermediate layer 22, and an output layer 23. The intermediate layer 22 has multiple intermediate layers 22(1) to 22(n). These intermediate layers 22(i) are, for example, convolutional layers and pooling layers.

[0078] Model M1 performs functions such as outputting similarity to good products, detecting anomalies (in this case, outputting a score and a heatmap), or detecting defects based on image recognition (recognizing images corresponding to defects) based on at least one of the color image and shape image of the object OB to be judged (hereinafter collectively referred to as "images"). For example, a score and a heatmap are output based on the features output from the intermediate layer 22. Here, for the sake of clarity, the recognition and detection results from the output layer 23 are assumed to be output from the intermediate layer 22, which then outputs features, and ultimately a score and a heatmap. However, only one of these may be output, or the output from the output layer 23 may be based on the features output from the intermediate layer 22.

[0079] Model M1 is a pre-trained neural network model that is generated by inputting multiple images. The following various models (1) to (4) can be used with Model M1 to determine whether an image is good or bad.

[0080] (1) A model that outputs the similarity to good products. As Model M1, a neural network model in which good products themselves are mapped can be used to obtain the similarity between good products and the object to be judged, OB. A general example of this Model M1 is a CNN (Convolutional Neural Network).

[0081] In this case, during the preparation step (step S11), a model M1 is formed by mapping images of good products of object OB using deep learning. As a result, the similarity of object OB can be calculated based on the distance between the images of good products mapped to model M1 and the input images of object OB.

[0082] When an image of the object to be judged OB is input to the trained model M1, the model M1 outputs the similarity between the good product and the object to be judged OB. For example, if this similarity is greater than a predetermined threshold, the processor 11 judges the object to be judged OB to be a good product, and if the similarity is less than or equal to the predetermined threshold, it judges the object to be judged OB to be a defective product.

[0083] (2) A model that outputs the results of anomaly detection (score, heat map) based on the characteristics of the object to be judged (OB). Model M1 is a neural network model in which the features of object OB, which is judged to be a good product, are mapped. Based on the features of object OB (in the hidden layer 22), the anomaly detection result (score or heatmap) can be obtained. This model M1 is based on the fact that similar images tend to have similar features from the hidden layer 22. Examples of models M1 include Mahalanobis AD, SPADE, PaDiM, PatchCore, and FastFlow.

[0084] In this case, during the preparation step (step S11), a model M1 is formed by deep learning using images of good products of object OB to be judged, mapping the features of good products. Since this model M1 functions as a feature extractor, it is not necessary to map the good products themselves in deep learning; it is sufficient if the features of good products are extracted. As a result of deep learning, anomaly detection (calculation of a score, formation of a heatmap) becomes possible based on the distance between the features of good products mapped to model M1 and the features of object OB to be judged from the intermediate layer 22.

[0085] Furthermore, anomaly detection can be performed based on the feature quantities of the object OB from the intermediate layer 22, or these feature quantities can be input into another model M1 for anomaly detection. FastFlow can be cited as an example of this approach.

[0086] When an image of the object OB is input to the trained model M1, at least one of a score and a heatmap is output based on the features of the object OB from the intermediate layer 22. The score is, for example, a score for the entire image, or an anomaly score representing the degree of abnormality from the perspective of the characteristics of a good product. The heatmap divides the parts of the object OB according to the degree of the score. Note that the heatmap is often used in the process of determining anomaly detection and is not necessarily output as the final result of anomaly detection.

[0087] The processor 11 determines whether the object OB is good or bad based on the acquired score or heatmap. For example, the processor 11 determines the object OB is good if the score (abnormal score) is less than a predetermined threshold, and determines the object OB is defective if the score is equal to or greater than the predetermined threshold. When using a heatmap, the processor 11 determines the object OB is good if the area (or number of pixels) where the score is greater than or equal to a certain value is less than a predetermined threshold, and determines the object OB is defective if the area where the score is greater than or equal to a certain value is equal to or greater than the predetermined threshold.

[0088] Mahalanobis AD treats the features from the hidden layer 22 as a multivariate normal distribution, calculates the Mahalanobis distance for each hidden layer 22(i), sums them up, and outputs a distance (score) of 1.

[0089] SPADE, PaDiM, and PatchCore form heatmaps based on features from the hidden layer 22. Of these, SPADE handles features from the hidden layer 22 on a pixel-by-pixel basis and creates a heatmap by comparing them using the kNN distance. PaDiM handles features from the hidden layer 22 on a pixel-by-pixel basis and creates a heatmap by comparing them using the mean and covariance. PatchCore selects features from the hidden layer 22 and creates a heatmap based on the nearest neighbor values.

[0090] FastFlow also extracts features from the 22 hidden layers and outputs anomaly detection results (e.g., score, heatmap) based on these features. More specifically, it uses a CNN-based or Transformer-based model as model M1 to create the heatmap. However, while Mahalanobis AD, SPADE, PaDiM, and PatchCore do not require transfer learning, FastFlow does.

[0091] (3) A model that outputs defect detection results based on image recognition. As Model M2, an image recognition model for detecting defects in the object OB can be used. As Model M1, examples include YOLO (You Only Look Once), semantic segmentation, and instance segmentation models.

[0092] In this case, during the preparation step (step S11), a model M1 capable of detecting defects and their types is formed by deep learning using images of defects (e.g., chips, foreign objects) to be detected in the object OB to be judged.

[0093] When an image of the object OB to be judged is input to model M1, the type of defect (e.g., foreign object, chip) detected from the image is output. In some cases, the location (region) of the defect is also output. For example, if no defect is detected, the processor 11 judges the object OB to be judged as a good product, and if a defect is detected, it judges the object OB to be a defective product. If the (region) of the defect is output, the processor 11 may judge the object OB to be a good product if the area of ​​the defect region is smaller than a predetermined threshold, and judge the object OB to be a defective product if the area of ​​the defect region is greater than or equal to the predetermined threshold.

[0094] (4) Combined use of models In the above, the quality of an image is determined using one of the following models M1: one for similarity output, one for anomaly detection (score, heatmap), or one for image recognition. Multiple models M1 can also be combined to determine quality. For example, similarity, anomaly detection, or image recognition can be combined. This can improve the accuracy of the determination.

[0095] Model M1 (for example, a general CNN or PatchCore), which outputs similarity or anomaly detection results, is basically formed by learning based on images of good products. Therefore, it is easy to determine that object OB containing defects with a different color from good products (e.g., a black screw or a white piece of paper on a gray casting) is a defective product. However, it is not easy to determine that object OB containing defects with a similar color to good products (e.g., a gray chip in a part of a gray casting) is a defective product.

[0096] Therefore, by combining Model M1 for similarity output or anomaly detection with Model M1 for image recognition, the accuracy of the judgment can be improved. For example, by performing a good or bad judgment using image recognition, only objects OB that did not have defects detected can be judged as good or bad based on their features.

[0097] [Additional Note 1] In the above-described embodiment, one or both of the color reflection image and the normal image may be divided, and the quality of each part of the object OB to be judged may be determined. For example, the color reflection image and the normal image may be divided into four parts, and the quality of a total of eight images may be used to determine whether they are good or bad. Note that the number of divisions is not limited to four, and may be more or less. For example, the processor 11 may divide the albedo image and the normal vector image into six or nine parts.

[0098] By dividing the image and determining the quality of each part of the object OB, the accuracy of determining the quality of the object OB can be improved. For example, if all of the divided images of the object OB are determined to be good, the object OB may be determined to be good, and if any of the divided images of the object OB are determined to be defective, the object OB may be determined to be defective.

[0099] [Additional Note 2] In the embodiments described above, one or both of the color reflection image and the normal image may be masked. For example, areas of the color reflection image and normal image that do not require determination (e.g., the background area) may be filled with a pre-prepared mask image. This improves the accuracy of the determination. This masking process may be performed on the divided color reflection image and normal image.

[0100] [Additional Note 3] Collecting images for training is not always easy, as it requires a large number of images. Therefore, it is acceptable to increase the number of images by processing images of good quality products that have been photographed.

[0101] (Examples) The following describes embodiments of the present invention. Here, the object to be judged OB was a casting, and its appearance was judged by the appearance judgment device 10. Good and defective castings were photographed by the imaging unit MP, and multiple images IM were obtained to generate an image (in this case, a color reflection image). The light source vector L(i) of the light source LS(i) was estimated by the process shown in Figure 5. Furthermore, castings with foreign objects (screws and pieces of paper) placed on them were considered defective.

[0102] The generated images were input into image recognition model M1 (semantic / instance segmentation, YOLO model) and anomaly detection model M1 (MahalanobisAD and PatchCore) to determine their visual quality. Although both semantic and instance segmentation were used to determine visual quality, the results were similar, so they are collectively referred to as "semantic / instance segmentation."

[0103] Model M1 for image detection was made capable of detecting defects in castings (foreign objects such as screws, chips) through pre-training based on images of such defects. Model M1 for anomaly detection was made capable of outputting a score or heatmap through pre-training based on images of good products.

[0104] Figure 8 shows an example of the visual inspection result. Images A1, A2, and A3 are color images of the object OB to be inspected, representing a good product, a defective product containing a foreign object (screw), and a defective product containing a chip, respectively. Images B1 to B3 represent the results of inspecting images A1, A2, and A3 using semantic / instance segmentation, respectively. Similarly, images C1 to C3, D1 to D3, and E1 to E3 represent the results of inspecting images A1 to A3 using YOLO, MahalanobisAD, and PatchCore, respectively.

[0105] As shown in images B1-B3 and C1-C3, the semantic / instance segmentation and YOLO systems used for image recognition recognized image A2, which contains foreign matter, and image A3, which contains chips, as "foreign matter" and "chips," respectively. Furthermore, YOLO identified the locations of the "foreign matter" and "chips" with rectangular bounding boxes. This demonstrates that semantic / instance segmentation and YOLO can determine the quality of the castings.

[0106] As shown in images D1-D3, Mahalanobis AD was able to distinguish between good and defective products based on the magnitude of the score. In other words, by comparing the score output by Mahalanobis AD with a threshold (for example, 120), a product can be determined to be defective if the score is greater than the threshold.

[0107] As shown in images E1-E3, PatchCore distinguishes between good and defective products using a heatmap. Image A2 was identified as abnormal based on the heatmap, but image A3 could not be detected as abnormal because the color of the chipped area and the surrounding areas were similar.

[0108] As shown in images A1 to E1, all of the objects OB, which were to be judged as good products, were judged as good products. When the appearance was judged for 100 good products, all of them were judged as good products.

[0109] As previously mentioned, Mahalanobis AD was able to identify both image A2 (defective product containing foreign matter) and A3 (defective product containing chips) as defective. On the other hand, PatchCore was able to identify image A2 as defective, but not image A3. This is likely because the "chip" in image A3 is similar in color to the casting itself.

[0110] However, even in such cases, it can be seen that the accuracy of pass / fail judgment can be improved by using PatchCore in conjunction with semantic / instance segmentation or YOLO for image recognition. For example, PatchCore can be used to determine the pass / fail status of an object OB that was not found to have defects by semantic / instance segmentation or YOLO.

[0111] In the above embodiments, the appearance is determined using a color reflection image. Alternatively, the appearance may be determined using a normal image, or both a color reflection image and a normal image.

[0112] Figure 9 shows another example of the appearance judgment result, illustrating an example of the appearance judgment result using both color reflection images and normal images. Images Q1, Q2, and Q3 in Figure 9 are color reflection images of the object OB (casting) to be judged, representing a good product, a defective product with foreign matter, and a defective product with chips, respectively. Here, image Q2 represents a state where foreign matter with a color close to the base color of the casting is attached to the casting. Images R1, R2, and R3 are normal images of the object OB (casting) to be judged, corresponding to images Q1, Q2, and Q3, respectively. Images S1, S2, and S3 represent the results (heatmaps) of the judgment of images R1, R2, and R3 by PatchCore, respectively. Here, arrow F points to the defective area (foreign matter, chip).

[0113] As previously mentioned, when the color difference between defective areas (e.g., foreign matter, chips) and non-defective areas is small (similar in color), it becomes difficult to detect anomalies using color reflection images. In this case, in images Q2 and Q3 (color reflection images), the defective areas (foreign matter, chips) are somewhat unclear on the image and difficult to detect. In contrast, in images R2 and R3 (normal images), the defective areas are clearly identified as convex areas (adhered foreign matter) and concave areas (chips). As a result, in images S2 and S3, it was possible to detect the defective areas using heat maps.

[0114] In this way, the appearance of the object OB can be determined using the normal image. Since the normal image represents information about the irregularities of the object OB, even if the colors of the defective and non-defective areas are similar, the defective areas can be detected as irregularities. Furthermore, using both the color reflection image and the normal image allows for more reliable detection of defective areas.

[0115] As described above, in this embodiment, the quality of the object OB is determined based on at least one of the color reflection image and the normal image of the object OB.

[0116] [summary] (1) The appearance determination device of embodiment 1 comprises one or more processors, the processors perform a determination step of determining whether the object to be determined is good or bad based on at least one of a color image representing the color optical image of the object to be determined, generated by illuminance difference stereo, and a shape image representing the shape of the object to be determined, generated by illuminance difference stereo.

[0117] (2) The appearance determination device of Embodiment 2 is the appearance determination device of Embodiment 1, wherein the shape image is a normal image representing the distribution in the normal direction on the surface of the object to be determined, and the color image is a color reflection image representing the distribution of reflectance on the surface of the object to be determined in a first wavelength range, a second wavelength range, and a third wavelength range, which are different from each other.

[0118] (3) The appearance determination device of embodiment 3, in the appearance determination device of embodiment 1 or embodiment 2, the processor performs a generation step of generating a color image and a shape image based on a plurality of images of the object to be determined irradiated with light from each of a plurality of light sources having different orientations, and in the determination step, determines whether the object to be determined is good or bad based on at least one of the generated color image and the shape image.

[0119] (4) The appearance determination device of Embodiment 4 is an appearance determination device of Embodiments 1 to 3 in which each of the plurality of images is an image taken using light including light in the first wavelength range, the second wavelength range, and the third wavelength range, and the processor performs the following steps in the generation step: extracting a plurality of first images in the first wavelength range, a plurality of second images in the second wavelength range, and a plurality of third images in the third wavelength range from the plurality of images; generating a first reflection image representing the reflectance distribution on the surface of the object to be determined based on the plurality of first images; generating a second reflection image representing the reflectance distribution on the surface of the object to be determined based on the plurality of second images; generating a third reflection image representing the reflectance distribution on the surface of the object to be determined based on the plurality of third images; and synthesizing the first reflection image, the second reflection image, and the third reflection image to generate the color image, which is the color reflection image.

[0120] (5) The appearance determination device of embodiment 5 is the appearance determination device of embodiment 1 to 4, wherein the processor performs the steps of: converting each of the plurality of images to grayscale in the generation step; and generating the shape image, which is the normal image, based on the plurality of grayscaled images, according to claim 3.

[0121] (6) The appearance determination device of embodiment 6 is an appearance determination device of embodiments 1 to 5 in which each of the plurality of images is an image taken using light including light in the first wavelength range, the second wavelength range, and the third wavelength range, and the processor performs the following steps in the generation step: extracting a plurality of first images in the first wavelength range, a plurality of second images in the second wavelength range, and a plurality of third images in the third wavelength range from the plurality of images; determining a first distribution of the normal direction on the surface of the object to be determined based on the plurality of first images; determining a second distribution of the normal direction on the surface of the object to be determined based on the plurality of second images; determining a third distribution of the normal direction on the surface of the object to be determined based on the plurality of third images; and generating the shape image, which is the normal image, by averaging the normal directions in the first distribution, the second distribution, and the third distribution.

[0122] (7) The appearance determination device of embodiment 6 is an appearance determination device of embodiment 1 to 5 in which the processor performs a step of determining the orientation of each of the plurality of light sources, and in the generation step, generates the color image and the shape image based on the plurality of images and the determined orientation.

[0123] (8) The appearance determination device of embodiment 8 is the appearance determination device of embodiment 7, wherein the processor performs the steps of: determining a plurality of values ​​for the orientation of the 1 light source based on a plurality of images of a reference object irradiated with light from 1 of the plurality of light sources in the step of determining the orientation; excluding values ​​from the plurality of values ​​that are more than a predetermined value away from the average value of the plurality of values; and determining the orientation of the 1 light source based on the plurality of values ​​from which the values ​​more than a predetermined value away have been excluded.

[0124] (9) The appearance determination device of embodiment 9 is an appearance determination device of embodiments 1 to 8 in which the object to be determined is a casting.

[0125] (10) The appearance determination device of embodiment 10, in the appearance determination device of embodiments 1 to 9, the processor performs the steps of: inputting at least one of the color image and the shape image into a neural network model on which the features of the object to be determined to be good are mapped, to obtain a score based on the feature quantities of the intermediate layer or a heat map that classifies the score; and determining whether the object to be determined is good or bad based on the obtained score or heat map.

[0126] (11) The appearance determination device of embodiment 11, in the appearance determination device of embodiment 10, the processor determines in the determination step that when the score is less than a predetermined threshold, the object to be determined OB is a good product, and when the score is equal to or greater than a predetermined threshold, the object to be determined OB is a defective product, or when the area where the score is equal to or greater than a certain value is less than a predetermined threshold, the object to be determined is a good product, and when the area where the score is equal to or greater than a certain value is equal to or greater than a predetermined threshold, the object to be determined is a defective product.

[0127] (12) The appearance determination device of embodiment 12, in the appearance determination device of embodiments 1 to 9, the processor performs the steps of: inputting at least one of the color image and the shape image of the object to be determined to a neural network model for image recognition that detects defects in the object to be determined, and obtaining a recognition result; and determining whether the object to be determined is good or bad based on the obtained recognition result.

[0128] (13) The appearance determination method of embodiment 13 includes an input step of inputting a color image representing the optical image of the object to be determined, which is generated by illuminance difference stereo, and a shape image representing the shape of the object to be determined, which is generated by the illuminance difference stereo; and a determination step of determining whether the object to be determined is good or bad based on the color image and the shape image.

[0129] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of Symbols]

[0130] 1. Appearance Judgment System 10. Appearance determination device 11 processors 12 Primary Memory 13 Secondary memory 16 bus 21 Input Layer 22 Middle Class 23 Output Layer P1 Appearance Judgment Program M1 Model

Claims

1. Equipped with one or more processors, The aforementioned processor, A process for determining the orientation of each of multiple light sources that emit light from a first wavelength range, a second wavelength range, and a third wavelength range, each with different orientations and different directions from one another, A generation step of generating a color reflection image representing the distribution of reflectance on the surface of the object to be determined in the first wavelength range, the second wavelength range, and the third wavelength range, and a normal image representing the distribution in the normal direction on the surface of the object to be determined, based on a plurality of images of the object to be determined irradiated with light from each of the plurality of light sources, A determination step in which the quality of the object to be judged is determined based on the color reflection image and the normal image, Execute, In the process of determining the orientation, the processor A step of determining multiple values ​​for the direction of the one light source based on multiple images of a reference object illuminated by light from one of the multiple light sources, A step of excluding values ​​from the plurality of values ​​that are more than a predetermined value away from the average value of the plurality of values, A step of determining the direction of the light source 1 based on a plurality of values ​​from which values ​​that are farther than the predetermined value have been excluded, An appearance determination device that performs this task.

2. Each of the aforementioned plurality of images is an image taken using light that includes light in the first wavelength range, the second wavelength range, and the third wavelength range. The processor, in the generation process, The process involves extracting from the plurality of images a plurality of first images in the first wavelength range, a plurality of second images in the second wavelength range, and a plurality of third images in the third wavelength range. A step of generating a first reflection image that represents the distribution of reflectance on the surface of the object to be judged, based on the plurality of first images, A step of generating a second reflection image that represents the distribution of reflectance on the surface of the object to be judged, based on the plurality of second images, A step of generating a third reflection image that represents the distribution of reflectance on the surface of the object to be judged, based on the plurality of third images, A step of generating the color reflection image by combining the first reflection image, the second reflection image, and the third reflection image, An appearance determination device according to claim 1, which performs the following actions.

3. The processor, in the generation process, A step of converting each of the aforementioned multiple images to grayscale, A step of generating the normal image based on the grayscale-converted plurality of images, An appearance determination device according to claim 1, which performs the following actions.

4. Each of the aforementioned plurality of images is an image taken using light that includes light in the first wavelength range, the second wavelength range, and the third wavelength range. The processor, in the generation process, The process involves extracting from the plurality of images a plurality of first images in the first wavelength range, a plurality of second images in the second wavelength range, and a plurality of third images in the third wavelength range. A step of determining a first distribution in the normal direction on the surface of the object to be judged based on the plurality of first images, A step of determining a second distribution in the normal direction on the surface of the object to be judged based on the plurality of second images, A step of determining a third distribution in the normal direction on the surface of the object to be judged based on the plurality of third images, A step of generating the normal image by averaging the normal directions in the first distribution, the second distribution, and the third distribution, The appearance determination device according to claim 3, which performs the following actions.

5. The appearance determination device according to claim 3, wherein the object to be determined is a casting.

6. In the determination step, the processor The process involves inputting at least one of the color reflection image and the normal image into a neural network model on which the features of the object to be judged as good are mapped, and obtaining a score based on the features of the intermediate layer or a heatmap that classifies the score. A step of determining whether the object to be judged is good or bad based on the acquired score or heat map, An appearance determination device according to any one of claims 1 to 5, which performs the following:

7. In the determination step, the processor When the score is less than a predetermined threshold, the object to be judged is judged as a good product, and when the score is equal to or greater than the predetermined threshold, the object to be judged is judged as a defective product, or When the area where the score exceeds a certain value is smaller than a predetermined threshold, the object to be judged is determined to be a good product; when the area where the score exceeds a certain value is greater than or equal to a predetermined threshold, the object to be judged is determined to be a defective product. The appearance determination device according to claim 6.

8. In the determination step, the processor The process involves inputting at least one of the color reflection image and the normal image of the object to be judged into a neural network model for image recognition that detects defects in the object to be judged, and obtaining a recognition result. A step of determining whether the object to be judged is good or bad based on the recognition result obtained above, An appearance determination device according to any one of claims 1 to 5, which performs the following:

9. One or more processors, A process for determining the orientation of each of multiple light sources that emit light from a first wavelength range, a second wavelength range, and a third wavelength range, each with different orientations and different directions from one another, A generation step of generating a color reflection image representing the distribution of reflectance on the surface of the object to be determined in the first wavelength range, the second wavelength range, and the third wavelength range, and a normal image representing the distribution in the normal direction on the surface of the object to be determined, based on a plurality of images of the object to be determined irradiated with light from each of the plurality of light sources, A determination step in which the quality of the object to be judged is determined based on the color reflection image and the normal image, Execute, In the process of determining the orientation, the processor A step of determining multiple values ​​for the direction of the one light source based on multiple images of a reference object illuminated by light from one of the multiple light sources, A step of excluding values ​​from the plurality of values ​​that are more than a predetermined value away from the average value of the plurality of values, A step of determining the direction of the light source 1 based on a plurality of values ​​from which values ​​that are farther than the predetermined value have been excluded, A method for determining appearance, which is used to perform this task.

Images