Appearance determination device, appearance determination method, and program

Abstract

To properly determine whether an object is non-defective or not in an appearance test of the object.SOLUTION: A processor (11) executes: a specifying step (M11) of specifying similarities between an object to be determined and a plurality of objects that have been verified to be non-defective, using a first image representing an optical image of an object and a second image representing a shape of an object; and a determination step (M12) of determining whether the object to be determines is non-defective or not on the basis of the similarities.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for inspecting the appearance of an object. [Background technology]

[0002] Visual inspection of objects such as castings has been mainly performed by determining the presence or absence of predetermined defects. In this case, if the object has a predetermined defect, the object is determined to be defective, and if the object does not have the predetermined defect, the object is determined to be non-defective. Template matching, for example, is widely used as a method for detecting defects in an object based on an image that includes the object as a subject. Patent Document 1 also describes the use of photometric stereo as a technology for visual inspection. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-015741 A Summary of the Invention [Problem to be solved by the invention]

[0004] Whether an object is good or not should be determined based on the structure of the object as a whole. That is, there are objects that should be judged as good even if a predetermined defect or a structure similar thereto exists, and conversely, there are objects that should be judged as defective even if a predetermined defect or a structure similar thereto does not exist. For this reason, in a visual inspection for judging the presence or absence of a predetermined defect, an object that should be judged as good may be judged as defective, or an object grain that should be judged as defective may be judged as good. Although it is possible to detect each defect with high accuracy by using an image obtained by using the photometric stereo method, it is not possible to solve such a problem since the presence or absence of a defect determines whether an object is good or not.

[0005] An object of one aspect of the present invention is to provide a technique that can more appropriately determine whether an object to be inspected is a non-defective product in a visual inspection. [Means for solving the problem]

[0006] In order to solve the above problems, an appearance judgment device according to one embodiment of the present invention includes one or more processors, which execute an identification step of identifying a similarity between the object to be judged and multiple objects that have been previously confirmed to be good products using a first image representing an optical image of the object and a second image representing the shape of the object, and a judgment step of judging whether the object to be judged is a good product based on the similarity.

[0007] The appearance judgment device according to each aspect of the present invention may be realized by a computer. In this case, the program for the appearance judgment device that causes a computer to operate as each part (software element) of the appearance judgment device to realize the appearance judgment device on a computer, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention. Effect of the Invention

[0008] According to one aspect of the present invention, it is possible to more appropriately determine whether an object is a non-defective item during visual inspection of the object. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an appearance assessment device according to a first embodiment of the present invention. [Diagram 2] 1 is a flowchart showing an appearance determination method according to a first embodiment of the present invention. [Diagram 3] 1 is a diagram illustrating an outline of a configuration of an appearance determination system according to a first embodiment of the present invention. [Figure 4] 2 is a flowchart illustrating an appearance determination method performed by the appearance determination system according to the first embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram illustrating an albedo image according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a diagram illustrating an example of a normal vector image according to the first embodiment of the present invention. [Figure 7] FIG. 4 is a diagram showing an example of a cut-out process according to the first embodiment of the present invention. [Figure 8] FIG. 1 is a diagram illustrating a trained model according to a first embodiment of the present invention. [Figure 9] FIG. 4 is a diagram showing an example of an abnormality score according to the first embodiment of the present invention. [Figure 10] 10 is a flowchart showing the flow of a learning method according to a second embodiment of the present invention. [Figure 11] 10 is a flowchart illustrating an appearance determination method performed by the appearance determination system according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Embodiment 1] An embodiment of the present invention will be described in detail below. Fig. 1 is a block diagram showing a configuration of an appearance judgment device 10 according to an embodiment of the present invention. The appearance judgment device 10 judges whether an object to be judged for appearance is a non-defective product (whether there is no defect). The appearance judgment device 10 is, for example, a personal computer.

[0011] Appearance judgment device 10 includes one or more processors 11. Processor 11 executes appearance judgment method M1. Fig. 2 is a flowchart showing the flow of appearance judgment method M1 executed by processor 11. Appearance judgment method M1 includes an identifying step M11 and a judging step M12. Identifying step M11 is a step of identifying the degree of similarity between the object to be judged and multiple objects that have been confirmed to be non-defective, using a first image representing an optical image of the object and a second image representing the shape of the object.

[0012] The object is an object to be subjected to appearance judgment. An example of the object is a cast product or a mold used in casting. Note that the object is not limited to a cast product or a mold, and may be another article.

[0013] The first image is an image that represents an optical image of an object. The first image is, for example, a captured image that is obtained by capturing an image of the object and is expressed in an RGB color space. The first image is, for example, an image that is obtained by processing the captured image of the object, for example, an image that reduces the influence of one or both of the specular reflection component and the shadow from the captured image of the object. For example, the image that reduces the influence of one or both of the specular reflection component and the shadow is generated by photometric stereo. Photometric stereo is one of the three-dimensional measurement techniques that captures images illuminated from a plurality of different lighting directions and obtains the normal vector of the object from the shadow information. Note that the first image is not limited to the above-mentioned image, and may be, for example, an image that has been processed by performing tilt correction, color correction, etc. on the captured image of the object.

[0014] The second image is an image that represents the shape of the object. As an example, the second image is an image that represents the normal direction at each point on the surface of the object. As an example, the image that represents the normal direction is generated by photometric stereo. Note that the second image is not limited to the above-mentioned image, and may be another image. As an example, the second image may be an image that represents the distance from the camera to each point on the surface of the object, generated by a distance sensor (depth sensor).

[0015] A good product is an object that does not contain defects. An example of a good product is a casting that does not contain defects such as cracks, dents, protrusions, sand inclusions, etc. Some good products differ from other good products in terms of the unevenness of the casting surface, the color unevenness, the presence or absence of burrs, etc.

[0016] An object that has been confirmed in advance to be a good product is, for example, an object that has been determined to be a good product by visual inspection by an inspector. In the following explanation, for convenience of explanation, an object that has been confirmed in advance to be a good product is also referred to as a "good product object." In addition, in the following explanation, an object that is the subject of appearance judgment is also referred to as an "object to be judged."

[0017] The similarity is information that indicates the degree of similarity between an object to be judged and a good object. As an example, the similarity is information that is specified based on the distance between objects when a plurality of objects are mapped in a predetermined feature space. As an example, the processor 11 specifies the similarity by a method based on k-nearest neighbor, LOF (Local Outlier Factor), z-score, Hotelling's theory, Mahalanobis distance, etc. In the case of LOF, as an example, the similarity is the density of points corresponding to good objects in the vicinity of points corresponding to the object to be judged.

[0018] The judgment step M12 is a step for judging whether or not the object to be judged is a good product based on the similarity identified in the identification step M11. As one example, the processor 11 judges whether or not the object to be judged is a good product by calculating a numerical value (such as an abnormality score) that indicates the degree to which the object to be judged is a good product (or the degree to which it is not a good product). As another example, the processor 11 may judge that the object to be judged is a good product when the similarity identified in the identification step M11 is equal to or greater than a predetermined threshold, and may judge that the object to be judged is not a good product when the similarity is less than the threshold.

[0019] According to the above configuration, the appearance judgment device 10 uses the first image representing an optical image of the object and the second image representing the shape of the object to determine the similarity between the judgment object and multiple non-defective objects, and judges whether the judgment object is a non-defective object based on the determined similarity. This makes it possible to more appropriately judge whether the judgment object is a non-defective object.

[0020] [System Configuration] 3 is a diagram showing a schematic configuration of an appearance judgment system 1 according to this embodiment. The appearance judgment system 1 is a system that judges the appearance of a judgment target object 2 that is to be judged. In addition to an appearance judgment device 10, the appearance judgment system 1 includes cameras 21-23, light sources 311-318, 321-328, 331-338, a rotating table 40, an installation jig 50, and a robot arm 60.

[0021] The cameras 21-23 and the light sources 311-318, 321-328, 331-338 are equipment that capture images for photometric stereo. The cameras 21-23 are imaging devices that capture images of the determination object 2. The cameras 21-23 are installed at positions and orientations that capture images of the determination object 2 from different directions. In the following description, when there is no need to distinguish between the cameras 21-23, they will be referred to as "camera 20."

[0022] The light sources 311 to 318, 321 to 328, and 331 to 338 are lighting sources used when the cameras 21 to 23 photograph the determination object 2. The light sources 311 to 318 are lighting sources used when the camera 21 photographs the object. The light sources 311 to 318 are installed at positions and orientations that illuminate the object from different directions. The light sources 321 to 328 are lighting sources used when the camera 22 photographs the object. The light sources 321 to 328 are installed at positions and orientations that illuminate the object from different directions. The light sources 331 to 338 are lighting sources used when the camera 23 photographs the object. The light sources 331 to 338 are installed at positions and orientations that illuminate the determination object 2 from different directions. In the following description, when it is not necessary to distinguish between the light sources 311 to 318, 321 to 328, and 331 to 338, they are referred to as "light sources 30." In the example of FIG. 3, eight light sources 30 are provided for one camera 20.

[0023] The rotating table 40 is a mechanism for rotating the object to be determined 2. The installation jig 50 is a support member for supporting the object to be determined 2 rotated by the rotating table 40. The robot arm 60 is a mechanism for changing the posture of the object to be determined 2.

[0024] [Configuration of Appearance Evaluation Device] Appearance judgment device 10 is realized using a general-purpose computer. As shown in Fig. 1, appearance judgment device 10 includes processor 11, primary memory 12, secondary memory 13, input / output IF 14, communication IF 15, and bus 16. Processor 11, primary memory 12, secondary memory 13, input / output IF 14, and communication IF 15 are connected to each other via bus 16.

[0025] The secondary memory 13 stores an appearance judgment program P1, maps LM11, and LM12. The processor 11 loads the appearance judgment program P1, maps LM11, and LM12 stored in the secondary memory 13 onto the primary memory 12. The processor 11 then executes each step included in the appearance judgment method M1 according to instructions included in the appearance judgment program P1 loaded onto the primary memory 12. The maps LM11 and LM12 loaded onto the primary memory 12 are used when the processor 11 executes a specific step M11 of the appearance judgment method M1. Note that the appearance judgment program P1 being stored in the secondary memory 13 refers to the source code or an executable file obtained by compiling the source code being stored in the secondary memory 13. Note also that the maps LM11 and LM12 being stored in the secondary memory 13 refers to the parameters defining the maps LM11 and LM12 being stored in the secondary memory 13.

[0026] Examples of devices that can be used as the processor 11 include a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The processor 11 is also called a "computing device."

[0027] Moreover, examples of devices that can be used as the primary memory 12 include, for example, a semiconductor RAM (Random Access Memory). The primary memory 12 is sometimes called a "main memory device." Examples of devices that can be used as the secondary memory 13 include, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), an optical disk drive (ODD), a floppy (registered trademark) disk drive (FDD), or a combination of these. The secondary memory 13 is sometimes called an "auxiliary memory device." The secondary memory 13 may be built in the appearance judging device 10, or may be built in another computer (for example, a computer constituting a cloud server) connected to the appearance judging device 10 via the input / output IF 14 or the communication IF 15. In this embodiment, the storage in the appearance judging device 10 is realized by two memories (the primary memory 12 and the secondary memory 13), but is not limited to this. That is, the storage in the appearance judging device 10 may be realized by one memory. In this case, for example, one storage area of ​​the memory may be used as the primary memory 12 and another storage area of ​​the memory may be used as the secondary memory 13.

[0028] An input device and / or an output device are connected to the input / output IF14. Examples of the input / output IF14 include interfaces such as Universal Serial Bus (USB), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), and Peripheral Component Interconnect (PCI). Examples of input devices connected to the input / output IF14 include cameras 21 to 23. Data acquired from the cameras 21 to 23 in the appearance determination method M1 is input to the appearance determination device 10 and stored in the primary memory 12. Examples of other input devices connected to the input / output IF14 include a keyboard, a mouse, a touchpad, a microphone, or a combination of these. Examples of output devices connected to the input / output IF14 include a robot arm 60. Examples of other output devices connected to the input / output IF14 include a display, a projector, a printer, a speaker, a headphone, or a combination of these. Information provided to a user in the appearance determination method M1 is output from the appearance determination device 10 via these output devices. The appearance assessment device 10 may incorporate a keyboard that functions as an input device and a display that functions as an output device, like a laptop computer, or may incorporate a touch panel that functions as both an input device and an output device, like a tablet computer.

[0029] The communication IF 15 is connected to other computers by wire or wirelessly via a network. Examples of the communication IF 15 include interfaces such as Ethernet (registered trademark) and Wi-Fi (registered trademark). Available networks include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a global area network (GAN), or an internetwork including these networks. The internetwork may be an intranet, an extranet, or the Internet. Data acquired by the appearance judging device 10 from other computers in the appearance judging method M1, and data provided by the appearance judging device 10 to other computers in the appearance judging method M1 are transmitted and received via these networks. The cameras 21 to 23 and the appearance judging device 10 may be connected to each other by the input / output IF 14 or by the communication IF 15.

[0030] In this embodiment, a configuration is adopted in which the appearance determination method M1 is executed using a single processor (processor 11), but the present invention is not limited to this. That is, a configuration may be adopted in which the appearance determination method M1 is executed using multiple processors. In this case, the multiple processors that execute the appearance determination method M1 in cooperation with each other may be provided in a single computer and configured to be able to communicate with each other via a bus, or may be provided in a distributed manner in multiple computers and configured to be able to communicate with each other via a network. As an example, a processor built in a computer that constitutes a cloud server and a processor built in a computer owned by a user of the cloud server may execute the appearance determination method M1 in cooperation with each other.

[0031] In addition, in this embodiment, the maps LM11 and LM12 are stored in a memory (secondary memory 13) built in the same computer as the processor (processor 11) that executes the appearance determination method M1, but the present invention is not limited to this. That is, the maps LM11 and LM12 may be stored in a memory built in a computer different from the processor that executes the appearance determination method M1. In this case, the computer in which the memory that stores the maps LM11 and LM12 is built in is configured to be able to communicate with the computer in which the processor that executes the appearance determination method M1 is built in via a network. As an example, the maps LM11 and LM12 are stored in a memory built in a computer that constitutes a cloud server, and a processor built in a computer owned by a user of the cloud server executes the appearance determination method M1.

[0032] In addition, in this embodiment, a configuration is adopted in which the maps LM11 and LM12 are stored in a single memory (secondary memory 13), but the present invention is not limited to this. That is, a configuration may be adopted in which the maps LM11 and LM12 are distributed and stored in multiple memories. In this case, the multiple memories that store the maps LM11 and LM12 may be provided in a single computer (which may or may not be a computer with a built-in processor that executes the appearance determination method M1), or may be distributed and provided in multiple computers (which may or may not include a computer with a built-in processor that executes the appearance determination method M1). As an example, a configuration in which the maps LM11 and LM12 are distributed and stored in memories built into each of the multiple computers that constitute the cloud server is possible.

[0033] The maps LM11 and LM12 are maps for mapping the first image and the second image of the object to points in the feature space. The map LM11 is, for example, a trained model that is trained to input a first image of the object and embed the object in a predetermined embedding space (feature space). The map LM12 is, for example, a trained model that is trained to input a second image of the object and embed the object in a predetermined embedding space (feature space). As the maps LM11 and LM12, for example, algorithms such as neural network models such as convolutional neural networks and recurrent neural networks, regression models such as linear regression, or tree models such as regression trees can be used.

[0034] Maps LM11 and LM12 are, as an example, maps constructed by metric learning. Metric learning is a method for learning a transformation for embedding samples in an input space into a different embedding space so that the similarity between samples in the input space corresponds to a measure such as Euclidean distance or cosine similarity.

[0035] [Operation of Appearance Judgment System] 4 is a flowchart illustrating an appearance judgment method performed by the appearance judgment system 1. Note that some steps may be executed in parallel or in a different order. First, an administrator of the appearance judgment system 1 or the like turns on the power of each device included in the appearance judgment system 1 and sets the judgment target object 2 on the installation jig 50.

[0036] (Step S11) In step S11, the processor 11 acquires captured images of the object 2 to be determined by the multiple cameras 20 while switching the illumination. When eight light sources 30 are associated with each of the three cameras 20, the processor 11 generates eight captured images by capturing images while switching the illumination using the eight light sources 30 in sequence for each camera 20. By capturing images using the eight light sources 30 for each of the three cameras 20, the processor 11 acquires 24 captured images.

[0037] (Step S12) In step S12, the processor 11 calculates a light source vector for each of the 24 captured images. The light source vector is a vector that represents the direction from the light source 30 to the determination target object 2. When calculating the light source vector, the processor 11 may perform the calculation after painting unnecessary parts (such as the background) in the captured image with a mask image prepared in advance. By the processor 11 performing the mask process, it is possible to make the calculation result more stable. Note that, when the light source vector of each captured image is prepared in advance, the processor 11 may read the prepared value without performing the calculation process of the light source vector.

[0038] (Step S13) In step S13, the processor 11 generates an image (hereinafter also referred to as an "albedo image") in which the influence of one or both of the specular reflection component and the shadow is reduced in the captured image of the object based on the eight captured images taken by one camera 20 and the light source vector calculated in step S12. The albedo image is an example of the "first image" according to this specification. As an example, the albedo image is an image expressed in the RGB color space. As a method for generating the albedo image, for example, the methods described in "Reflectance Map Techniques for Analyzing Surface Defects in Metal Castings, "https: / / dspace.mit.edu / handle / 1721.1 / 16353" and "Determining shape and reflectance using multiple images, https: / / dspace.mit.edu / handle / 1721.1 / 16136" are used. Note that the method for generating the albedo image is not limited to the above-mentioned method, and the processor 11 may generate the albedo image by another method.

[0039] The processor 11 generates an albedo image based on eight captured images taken by each of the multiple cameras 20. In other words, the processor 11 generates an albedo image for each camera 20. When the number of cameras 20 is three, the processor 11 generates three albedo images.

[0040] 5 is a diagram illustrating an example of an albedo image. In the figure, an albedo image Img11 is an albedo image generated by the processor 11 from a captured image of a non-defective object. Albedo images Img12 to Img14 are albedo images generated by the processor 11 from captured images of non-defective (defective) objects.

[0041] As shown in Fig. 5, the size of the defect contained in the target object is reflected in the albedo image. In the example of Fig. 5, the defect in albedo image Img13 is larger than the defect in albedo image Img12, and the defect in albedo image Img14 is larger than the defect in albedo image Img13.

[0042] (Step S14) In step S14, the processor 11 generates a normal vector image based on eight captured images captured by one camera 20. The normal vector image is an image generated by photometric stereo and represents the normal direction at each point on the surface of the object to be determined 2. The normal vector image is an example of a "second image" according to this specification. As an example, the normal vector image is an image that represents the normal direction at each point in RGB.

[0043] 6 is a diagram illustrating an example of a normal vector image. In the figure, a normal vector image Img21 is a normal vector image generated by the processor 11 from a captured image of a non-defective object. Normal vector images Img22 to Img24 are normal vector images generated by the processor 11 from captured images of non-defective (defective) objects.

[0044] As shown in Fig. 6, the thickness of a defect contained in an object is shown in a normal vector image. In the example of Fig. 6, the thickness of the defect in normal vector image Img23 is thicker than the thickness of the defect in normal vector image Img22, and the thickness of the defect in normal vector image Img24 is thicker than the thickness of the defect in normal vector image Img23.

[0045] (Step S15) In step S15 of FIG. 4, processor 11 determines whether or not image capture has been completed for the object to be determined. As an example, processor 11 determines that image capture has been completed when image capture of the object from six directions has been completed. On the other hand, processor 11 determines that image capture has not been completed when image capture has only been performed from three directions. If image capture has been completed (step S15; YES), processor 11 proceeds to processing of step S17. On the other hand, if image capture has not been completed (step S15; NO), processor 11 proceeds to processing of step S16.

[0046] (Step S16) In step S16, the processor 11 controls the robot arm 60 to rotate the object to be determined 2 and change the posture of the object to be determined 2. After completing the process of step S16, the processor 11 returns to step S11 and similarly captures images of the remaining three directions. In this manner, in steps S11 to S16, the appearance determination system 1 captures images of three sides of the object to be determined 2 to generate three captured images, and then captures the remaining three sides by flipping the images upside down and rotating them.

[0047] Processor 11 repeatedly executes steps S11 to S16, thereby generating an albedo image and a normal vector image for each of the six directions. In other words, processor 11 generates six albedo images and six normal vector images for one determination object 2.

[0048] At this time, the processor 11 may execute a predetermined mask process on the albedo image and the normal vector image. As an example, the processor 11 may execute a mask process on the albedo image and the normal vector image, by matching them with a mask image prepared in advance and painting out unnecessary parts (such as the background).

[0049] The processor 11 may also execute a predetermined mask process on one or both of the albedo image and the normal vector image, and execute a process of cutting out an image of an area including an area that is not filled in by the mask process. In this case, the processor 11 handles the cut-out image as an albedo image and a normal vector image in the subsequent process. In other words, the processor 11 specifies the similarity using the cut-out image.

[0050] 7 is a diagram showing an example of an image cut-out process performed by the processor 11. In the figure, an image Img31 is an example of an albedo image that has been subjected to a mask process. The processor 11 executes a process of cutting out a rectangular area Img312 that includes an area other than the mask area Img311 from the image Img31. In subsequent processing, the processor 11 handles the image of the rectangular area Img312 as an albedo image.

[0051] (Step S17) In step S17, the processor 11 performs a process of embedding the determination object and one or more non-defective objects in a feature space. As an example, the processor 11 maps the albedo images (first images) of the non-defective objects to a plurality of first points on the first feature space, and maps the normal vector images (second images) of the non-defective objects to a plurality of second points on the second feature space. The processor 11 also maps the albedo image (first image) of the determination object to a third point on the first feature space, and maps the normal vector image (second image) of the determination object to a fourth point on the second feature space. Step S17 is an example of a "mapping step" according to the present specification. As an example, the second feature space is a feature space different from the first feature space.

[0052] In step S17, the map LM11 that maps the albedo image (first image) to a point on the first feature space and the map LM12 that maps the normal vector image (second image) to a point on the second feature space are, for example, maps constructed by metric learning. For example, the maps LM11 and LM12 are modified terminal layers of a conventional CNN to have embedded data defined to a prescribed size. The maps LM11 and LM12 are trained so that the distance between the embedded data in the feature space of samples of the same category is close, and the distance between samples of different categories is large. In the following description, when there is no need to distinguish between the maps LM11 and LM12, they are referred to as the "trained model LM1".

[0053] 8 is a diagram showing an example of the trained model LM1. As described above, input data is input to the trained model LM1. The input data is the albedo image generated by the processor 11 in step S13 of FIG. 4, or the normal vector image generated in step S14.

[0054] The trained model LM1 consists of, for example, multiple convolutional layers, pooling layers, and combination layers. In the convolutional layer, the input data is convoluted by filtering. The convolutional data is pooled in the pooling layer. This improves the model's ability to recognize changes in the position of features in the data. The pooled data is processed in the combination layer and converted into output data of the trained model LM1, i.e., embedding data that represents the object in the embedding space, and output.

[0055] That is, the input data input to the trained model LM1 is passed through each layer shown in Fig. 8 in the order shown, and the embedded data is output. The embedded data, which is the output of the trained model LM1, includes, as an example, a plurality of floating-point numbers. The number of floating-point numbers included in the embedded data is the number of dimensions of the feature space.

[0056] The configuration of the trained model LM1 is not limited to that shown in Fig. 8. The trained model LM1 may be configured, for example, based on a general-purpose CNN, with layers near the end deleted and with a convolutional layer, a pooling layer, a dropout layer, a fully connected layer, and an L2 regularization layer added.

[0057] In step S17, when mapping the determination object to the feature space, the processor 11 inputs the albedo image generated in step S13 to the map LM11, and obtains embedded data (data representing the third point) representing the object in the feature space output by the map LM11. The processor 11 also inputs the normal vector image generated in step S14 to the map LM12, and obtains embedded data (data representing the fourth point) representing the object in the feature space output by the LM12.

[0058] 4, when mapping a non-defective object into the feature space, the processor 11 inputs an albedo image of the non-defective object into the map LM11, and obtains embedded data (data representing a first point) representing the non-defective object in the feature space output by the map LM11. The processor 11 also inputs a normal vector image of the non-defective object into the map LM12, and obtains embedded data (data representing a second point) representing the non-defective object in the feature space output by the map LM12.

[0059] In this embodiment, the albedo image and normal vector image of the non-defective object are generated in advance by, for example, the appearance judgment device 10 or another device. The method of generating the albedo image and normal vector image of the non-defective object is similar to the method of generating the albedo image and normal vector image of the judgment object, and for example, the appearance judgment system 1 generates the albedo image and normal vector image of the non-defective object by executing steps similar to steps S11 to S16 in Fig. 4.

[0060] (Step S18) In step S18 of FIG. 4, the processor 11 specifies the similarity between the object to be determined and the non-defective object. As an example, the processor 11 specifies the similarity based on the distance between the first point (point representing the albedo image of the non-defective object) and the third point (point representing the albedo image of the object to be determined) in the first feature amount space, and the distance between the second point (point representing the normal vector image of the non-defective object) and the fourth point (point representing the normal vector image of the object to be determined) in the second feature amount space. In this operation example, the processor 11 specifies the similarity for each of a total of 12 images, including 6 albedo images and 6 normal vector images, for one object to be determined. In other words, the processor 11 specifies 12 similarities for one object to be determined.

[0061] For example, the similarity is determined by k-nearest neighbor method, LOF, or a method based on Mahalanobis distance. LOF is a method for detecting anomalies in consideration of data distribution by focusing on local density between the object and k (k is an integer equal to or greater than 2) neighboring samples. For example, the processor 11 determines, as the similarity, a value determined by the LOF or k-nearest neighbor method based on the density of points (first point, third point) corresponding to multiple non-defective objects in the vicinity of a point (second point, fourth point) corresponding to the object to be determined, or the distance between the points (second point, fourth point) corresponding to the object to be determined and the points (first point, third point) corresponding to multiple non-defective objects.

[0062] When LOF is used, for example, the higher the density, the higher the similarity, and the lower the density, the lower the similarity. When the k-nearest neighbor method is used, for example, the processor 11 calculates the k-nearest neighbors of the object to be determined in a feature space in which points corresponding to the non-defective object are embedded, and specifies the similarity based on the average value, maximum value, minimum value, etc. of the distance from the non-defective object. For example, the processor 11 specifies the similarity as lower the average value of the distance, and as higher the average value.

[0063] As another example of using the k-nearest neighbor method, the processor 11 may specify the number of samples for each class among the k nearest samples of the sample (second point) representing the object to be determined. In this case, the number of samples of the class corresponding to the non-defective object is treated as the similarity. In this case, as an example, the more samples of the class corresponding to the non-defective object, the higher the similarity, and the fewer the number of samples, the lower the similarity.

[0064] (Step S19) In step S19, processor 11 judges whether the object to be judged is a good product based on the multiple similarities identified in step S18. As an example, processor 11 may judge that the object to be judged is a good product when all of the multiple similarities identified in step S18 are equal to or greater than a predetermined threshold. As another example, processor 11 calculates a numerical value (anomaly score) representing the degree to which the object to be judged is a good product based on the multiple similarities. As an example, processor 11 calculates the anomaly score based on the minimum, maximum, average, etc. of the multiple similarities identified. As an example, the smaller the anomaly score, the more likely the object to be judged is a good product.

[0065] (Step S20) 4, the processor 11 outputs the determination result. As an example, the processor 11 outputs the determination result by displaying it on a display.

[0066] Fig. 9 is a diagram showing an example of an anomaly score calculated by processor 11. In graph 101 in Fig. 9, the horizontal axis indicates the height of a defect contained in the object to be determined, and the vertical axis indicates the anomaly score calculated by processor 11. As shown in the figure, the higher the height of the defect, the higher the value of the anomaly score calculated by processor 11. Also, in graph 102, the horizontal axis indicates the area of ​​a defect contained in the object to be determined, and the vertical axis indicates the anomaly score calculated by processor 11. As shown in the figure, the larger the area of ​​the defect, the higher the value of the anomaly score calculated by processor 11.

[0067] As described above, according to this embodiment, the appearance judgment device 10 determines the similarity between the judgment object and a non-defective object using not only the optical image of the object but also the second image representing the shape of the object. Even if the object contains a defect that does not appear in the optical image, the appearance judgment device 10 determines the similarity using both the optical image and the second image representing the shape of the object, thereby making it possible to more appropriately judge whether the judgment object is a non-defective object.

[0068] In the present embodiment, the appearance judgment device 10 specifies the similarity between the judgment target and a plurality of non-defective targets. By using various images of non-defective targets, such as images of non-defective targets including burrs, as images of the non-defective targets (albedo images, normal vector images), it is possible to more appropriately judge whether the target is good or bad, for example, to judge an object including burrs as a non-defective target.

[0069] Furthermore, according to this embodiment, the appearance determination system 1 captures the object 2 to be determined from three directions at once, thereby shortening the time required for determination. Furthermore, according to this embodiment, by utilizing photometric stereo, it is possible to generate an albedo image and a normal vector image with a simple system configuration using a general-purpose camera.

[0070] [Embodiment 2] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0071] An appearance judgment system 1B according to the second embodiment includes an appearance judgment device 10B instead of the appearance judgment device 10 according to the first embodiment. The appearance judgment device 10B has a similar configuration to the appearance judgment device 10 shown in Fig. 1. In other words, the appearance judgment device 10B includes a processor 11, a primary memory 12, a secondary memory 13, an input / output IF 14, and a communication IF 15. These components are similar to those in the above-described first embodiment, and description thereof will not be repeated.

[0072] The secondary memory 13 of the appearance judgment device 10B stores a map construction program P2 and a learning data set LD1 in addition to the appearance judgment program P1, maps LM11, and LM12. The processor 11 loads the map construction program P2 stored in the secondary memory 13 onto the primary memory 12. The processor 11 then executes each step included in the map construction method M2 according to instructions included in the map construction program P2 loaded onto the primary memory 12. The learning data set LD1 is used when the processor 11 executes the map construction method M2. Note that the map construction program P2 being stored in the secondary memory 13 refers to the source code, or an executable file obtained by compiling the source code, being stored in the secondary memory 13.

[0073] (Map building method flow) In this embodiment, the appearance judgment device 10B executes a process of constructing maps LM11 and LM12. Fig. 10 is a flowchart illustrating the flow of a map construction method M2 performed by the appearance judgment system 1B. Note that some steps may be executed in parallel or in a different order.

[0074] In steps S31 to S38, the processor 11 collects a learning data set LD1. The learning data set LD1 is used in the generation process of the maps LM11 and LM12. The learning data set LD1 is a collection of data combining an albedo image of an object, a normal vector image, and a class representing a category of the object or image. As an example, the category represents the shooting direction of a captured image. Hereinafter, an object from which learning data is collected is also referred to as a "sample." The object from which learning data is collected is a non-defective object.

[0075] The processing in steps S31 to S36 is generally similar to steps S11 to S16 in Fig. 4, except that the object to be imaged changes from a "determination object" to an "object." The following will mainly explain the differences from the flowchart in Fig. 4.

[0076] (Step S37) In step S37, processor 11 determines whether or not imaging of all samples has been completed. If imaging has been completed (step S37; YES), processor 11 proceeds to the process of step S38. On the other hand, if there are samples that have not yet been imaged (step S37; NO), processor 11 returns to the process of step S31 and images the next sample. Appearance determination system 1 executes the processes of steps S31 to S36 for each of the multiple samples, and generates an albedo image and a normal vector image of the object.

[0077] (Step S38) In step S38, the processor 11 performs data augmentation. As an example, the processor 11 performs various processes such as rotation, shift, projection transformation, and noise addition on the image generated as the learning data to generate simulated data. Note that, if there is sufficient learning data, the processor 11 does not need to perform the data augmentation process in step S38.

[0078] (Step S39) In step S39, the processor 11 constructs a map LM11 that maps the albedo image of the object to the feature space and a map LM12 that maps the normal vector image of the object to the feature space by metric learning. At this time, the processor 11 trains the maps LM11 and LM12 so that the distance between the embedded data in the feature space between samples of the same category is close, and the distance between samples of different categories is large.

[0079] Processor 11 performs appearance determination of the object to be determined using the map generated in step S39. The method of appearance determination is the same as the method shown in the first embodiment described above.

[0080] (Effects of the embodiment) According to this embodiment, the appearance judgment device 10B uses an albedo image and a normal vector image of an object to construct a map that maps the albedo image and the normal vector image in a feature space. By using this map, the appearance judgment device 10B can more appropriately judge whether the object is a non-defective item.

[0081] [Embodiment 3] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0082] An appearance judgment system 1C according to the present embodiment includes an appearance judgment device 10C instead of the appearance judgment device 10 according to the first embodiment. The appearance judgment device 10C has a similar configuration to the appearance judgment device 10 shown in Fig. 1. In other words, the appearance judgment device 10C includes a processor 11, a primary memory 12, a secondary memory 13, an input / output IF 14, and a communication IF 15. These components are similar to those in the first embodiment described above, and description thereof will not be repeated.

[0083] The secondary memory 13 according to this embodiment stores eight maps, LM21 to LM28. The maps LM21 to LM24 are maps for mapping divided images, obtained by dividing an albedo image into a plurality of regions, onto points in a first feature space. The maps LM25 to LM28 are maps for mapping divided images, obtained by dividing a normal vector image into a plurality of regions, onto points in a second feature space.

[0084] Fig. 11 is a flowchart illustrating an appearance judging method performed by the appearance judging system 1C according to embodiment 3. Note that some steps may be executed in parallel or in a different order. In step S15 of Fig. 11, when the photographing is completed (YES in step S15), the processor 11 performs the process of step S21 before step S17.

[0085] (Step S20) In step S20, the processor 11 divides each of the six albedo images generated in step S13 and the six normal vector images generated in step S14 into a plurality of regions to generate divided images. As an example, the processor 11 divides each of the six albedo images and the six normal vector images into four regions. In other words, the processor 11 generates 24 albedo images (divided images obtained by dividing the albedo image generated in step S13 into four) and 24 normal vector images (divided images obtained by dividing the normal vector image generated in step S14 into four). Note that the number of divisions is not limited to four, and may be more or less than this. For example, the processor 11 may divide the albedo images and the normal vector images into six or nine regions.

[0086] In step S17C (an example of a mapping step), the processor 11 maps the divided images obtained by dividing the albedo image (first image) into a plurality of regions to points in the feature space corresponding to each of the plurality of regions, and maps the divided images obtained by dividing the normal vector image (second image) into a plurality of regions to points in the feature space corresponding to each of the plurality of regions. The maps LM21 to LM24 correspond to the divided images of the albedo image divided into four, respectively. As an example, the processor 11 maps the upper right divided image of the divided images divided into four to the first feature space using the map LM21. Similarly, the processor 11 maps the lower right divided image of the divided images divided into four to the first feature space using the map LM22. As an example, the processor 11 maps the upper left divided image of the divided images divided into four to the first feature space using the map LM23. Furthermore, as an example, the processor 11 maps the lower left divided image out of the four divided images into the first feature amount space using the map LM24.

[0087] Moreover, the maps LM25 to LM28 correspond to the divided images of the normal vector image divided into four. As an example, the processor 11 maps the upper right divided image of the divided images of the normal vector image divided into four to the second feature space using the map LM25. Similarly, the processor 11 maps the lower right divided image of the divided images divided into four to the second feature space using the map LM26. As an example, the processor 11 maps the upper left divided image of the divided images divided into four to the second feature space using the map LM117. As an example, the processor 11 maps the lower left divided image of the divided images divided into four to the second feature space using the map LM28.

[0088] In step S18C, processor 11 identifies the similarity for each of the divided images of the first image and the divided images of the second image. In addition, in step S19C, processor 11 determines whether the object to be determined is a non-defective product based on the multiple similarities identified in step S18C. As an example, processor 11 may calculate an abnormality score for the object to be determined based on the minimum, maximum, and average values ​​of the multiple similarities identified.

[0089] According to this embodiment, the processor 11 can reduce the unit of the image to be input to the map by dividing the image for discrimination (albedo image and normal vector image). This allows evaluation of even more detailed parts. In particular, in a CNN-based neural network, information on detailed parts is lost when undergoing convolution and pooling, but this effect can be reduced by dividing the image to make it smaller.

[0090] In this embodiment, if the image for discrimination is divided without including overlapping areas, defects across the divided boundary parts are difficult to detect. Therefore, the processor 11 may divide the captured image so that the multiple divided images include overlapping areas. In addition, the processor 11 may perform judgment using images divided at different resolutions, as an example. By the processor 11 generating divided images by overlapping some areas or dividing the image at different resolutions, it is possible to more appropriately judge whether the product is good or not even if a defect is included in the boundary part.

[0091] [Embodiment 4] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0092] An appearance judgment system 1D according to the fourth embodiment includes an appearance judgment device 10D instead of the appearance judgment device 10 according to the first embodiment. The appearance judgment device 10D has a similar configuration to the appearance judgment device 10 shown in Fig. 1. In other words, the appearance judgment device 10D includes a processor 11, a primary memory 12, a secondary memory 13, an input / output IF 14, and a communication IF 15. These components are similar to those in the first embodiment described above, and description thereof will not be repeated.

[0093] The secondary memory 13 of the appearance judgment device 10B stores an appearance judgment program P4 and a map LM41 instead of the appearance judgment program P1, the maps LM11, and LM12. The map LM41 is a map that maps an albedo image (first image) and a normal vector image (second image) to points on a feature space. In other words, in this embodiment, the first feature space in which the first image is mapped and the second feature space in which the second image is mapped are the same feature space. The map 41 is, for example, a learned model constructed by metric learning.

[0094] In this embodiment, the processor 11 uses the map LM41 to map the albedo image of the object onto the feature space, and also maps the normal vector image of the object onto the same feature space as the albedo image. The map LM41 plots similar objects at close distances, and dissimilar objects at far distances. Since the albedo image and the normal vector image have completely different colors, the albedo image and the normal vector image are plotted apart in one feature space.

[0095] The processor 11 determines the similarity based on the distance between the albedo image of the object to be determined and the albedo image of the non-defective object in the feature space, and determines the similarity based on the distance between the normal vector image of the object to be determined and the non-defective vector image of the non-defective object. The process of determining the similarity is the same as the process shown in the above-mentioned embodiment 1, and the description thereof will not be repeated.

[0096] [Embodiment 5] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0097] An appearance judgment system 1E according to the fifth embodiment includes an appearance judgment device 10E instead of the appearance judgment device 10C according to the third embodiment. The appearance judgment device 10E has a configuration similar to that of the appearance judgment device 10 shown in Fig. 1. In other words, the appearance judgment device 10E includes a processor 11, a primary memory 12, a secondary memory 13, an input / output IF 14, and a communication IF 15. These components are similar to those of the first embodiment described above, and description thereof will not be repeated.

[0098] The secondary memory 13 according to this embodiment stores a map LM51 instead of the maps LM21 to LM28 according to the third embodiment. The map LM51 is a map that maps divided images obtained by dividing an albedo image (first image) into a plurality of regions and divided images obtained by dividing a normal vector image (second image) into a plurality of regions to points on a feature space. In other words, in this embodiment, the first feature space in which the divided images obtained by dividing a first image into a plurality of regions are mapped and the second feature space in which the divided images obtained by dividing a second image into a plurality of regions are mapped are the same feature space. The map LM51 is, for example, a learned model constructed by metric learning.

[0099] The processor 11 determines the similarity based on the distance between the divided images of the albedo image of the object to be determined and the divided images of the albedo image of the non-defective object in the feature space, and also determines the similarity based on the distance between the divided images of the normal vector image of the object to be determined and the normal vector image of the non-defective object in the feature space. As an example, when 6 albedo images and 6 normal vector images, totaling 12 images, are each divided into 4, the processor 11 determines the similarity for each divided image, i.e., a total of 48 similarities.

[0100] In this embodiment, similarly to the above-described embodiment 3, the processor 11 may divide the captured image so that the divided images include overlapping areas. Also, as one example, the processor 11 may perform the judgment using images divided at different resolutions. The processor 11 may generate divided images by overlapping some areas or divide the image at different resolutions, so that the quality judgment can be performed more appropriately even if a defect is included in the boundary portion.

[0101] [Embodiment 6] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0102] An appearance judgment system 1F according to the sixth embodiment includes an appearance judgment device 10F instead of the appearance judgment device 10 according to the first embodiment. The appearance judgment device 10F has a similar configuration to the appearance judgment device 10 shown in Fig. 1. In other words, the appearance judgment device 10F includes a processor 11, a primary memory 12, a secondary memory 13, an input / output IF 14, and a communication IF 15. These components are similar to those in the first embodiment described above, and description thereof will not be repeated.

[0103] In the above-mentioned first embodiment, the processor 11 specifies the similarity using the albedo image and the normal vector image, and uses an image representing the normal direction at each point on the surface of the object as the normal vector image (second image). The second image is not limited to the images shown in the above-mentioned embodiments, and may be other images. As an example, the second image may include a normal vector image representing the normal vector at each point with each RGB component value, and an image representing an edge included in the normal vector image. As an example, the processor 11 performs a conversion process from the normal vector to height information, or a sharpening process of the normal vector, to generate image data representing the edge of the normal vector. As an example, the processor 11 performs an outline detection process using a Sobel filter to generate image data representing the edge.

[0104] Also, the second image may be, for example, an image obtained by combining a normal vector image that represents a normal vector at each point by each RGB component value and an image that represents an edge included in the normal vector image. Also, the second image may be, for example, image data that represents an edge included in the normal vector image.

[0105] Since the normal vector image, which expresses the normal direction at each point using RGB, does not include height information, the sensitivity tends to be low when the height of the defect is low, such as when a flat foreign object is attached. In this embodiment, by converting the normal vector image into height information or sharpening the normal vector image, it is possible to appropriately determine pass / fail for objects that include low-height defects.

[0106] [Additional Note 1] In the above-described embodiment, one or both of the first image and the second image may be images generated using an image obtained by performing a predetermined mask process on the captured image of the object. As an example, the processor 11 may perform a process of filling in the background area with a mask image prepared in advance. If the mask process is not performed, the background included in the captured image may become noise, and the judgment may not be performed appropriately. By the processor 11 performing the mask process on the captured image, it is possible to suppress fluctuations due to noise in areas other than the area to be judged. Furthermore, by the processor 11 performing the mask process, it is possible to reduce the amount of calculation related to the judgment.

[0107] [Additional Note 3] In the above-described second embodiment, the processor 11 learns data of a non-defective object, but may also learn data of an object having a defect. In this case, the processor 11 may associate a label representing the type, size, position, etc. of the defect with each pixel by semantic segmentation. Specifically, the trained model constructed by machine learning may be one that estimates only the position of the defect, such as YOLO, or may be one that estimates even a specific outline, such as U-NET and MRCNN.

[0108] 〔summary〕 The appearance evaluation device of aspect 1 includes one or more processors, which execute an identification step of identifying a similarity between the object to be evaluated and a plurality of objects that have been previously confirmed to be good products using a first image representing an optical image of the object and a second image representing the shape of the object, and a judgment step of judging whether the object to be evaluated is a good product based on the similarity.

[0109] According to the above configuration, the appearance evaluation device uses a first image representing an optical image of the object and a second image representing the shape of the object to determine the similarity between the object to be evaluated and multiple objects that have been confirmed to be good products, and determines whether the object to be evaluated is good or bad based on the determined similarity. This makes it possible to more appropriately determine whether the object is good or bad.

[0110] The appearance judgment device according to aspect 2 has the following features in addition to the features of the appearance judgment device according to aspect 1. That is, in the appearance judgment device according to aspect 2, the second image is an image representing a normal direction at each point on the surface of the object.

[0111] According to the above configuration, the appearance evaluation device uses a first image representing an optical image of the object and a second image representing the normal direction at each point on the surface of the object to determine the similarity between the object to be evaluated and an object confirmed to be a good product, and determines whether the object to be evaluated is a good product based on the determined similarity. This makes it possible to more appropriately determine whether the object is a good product.

[0112] The appearance judging device according to aspect 3 has the following features in addition to the features of the appearance judging device according to aspect 1 or 2. That is, in the appearance judging device according to aspect 3, the processor further executes a mapping step of mapping the first images of a plurality of objects previously confirmed to be non-defective to a plurality of first points in a first feature space and mapping the second images of the plurality of objects to a plurality of second points in a second feature space, mapping the first image of an object to be judged to a third point in the first feature space and mapping the second image of the object to be judged to a fourth point in the second feature space, and the processor specifies the similarity based on the distance between the first point and the third point in the first feature space and the distance between the second point and the fourth point in the second feature space in the specifying step.

[0113] According to the above configuration, the appearance judgment device judges whether or not the object to be judged is a non-defective product based on the distance between the objects embedded in the feature space, thereby enabling the appearance judgment device to more appropriately judge whether or not the object to be judged is a non-defective product.

[0114] The appearance judging device according to aspect 4 has the following features in addition to the features of the appearance judging device according to aspect 3. That is, in the appearance judging device according to aspect 4, the first feature amount space and the second feature amount space are the same feature amount space.

[0115] According to the above configuration, the appearance judgment device judges whether or not the object to be judged is a pass / fail product based on the distance between the objects embedded in one feature space, thereby enabling the appearance judgment device to more appropriately judge whether or not the object to be judged is a pass / fail product.

[0116] The appearance judging device according to aspect 5 has the following features in addition to the features of the appearance judging device according to aspect 3. That is, in the appearance judging device according to aspect 5, the second feature space is a feature space different from the first feature space.

[0117] According to the above configuration, the appearance judgment device judges whether or not the object to be judged is a pass / fail product based on the distance between the objects embedded in one feature space, thereby enabling the appearance judgment device to more appropriately judge whether or not the object to be judged is a pass / fail product.

[0118] The appearance judgment device according to aspect 6 has the following features in addition to the features of the appearance judgment device according to any one of aspects 3 to 5. That is, in the appearance judgment device according to aspect 6, in the mapping step, the map that maps the first image to points in the first feature space and the map that maps the second image to points in the second feature space are maps constructed by metric learning.

[0119] According to the above configuration, the appearance judging device maps an object into a feature space using a map constructed by metric learning, thereby enabling the appearance judging device to more appropriately judge whether the object being judged is a non-defective product.

[0120] The appearance judgment device according to aspect 7 has the following features in addition to the features of the appearance judgment device according to any one of aspects 3 to 6. That is, in the appearance judgment device according to aspect 7, in the identification step, the processor determines as the similarity a value identified by LOF or k-nearest neighbor based on the density of points corresponding to a plurality of objects that have been confirmed to be good products in the vicinity of a point corresponding to the object to be judged, or the distance between the point corresponding to the object to be judged and a point corresponding to a plurality of objects that have been confirmed to be good products.

[0121] According to the above configuration, the appearance judging device specifies the similarity by the LOF or k-nearest neighbor method, which allows the appearance judging device to more appropriately judge whether the target object is a non-defective product.

[0122] The appearance judgment device according to aspect 8 has the following features in addition to the features of the appearance judgment device according to any one of aspects 3 to 7. That is, in the appearance judgment device according to aspect 8, in the mapping step, the processor maps divided images obtained by dividing the first image into a plurality of regions to points in a feature space corresponding to each of the plurality of regions, and maps divided images obtained by dividing the second image into a plurality of regions to points in a feature space corresponding to each of the plurality of regions, in the identifying step, identifies the similarity for each of the divided images of the first image and the divided images of the second image, and in the judging step, judges whether the object to be judged is a non-defective product based on the multiple similarities identified in the identifying step.

[0123] According to the above configuration, the appearance judging device determines the similarity between the divided images obtained by dividing the first image and the divided images obtained by dividing the second image in the feature space, which allows the appearance judging device to more appropriately judge whether the judgment target is a non-defective product than when the image is not divided.

[0124] The appearance judgment device according to aspect 9 has the following features in addition to the features of the appearance judgment device according to any one of aspects 1 to 8. That is, in the appearance judgment device according to aspect 9, the first image is an image in which the influence of one or both of the specular reflection component and the shadow in the captured image of the object is reduced.

[0125] According to the above configuration, the appearance judgment device determines the similarity using the first image, which is an image in which the effects of one or both of the specular reflection component and the shadow are reduced in the captured image of the object. If specular reflection occurs or there is a shadow, the original color may not be captured, and an appropriate judgment result may not be obtained. However, according to the above configuration, the appearance judgment device determines the similarity using the image in which the effects of one or both of the specular reflection component and the shadow are reduced, so that it can more appropriately judge whether the object is a non-defective product.

[0126] The appearance judgment device according to aspect 10 has the following features in addition to the features of the appearance judgment device according to any one of aspects 1 to 9. That is, in the appearance judgment device according to aspect 10, one or both of the first image and the second image are images generated by photometric stereo.

[0127] According to the above configuration, the appearance judgment device uses one or both of the first and second images generated by photometric stereo to determine the similarity between the object to be judged and an object that has been confirmed to be a good product, and judges whether the object to be judged is a good product based on the determined similarity, thereby making it possible to more appropriately judge whether the object is a good product.

[0128] The appearance judgment device according to aspect 11 has the following features in addition to the features of the appearance judgment device according to any one of aspects 1 to 10. That is, in the appearance judgment device according to aspect 11, in the identification step, the processor performs a predetermined masking process on one or both of the first image and the second image, cuts out an image of an area including an area that is not filled in by the masking process, and determines the similarity using the cut-out image.

[0129] According to the above configuration, the appearance judgment device can improve the accuracy of the similarity determination process by cutting out an image of an area that includes areas that are not filled in by mask processing and determining the similarity using the cut-out image.

[0130] The appearance judgment device according to aspect 12 has the following features in addition to the features of the appearance judgment device according to any one of aspects 1 to 11. That is, in the appearance judgment device according to aspect 12, one or both of the first image and the second image are images generated using an image obtained by performing a predetermined mask processing on a captured image of the object.

[0131] According to the above configuration, the appearance determining device can remove background noise and the like from the captured image, thereby reducing the amount of calculation required for determination.

[0132] The appearance assessment method of aspect 13 includes an identification step in which one or more processors use a first image representing an optical image of the object and a second image representing the shape of the object to determine the similarity between the object to be assessed and an object that has been previously confirmed to be a good product, and a determination step in which the object to be assessed is determined to be a good product based on the similarity.

[0133] According to the above configuration, it is possible to more appropriately determine whether or not an object is a non-defective item.

[0134] The program of aspect 14 causes a computer to execute an identification step of identifying a similarity between an object to be judged and an object that has been previously confirmed to be a good product using a first image representing an optical image of the object and a second image representing the shape of the object, and a judgment step of judging whether the object to be judged is a good product based on the similarity.

[0135] According to the above configuration, it is possible to more appropriately determine whether or not an object is a non-defective item.

[0136] [Software implementation example] The functions of the appearance assessment device 10 (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and by a program for causing a computer to function as each control block of the device.

[0137] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0138] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0139] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0140] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may be executed by the control device or another device (for example, an edge computer or a cloud server).

[0141] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0142] 1 Appearance Judgment System 10 Appearance inspection device 11 Processors 12 Primary Memory 13 Secondary Memory 14 Input / Output Interface 15 Communication Interface 20 Camera 311~318, 321~328, 331~338 Lighting 40 Rotating table 50 Installation jig 60 Robot Arm 2. Object

Claims

1. One or more processors; The processor, a specifying step of specifying a similarity between the object to be judged and a plurality of objects that have been confirmed to be non-defective, using a first image that represents an optical image of the object and a second image that represents a shape of the object; a determination step of determining whether or not the object to be determined is a non-defective product based on the similarity; Mapping the first images of a plurality of objects previously confirmed to be non-defective onto a plurality of first points in a first feature space, and mapping the second images of the plurality of objects onto a plurality of second points in a second feature space; a mapping step of mapping the first image of the object to be determined to a third point in the first feature space and mapping the second image of the object to be determined to a fourth point in the second feature space; Run In the identifying step, the processor identifies the similarity based on a distance between the first point and the third point in the first feature space and a distance between the second point and the fourth point in the second feature space. An appearance assessment device comprising:

2. The second image is an image representing a normal direction at each point on a surface of an object.

2. The appearance determining device according to claim 1.

3. The first feature space and the second feature space are the same feature space. The appearance assessment device according to claim 1 or 2.

4. The second feature space is a feature space different from the first feature space. The appearance determining device according to claim 1 .

5. In the mapping step, a map that maps the first image to points in the first feature space and a map that maps the second image to points in the second feature space are maps constructed by metric learning.

5. The appearance determining device according to claim 1,

6. In the identification step, the processor determines, as the similarity, a value determined based on the density of points corresponding to a plurality of objects confirmed to be good products in the vicinity of a point corresponding to the object to be judged by Local Outlier Factor (LOF) or k-nearest neighbor, or the distance between the point corresponding to the object to be judged and a point corresponding to a plurality of objects confirmed to be good products.

6. The appearance determining device according to claim 1,

7. In the mapping step, the processor maps divided images obtained by dividing the first image into a plurality of regions onto points in a feature space corresponding to each of the plurality of regions, and maps divided images obtained by dividing the second image into a plurality of regions onto points in a feature space corresponding to each of the plurality of regions; In the specifying step, the similarity is specified for each of the divided images of the first image and the divided images of the second image; In the judging step, it is judged whether the object to be judged is a non-defective product based on the plurality of similarities identified in the identifying step. The appearance determining device according to claim 1 .

8. The first image is an image in which the influence of one or both of the specular reflection component and the shadow in the captured image of the object is reduced. The appearance determining device according to claim 1 .

9. One or both of the first image and the second image are images generated by photometric stereo.

9. The appearance determining device according to claim 1,

10. An appearance judgment device as described in any one of claims 1 to 9, wherein, in the identification step, the processor performs a predetermined masking process on one or both of the first image and the second image, cuts out an image of an area including areas that have not been filled in by the masking process, and identifies the similarity using the cut-out image.

11. One or both of the first image and the second image are images generated using an image obtained by performing a predetermined mask processing on the captured image of the object. The appearance determining device according to claim 1 .

12. One or more processors, a specifying step of specifying a similarity between the object to be judged and a plurality of objects that have been confirmed to be non-defective, using a first image that represents an optical image of the object and a second image that represents a shape of the object; a determination step of determining whether or not the object to be determined is a non-defective product based on the similarity; Mapping the first images of a plurality of objects previously confirmed to be non-defective onto a plurality of first points in a first feature space, and mapping the second images of the plurality of objects onto a plurality of second points in a second feature space; a mapping step of mapping the first image of the object to be determined to a third point in the first feature space and mapping the second image of the object to be determined to a fourth point in the second feature space; Including, In the identifying step, the processor identifies the similarity based on a distance between the first point and the third point in the first feature space and a distance between the second point and the fourth point in the second feature space. A method for determining appearance comprising:

13. A computer comprising: a specifying step of specifying a similarity between the object to be judged and a plurality of objects that have been confirmed to be non-defective, using a first image that represents an optical image of the object and a second image that represents a shape of the object; a determination step of determining whether or not the object to be determined is a non-defective product based on the similarity; Mapping the first images of a plurality of objects previously confirmed to be non-defective onto a plurality of first points in a first feature space, and mapping the second images of the plurality of objects onto a plurality of second points in a second feature space; a mapping step of mapping the first image of the object to be determined to a third point in the first feature space and mapping the second image of the object to be determined to a fourth point in the second feature space; A program for executing In the identifying step, the computer identifies the similarity based on a distance between the first point and the third point in the first feature space and a distance between the second point and the fourth point in the second feature space. A program characterized by:

Images