Visual inspection system, visual inspection device, visual inspection method, and program

Abstract

The visual inspection system (1) includes: a feature detection means for detecting the defect-likelihood score of an input visual image of an inspection object as a feature using a model to be verified that outputs a defect-likelihood score, which is an index representing the possibility that the input visual image of the inspection object is a defective product; a feature distribution information generation means for generating feature distribution information indicating the distribution of the defect-likelihood scores of the visual image; a pseudo image generation means for generating, from the original image, a pseudo image corresponding to a feature space with a sparse distribution density in the feature distribution information generated from the original image including good product images and defective product images in which the appearances of the good product and defective product to be inspected are respectively captured; and a pseudo image information generation means for generating pseudo image information indicating the image processing settings used when the pseudo image generation means generated the pseudo image; the pseudo image generation means references the pseudo image information, calculates image processing settings for generating a pseudo image corresponding to the feature space, and generates the pseudo image based on the calculated image processing settings.

Description

[Technical Field]

[0001] The present disclosure relates to a visual inspection system, a visual inspection device, a visual inspection method, and a program. [Background technology]

[0002] To build an appearance inspection system that inspects the appearance of industrial products, it is necessary to train a learning model to determine whether an industrial product is good or bad, for example, using multiple images of good and defective industrial products. However, collecting the images of good and defective industrial products necessary to generate a high-quality learning model requires a great deal of time and effort. Therefore, conventional appearance inspection systems use a method to efficiently automatically generate pseudo-image data from a small number of original image data.

[0003] For example, Patent Document 1 discloses a system that calculates a boundary surface from the feature distribution of original image data and generates pseudo images of good and bad products without affecting the rough position of the boundary surface. As a result, the pseudo image data can be used as learning data that maintains to some extent the bias of the feature distribution of the original image data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-125003 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Patent Document 1 generates pseudo images so as to maintain the bias of feature distribution to some extent. That is, even when it is desired to generate pseudo images mainly for a feature space with a sparse distribution density, pseudo images are generated not only for the feature space with a sparse distribution density but also for feature spaces other than the sparse distribution density. Therefore, it is difficult for the technology disclosed in Patent Document 1 to efficiently generate pseudo images corresponding to a feature space with a sparse distribution density.

[0006] The present disclosure has been made in consideration of the above-described circumstances, and aims to efficiently generate a pseudo image corresponding to a feature space with a sparse distribution density. [Means for solving the problem]

[0007] To achieve the above object, the appearance inspection system according to the present disclosure is an appearance inspection system for inspecting the appearance of an inspection object, and includes a feature detection means, a feature distribution information generation means, a pseudo image generation means, and a pseudo image information generation means. The feature detection means detects a defect-likelihood score of an input appearance image of the inspection object as a feature using a model to be verified that outputs a defect-likelihood score, which is an index representing the likelihood that the input appearance image of the inspection object is a defective product. The feature distribution information generation means generates feature distribution information indicating a distribution state of the defect-likelihood scores of the appearance image detected by the feature detection means. The pseudo image generation means generates a pseudo image from the original image, corresponding to a feature space with a sparse distribution density in the feature distribution information generated from the original image including good product images and defective product images captured of the appearances of good and defective products, respectively, of the inspection object. The pseudo image information generation means generates pseudo image information indicating the image processing settings used when the pseudo image generation means generated the pseudo image. The pseudo image generating means also refers to the pseudo image information, calculates image processing settings for generating a pseudo image corresponding to the feature space, and generates the pseudo image based on the calculated image processing settings. [Effects of the Invention]

[0008] According to the present disclosure, the visual inspection system references pseudo image information indicating image processing settings for the generated pseudo image, calculates image processing settings for generating a pseudo image corresponding to a feature space, and generates a pseudo image corresponding to a feature space with a sparse distribution density based on the calculated image processing settings. This makes it possible to efficiently generate a pseudo image corresponding to a feature space with a sparse distribution density. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a configuration example of a visual inspection system according to a first embodiment of the present disclosure. [Figure 2] Block diagram showing an example of the hardware configuration of a visual inspection device [Figure 3A] FIG. 10 is a diagram showing an example of a score region with sparse distribution density in feature distribution information from an original image. [Figure 3B] A diagram showing an example of class setting in feature distribution information from an original image. [Figure 4] FIG. 10 is a diagram showing an example of pseudo image information. [Figure 5A] FIG. 10 is a diagram showing an example of feature distribution information for an original image and a pseudo image. [Figure 5B] FIG. 10 is a diagram showing another example of feature distribution information based on an original image and a pseudo image. [Figure 6] Flowchart showing the flow of threshold setting processing [Figure 7] Flowchart showing the flow of pseudo image generation processing [Figure 8] A diagram for explaining the procedure for generating a pseudo image. [Figure 9] FIG. 10 is a block diagram illustrating a configuration example of a visual inspection system according to a second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present disclosure provides a visual inspection system, a visual inspection device, a visual inspection method, and a program for determining whether an object to be inspected is good or bad based on an appropriate threshold value.

[0011] (Embodiment 1) 1, the visual inspection system 1 according to the first embodiment of the present disclosure includes at least one visual inspection device 100 that performs a visual inspection of industrial products, and at least one operation terminal 200 that is operated by a user of the visual inspection system 1. The visual inspection device 100 and the operation terminal 200 are communicably connected via any network. The network may be configured, for example, by an Internet network, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a dedicated communication network, or the like, and may include various network relay devices such as antennas, gateways, routers, and hubs.

[0012] The visual inspection device 100 is a general-purpose computer device such as a personal computer or server computer, and determines whether the appearance of an industrial product is acceptable or not based on an appearance image of the industrial product being inspected. The visual inspection device 100 sets, for example, the number of pseudo images to be generated and a threshold value used for determining whether the product is acceptable or not, based on operations performed by a user of the visual inspection system 1 via an operation terminal 200. The visual inspection device 100 may be implemented as an industrial personal computer (IPC) or programmable logic controller (PLC) equipped with computing resources capable of executing each process and a memory area for storing multiple image data. The visual inspection device 100 may also be logically implemented using cloud computing.

[0013] The operation terminal 200 is a general-purpose computer device such as a personal computer, tablet terminal, or smartphone used by a user of the visual inspection system 1. The operation terminal 200 displays, for example, various information acquired from the visual inspection apparatus 100, an operation image for specifying a threshold value used for determining pass / fail, etc. The operation terminal 200 also supplies an operation signal to the visual inspection apparatus 100 based on an operation by the user.

[0014] (Example of hardware configuration of visual inspection device 100) Next, we will explain an example of the hardware configuration of the visual inspection device 100. As shown in Fig. 2, the visual inspection device 100 physically includes a processor 101, a memory 102, a storage 103, a display device 104, an input / output interface (input / output I / F (InterFace)) 105, and a communication interface (communication I / F) 106. These components are electrically connected to each other via a bus line 107.

[0015] The processor 101 is an arithmetic device that controls the overall operation of the visual inspection apparatus 100. The processor 101 is a general-purpose processor such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit). The processor 101 is not limited to a general-purpose processor, and may be a dedicated processor configured using an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0016] The memory 102 is a main storage device and includes a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 101 reads programs and various data from the ROM or the storage 103 onto the RAM and executes processing, thereby realizing overall control and functions of the appearance inspection apparatus 100.

[0017] The storage 103 is an auxiliary storage device that stores programs and various data required for program execution, and includes non-volatile storage devices such as HDDs (Hard Disk Drives) and SSDs (Solid State Drives). The storage 103 stores, for example, an OS (Operating System), which is basic software that controls the entire visual inspection apparatus 100, and applications that run on the OS and provide various functions. The storage 103 also stores a verification target model, multiple image data, and the like. Note that some of the programs and various data required for program execution may be stored in ROM.

[0018] The display device 104 is an image display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), or an organic EL (Electro-Luminescence) display, and displays various images under the control of the processor 101. For example, the display device 104 displays an image related to the pseudo-image generation process on a display screen. The display device 104 may be connected to the input / output interface 105 for use.

[0019] The input / output interface (input / output I / F) 105 is an interface for connecting to input devices such as a keyboard and a mouse for inputting operation signals, and output devices such as a speaker for outputting audio data. The input / output interface 105, for example, receives operation data input by a user via the input device and outputs data such as moving images and audio to the output device. Note that if user operation is not required, the visual inspection apparatus 100 does not need to include the display device 104 and the input / output interface 105.

[0020] The communication interface (communication I / F) 106 is an interface that connects to a communication network and enables the visual inspection apparatus 100 to perform data communication with the operation terminal 200. The communication interface 106 includes, for example, a network board, a LAN module, and the like.

[0021] (Example of functional configuration of visual inspection device 100) Next, we will explain the functional configuration of the visual inspection device 100. The visual inspection device 100 functionally comprises a control unit 110, a storage unit 120, a display unit 130, an input / output unit 140, and a communication unit 150, as shown in FIG.

[0022] The control unit 110 controls the overall functions of the appearance inspection apparatus 100. The control unit 110 is realized, for example, by a process in which the processor 101 shown in FIG. 2 executes a program loaded from the storage 103 to the RAM of the memory 102. The control unit 110 may also be realized by hardware using a dedicated integrated circuit such as an ASIC or FPGA. The control unit 110 includes a feature amount detection unit 111 that detects feature amounts of the appearance image, a feature amount distribution information generation unit 112 that generates feature amount distribution information indicating the feature amount distribution of the appearance image, a pseudo image generation unit 113 that generates a pseudo image based on the original image, a pseudo image information generation unit 114 that generates pseudo image information indicating image processing settings for the pseudo image, and a determination unit 115 that determines whether the appearance of an industrial product is good or bad.

[0023] The feature detection unit 111 detects feature quantities of the appearance image of the industrial product to be inspected. The feature detection unit 111 inputs the appearance image of the industrial product to be inspected into the verification target model stored in the storage unit 120 and detects the defectiveness score obtained as a feature quantity of the appearance image of the industrial product. The defectiveness score is an index representing the possibility of a defective product, and is a prediction score in which a higher value indicates that the inspection target is more likely to be defective. In the following description, the defectiveness score may be simply referred to as a "score." The defectiveness score is a scalar value within a certain range, for example, from 1 to 100. Note that the defectiveness score may be expressed as an N-dimensional vector associated with N label candidates assigned to the input image, where N is a natural number. Note that instead of the defectiveness score, a "goodness score," which is an index representing the possibility of a good product, may be used to detect feature quantities of the appearance image of the industrial product. The feature detection unit 111 is an example of a feature detection means.

[0024] The feature distribution information generation unit 112 generates feature distribution information indicating the distribution of features of the appearance image of the industrial product. The feature distribution information generation unit 112 generates feature distribution information indicating the distribution of defect-likeness scores obtained by inputting each of original images, including, for example, good product images of good industrial products and defective product images of defective industrial products, into the model to be verified. The feature distribution information generation unit 112 also generates feature distribution information indicating the distribution of defect-likeness scores obtained by inputting pseudo images generated by the pseudo image generation unit 113 into the model to be verified, in addition to the distribution of defect-likeness scores of the original images. The feature distribution information generation unit 112 generates the feature distribution information in the form of a histogram so that it can be displayed in a visualized format that is easy for the user to visually understand. The feature distribution information generation unit 112 is an example of a feature distribution information generation means.

[0025] The feature distribution information generating unit 112 generates feature distribution information indicating the distribution of defectiveness scores obtained by inputting original images into a verification target model, for example, in the form of a histogram as shown in Fig. 3A. In the histogram shown in Fig. 3A, the classes of non-defective images are hatched with an upper right diagonal line, and the classes of defective images are hatched with a lower right diagonal line.

[0026] The pseudo image generation unit 113 generates a pseudo image based on an original image. Specifically, the pseudo image generation unit 113 generates a pseudo image from the original image, which is an appearance image corresponding to a feature space with a sparse distribution density in feature distribution information generated from original images including good product images and defective product images that capture the appearances of good and defective industrial products, respectively. When generating a pseudo image, the pseudo image generation unit 113 randomly selects, for example, one good product image and one defective product image from the original images stored in the storage unit 120, and generates a pseudo image by combining these two images. The pseudo image generation unit 113 reselects the images when generating the next pseudo image. The pseudo image generation unit 113 is an example of a pseudo image generation means.

[0027] The pseudo image generation unit 113 divides a feature space with a sparse distribution density in the feature distribution information generated from the original image, for example, a score region where the defect-likeness scores of non-defective images and defective images are not distributed in the histogram shown in FIG. 3B, into M equal parts using an arbitrary natural number M, and sets M classes. The pseudo image generation unit 113 generates a pseudo image having a defect-likeness score belonging to each of the set classes. That is, the pseudo image generation unit 113 references the pseudo image information generated by the pseudo image information generation unit 114, calculates image processing settings for generating a pseudo image corresponding to the sparsely distributed feature space, and generates a pseudo image using the calculated image processing settings.

[0028] The pseudo image information generation unit 114 generates pseudo image information indicating the image processing settings of the pseudo image. The pseudo image information is information indicating what kind of image processing the pseudo image generation unit 113 used to generate the pseudo image. The pseudo image information can be represented in a table format as shown in FIG. 4, and includes, for example, the following items: "pseudo image identification information," "original image identification information," "likelihood score of defective product," and "processing data." The "pseudo image identification information" is information for identifying the pseudo image. The "original image identification information" is information for identifying the good product image and the defective product image used to generate the pseudo image. The "defective product likelihood score" is a defective product likelihood score obtained by inputting the pseudo image into the model to be verified. The "processing data" indicates the image processing settings used when generating the pseudo image, including, for example, a "paste area" indicating the area in the good-product image where the defective portion is pasted, a "cut area" indicating the area in the defective-product image where the defective portion is cut out, a "magnification factor" indicating the ratio of enlargement or reduction of the defective portion, a "rotation angle" indicating the angle when the defective portion is rotated around a certain point, and a "hue" indicating the hue, brightness, and saturation of the paste area and the defective portion. In the pseudo image information shown in FIG. 4, the "paste area" and "cut area" are expressed by the coordinates of the lower left corner and the upper right corner of each area, i.e., two points on the diagonal line of each area, when the lower left corner of the good-product image and the defective-product image is set as the origin (0,0). Note that the paste area and cut area are not limited to rectangular areas; they may be circular areas specified by a center point and a radius, or polygonal areas specified by multiple points. The pseudo image information is updated each time the pseudo image generating unit 113 generates a pseudo image and stored in the storage unit 120. The pseudo image information generating unit 114 is an example of a pseudo image information generating means.

[0029] Here, a method will be described in which the pseudo image generation unit 113 refers to the pseudo image information and calculates image processing settings for generating a pseudo image corresponding to a feature space with a sparse distribution density. The pseudo image generation unit 113 calculates the image processing settings using, for example, Bayesian optimization. More specifically, the pseudo image generation unit 113 calculates the image processing settings by using j (j is a natural number) parameters of the processing data of the i-th image processing setting in the pseudo image information including i (i is a natural number) image processing settings, such as "paste area", "cut area", "magnification", "rotation angle", and "color" shown in FIG. 4, as Z i,1 ,Z i,2 ,…,Z i,j The defect likelihood score of the i-th pseudo image calculated based on the parameters of the processing data of the i-th image processing setting is F i,j (Z i,1 ,Z i,2 ,…,Z i,j ), the target value of the defect likelihood score corresponding to the sparse feature space is S, and the objective function is G i,j (Z i,1 ,Z i,2 ,…,Z i,j The pseudo image generation unit 113 calculates the defect likelihood score F i+1,j (Z i+1,1 ,Z i+1,2 ,…,Z i+1,j ) is close to the target value S. i+1,1 ,Z i+1,2 ,…,Z i+1,j That is, the pseudo image generating unit 113 calculates G i+1,j (Z i+1,1 ,Z i+1,2 ,…,Z i+1,j ) approaches 0. i+1,1 ,Z i+1,2 ,…,Z i+1,j Calculate. G i+1,j (Z i+1,1 ,Z i+1,2 ,…,Z i+1,j ) = |F i+1,j (Z i+1,1 ,Z i+1,2 ,…,Z i+1,j) - S| ……(1)

[0030] The defect likelihood score F for the actual i+1th pseudo image i+1,j (Z i+1,1 ,Z i+1,2 ,…,Z i+1,j ) is the parameter Z of the processing data for the image processing settings of the i+1th pseudo image i+1,1 ,Z i+1,2 ,…,Z i+1,j is calculated, and the parameter Z of the processing data of the calculated image processing setting is i+1,1 ,Z i+1,2 ,…,Z i+1,j After the (i+1)th pseudo image is generated, the feature detection unit 111 reads the model to be verified stored in the storage unit 120 and inputs the (i+1)th pseudo image into the model to be verified to obtain it. Here, if there is no pseudo image information for the pseudo image generation unit 113 to refer to, such as when generating a pseudo image for the first time, the pseudo image generation unit 113 can randomly calculate initial image processing settings. Alternatively, the user may set the initial image processing settings in advance, or the user may determine in advance the range of parameters of the processing data for the initial image processing settings when the pseudo image generation unit 113 randomly calculates the initial image processing settings. Even in such cases, the amount of pseudo image information that can be referenced increases as pseudo image generation is repeated, so that pseudo images corresponding to feature spaces with sparse distribution density can be generated without any particular problems.

[0031] Furthermore, it is preferable that the pseudo image generated by the pseudo image generation unit 113 is a new pseudo image that does not overlap with any pseudo image already generated. That is, it is preferable that the pseudo image generation unit 113 references the pseudo image information generated by the pseudo image information generation unit 114, calculates new image processing settings for generating a pseudo image that corresponds to a feature space with a sparse distribution density and does not overlap with any pseudo image already generated, and generates the new pseudo image using the calculated image processing settings. Note that "does not overlap" in this case means that the image processing settings do not overlap with any pseudo image already generated. Even if a pseudo image is generated using image processing settings different from those of the pseudo image already generated, the resulting defectiveness score may overlap with that of the pseudo image already generated. Details of the pseudo image generation procedure will be described later.

[0032] Returning to FIG. 1, the determination unit 115 determines whether the appearance of an industrial product is good or bad. The determination unit 115 sets a threshold for the feature specified by the user, i.e., the defectiveness score, according to the distribution of the original image and the pseudo image generated by the feature distribution information generation unit 112, and determines whether the appearance of the industrial product being inspected is good or bad based on the set threshold. For example, if the threshold for the defectiveness score is set to 60, the product is determined to be good if its defectiveness score is 1 to 59, and is determined to be defective if its score is 60 to 100. The threshold may be two or more values ​​rather than one value. For example, if a defectiveness score of 1 to 40 is determined to be good and a score of 60 to 100 is determined to be defective, the thresholds may be set to 41 and 59, and products 41 to 59 may be classified as "impossible to determine whether they are good or bad." The determination unit 115 is an example of a determination means.

[0033] The storage unit 120 stores the control program executed by the control unit 110, various data, etc. The storage unit 120 is realized, for example, by the memory 102 and storage 103 shown in Fig. 2. The storage unit 120 has, for example, a verification target model storage unit 121, an original image storage unit 122, a pseudo image storage unit 123, and a pseudo image information storage unit 124.

[0034] The verification target model storage unit 121 stores a verification target model. The verification target model is a trained model that outputs a defect likelihood score, which is an index representing the possibility that an inputted appearance image of an industrial product is a defective product. The verification target model is generated using a known machine learning algorithm, such as a convolutional neural network (CNN) that locally extracts feature amounts.

[0035] The original image storage unit 122 stores original images obtained by capturing images of the exterior of industrial products. The original images include non-defective product images, which are captured images of non-defective products, and defective product images, which are captured images of defective products. The pseudo image storage unit 123 stores pseudo images generated by the pseudo image generation unit. The pseudo image information storage unit 124 stores pseudo image information that indicates image processing settings for the pseudo images generated by the pseudo image information generation unit 114.

[0036] The display unit 130 displays various pieces of information under the control of the control unit 110. The display unit 130 is realized, for example, by the display device 104 shown in FIG. 2. The input / output unit 140 inputs various pieces of data from an input device and outputs various pieces of data to an output device under the control of the control unit 110. The input / output unit 140 is realized, for example, by the input / output interface 105 shown in FIG. 2. The communication unit 150 transmits and receives various pieces of data to the operation terminal 200 under the control of the control unit 110. The communication unit 150 is realized, for example, by the communication interface 106 shown in FIG. 2. The communication unit 150 transmits, for example, feature amount distribution information to the operation terminal 200 and receives various operation instructions from the operation terminal 200. The communication unit 150 is an example of a threshold acquisition means.

[0037] (Example of functional configuration of operation terminal 200) Next, an example of the functional configuration of the operation terminal 200 will be described with reference to Fig. 1. Note that an example of the hardware configuration of the operation terminal 200 is similar to the example of the hardware configuration of the appearance inspection apparatus 100 shown in Fig. 2, and therefore description thereof will be omitted. The operation terminal 200 includes a control unit 210, a storage unit 220, a display unit 230, an input / output unit 240, and a communication unit 250.

[0038] The control unit 210 is realized by a processor and a memory, and controls the overall functions of the operation terminal 200. The control unit 210, for example, controls the display of various information on the display unit 230, controls input and output on the input / output unit 240, and controls communication between the communication unit 250 and the appearance inspection apparatus 100.

[0039] The memory unit 220 stores a control program executed by the control unit 210, various data, and the like. The memory unit 220 stores, for example, feature distribution information provided by the visual inspection apparatus 100. The display unit 230 displays various information under the control of the control unit 210. The display unit 230 displays, for example, feature distribution information provided by the visual inspection apparatus 100. The input / output unit 240 inputs various data from an input device and outputs the data to an output device under the control of the control unit 210. The input / output unit 240 accepts user instructions such as starting pseudo image generation, the number of pseudo images to be generated, and a pass / fail judgment threshold. The communication unit 250 transmits and receives various data to the operation terminal 200 under the control of the control unit 210. The communication unit 250 receives, for example, feature distribution information from the visual inspection apparatus 100 and transmits various operation instructions to the visual inspection apparatus 100. The input / output unit 240 and the communication unit 250 are examples of threshold acquisition means.

[0040] The display unit 230 displays feature distribution information expressed as a histogram, as shown in FIG. 5A . When the user hovers the mouse cursor over each class of the pseudo image hatched with dots, a pop-up displays the pseudo image and the resemblance score corresponding to that class. That is, the display unit 230 displays the feature distribution information, the pseudo image, and the resemblance score in association with each other. Note that the display of the pseudo image and the resemblance score may be in any display format, regardless of the pop-up display, as long as the pseudo image and the resemblance score are displayed in association with each other. By checking the pseudo image and the resemblance score in the pop-up display, the user can determine in which distribution of feature values ​​the displayed pseudo image is located, and can set an appropriate threshold for determining whether the product is good or bad based on the displayed content. Note that the display unit 230 is an example of a display means.

[0041] When the output of the model to be inspected is feature information expressed as an N-dimensional vector, the display unit 230 displays feature distribution information to which a pull-down display has been added, which allows the contents of the pop-up display of each class of the histogram to be switched for each feature, as shown in FIG. 5B.

[0042] Next, we will explain the operation of the visual inspection system 1 having the above configuration. The flowchart shown in Fig. 6 is a flowchart related to a threshold setting process executed by the visual inspection apparatus 100, which is an example of the operation of the visual inspection system 1. This operation corresponds to the visual inspection method of the visual inspection apparatus 100 according to this embodiment.

[0043] The control unit 110 of the visual inspection apparatus 100 starts the threshold setting process in response to, for example, an operation input of a start instruction by the user of the operation terminal 200.

[0044] When the threshold setting process starts, first, the control unit 110 generates feature distribution information for the original image (step S101). The feature distribution information generation unit 112 of the control unit 110 reads out the good product images and defective product images included in the original image and the model to be verified from the storage unit 120. Then, the feature distribution information generation unit 112 inputs the good product images and defective product images to the model to be verified, respectively, and acquires a defect likelihood score for each image. The feature distribution information generation unit 112 generates feature distribution information for the original image based on the defect likelihood scores of each of the acquired good product images and defective product images. As the feature distribution information, the feature distribution information generation unit 112 generates, for example, a histogram that visually represents the distribution of defect likelihood scores.

[0045] Next, the control unit 110 identifies a score region in the histogram where the distribution density of the defectiveness scores is sparse, and divides the region (step S102). The feature amount distribution information generation unit 112 identifies a score region in the histogram showing the distribution state of the defectiveness scores of the original image where the distribution density of the defectiveness scores is sparse, and divides the region into M score intervals, where M is a natural number.

[0046] Next, the control unit 110 sets the number of pseudo images to be generated corresponding to score regions with sparse distribution densities (step S103). The feature distribution information generation unit 112 sets the number of pseudo images to be generated, for example, according to a setting instruction from the user. This makes it possible to uniformly generate the number of pseudo images desired by the user for score regions with sparse distribution densities. Here, the number of pseudo images to be generated may be set for each score section. In this case, it is possible to generate the number of pseudo images desired by the user for each score section for score regions with sparse distribution densities.

[0047] After setting the number of pseudo images to be generated in step S103, the control unit 110 executes pseudo image generation processing (step S104).

[0048] Here, the pseudo image generation process (step S104), which is a subroutine of the threshold setting process, will be described with reference to FIG.

[0049] When the control unit 110 starts the pseudo image generation process, it first selects a good product image and a defective product image (step S201). The pseudo image generation unit 113 of the control unit 110 selects one good product image and one defective product image from the original images stored in the original image storage unit 122 of the storage unit 120, which are to be used to generate the pseudo image.

[0050] Next, the control unit 110 acquires a defective part image from the defective product image (step S202). The pseudo image generation unit 113 acquires a defective part image DPa by, for example, cutting out an image capturing the defective part from the defective product image DIa shown in FIG. 8. When cutting out the defective part image DPa from the defective product image DIa, the pseudo image generation unit 113 recognizes the defective part area by, for example, edge detection, and then trims the recognized area to acquire the defective part image DPa capturing only the defective part. Note that, instead of executing the process of step S202, the pseudo image generation unit 113 may use a defective part image DPa prepared in advance by a user, as necessary. That is, the user may visually detect the defective part in the defective product image DIa in advance, manually cut out the defective part from the defective product image DIa, and then perform appropriate processing, such as trimming, to obtain an image capturing the defective part, which may then be used as the defective part image DPa in subsequent processes.

[0051] Next, the control unit 110 extracts feature amounts of the non-defective image and the defective portion image (step S203). The pseudo image generation unit 113 extracts color elements such as hue, brightness, and saturation of the non-defective image GIa and the defective portion image DPa shown in FIG.

[0052] Subsequently, the control unit 110 refers to the pseudo image information (step S204). The pseudo image generating unit 113 reads out and refers to the pseudo image information stored in the pseudo image information storage unit 124 of the storage unit 120.

[0053] Next, the control unit 110 calculates image processing settings (step S205). The pseudo image generation unit 113 calculates image processing settings for generating a pseudo image corresponding to a feature space with a sparse distribution density, based on the feature amounts of the good product image and the defective part image extracted in step S203 and the pseudo image information read out in step S204. The pseudo image generation unit 113 calculates, as the image processing settings, for example, an attachment area where the defective part image is to be attached in the good product image GIa shown in FIG. 8, a magnification and a rotation angle of the defective part image DPa, etc.

[0054] Next, the control unit 110 processes the non-defective product image and the defective area image based on the image processing settings calculated in step S205 (step S206). The pseudo image generation unit 113 generates a non-defective product image GIb by removing unnecessary background around the industrial product, for example, by extracting the outline of the industrial product in the non-defective product image GIa shown in Fig. 8. The pseudo image generation unit 113 also performs deformations such as rotation and enlargement / reduction, and color adjustments on the defective area image DPa to match the pasted area of ​​the non-defective product image GIb shown in Fig. 8, for example, to generate the defective area image DPb.

[0055] Next, the control unit 110 combines the good product image and the defective part image (step S207). The pseudo image generation unit 113 combines the good product image and the defective part image processed based on the image processing settings in step S206. The pseudo image generation unit 113 generates a pseudo image PIa by combining, for example, the good product image GIb and the defective part image DPb shown in FIG. 8. Furthermore, the pseudo image generation unit 113 generates a more natural pseudo image PIb by adjusting the transmittance, color, etc. of the edge portion of the combined portion. After executing the processing of step S207, the control unit 110 ends the pseudo image generation processing.

[0056] 6, after executing the pseudo image generation process in step S104, the control unit 110 acquires a defectiveness score of the new pseudo image (step S105). The feature amount detection unit 111 of the control unit 110 reads out the model to be verified stored in the storage unit 120, inputs the newly generated pseudo image into the model to be verified, and acquires its defectiveness score.

[0057] Next, the control unit 110 updates the pseudo image information (step S106). The pseudo image information generation unit 114 of the control unit 110 generates pseudo image information indicating the image processing settings and defectiveness score of the newly generated pseudo image, and adds it to the existing pseudo image information to update the pseudo image information.

[0058] After executing the process of step S106, control unit 110 determines whether the set number of pseudo images to be generated has been reached (step S107). Control unit 110, for example, refers to a counter that counts the number of pseudo images to be generated, and determines whether the set number of pseudo images to be generated has been reached. If it is determined that the set number of pseudo images to be generated has not been reached (step S107: NO), control unit 110 returns the process to step S104, and repeats the series of processes from step S104 to step S107.

[0059] On the other hand, if it is determined that the set number of pseudo images to be generated has been reached (step S107: YES), control unit 110 generates feature amount distribution information based on the original image and the pseudo images (step S108). Feature amount distribution information generation unit 112 of control unit 110 adds, for example, the distribution of defectiveness scores of the pseudo images generated by pseudo image generation unit 113 to the feature amount distribution information based on the original image generated in step S101, thereby generating new feature amount distribution information based on the original image and the pseudo images.

[0060] Next, the control unit 110 supplies the feature distribution information to the operation terminal 200 (step S109). The control unit 110 transmits the feature distribution information based on the original image and the pseudo image generated by the feature distribution information generation unit 112 to the operation terminal 200 via the communication unit 150.

[0061] Next, the control unit 110 sets a threshold value for determining whether the image is good or bad (step S110). The control unit 110 sets a threshold value for determining whether the image is good or bad, based on a threshold setting instruction input by the user via the input / output unit 240 of the operation terminal 200, in accordance with the feature distribution information supplied to the operation terminal 200.

[0062] After executing the process of step S110, the control unit 110 ends the threshold setting process. The determination unit 115 of the control unit 110 determines whether the industrial product is good or bad by comparing the defectiveness score obtained by inputting the appearance image of the industrial product to be inspected into the model to be verified with the threshold set by the threshold setting process.

[0063] As described above, in the visual inspection system 1 according to this embodiment, the visual inspection device 100 uses a verification target model that outputs a defect likelihood score for an input image of the appearance of an industrial product to detect, as features, the defect likelihood scores of pre-prepared images of the appearances of a good product and a defective product, respectively, and generates, from the original image, a pseudo image corresponding to a feature space with a sparse distribution density in feature distribution information indicating the distribution of the feature values. The visual inspection device 100 also generates pseudo image information indicating the image processing settings used when the pseudo image was generated, calculates image processing settings for generating the pseudo image corresponding to the feature space by referring to the pseudo image information, and generates the pseudo image based on the calculated image processing settings. This allows the visual inspection device 100 to efficiently and effectively generate a pseudo image corresponding to a sparsely distributed feature space. Furthermore, by generating a pseudo image corresponding to a feature space with a sparse distribution density, the visual inspection device 100 can naturally obtain a pseudo image that approximates a marginal sample, which is a product sample that indicates the limit of whether an industrial product is "good" or "defective."

[0064] Furthermore, the visual inspection device 100 displays feature distribution information, pseudo images, and defectiveness scores in association with each other, accepts an input operation of a threshold for determining the pass / fail of an inspection object, and determines the pass / fail of the inspection object based on this threshold. This allows the user to easily determine to which feature distribution the pseudo image approximating the marginal sample belongs, and also allows the user to appropriately set the threshold for pass / fail determination. Therefore, the visual inspection system 1 according to this embodiment can improve the accuracy of the visual inspection of the inspection object.

[0065] The visual inspection system 1 according to this embodiment generates multiple pseudo images with uniform feature values ​​of the boundary between good and defective products that are not visible in the original image, based on the feature value distribution of the original image. Therefore, the generated pseudo images can be used as verification data with a uniform distribution. Furthermore, by using the pseudo images as learning information, it is possible to improve the inference performance of the model to be verified, which is a trained model.

[0066] (Embodiment 2) In the above-described first embodiment, the visual inspection system 1 has been described as including the visual inspection device 100 and the operation terminal 200. However, the device configuration of the visual inspection system 1 is not limited to this, and for example, either one of the devices may have all of the functions that the visual inspection device 100 and the operation terminal 200 each have.

[0067] 9, the appearance inspection system 1 may include only an appearance inspection device 300 that combines the functions of the appearance inspection device 100 and the operation terminal 200 described in the first embodiment. The appearance inspection device 300 is an apparatus in which the functions of the appearance inspection device 100 are added to the operation terminal 200. It is desirable that the appearance inspection device 300 has sufficient processing capabilities and storage resources equivalent to those of the appearance inspection device 100.

[0068] The appearance inspection apparatus 300 functionally includes a control unit 310, a storage unit 320, a display unit 330, an input / output unit 340, and a communication unit 350. The control unit 310 includes a feature detection unit 311, a feature distribution information generation unit 312, a pseudo image generation unit 313, a pseudo image information generation unit 314, and a determination unit 315. The storage unit 320 includes a verification target model storage unit 321, an original image storage unit 322, a pseudo image storage unit 323, and a pseudo image information storage unit 324.

[0069] The control unit 310, the memory unit 320, the display unit 330, the input / output unit 340, and the communication unit 350 are similar to the control unit 110, the memory unit 120, the display unit 130, the input / output unit 140, and the communication unit 150 in the appearance inspection apparatus 100 of the first embodiment, and therefore detailed description thereof will be omitted. The feature amount detection unit 311, the feature amount distribution information generation unit 312, the pseudo image generation unit 313, the pseudo image information generation unit 314, and the determination unit 315 included in the control unit 310 are similar to the feature amount detection unit 111, the feature amount distribution information generation unit 112, the pseudo image generation unit 113, the pseudo image information generation unit 114, and the determination unit 115 in the first embodiment. Furthermore, the verification target model memory unit 321, original image memory unit 322, pseudo image memory unit 323, and pseudo image information memory unit 324 of the memory unit 320 are the same as the verification target model memory unit 121, original image memory unit 122, pseudo image memory unit 123, and pseudo image information memory unit 124 of embodiment 1.

[0070] With such an apparatus configuration, a user of the appearance inspection system 1 can operate the functions of the entire system on the appearance inspection apparatus 300 without using the appearance inspection apparatus 100. Furthermore, similar to the first embodiment, the appearance inspection apparatus 300 can improve the accuracy of the appearance inspection of the inspection target, and can also reduce the number of devices and eliminate communication delays.

[0071] The present disclosure is not limited to the above-described embodiments, and various modifications and applications are possible without departing from the gist of the present disclosure.

[0072] In the above embodiment, the visual inspection of the industrial product to be inspected is performed using a model to be verified that is stored in advance in the visual inspection apparatus 100. However, this is not limiting. For example, a trained model that has undergone machine learning using training information generated based on inspection information acquired from an inspection information source provided in another visual inspection system may be imported as the model to be verified in the visual inspection system 1, and the model to be verified may be updated by re-learning using training information stored in the visual inspection apparatus 100 or acquired from an external source.

[0073] In the above embodiment, for example, the control program executed by the processor 101 that realizes the control unit 110 of the visual inspection apparatus 100 is stored in advance mainly in the storage 103. However, the present disclosure is not limited to this, and the control program for executing the various processes described above may be implemented in an existing general-purpose computer, framework, workstation, etc., to function as a device equivalent to the visual inspection apparatus 100 according to the above embodiment. The same applies to the control program executed by the processor that realizes the control unit 210 of the operation terminal 200.

[0074] Such programs may be provided in any manner, for example, by storing them on a computer-readable recording medium (such as a flexible disk, a CD (Compact Disc)-ROM, or a DVD (Digital Versatile Disc)-ROM) and distributing them, or by storing the programs in storage on a network such as the Internet and providing them by downloading them.

[0075] Furthermore, when the above processing is performed by sharing the work between the OS and the application program, or by cooperation between the OS and the application program, only the application program may be stored on a recording medium, storage, etc. It is also possible to superimpose the program on a carrier wave and distribute it over a network. For example, the program may be posted on a bulletin board system (BBS) on a network and distributed over the network. The program may then be launched and run under the control of the OS in the same way as other application programs, thereby enabling the above processing to be performed.

[0076] The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are considered to be within the scope of the present disclosure. [Explanation of symbols]

[0077] 1...visual inspection system, 100...visual inspection device, 101...processor, 102...memory, 103...storage, 104...display device, 105...input / output interface, 106...communication interface, 107...bus line, 110...control unit, 111...feature detection unit, 112...feature distribution information generation unit, 113...pseudo image generation unit, 114...pseudo image information generation unit, 115...judgment unit, 120...storage unit, 121...verification target model storage unit, 122...original image storage unit, 123...pseudo image storage unit, 124...pseudo image information storage unit, 130...display unit, 140...input / output unit, 150...communication unit, 200...operation terminal End, 210...control unit, 220...memory unit, 230...display unit, 240...input / output unit, 250...communication unit, 300...appearance inspection device, 310...control unit, 311...feature detection unit, 312...feature distribution information generation unit, 313...pseudo image generation unit, 314...pseudo image information generation unit, 315...judgment unit, 320...memory unit, 321...verification target model memory unit, 322...original image memory unit, 323...pseudo image memory unit, 324...pseudo image information memory unit, 330...display unit, 340...input / output unit, 350...communication unit, DIa...defective product image, DPa, DPb...defective area image, GIa, GIb...good product image, PIa, PIb...pseudo image.

Claims

1. An appearance inspection system for inspecting the appearance of an inspection object, a feature quantity detection means for detecting a defect-likelihood score of the input appearance image of the inspection object as a feature quantity, using a verification target model that outputs a defect-likelihood score, which is an index representing the possibility that the input appearance image of the inspection object is a defective product; a feature amount distribution information generating means for generating feature amount distribution information indicating a distribution state of the defectiveness scores of the appearance images detected by the feature amount detecting means; a pseudo image generating means for generating a pseudo image corresponding to a feature space with a sparse distribution density from original images containing good product images and defective product images obtained by capturing the appearances of the good products and defective products to be inspected, in feature distribution information generated from the original images; a pseudo image information generating means for generating pseudo image information indicating image processing settings when the pseudo image generating means generated the pseudo image, the pseudo image generating means refers to the pseudo image information, calculates image processing settings for generating a pseudo image corresponding to the feature amount space, and generates the pseudo image based on the calculated image processing settings. Visual inspection system.

2. a display means for displaying the feature distribution information, the pseudo image, and the defective product likelihood score in association with each other; a threshold value acquiring means for accepting an input operation of a threshold value for determining whether the inspection object is good or bad and acquiring the threshold value; and a determination means for determining whether the inspection object is good or bad based on the threshold value. The visual inspection system according to claim 1 .

3. the pseudo image generating means calculates new image processing settings for generating a pseudo image that does not overlap with any pseudo image already generated, and generates the new pseudo image based on the calculated image processing settings; 3. The visual inspection system according to claim 1.

4. the feature distribution information generating means generates the feature distribution information as a histogram; 3. The visual inspection system according to claim 1.

5. the pseudo image generating means adjusts at least one element of the area, color, and rotation angle included in the image processing settings to paste the defective portion included in the defective product image onto the paste portion of the non-defective product image, thereby generating the pseudo image.

3. The visual inspection system according to claim 1.

6. An appearance inspection device that inspects the appearance of an inspection object, a feature quantity detection means for detecting a defect-likelihood score of the input appearance image of the inspection object as a feature quantity, using a verification target model that outputs a defect-likelihood score, which is an index representing the possibility that the input appearance image of the inspection object is a defective product; a feature amount distribution information generating means for generating feature amount distribution information indicating a distribution state of the defectiveness scores of the appearance images detected by the feature amount detecting means; a pseudo image generating means for generating a pseudo image corresponding to a feature space with a sparse distribution density from original images containing good product images and defective product images obtained by capturing the appearances of the good products and defective products to be inspected, in feature distribution information generated from the original images; a pseudo image information generating means for generating pseudo image information indicating image processing settings when the pseudo image generating means generated the pseudo image, the pseudo image generating means refers to the pseudo image information, calculates image processing settings for generating a pseudo image corresponding to the feature amount space, and generates the pseudo image based on the calculated image processing settings. Visual inspection equipment.

7. An appearance inspection method performed by an appearance inspection system that inspects the appearance of an inspection object, comprising: The computer included in the visual inspection system detecting a defect-likelihood score of the input appearance image of the inspection object as a feature quantity using a verification target model that outputs a defect-likelihood score, which is an index representing the possibility that the input appearance image of the inspection object is a defective product; generating feature distribution information indicating a distribution state of the defectiveness scores of the appearance images; generating a pseudo image corresponding to a feature space with a sparse distribution density from original images containing good product images and defective product images obtained by capturing the appearances of the good product and defective product to be inspected, generating pseudo-image information indicating image processing settings used to generate the pseudo-image; When generating a pseudo image corresponding to the feature space, the computer refers to the pseudo image information, calculates image processing settings for generating the pseudo image corresponding to the feature space, and generates the pseudo image based on the calculated image processing settings. Visual inspection method.

8. Computer, a feature detection means for detecting a defect-likelihood score of an input appearance image of an inspection object as a feature, using a verification target model that outputs a defect-likelihood score, which is an index representing the possibility that the input appearance image of the inspection object is a defective product; a feature amount distribution information generating means for generating feature amount distribution information indicating a distribution state of the defectiveness scores of the appearance images detected by the feature amount detecting means; a pseudo image generating means for generating, from the original images, a pseudo image corresponding to a feature space with a sparse distribution density in feature distribution information generated from the original images including good product images and defective product images obtained by capturing the appearances of the good product and defective product to be inspected, the pseudo image generating means functions as a pseudo image information generating means for generating pseudo image information indicating image processing settings when the pseudo image generating means generated the pseudo image; the pseudo image generating means refers to the pseudo image information, calculates the image processing settings for generating a pseudo image corresponding to the feature amount space, and generates the pseudo image based on the calculated image processing settings. program.

Images