Image creation method and visual inspection device

JP2023183808A5Active Publication Date: 2024-11-29HITACHI LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2022097546
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2022-06-16
Publication Date
2024-11-29
Estimated Expiration
2042-06-16

AI Technical Summary

Technical Problem

Machine learning models for defect detection in manufacturing are prone to inaccurate determinations due to insufficient variation in training images, especially when defect frequencies vary, leading to false positives or negatives.

Method used

An appearance inspection apparatus that generates additional images based on statistical distributions of characteristics such as deformation, brightness, contrast, and noise intensity to enrich training data, using machine learning to create a learned model for accurate defect detection.

Benefits of technology

The method enhances the variation of training data, reducing the likelihood of incorrect defect judgments by improving the model's ability to distinguish between normal and abnormal conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To prevent determination of the presence or absence of an abnormality by machine learning from becoming inaccurate.SOLUTION: A visual inspection device comprises a processor, and the processor acquires a plurality of appearance images capturing the appearance of a test object, creates a statistical distribution representing variations in characteristics of the appearance images when the plurality of appearance images are defined as population, creates an additional image capturing the appearance based on the variations indicated by the statistical distribution, and generates a learned model through machine learning using learning data including the plurality of appearance images and the additional image.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image generation method and an appearance inspection apparatus.

Background Art

[0002] In a manufacturing line or the like for manufacturing products, it is performed to recognize whether there are defects in the products or their parts during manufacturing by image recognition technology, and to detect the defects at an early stage. In particular, due to the recent progress of machine learning, detecting defects by machine learning has also been actively carried out. In that case, after various learning images are learned by a machine learning model such as a CNN (Convolutional Neural Network), a learned machine learning model determines whether there are defects in the product based on an image of an actual product.

[0003] However, if the variations of the learning images are few, the learning of the machine learning model becomes insufficient, and the determination of whether there are defects becomes inaccurate. Therefore, in Patent Document 1, a technique is proposed to increase the variations of the learning images by extracting scratches from the learning images, generating partial images obtained by variously deforming the scratches, and synthesizing them with a synthesis destination image to obtain new learning images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since the frequency of occurrence of defects in products varies depending on each defect, if the variations of the learning images are increased without considering the frequency, the machine learning model may misdetect defects or miss defects, and the determination result of the machine learning model may become inaccurate.

[0006] This invention has been made in view of the above circumstances, and aims to suppress inaccuracies in the determination of the presence or absence of anomalies by machine learning. [Means for solving the problem]

[0007] This application includes several means to solve at least some of the above problems, and some examples are as follows.

[0008] To solve the above problems, an appearance inspection apparatus according to one aspect of the present invention is an appearance inspection apparatus equipped with a processor, the processor acquires a plurality of appearance images that capture the appearance of an object to be inspected, generates a statistical distribution that represents the variability of the features of each appearance image when the plurality of appearance images are used as a population, generates an additional image that captures the appearance based on the variability shown in the statistical distribution, and generates a trained model by machine learning using training data that includes the plurality of appearance images and the additional image. [Effects of the Invention]

[0009] According to the present invention, it is possible to suppress inaccuracies in the determination of whether or not an anomaly is present using machine learning.

[0010] Other issues, configurations, and effects not mentioned above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a schematic diagram showing an example of the functional configuration of a visual inspection device. [Figure 2] Figure 2 is a schematic diagram showing an example of a flowchart of the visual inspection method according to the first embodiment. [Figure 3] Figure 3 is a flowchart showing an example of the additional image generation process according to the first embodiment. [Figure 4] Figure 4 is a schematic diagram illustrating an example of the additional image generation process according to the first embodiment. [Figure 5]Figure 5 is a schematic diagram illustrating an example of the additional image generation process according to the first embodiment, where the position where deformation occurred in the part is adopted as a characteristic of the external image. [Figure 6] Figure 6 is a schematic diagram illustrating an example of the additional image generation process according to the first embodiment, where the overall brightness of the exterior image is adopted as a characteristic of the exterior image. [Figure 7] Figure 7 is a schematic diagram illustrating an example of the additional image generation process according to the first embodiment, where the contrast of the exterior image is adopted as a characteristic of the exterior image. [Figure 8] Figure 8 is a schematic diagram illustrating an example of the additional image generation process according to the first embodiment, where the noise intensity of the appearance image is used as a characteristic of the appearance image. [Figure 9] Figure 9 is a schematic diagram illustrating an example of a method for generating a trained model according to the first embodiment. [Figure 10] Figure 10 is a schematic diagram showing an example of an inspection method according to the first embodiment. [Figure 11] Figure 11 is a schematic diagram showing an example of the display unit according to the first embodiment. [Figure 12] Figure 12 is a schematic diagram illustrating an example of the additional image generation process in the second embodiment. [Figure 13] Figure 13 is a schematic diagram illustrating an example of a method for generating a trained model in the third embodiment. [Figure 14] Figure 14 is a schematic diagram illustrating an example of the inspection method in the third embodiment. [Figure 15] Figure 15 shows an example of the hardware configuration of the visual inspection apparatus according to the first to third embodiments. [Modes for carrying out the invention]

[0012] Hereinafter, an embodiment of the present invention will be described based on the drawings. In all the drawings for explaining the embodiment, the same members are generally denoted by the same reference numerals, and the repeated description thereof will be omitted as appropriate. Further, in the following embodiments, it is needless to say that the constituent elements (including element steps, etc.) are not necessarily essential except in the case where it is particularly specified or considered to be clearly essential in principle. Also, when it is said that "consisting of A", "composed of A", "having A", or "including A", it is needless to say that other elements are not excluded except in the case where it is particularly specified that only that element is present. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of the constituent elements, etc., it is to include those that are substantially approximated or similar to the shape, etc. except in the case where it is particularly specified and in the case where it is clearly not so in principle.

[0013] <First Embodiment> FIG. 1 is a schematic diagram showing an example of the functional configuration of the appearance inspection apparatus 100 according to the present embodiment.

[0014] The appearance inspection apparatus 100 is, for example, an apparatus that inspects whether there is an abnormality in the appearance of a part as an inspection object, and includes a processing unit 110, a storage unit 120, an input unit 130, a display unit 140, and an imaging unit 150.

[0015] The input unit 130 is an input device such as a keyboard or a mouse for receiving various inputs from the user. The display unit 140 is, for example, a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display that displays the inspection result of the appearance of the part.

[0016] The imaging unit 150 is an imaging device such as a camera that images the appearance of the part as the inspection object and stores an appearance image 121a showing the appearance in the storage unit 120.

[0017] The memory unit 120 is a functional unit that stores the appearance image DB (Database) 121, the extended learning image DB 122, the learned parameter DB 123, and the program 124, respectively.

[0018] The external image DB121 is a database that stores external images 121a of parts captured by the imaging unit 150 and their attribute information 121b. The attribute information 121b includes information indicating whether the part captured in the external image 121a is normal or abnormal, the type of abnormality, and the location where the abnormality occurred in the part.

[0019] The extended training image DB122 is a database that stores extended training images 122a and attribute information 122b. Extended training images 122a are training data used when a machine learning model that inspects whether there are abnormalities in a part is being trained. As an example, extended training image 122a is an image that includes the aforementioned appearance image 121a and additional images generated by the image generation unit 113 described later. In this way, by using additional images in addition to the appearance image 121a as training data, the variety of training data can be increased.

[0020] Attribute information 122b includes information indicating whether the component shown in the extended learning image 122a is normal or abnormal, the type of abnormality, and the location where the abnormality occurred in the component.

[0021] The trained parameter DB123 is a database that stores the internal parameters of a machine learning model trained using the extended training image 122a as training data.

[0022] Program 124 is a visual inspection program according to this embodiment. The various functions of the processing unit 110 are realized when the visual inspection device 100 executes Program 124.

[0023] The processing unit 110 is a functional unit that controls various parts of the visual inspection device 100. For example, the processing unit 110 includes an image acquisition unit 111, a statistical distribution generation unit 112, an image generation unit 113, a learning unit 114, and an inspection unit 115.

[0024] The image acquisition unit 111 is a functional unit that acquires the exterior image 121a from the exterior image DB 121.

[0025] The statistical distribution generation unit 112 is a processing unit that generates a statistical distribution representing the variability of the features of each appearance image 121a when the population consists of multiple appearance images 121a. Examples of these features include the amount of deformation of the part being inspected, or the location where deformation has occurred on the part, as described later. The brightness, contrast, and noise intensity of each appearance image 121a are also examples of features.

[0026] The image generation unit 113 is a functional unit that generates additional images of the appearance of parts based on the variability of features shown in the statistical distribution generated by the statistical distribution generation unit 112, and stores these additional images as extended learning images 122a in the extended learning image database.

[0027] The learning unit 114 is a functional unit that generates a trained model using machine learning with the extended training image 122a as training data.

[0028] The inspection unit 115 is a functional unit that uses a learned model to inspect whether there are any abnormalities in the appearance of the component to be inspected. The inspection unit 115 may also instruct the display unit 140 to display the inspection results, so that the user who looks at the display unit 140 can understand the inspection results.

[0029] Figure 2 is a schematic diagram showing an example of a flowchart of the visual inspection method according to this embodiment.

[0030] First, the image acquisition unit 111 acquires one or more appearance images 121a from the appearance image database 121 (step S21). In this example, the image acquisition unit 111 randomly acquires one or more normal appearance images 121a from all the appearance images 121a stored in the appearance image database 121. The image acquisition unit 111 can determine whether an appearance image 121a is normal based on the attribute information 121b corresponding to that appearance image 121a. Furthermore, the image acquisition unit 111 also acquires attribute information 121b corresponding to each of the acquired appearance images 121a from the appearance image database 121.

[0031] Next, the statistical distribution generation unit 112 and the image generation unit 113 perform the additional image generation process (step S22). Details of this generation process will be described later.

[0032] Next, the learning unit 114 generates a trained model using machine learning with the extended training image 122a as training data (step S23). Details of this step will be described later.

[0033] Next, the inspection unit 115 uses a trained model to inspect the part for any abnormalities (step S24). Here, the inspection unit 115 acquires an inspection image of the part captured by the imaging unit 150 and checks whether there are any abnormalities in the part shown in the inspection image. Details will be described later.

[0034] This concludes the basic processing of the visual inspection method according to this embodiment.

[0035] Next, the process of generating additional images in step S22 will be described. Figure 3 is a flowchart showing an example of the process of generating additional images. Figure 4 is a schematic diagram illustrating an example of the process of generating additional images. Note that the process of generating additional images is just one example of an image generation method.

[0036] First, as shown in Figure 3, the statistical distribution generation unit 112 generates a statistical distribution that represents the variability of the features of each appearance image 121a when each appearance image 121a acquired in step S21 is used as the population (step S31).

[0037] In the example in Figure 4, each appearance image 121a, which forms the population of the statistical distribution, is represented by an image set 401. The statistical distribution generation unit 112 generates a reference value image 402 from each appearance image 121a included in the image set 401. In this example, the statistical distribution generation unit 112 calculates the median of the pixel values ​​of each appearance image 121a for each position and generates a median image as the reference value image, where the pixel value at each position is the median. Alternatively, the average value of the pixel values ​​may be used instead of the median. The median may be calculated at the position of each pixel or in a region containing multiple pixels. Next, the statistical distribution generation unit 112 finds the difference in pixel values ​​between each appearance image 121a and the reference value image 402 for each position. The difference calculated at a certain position is the amount of deformation of the part shown in the appearance image 121a at that position, relative to the reference value image 402. Finally, the statistical distribution generation unit 112 finds a cumulative value by summing the differences of each appearance image 121a for each position. Locations with large cumulative values ​​correspond to locations where the variation in deformation is large when the image set 401 is considered the population. Conversely, locations with small cumulative values ​​correspond to locations where the variation in deformation is small when the image set 401 is considered the population. The statistical distribution generation unit 112 generates a statistical distribution 403 by mapping these cumulative values. In the statistical distribution 403, locations with darker colors correspond to locations with large variation in deformation, and locations with lighter colors correspond to regions with small variation in deformation.

[0038] Here, the statistical distribution generation unit 112 generates a statistical distribution 403 that shows the variation in the amount of deformation for each position of the part, but the statistical distribution generation unit 112 may generate multiple types of statistical distributions 403. For example, the statistical distribution generation unit 112 may generate a statistical distribution that shows the variation in the shape of the part, as described later, or it may generate statistical distributions that show the variation in brightness, contrast, and noise intensity of each appearance image 121a.

[0039] Refer to Figure 3 again. Next, the image generation unit 113 selects one or more of the multiple statistical distributions generated by the statistical distribution generation unit 112 (step S32). For the sake of simplicity, we will explain using the example where the image generation unit 113 selects the statistical distribution 403 that shows the variation in the amount of deformation mentioned above.

[0040] Next, the image generation unit 113 generates an additional image based on the selected statistical distribution 403 (step S33). The method for generating the additional image will be explained with reference to Figure 4.

[0041] As shown in statistical distribution 403, the variation in the amount of deformation differs depending on the position of the part. The appearance image 112a, which captures a part deformed at a position with high variation, is statistically fewer in number and less variation in the appearance image DB 121 compared to the appearance image 112a, which captures a part deformed at a position with low variation.

[0042] Therefore, the image generation unit 113 generates additional images 404 by applying a deformation process to the appearance images 121a included in the appearance image DB 121, thereby increasing the variation of the deformation. The amount of deformation and the number of additional images 404 are determined by the image generation unit 113 based on the statistical distribution 403. For example, the image generation unit 113 increases the amount of deformation and the number of additional images 404 for distribution regions with large variability in the statistical distribution 403. The appearance image 121a to be subjected to deformation may be a single appearance image 121a arbitrarily selected from the appearance image DB 121, or it may be multiple appearance images 121a. This makes it possible to increase the variation of additional images 404 with large variability.

[0043] The image generation unit 113 ensures that the amount of deformation in the additional image 404 falls within the range of the statistical distribution 403 of the normal appearance image 121a. This allows the parts captured in the additional image 404 to be considered normal. The appearance image 121 that becomes the additional image 404 through the deformation process may be either normal or abnormal. The image generation unit 113 may also generate an abnormal additional image 404 by applying deformation processing to the normal or abnormal appearance image 121 in this manner. The same applies to the examples in Figures 5 to 8 described later.

[0044] Refer to Figure 3 again. Next, the image generation unit 113 saves all appearance images 121a included in the appearance image DB and all additional images 404 generated in step S33 to the extended learning image DB 122 (step S34). The image generation unit 113 also saves the attribute information 121b of the appearance image 121a to the extended learning image DB 122, as well as the attribute information 122b of the additional image 404 to the extended learning image DB 122. Since the normal appearance image 121a was used as image set 401, the parts shown in the additional image 404 can also be considered normal, as mentioned above. Therefore, the attribute information 122b of the additional image 404 is information indicating that the parts are normal.

[0045] This completes the basic processing for generating additional images.

[0046] As described above, the appearance image 121a and additional image 404 in the extended learning image DB122 become the training data when the learning unit 114 generates a trained model. In this embodiment, as mentioned above, the number of additional images 404 in the distribution region where the variation in deformation amount is large in the statistical distribution 403 is increased, so the variation of the training data in that distribution region increases. As a result, the learning unit 114 can accurately learn the discrimination boundary that distinguishes between normal and abnormal based on the training data. As a result, the possibility that the inspection unit 115 will mistakenly judge a part that has been greatly deformed within the normal range as abnormal can be reduced, and inaccuracies in the determination of the presence or absence of abnormality by machine learning can be suppressed.

[0047] Figure 5 is a schematic diagram illustrating an example of the process for generating an additional image 404 when the location where deformation occurred in the part is selected as a characteristic of the external image 121a.

[0048] In the example shown in Figure 5, multiple appearance images 121a that form the population of the statistical distribution are represented by an image set 501. Then, in step S31, the statistical distribution generation unit 112 extracts the shape of each appearance image 121a included in the image set 501 using image processing such as contour extraction. Next, the statistical distribution generation unit 112 generates a median image, which represents the median of the part shapes in the image set 501, as a reference value image 502. Alternatively, the statistical distribution generation unit 112 may generate an average value image, which represents the average value of the part shapes across the image set 501, instead of the median image.

[0049] Furthermore, the statistical distribution generation unit 112 calculates the difference in pixel values ​​between each appearance image 121a included in the image set 501 and the reference value image 502 for each position, thereby calculating the positions where deformation occurred in the parts relative to the reference value image 502 for each appearance image 121a. The statistical distribution generation unit 112 then generates a statistical distribution 503 that shows the variability of the deformation positions when the image set 501 is used as the population.

[0050] In this example, an abnormality template 505 containing images of various defects such as scratches and stains is pre-stored in the storage unit 120. Then, in step S33, the image generation unit 113 processes the appearance image 121a included in the appearance image DB 121 using image processing, etc., to generate images in which the shape of the part is deformed in various ways within the normal range. For example, the image generation unit 113 generates more images with greater deformation in distribution regions where the variability in the statistical distribution 503 is large.

[0051] The image generation unit 113 may obtain any one appearance image 121a from the appearance image DB 121 and perform the above image processing on that appearance image 121a. Alternatively, the image generation unit 113 may obtain multiple appearance images 121a from the appearance image DB 121 and perform image processing on each of the appearance images 121a. Then, the image generation unit 113 generates an additional image 504 by superimposing (combining) the defective image of the abnormal template 505 onto the image that has undergone the image processing in this manner.

[0052] The position where the defect image is superimposed on the appearance image 121a is the position where deformation occurred in that appearance image 121a. For example, if deformation occurs at the edge of a part in a certain appearance image 121a, the image generation unit 113 superimposes the defect image on that edge.

[0053] The image generation unit 113 may perform rotation, enlargement, or reduction, or a combination thereof, on the defective image of the abnormal template 505, and overlay the processed image onto the appearance image 121a.

[0054] Although additional image 504 contains a defective image, the image on which additional image 504 is based is an image in which the shape of the part is deformed within the normal range. Therefore, the attribute information 122b of additional image 504 saved in the extended learning image DB 122 in step S34 will be information indicating that it is normal.

[0055] The appearance image DB121 includes appearance images 121a where the location of deformation in the part varies considerably. However, similar to the example in Figure 4, appearance images 121a in the distribution region with high variability in the statistical distribution 503 are thought to be fewer in number and less varied compared to appearance images 121a in the distribution region with low variability.

[0056] Therefore, by generating an additional image 504 as in this example, the variety of images can be increased, enriching the variety of training data. Furthermore, by overlaying the defect image onto the appearance image 121a, the combinations of deformations and defects become more diverse, further increasing the variety of training data.

[0057] This allows the learning unit 114 to accurately learn a discrimination boundary that distinguishes between normal and abnormal based on learning data in which defects exist at the locations where deformation occurs. As a result, the possibility of the inspection unit 115 mistakenly identifying shape variations within the normal range as abnormal is reduced.

[0058] Figure 6 is a schematic diagram illustrating an example of the additional image generation process when the overall brightness of the exterior image 121a is adopted as a characteristic of the exterior image 121a.

[0059] In the example shown in Figure 6, multiple appearance images 121a, which form the population of the statistical distribution, are represented by an image set 601. Then, in step S31, the statistical distribution generation unit 112 calculates the average brightness obtained by averaging the brightness of all appearance images 121a across the image set 601 as the reference brightness. Alternatively, the statistical distribution generation unit 112 may calculate the median brightness in the image set 601 as the reference brightness instead of the average brightness.

[0060] Furthermore, the statistical distribution generation unit 112 calculates the difference between the overall brightness of the image and the reference brightness for each appearance image 121a included in the image set 601, and generates a statistical distribution 602 that shows the variability of that difference. The horizontal axis of the statistical distribution 602 is the difference between the reference brightness and the brightness, and the vertical axis is the number of appearance images 121a.

[0061] Then, in step S33, the image generation unit 113 appropriately selects an appearance image 121a from the appearance image DB 121. The number of appearance images 121a to be selected may be one or multiple. The image generation unit 113 then performs a brightness correction process on the selected appearance images 121a to generate various additional images 604 in a number corresponding to the statistical distribution 602, such that the difference between the reference brightness and the brightness falls within the range of the distribution in the statistical distribution 602.

[0062] For example, the image generation unit 113 increases the number of additional images 604 in distribution regions of the statistical distribution 602 where the number of appearance images 121a is small. This increases the variety of images that are few in number in the statistical distribution 602.

[0063] Furthermore, since the difference between the brightness of the additional image 604 and the reference brightness falls within the range of the distribution in the statistical distribution 602 of the normal appearance image 121a, the parts shown in the additional image 604 can be considered normal. Therefore, the attribute information 122b of the additional image 604 saved in the extended learning image DB 122 in step S34 will be information indicating that it is normal.

[0064] In this example, the statistical distribution 602 increases the variety of images with fewer images, resulting in a richer variety of brightness in the training data and a reduction in brightness bias in the training data. As a result, the learning unit 114 can learn the appearance of a normal part by taking into account the color of the part being inspected. Consequently, the inspection unit 115 can reduce the possibility of mistakenly identifying a normal part as abnormal due to color differences.

[0065] Figure 7 is a schematic diagram illustrating an example of the additional image generation process when the contrast of the exterior image 121a is adopted as a feature of the exterior image 121a.

[0066] In the example shown in Figure 7, multiple appearance images 121a that form the population of the statistical distribution are represented by an image set 701. Then, in step S31, the statistical distribution generation unit 112 calculates a luminance histogram 702 for each appearance image 121a included in the image set 701, relating the luminance value to the number of pixels. Next, the statistical distribution generation unit 112 calculates an average luminance histogram 703 as a reference histogram, which is obtained by averaging the luminance histograms 702 in the image set 701.

[0067] Furthermore, the statistical distribution generation unit 112 calculates a reference contrast based, for example, on the difference between the maximum and minimum brightness in the reference histogram 703. Similarly, the statistical distribution generation unit 112 calculates the contrast of each appearance image 121a included in the image set 701 based on the difference between the maximum and minimum brightness in each brightness histogram 702. Then, the statistical distribution generation unit 112 calculates the difference between the contrast of each appearance image 121a and the reference contrast and generates a statistical distribution 704 that shows the variability of that difference. The horizontal axis of the statistical distribution 704 is the difference between the reference contrast and the contrast, and the vertical axis is the number of appearance images 121a.

[0068] Then, in step S33, the image generation unit 113 appropriately selects an appearance image 121a from the appearance image DB 121. The number of appearance images 121a to be selected may be one or multiple. Then, the image generation unit 113 performs contrast correction processing on the selected appearance images 121a to generate various additional images 705 in a number corresponding to the statistical distribution 704, such that the difference between the reference contrast and the contrast falls within the range of the distribution in the statistical distribution 704.

[0069] As an example, the image generation unit 113 increases the number of additional images 705 that have such variability in distribution regions where the number of appearance images 121a is small in the statistical distribution 704. This makes it possible to increase the variety of images that are few in number in the statistical distribution 704.

[0070] Furthermore, since the difference between the contrast of the additional image 705 and the reference contrast falls within the range of the distribution in the statistical distribution 704 of the normal appearance image 121a, the parts shown in the additional image 705 can be considered normal. Therefore, the attribute information 122b of the additional image 705 saved in the extended learning image DB 122 in step S34 will be information indicating that it is normal.

[0071] In this example, the statistical distribution 704 increases the variety of images with fewer images, resulting in a richer variety of contrasts in the training data and a reduction in contrast bias in the training data. Therefore, the learning unit 114 can learn the appearance of normal parts by taking image contrast into consideration. As a result, the inspection unit 115 can reduce the possibility of mistakenly identifying normal parts as abnormal due to differences in image contrast.

[0072] Figure 8 is a schematic diagram illustrating an example of the additional image generation process when the noise intensity of the appearance image 121a is adopted as a characteristic of the appearance image 121a.

[0073] In the example shown in Figure 8, multiple appearance images 121a that form the population of the statistical distribution are represented by an image set 801. Then, in step S31, the statistical distribution generation unit 112 generates a denoised image 802 by removing the noise from each of the appearance images 121a included in the image set 801.

[0074] Next, the statistical distribution generation unit 112 generates a difference image between each appearance image 121a of the image set 801 and its corresponding denoised image 802, and calculates the average noise intensity of the entire image of the difference image. Then, the statistical distribution generation unit 112 calculates the average of the average noise intensities in the image set 801 as the reference noise intensity. Alternatively, the median of the average noise intensities in the image set 801 may be used as the reference noise intensity. Furthermore, the statistical distribution generation unit 112 calculates the difference between the average noise intensity of each appearance image 121a and the reference noise intensity, and generates a statistical distribution 803 that shows the variability of this difference. In the statistical distribution 803, the horizontal axis is the difference between the reference noise intensity and the average noise intensity, and the vertical axis is the number of appearance images 121a.

[0075] Then, in step S33, the image generation unit 113 appropriately selects an appearance image 121a from the appearance image DB 121. The number of appearance images 121a to be selected may be one or multiple. Then, the image generation unit 113 applies noise processing to the selected appearance images 121a to generate various additional images 804 in a number corresponding to the statistical distribution 803, such that the difference between the reference noise intensity and the average noise intensity falls within the range of the distribution in the statistical distribution 803.

[0076] As an example, the image generation unit 113 increases the number of additional images 804 in distribution regions of the statistical distribution 803 where the number of appearance images 121a is small. This increases the variety of images with a small number of occurrences in the statistical distribution 803.

[0077] Furthermore, the difference between the average noise intensity of the additional image 804 and the reference noise intensity falls within the range of the distribution in the statistical distribution 803 of the normal appearance image 121a, so the parts shown in the additional image 804 can be considered normal. Therefore, the attribute information 122b of the additional image 804 saved in the extended learning image DB 122 in step S34 will be information indicating that it is normal.

[0078] In this example, the statistical distribution 803 shows an increase in the variety of images with fewer images, resulting in a richer variety of average noise intensity in the training data and a reduction in bias in the average noise intensity of the training data. Therefore, the learning unit 114 can learn the appearance of normal parts by taking into account the average noise intensity of the images. As a result, the inspection unit 115 can reduce the possibility of mistakenly identifying normal parts as abnormal due to changes in noise intensity caused by the imaging environment.

[0079] Next, we will explain how to generate the trained model in step S23 of Figure 2.

[0080] Figure 9 is a schematic diagram illustrating an example of a method for generating a trained model.

[0081] First, the learning unit 114 retrieves one or more extended learning images 122a from the extended learning image DB 122. Here, the set of extended learning images 122a obtained in this way is called the learning image set 901.

[0082] Next, the learning unit 114 inputs each extended learning image 122a from the learning image set 901 as training data to a machine learning model 902 such as a CNN. Based on its internal parameters, the machine learning model 902 determines whether the parts depicted in the extended learning images 122a are normal or abnormal, and outputs an estimated evaluation value 903 that includes the determination result. In addition to the determination result of whether it is normal or abnormal, the estimated evaluation value 903 also includes the type of abnormality and the location where the abnormality occurred.

[0083] Next, the learning unit 114 calculates the error between the estimated evaluation value 903 and the attribute information 122b, and updates the internal parameters of the machine learning model 902 to minimize this error. Then, the learning unit 114 stores the updated internal parameters in the trained parameter DB 123.

[0084] Next, the machine learning model 902 outputs an estimated evaluation value 903 using the internal parameters stored in the trained parameter database. The machine learning model 902 that outputs an estimated evaluation value 903 using the internal parameters stored in the trained parameter database in this way is a trained model.

[0085] This completes the basic processing for generating a trained model. In this example, an extended training image 122a is selected from the training image set 901 from the extended training image DB122, which has increased variation due to additional images corresponding to one of the aforementioned statistical distributions 403, 503, 602, 704, or 803, and this is used as training data for the machine learning model 902. As a result, the machine learning model 902 learns from varied training data, which reduces the possibility of the trained machine learning model 902 making incorrect judgments.

[0086] Next, the inspection method in step S24 of Figure 2 will be explained.

[0087] Figure 10 is a schematic diagram illustrating an example of an inspection method. First, the inspection unit 115 acquires an inspection image 1001 of the part captured by the imaging unit 150.

[0088] Next, the inspection unit 115 reads internal parameters from the trained parameter DB 123 into the machine learning model 902, and then inputs the inspection image 1001 into the machine learning model 902. As a result, the machine learning model 902, being a trained model, determines whether the parts shown in the inspection image 1001 are normal or abnormal based on its internal parameters, and outputs an estimated evaluation value 903 that includes the determination result.

[0089] The inspection unit 115 then determines that there is no abnormality in the part (OK) if the estimated evaluation value 903 indicates that it is normal. On the other hand, if the estimated evaluation value 903 indicates that it is abnormal, the inspection unit 115 determines that there is an abnormality in the part (NG).

[0090] This completes the basic procedures for inspecting the parts.

[0091] Figure 11 is a schematic diagram showing an example of the display of the display unit 140. In this example, the display unit 140 displays the appearance image 121a from the appearance image DB 121. The display unit 140 may also display whether the appearance is normal or abnormal, as indicated by the attribute information 121b, and, in the case of an abnormality, the type of abnormality, such as "scratches," along with the appearance image 121a.

[0092] The display unit 140 also displays the extended learning image 122a from the extended learning image DB 122. In this case, the display unit 140 may also display whether the attribute information 122b indicates normal or abnormal, and in the case of abnormal, the type of abnormality, such as "stain," along with the extended learning image 122a.

[0093] Furthermore, the display unit 140 also displays the statistical distributions for both the appearance image DB 121 and the extended learning image DB 122. Here, the display unit 140 displays the statistical distributions 704 and 803 that it selected to generate additional images in the extended learning image DB 122. In this case, the additional image 705 generated using statistical distribution 704 and the additional image 804 generated using statistical distribution 803 will be included in the extended learning image 122a of the extended learning image DB 122.

[0094] In the appearance image DB121, as shown in the statistical distribution 704, the number of images decreases as the difference between the reference contrast and the contrast increases, resulting in a lack of variation in appearance images 121a. On the other hand, in the extended learning image DB122, as indicated by the upward arrow, the variability in the statistical distribution 704 is eliminated, and the number of images is almost uniform regardless of the magnitude of the contrast. The same applies to the statistical distribution 803. As a result, extended learning images 122a with rich variation can be obtained regardless of the magnitude of the contrast and average noise intensity. Consequently, by training the machine learning model 902 with extended learning images 122a as training data, a trained model with fewer misclassifications can be obtained.

[0095] Furthermore, the display unit 140 also displays the inspection results performed by the inspection unit 115. In this example, the display unit 140 displays the inspection image 1001 and the estimated evaluation value 903. The estimated evaluation value 903 includes the probability of normality and the probability of abnormalities such as "stains" and "scratches" being present. If there are abnormalities, the display unit 140 also displays the location of the defects.

[0096] This allows the user to understand the location of the defect and the type of anomaly.

[0097] <Second Embodiment> In the first embodiment, as shown in Figures 4 to 8, the image acquisition unit 111 acquired a normal appearance image 121a from the appearance image DB 121. In contrast, in this embodiment, the image acquisition unit 111 acquires an abnormal appearance image 121a from the appearance image DB 121 as follows.

[0098] Figure 12 is a schematic diagram illustrating an example of the additional image generation process in this embodiment.

[0099] First, in step S21 of Figure 1, the image acquisition unit 111 randomly acquires one or more abnormal appearance images 121a and their attribute information 121b from all the appearance images 121a stored in the appearance image DB 121. The acquired appearance images 121a are images that form the population of the statistical distribution, and below they will be represented as the image set 1201.

[0100] Next, in step S31, the statistical distribution generation unit 112 identifies the pixels at the location of the anomaly indicated by the attribute information 121b for each of the acquired appearance images 121a. Then, the statistical distribution generation unit 112 generates a statistical distribution 1202 that shows the distribution of the pixels identified in the image set 1201. This statistical distribution 1202 is a distribution in which the frequency of anomalies associated with deformation is represented by the intensity of the color, and the darker the color, the more frequently anomalies occur, the more easily deformation occurs, and the greater the variation in the amount of deformation.

[0101] The appearance image DB121 contains abnormal appearance images 121a with various deformation amounts, but most of the abnormal appearance images 121a have deformation amounts close to the median in the image set 1201, and the number of abnormal appearance images 121a with large deformation amounts is considered to be statistically small.

[0102] Therefore, in step S31, the image generation unit 113 appropriately selects normal appearance images 121a from the appearance image DB 121. The number of appearance images 121a to be selected may be one or multiple. Then, the image generation unit 113 performs image processing or other processing on the selected appearance images 121a to generate various additional images 1203 in a number corresponding to the statistical distribution 1202.

[0103] At this time, the image generation unit 113 increases the number of additional images 1203 in distribution regions where the variation in the amount of deformation is large in the statistical distribution 1202. This makes it possible to increase the variation of normal additional images 1203 in distribution regions where anomalies are likely to occur. The amount of deformation of the additional images 1203 is determined according to the statistical distribution 1202 of the abnormal image set 1201.

[0104] Then, in step S34, the image generation unit 113 saves all appearance images 121a and all additional images 1203 included in the appearance image DB 121 to the extended learning image DB 122. At this time, the image generation unit 113 also saves the attribute information of the appearance images 121a and additional images 1203 to the extended learning image DB 122.

[0105] As a result, the number of variations of normal extended learning images 122a in the regions prone to abnormalities, as described above, increases in the extended learning image DB 122. Consequently, the learning unit 114 can accurately learn the discrimination boundary that distinguishes between abnormal and normal, reducing the possibility of misjudgment by the inspection unit 115.

[0106] <Third Embodiment> In this embodiment, an example of using an autoencoder in the generation of a trained model in step S23 will be described.

[0107] Figure 13 is a schematic diagram illustrating an example of a method for generating a trained model in this embodiment.

[0108] In this embodiment, the learning unit 114 first acquires one or more normal extended learning images 122a from the extended learning image DB 122. The set of extended learning images 122a acquired in this way will be referred to as the learning image set 1302 below.

[0109] Next, the learning unit 114 inputs each extended training image 122a from the training image set 1302 as ground truth data into the autoencoder 1303. The autoencoder 1303 then processes the data based on its internal parameters and outputs a reconstructed image 1304. Since the autoencoder 1303 is a model that learns to produce the same image as the input image, it updates its internal parameters to minimize the error between the input image and the reconstructed image 1304. The learning unit 114 then stores the updated internal parameters in the trained parameter DB 123.

[0110] After this, the autoencoder 1303 outputs the reconstructed image 1304 using the internal parameters stored in the trained parameter DB. The autoencoder 1303 that outputs the reconstructed image 1304 using the internal parameters stored in the trained parameter DB in this manner is the trained model in this embodiment.

[0111] This concludes the basic process for generating a trained model using an autoencoder.

[0112] Next, the inspection method in step S24 of Figure 2 will be explained.

[0113] Figure 14 is a schematic diagram illustrating an example of the inspection method in this embodiment. First, the inspection unit 115 acquires an inspection image 1401 of the part captured by the imaging unit 150.

[0114] Next, the inspection unit 115 reads internal parameters from the learned parameter DB 123 into the autoencoder 1303, and then inputs the inspection image 1401 to the autoencoder 1303. As a result, the autoencoder 1303 outputs a reconstructed image 1304 based on its internal parameters.

[0115] As shown in Figure 13, the autoencoder 1303 learns from the normal extended learning image 122a as ground truth data, and therefore outputs a normal reconstructed image 1304 without any abnormalities. Therefore, even if the inspection image 1401 contains a foreign object, the normal reconstructed image 1304, from which the foreign object has been removed, is output. Thus, if the inspection image 1401 contains a foreign object, the difference image 1305, which is the difference between the inspection image 1401 and the reconstructed image 1304, will contain the foreign object.

[0116] Therefore, the inspection unit 115 determines that there is an abnormality in the part being inspected if the difference image 1305 contains foreign matter, and determines that the part is normal if the difference image 1305 does not contain foreign matter.

[0117] This completes the basic processing for inspecting components in this embodiment. According to this, the inspection unit 115 can check for any abnormalities in the component by taking the difference between the reconstructed image 1304 output by the autoencoder 1303 and the inspection image 1401.

[0118] <Hardware Configuration> Figure 15 shows an example of the hardware configuration of the visual inspection device 100 according to the first to third embodiments.

[0119] As shown in Figure 15, the visual inspection apparatus 100 comprises an imaging device 100a, a memory 100b, a processor 100c, a storage device 100d, a display device 100e, an input device 100f, and a reading device 100g. These devices are interconnected by a bus 100i.

[0120] The imaging device 100a is hardware for realizing the imaging unit 150 shown in Figure 1. For example, the imaging device 100a is a camera equipped with an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor for capturing images of the external appearance of a component.

[0121] Memory 100b is hardware that temporarily stores data, such as DRAM (Dynamic Random Access Memory), and program 124 is deployed on top of it.

[0122] The processor 100c is a CPU (Central Processing Unit) or GPU (Graphical Processing Unit) that controls each part of the visual inspection device 100. The processor 100c works in cooperation with the memory 100b to execute the program 124, thereby realizing the processing unit 110 shown in Figure 1.

[0123] The storage device 100d is a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores the program 124.

[0124] Alternatively, program 124 may be recorded on a computer-readable recording medium 100h, and the processor 100c may be instructed to read program 124 from the recording medium 100h.

[0125] Examples of recording media 100h include physical portable recording media such as CD-ROM (Compact Disc - Read Only Memory), DVD (Digital Versatile Disc), and USB (Universal Serial Bus) memory. Semiconductor memory such as flash memory or hard disk drives may also be used as recording media 100h.

[0126] The program 124 may be stored in a device connected to a public network, the internet, or a LAN (Local Area Network). In that case, the processor 100c can read and execute the program 124.

[0127] The storage unit 120 in Figure 1 is realized by a memory 100b and a storage device 100d.

[0128] The display device 100e is hardware such as a liquid crystal display or an organic EL display for realizing the display unit 140 in Figure 1. The input device 100f is hardware such as a keyboard or mouse for realizing the input unit 130 in Figure 1.

[0129] The reading device 100g is hardware such as a CD drive for reading data recorded on the recording medium 100h.

[0130] The effects described herein are merely illustrative and not limited to those described herein; other effects may also occur.

[0131] The present invention is not limited to the embodiments described above, and various modifications are included. For example, in the example of Figure 1, the appearance inspection device 100 is equipped with an imaging unit 150, but the imaging unit 150 may be provided outside the appearance inspection device 100. In this case, the imaging unit 150 and the appearance inspection device 100 are connected by a network (not shown), such as a LAN or the Internet, and the appearance inspection device 100 stores the appearance image 121a captured by the imaging unit 150 in the appearance image DB 121. By adopting such a configuration, the appearance inspection device 100 can realize a cloud service in which the learning unit 114 generates a trained model using an extended training image 122a including the appearance image 121a as training data, and outputs the internal parameters of the trained model.

[0132] Furthermore, the embodiments described above are explained in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to those comprising all the components described. It is also possible to replace parts of the configuration of one embodiment with those of another embodiment, and to add components from other embodiments to the configuration of one embodiment. Additionally, it is possible to add, delete, or replace parts of the configuration of each embodiment with those of other embodiments.

[0133] Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the above configurations and functions may be implemented in software by a processor interpreting and executing programs that implement each function. Information such as programs, decision tables, and files that implement each function can be stored in memory, storage devices such as HDDs and SSDs, or recording media such as IC (Integrated Circuit) cards, SD (Secure Digital) cards, and DVDs (Digital Versatile Discs). Also, control lines and information lines are shown only if deemed necessary for explanation, and not all control lines and information lines are necessarily shown in the actual product. In practice, almost all configurations can be considered interconnected. [Explanation of symbols]

[0134] 100...Visual inspection device, 110...Processing unit, 111...Image acquisition unit, 112...Statistical distribution generation unit, 113...Image generation unit, 114...Learning unit, 115...Inspection unit, 120...Storage unit, 121a...Visual image, 121b...Attribute information, 122a...Extended learning image, 122b...Attribute information, 124...Program, 130...Input unit, 140...Display unit, 150...Imaging unit, 401, 501, 601, 701, 801, 1201...Image set, 402...Reference value image, 403, 503, 602, 70 4, 803, 1202…Statistical distribution, 404, 504, 604, 705, 804, 1203…Additional images, 502…Reference image, 505…Anomaly template, 702…Brightness histogram, 703…Reference histogram, 802…Denoised image, 901…Training image set, 902…Machine learning model, 903…Estimated evaluation value, 1001…Inspection image, 1302…Training image set, 1303…Autoencoder, 1304…Reconstructed image, 1305…Difference image, 1401…Inspection image.

Claims

1. A visual inspection apparatus including a processor, The processor, Acquire multiple appearance images of the inspection target, generating a statistical distribution representing a variation in characteristics of each of the plurality of appearance images when the plurality of appearance images is a population; generating a larger number of additional images showing the appearance of a distribution region in which the statistical distribution indicates a larger variation; Generate a trained model by machine learning using training data including the plurality of appearance images and the additional image. Visual inspection equipment.

2. 2. The visual inspection apparatus according to claim 1, The feature is a deformation amount of the inspection object captured in the appearance image. Visual inspection equipment.

3. 2. The visual inspection apparatus according to claim 1, the feature is a position where a deformation has occurred in the inspection target captured in the appearance image, the processor generates the additional image by overlaying a defect at the location of the appearance image. Visual inspection equipment.

4. 2. The visual inspection apparatus according to claim 1, The processor increases the number of the additional images in a distribution region having a smaller number of the appearance images in the statistical distribution. Visual inspection equipment.

5. 5. The visual inspection apparatus according to claim 4, An appearance inspection apparatus, wherein the feature is any one of brightness, contrast, and noise intensity of the appearance image.

6. 2. The visual inspection apparatus according to claim 1, The processor further inspects the inspection object captured in the inspection image for an abnormality using the trained model. Visual inspection equipment.

7. 7. The visual inspection apparatus according to claim 6, The trained model is an autoencoder trained using the training data as correct answer data, the processor inputs the inspection image to the autoencoder, and determines whether a foreign object is included in a differential image obtained by taking a difference between the reconstructed image output from the autoencoder and the inspection image, thereby inspecting whether the inspection object has an abnormality. Visual inspection equipment.

8. 2. The visual inspection apparatus according to claim 1, Each of the plurality of appearance images is an image depicting the appearance of the normal test subject. Visual inspection equipment.

9. 2. The visual inspection apparatus according to claim 1, Each of the plurality of appearance images is an image capturing the appearance of the abnormal inspection target. Visual inspection equipment.

10. The visual inspection apparatus according to claim 8 or 9, The processor processes the image of the normal appearance of the test subject to generate the additional image showing normality. Visual inspection equipment.

11. 9. The visual inspection apparatus according to claim 8, The processor processes the image showing an abnormal or normal appearance of the inspection target to generate the additional image showing an abnormality. Visual inspection equipment.

12. The computer acquiring a plurality of appearance images showing the appearance of an object to be inspected; generating a statistical distribution representing a variance in characteristics of each of the plurality of appearance images when the plurality of appearance images is treated as a population; generating a greater number of additional images capturing the appearance of a distribution region in which the statistical distribution indicates a greater variance; generating a trained model by machine learning using training data including the plurality of appearance images and the additional image; An image generation method that performs the following: