Abnormality detection device, abnormality detection method, and abnormality detection program

Abstract

To provide an abnormality detection device that detects an abnormality of an inspection object based on an inspection image obtained by photographing the inspection object.SOLUTION: An abnormality detection device 100 for detecting an abnormality of an inspection object by using a neural network includes: a mask generation unit 102 that generates a mask area; a mask image generation unit 103 that generates a mask image; an interpolation image generation unit 104 that outputs an interpolation image by using a learned model that has learned to output the interpolation image using the mask image whose mask area position, size and the like have been changed based on a mask image of a normal image as an input; and an abnormality detection unit 110 that detects an abnormal area in an inspection image and determines an abnormality of the inspection object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an abnormality detection device, an abnormality detection method, and an abnormality detection program for detecting an abnormality of an inspection target based on an inspection image obtained by photographing the inspection target.

Background Art

[0002] In recent years, image conversion using a generative adversarial network (GAN) has been actively studied. For the problem of interpolating the masked region of an image, a technique called GLCIC (Globally and Locally Consistent Image Completion) has achieved extremely natural interpolation and has attracted attention.

[0003] Patent Document 1 discloses an abnormality detection device that uses a neural network to detect an abnormality of an inspection target based on an inspection image obtained by photographing the inspection target, the device including: a missing region adding image generation unit that generates a plurality of missing region adding images having a missing region from the inspection image; an interpolation image generation unit that uses a learned model trained to output an interpolation image in which the missing region is interpolated when the missing region adding image is input; a reconstructed image generation unit that synthesizes a plurality of interpolation images output from the interpolation image generation unit by inputting the plurality of missing region adding images generated by the missing region adding image generation unit to the interpolation image generation unit to generate a reconstructed image; and an abnormality determination unit that detects an abnormal region in the inspection image based on the difference between the reconstructed image and the inspection image and determines an abnormality of the inspection target.

[0004] According to the above disclosure, an abnormality of an inspection target can be detected based on an image obtained by photographing the inspection target.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] When detecting abnormalities such as scratches in the manufacturing line of precision instruments, it is necessary to detect abnormalities with high accuracy, including not only scratches that can be immediately recognized visually but also small scratches that cannot be seen visually. However, when considering even small scratches, the possible appearances of abnormalities are extremely diverse. Furthermore, it is difficult to intentionally create various abnormalities in the manufacturing line.

[0007] Therefore, the present invention provides a technique for automatically detecting abnormalities from an image, including small scratches, through learning using only normal images without abnormalities.

Means for Solving the Problems

[0008] The abnormality detection device in the present invention is an abnormality detection device that uses a neural network to detect an abnormality of an inspection object based on an inspection image obtained by photographing the inspection object, and includes a mask generation unit that generates a mask region to be given to the inspection image, a mask image generation unit that generates a mask image with the mask region given, and at least one of the position, number, size, shape, and color of the mask region in the mask image of the mask region is changed based on a mask image with the mask region given to a normal image without abnormalities among the inspection images. Using a learned model trained to output an interpolated image with the mask region interpolated, an interpolation image generation unit that outputs an interpolated image obtained by interpolating the mask region of the mask image output by the mask image generation unit, and a plurality of interpolation images generated by the interpolation image generation unit with mask regions given at different positions are input and synthesized, and an image synthesis unit that outputs a synthesized image, and an abnormality detection unit that detects an abnormal region in the inspection image based on the difference between the inspection image and the synthesized image and determines the abnormality of the inspection object.

[0009] The abnormality detection method in the present invention is an abnormality detection method that uses a neural network to detect abnormalities in an inspection target based on an inspection image obtained by photographing the inspection target. Based on a mask image obtained by assigning a mask region to a normal image without abnormalities among the inspection images, using, as an input, a mask image in which at least one of the position, number, size, shape, and color of the mask region in the mask image of the mask region is changed, a learning step of generating a learned model trained to output an interpolated image in which the mask region is interpolated, a mask generation step of generating a mask region to be assigned to the inspection image, a mask image generation step of generating a mask image with the mask region assigned, an interpolated image generation step of using the learned model to output an interpolated image obtained by interpolating the mask region of the mask image output by the mask image generation step, an image synthesis step of synthesizing a plurality of interpolated images generated in the interpolated image generation step with a plurality of mask images having the mask region assigned at different positions and outputting a synthesized image, and an abnormality detection step of detecting an abnormal region in the inspection image based on the difference between the inspection image and the synthesized image and determining the abnormality of the inspection target.

Advantages of the Invention

[0010] According to the present invention, it is possible to improve the accuracy of abnormality detection of an inspection target based on an image obtained by photographing the inspection target.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described. Note that the present invention is not limited to the following embodiments.

[0013] (Embodiment) In the present embodiment, the overall processing flow for detecting an abnormality in an inspection target based on an inspection image obtained by photographing the inspection target is described with reference to FIG. 2 in comparison with the processing flow using conventional machine learning.

[0014] FIG. 2(A) shows the flow of abnormality detection in the present embodiment, and FIG. 2(B) shows the flow of abnormality detection in conventional machine learning.

[0015] Generally, abnormality detection using machine learning consists of a training phase and an inspection phase (inference phase). In the training phase, training data for training is prepared, model development is performed, and the model is trained using the training data. In the inspection phase, an inspection is carried out using the trained learned model, and the inspection results are evaluated.

[0016] The process of anomaly detection in conventional machine learning will be described with reference to FIG. 2(B). In the process of conventional anomaly detection, images capturing various anomalies that may occur on the inspection target are prepared as training data (step S211), a model for detecting anomalies is developed (step S212), model training is performed using images with various anomalies (step S213), the test of the inspection image using the trained model is conducted (step S214), and the test results are evaluated (step S215). If the evaluation results are insufficient, images capturing anomalies that occur on the inspection target again are added as training data (step S211), or the model for detecting anomalies is modified and model development is restarted (step S212).

[0017] In the process of conventional anomaly detection, the possible anomalies are very diverse, and it is necessary to prepare a huge amount of training data assuming such diverse forms. Furthermore, when the evaluation results are insufficient, it is necessary to repeat the re-preparation of training data, the review of the model, model training, inspection tests, and evaluation, which requires a lot of processing time.

[0018] The process of anomaly detection in the present embodiment will be described with reference to FIG. 2(A). In the present embodiment, normal images without anomalies on the inspection target are prepared as training data (step S201), a model for reproducing the original normal images using images with a part of the normal images masked is developed (step S202), and model training is performed using images with various mask regions added to the normal images (step S203). In the inspection test, a part of the region of the inspection image is masked, and the masked region is interpolated using the trained model. By adding a mask region to the region containing an anomaly and performing interpolation of the mask image, an interpolated image in which the anomalous region is almost replaced by a normal region is generated. Since a significant difference occurs between the interpolated image and the original image in the region containing the anomaly, the anomaly is detected thereby (step S204). The detection results are evaluated (step S205). If the evaluation results are insufficient, the mask applied to the inspection image is changed, and the inspection test and evaluation are performed again.

[0019] In the abnormal detection process of this embodiment, it is only necessary to prepare normal images that do not contain abnormalities, and the number and labor of the training data to be prepared can be reduced compared with the prior art. In addition, since interpolation learning is performed using normal images, even for environmental changes such as color changes due to the aging of the imaging device that captures the inspection target, the color of the entire inspection target of the inspection image changes, and interpolation that takes into account the color change is performed. Therefore, it is possible to realize abnormal detection that maintains high accuracy even for environmental changes. Furthermore, when the evaluation result is insufficient, it is not necessary to perform re-learning of the model such as reviewing the model and model training, and it is only necessary to repeat the inspection test and its evaluation. Therefore, the processing time can be shortened, and by taking measures at the site such as the manufacturing line of precision equipment, it is possible to improve the accuracy of abnormal detection.

[0020] Hereinafter, the outline of the abnormal detection in this embodiment described with reference to FIG. 2(A) will be described in more detail by dividing it into a training phase and an inspection phase.

[0021] [1. Training Phase] The processing of the training phase for detecting abnormalities in the inspection target based on the inspection image obtained by photographing the inspection target in this embodiment will be described with reference to FIGS. 3 to 5.

[0022] [1-1. Configuration] FIG. 3 shows the configuration of the training device 300. The training device 300 includes an input unit 301 that inputs training images, a mask generation unit 302 that generates a mask area, a mask image generation unit 303 that generates a mask image with a mask area added to the training image, and a learning unit 310 that learns image interpolation using the mask image and the original training image that is not masked as inputs.

[0023] The training device 300 is composed of a memory and a processor, and the input unit 301, the mask generation unit 302, the mask image generation unit 303, and the learning unit 310 are realized by software in which the memory and the processor cooperate to operate. Note that the realization method of each of the input unit 301, the mask generation unit 302, the mask image generation unit 303, and the learning unit 310 is not limited to software, and a part of them may be realized by hardware.

[0024] The training images are a plurality of image data consisting only of normal images without abnormalities in the inspection target among the inspection images obtained by photographing the inspection target. The training images are input into the training device 300 via the input unit 301.

[0025] The mask generation unit 302 generates a mask area to be applied to the training image. The mask generation unit 302 generates various mask areas with the number, size, shape, and color of the mask areas changed. The number of mask areas is not limited to one, and may be two or more. The size of the mask area is, for example, 16 pixels × 16 pixels, 32 pixels × 32 pixels, 64 pixels × 64 pixels, etc. The shape of the mask area may be a rhombus, a star, etc. in addition to a square or a rectangle. The color of the mask area is composed of various colors with any one of the attributes of hue, saturation, and lightness changed. The color of the mask area is not limited to a single color, and may be a gradient, a striped pattern, or a color-divided lattice pattern. The mask generation unit 302 holds information on the number, size, shape, and color of the mask areas to be generated, and information on their combinations, and generates a large number of mask areas based on this.

[0026] The mask image generation unit 303 applies the mask area generated by the mask generation unit 302 to the training image. The mask image generation unit 303 sequentially generates mask images with the position of the mask area on the training image changed.

[0027] The learning unit 310 learns image interpolation using the mask image generated by the mask image generation unit 303 and the training image before the mask area is assigned as inputs. The learning unit 310 is configured by a learning model using a neural network. The learning model is configured by a generative adversarial network (GAN: Generative Adversarial Network). By using the generative adversarial network, interpolation of the mask area can be performed with high accuracy. In the present embodiment, as the generative adversarial network, a method based on GLCIC (Globally and Locally Consistent Image Completion) is adopted, and interpolation of the mask area can be performed with even higher accuracy. The learning model using GLCIC includes three networks, namely, an interpolation network (generator) that performs image interpolation, a global discrimination network (global discriminator) that evaluates whether the input image is real or fake, and a local discrimination network (local discriminator).

[0028] [1-2. Operations] The operation of the training device 300 will be described with reference to FIGS. 3 to 5. FIG. 4 is a flowchart showing the processing of the training device 300. FIG. 5 shows the training image that is input data to the learning unit 310 and the mask image generated by the mask image generation unit 303.

[0029] A training image is input to the training device 300 via the input unit 301 (step S401).

[0030] The mask generation unit 302 generates a mask area to be assigned to the training image (step S402). As the mask area, a monochromatic square area shown in FIGS. 5(B) to 5(D), a monochromatic rectangular area shown in FIGS. 5(E) to 5(G), an area composed of two areas of a square and a rectangle in monochromatic shown in FIGS. 5(H) to 5(J), or an area composed of a square with a color gradient shown in FIGS. 5(K) to 5(M) is generated. The mask generation unit 302 first generates a monochromatic square mask area on the mask image shown in FIGS. 5(B) to 5(D).

[0031] The mask image generation unit 303 assigns the mask area generated by the mask generation unit 302 to the training image (step S403). The mask image generated by the mask image generation unit 303 is shown in FIG. 5(B). The training image (normal image) shown in FIG. 5(A) and the mask image shown in FIG. 5(B) are input to the learning unit 310. The learning unit 310 learns image interpolation using the input training image and mask image (step S404).

[0032] If the position of the mask area on the mask image is determined and the final position on the mask image has not been reached (No in step S405), the position of the mask area on the mask image is changed (step S406). In the present embodiment, the position of the mask area on the mask image is changed from the upper left to the lower right so as not to overlap each other, but the change in the position of the mask area is not limited to this. For example, the mask areas may be shifted vertically and horizontally by several pixels from the upper left to the lower right of the training image so that the mask areas overlap each other. Also, the position of the mask area on the mask image may be shifted regularly or randomly.

[0033] The mask image generation unit 303 generates a mask image shown in FIG. 5(C) with the position of the mask area changed (step S403). The learning unit 310 learns image interpolation using the training image (normal image) shown in FIG. 5(A) and the mask image shown in FIG. 5(C) (step S405). Thereafter, the learning unit 310 sequentially learns image interpolation using the mask image with the position of the mask area changed and the training image (normal image). If, as a result of changing the position of the mask area and determining the position of the mask area on the mask image, the final position on the mask image has been reached (Yes in step S405), it is determined whether the processing has been completed for mask areas of all preset patterns that differ in number, size, shape, and color (step S407). The mask areas used in the learning phase are a large number of areas composed of the possible values of each of the number, size, shape, and color and their various combinations. If the processing for all mask areas has not been completed (No in step S407), the mask generation unit 302 generates a mask area by changing any one or more of the number, size, shape, and color of the mask area (step S408), and the mask generation unit 302 generates a mask area (step S402).

[0034] Next, the mask generation unit 302 generates a rectangular mask area in an intermediate color (represented by gray in FIG. 5) on the mask images shown in FIGS. 5(E) to (G) (step S402). Hereinafter, in the same manner as the processing related to FIGS. 5(C) to (D), the mask image generation unit 303 sequentially generates the mask images shown in FIGS. 5(E) to (G) (step S403). The learning unit 310 sequentially receives the training image (normal image) shown in FIG. 5(A) and the mask images shown in FIGS. 5(E) to (G), and learns image interpolation (step S404).

[0035] Next, the mask generation unit 302 generates a mask area composed of two areas, a square and a rectangle, in a single color shown in FIGS. 5(H) to (J) (step S402). The learning unit 310 sequentially receives the training image (normal image) shown in FIG. 5(A) and the mask images shown in FIGS. 5(H) to (J), and learns image interpolation (step S404).

[0036] Next, the mask generation unit 302 generates a mask region with gradation in the form of a square shown in FIGS. 5(K) to 5(M). This mask region is the last pattern of the mask regions generated and held by the mask generation unit 302. The learning unit 310 sequentially receives the training image (normal image) shown in FIG. 5(A) and the mask images shown in FIGS. 5(K) to 5(M), and learns image interpolation (step S404). After that, when the processing of all mask regions is completed (Yes in step S407), the processing of the training phase ends.

[0037] In the training phase, by learning image interpolation using a large number of mask regions with different positions, numbers, sizes, shapes, and colors, in the inspection phase, it becomes possible to perform anomaly detection using mask regions corresponding to the positions, numbers, sizes, shapes, and colors of abnormal regions on the inspection target, and the accuracy of anomaly detection can be improved.

[0038] [2. Inspection Phase] Based on the inspection image obtained by photographing the inspection target in the present embodiment, the processing of the inspection phase for detecting anomalies in the inspection target will be described with reference to FIGS. 1, 6 to 8.

[0039] [2-1. Configuration] FIG. 1 shows the configuration of the anomaly detection device 100. The anomaly detection device 100 includes an input unit 101 for inputting an inspection image, a mask generation unit 102 for generating a mask region, a mask image generation unit 103 for generating a mask image with a mask region added to the inspection image, an interpolation image generation unit 104 for interpolating the mask region of the mask image to generate an interpolation image, an image synthesis unit 105 for synthesizing a plurality of interpolation images, a smoothing processing unit 106 for performing smoothing processing on the synthesized image, and an anomaly detection unit 110 for detecting an abnormal region of the inspection image and determining an anomaly in the inspection target. The anomaly detection unit 110 is composed of a difference operation unit 111 and a determination unit 112.

[0040] The abnormality detection device 100 is composed of a memory and a processor. The input unit 101, mask generation unit 102, mask image generation unit 103, interpolation image generation unit 104, image synthesis unit 105, smoothing processing unit 106, and abnormality detection unit 110 are realized by software in which the memory and the processor cooperate to operate. Note that the implementation method of each of the input unit 101, mask generation unit 102, mask image generation unit 103, interpolation image generation unit 104, image synthesis unit 105, smoothing processing unit 106, and abnormality detection unit 110 is not limited to software, and a part of it may be realized by hardware.

[0041] The inspection image is image data obtained by photographing an inspection target, and is a plurality of image data including those with abnormalities in the inspection target. The inspection image is input to the abnormality detection device 100 via the input unit 101.

[0042] The mask generation unit 102 generates mask areas to be applied to the inspection image. The mask generation unit 102 generates various mask areas with the number, size, shape, and color of the mask areas changed. The number of mask areas is not limited to one, and may be 2 to 4. The size of the mask area is 16 pixels × 16 pixels, 32 pixels × 32 pixels, 64 pixels × 64 pixels, etc. The shape of the mask area may be a rhombus, a star, etc. in addition to a square or a rectangle. The color of the mask area is composed of various colors with any one of the attributes of hue, saturation, and lightness changed. The color of the mask area is not limited to a single color, and may be a gradient, a striped pattern, or a color-separated lattice pattern. The mask generation unit 102 holds information on the number, size, shape, and color of the mask areas to be generated, as well as information on their combinations, and generates a large number of mask areas based on this.

[0043] The mask image generation unit 103 applies the mask areas generated by the mask generation unit 102 to the inspection image. The mask image generation unit 103 generates a mask image with the position of the mask area on the inspection image changed.

[0044] The interpolation image generation unit 104 is configured by a learned model by the learning unit 310 that has learned in the training phase. The learned model is configured by an adversarial generation network. In the present embodiment, a method based on GLCIC is used as the adversarial generation network.

[0045] The image synthesis unit 105 synthesizes a plurality of interpolation images generated by the interpolation image generation unit 104 for different positions of the mask regions to generate a synthesized image.

[0046] The smoothing processing unit 106 performs smoothing of the synthesized image generated by the image synthesis unit 105. In the present embodiment, a Gaussian filter is used for smoothing, but it is not limited thereto. For example, an average value filter, a weighted average value filter, a median filter, etc. can also be used. Also, the smoothing processing unit 106 also performs smoothing of the inspection image input to the difference calculation unit 111.

[0047] The abnormality detection unit 110 detects an abnormal region in the inspection image and determines the abnormality of the inspection target. The difference calculation unit 111 compares the pixels at the same coordinates in the synthesized image output by the smoothing processing unit 106 and the inspection image to obtain the difference d for each pixel. The determination unit 112 determines an evaluation value according to the magnitude of the difference d and obtains a total evaluation value obtained by summing the evaluation values for each pixel.

[0048] Here, the difference d for each pixel can be calculated based on the difference in luminance values in the case of a grayscale image. d may be a simple difference value, an absolute value of the difference, or a squared value of the difference. Also, an index for evaluating the similarity of images, such as SSIM (the Structural SIMilarity index measure), may be used. Also, in the case of a color image, the difference d for each pixel can be calculated based on the color difference. For example, in the case of an RGB image, d can be calculated based on the difference value of each of the RGB values, the absolute value of the difference, the squared value of the difference, or the sum value or average value thereof. Also, d may be calculated using the CIEDE2000 color difference, which is considered to be relatively close to the human color discrimination range.

[0049] [2-2. Operation] The operation of the abnormality detection device 100 will be described with reference to FIGS. 1, 6 to 8. FIG. 6 is a flowchart showing the processing of the abnormality detection device 100. FIGS. 7 and 8 show the inspection image input to the abnormality detection device 100, the mask image generated by the mask image generation unit 103, the interpolated image generated by the interpolation image generation unit 104, and the composite image output by the image composite unit 105. In FIGS. 7 and 8, FIGS. 7(A) and 8(A) are the same image and are inspection images of an inspection target with an abnormality. FIGS. 7(B) and 8(B) are images (normal images) of the same inspection target without an abnormality. In the present embodiment, the operation when FIGS. 7(A) and 8(A), which are images of an inspection target with an abnormality, are input to the abnormality detection device 100 as inspection images will be described.

[0050] An inspection image shown in FIG. 7(A) is input to the abnormality detection device 100 via the input unit 101 (step S601).

[0051] The mask generation unit 102 generates a mask area to be applied to the inspection image (step S602). The mask generation unit 102 first sets, as the mask area to be set, based on the size, shape, and color information of the abnormality that can occur in the inspection target set by the user in advance. The mask generation unit 102 selects, as the size and shape of the mask area, those that are close to the set size and shape of the abnormality and include it, and selects, as the color of the mask area, a color different from the color of the inspection target, for example, a complementary color, to generate the mask area. The information on the size and shape of the abnormality of the inspection target and the color information of the inspection target by the user may use the information held by the mask generation unit 102 in advance, or each time the abnormality detection device 100 is operated, the user inputs the information and uses the input information. For example, as a result of operating the abnormality detection device 100, a determination result that there is no abnormality in the inspection target is output, and when performing the abnormality detection process again, the user inputs the information and uses the input information. The mask generation unit 102 generates monochromatic square mask areas shown in FIGS. 7(C), (E), (G), and (I).

[0052] The mask image generation unit 103 assigns the mask area generated by the mask generation unit 102 to the inspection image (step S603). The mask image generated by the mask image generation unit 103 is shown in FIG. 7(C).

[0053] The interpolation image generation unit 104 interpolates the mask area of the input mask image to generate an interpolation image and stores it in the memory (step S604). The interpolation image generated by the interpolation image generation unit 104 is shown in FIG. 7(D).

[0054] If the position of the mask area on the mask image is determined and the final position on the mask image has not been reached (No in step S605), the position of the mask area on the mask image is changed (step S606). In the present embodiment, the position of the mask area on the mask image is changed from the upper left to the lower right so as not to overlap each other, but the change in the position of the mask area is not limited to this. The mask areas may be shifted vertically and horizontally by several pixels from the upper left to the lower right of the training image so that they overlap each other, or the positions of the mask areas on the mask image may be shifted regularly, or may be shifted randomly.

[0055] The mask image generation unit 103 generates the mask image shown in FIG. 7(E) with the changed position of the mask area (step S603). The interpolation image generation unit 104 interpolates the mask area of the input mask image to generate the interpolation image shown in FIG. 7(F) and stores it in the memory (step S604).

[0056] Thereafter, while sequentially changing the position of the mask area until the mask area reaches the final position on the mask image, the interpolation image generation unit 104 generates an interpolated image obtained by interpolating the mask image and stores it in the memory. FIG. 7(G) shows the mask image when the mask image generation unit 103 assigns a mask area to an abnormal area on the inspection target. By performing interpolation of the mask area by the interpolation image generation unit 104, the area that was abnormal within the mask area is interpolated and replaced with a substantially normal area, and the interpolated image shown in FIG. 7(H) is generated. FIG. 7(I) shows the mask image when the mask area is moved and a mask area is assigned to an abnormal area on the inspection target at a position different from the position shown in FIG. 7(G). The interpolation image generation unit 104 performs interpolation on the mask area to generate the interpolated image shown in FIG. 7(J), in which the area that was abnormal within the mask area is interpolated and replaced with a substantially normal area.

[0057] Thereafter, when the position of the mask area on the mask image reaches the final position on the mask image (Yes in step S605), using the plurality of mask images generated while sequentially changing the position of the mask area as inputs, the interpolation image generation unit 104 generates and the memory stores a plurality of interpolated images, and the image composition unit 105 reads and composes them, and outputs the composite image shown in FIG. 7(K) (step S607).

[0058] The smoothing processing unit 106 smooths the composite image generated by the image composition unit 105 and the inspection image (step S608). The difference calculation unit 111 compares the pixels at the same coordinates in the composite image output by the smoothing processing unit 106 and the inspection image, and obtains the difference d for each pixel (step S609). The determination unit 112 determines an evaluation value according to the magnitude of the difference d, obtains the total evaluation value obtained by summing the evaluation values for each pixel, and stores the evaluation value for each pixel and the total evaluation value. The determination unit 112 determines whether the number of determinations performed has reached a preset determination count threshold. If not, it changes the number, size, shape, and color of the mask areas (step S612), and the mask generation unit 102 generates the changed mask areas (step S603). Note that the determination count threshold is appropriately set according to the abnormal state of the inspection target and the like.

[0059] When the total evaluation value exceeds the threshold Te1 in the process using the mask area before the change, that is, when there is a high possibility that an abnormal area is detected in the abnormality detection process using the mask area before the change, the mask generation unit 102 generates a new mask area with slight changes without significantly changing the size, shape, or color of the mask area before the change. On the other hand, when the total evaluation value is equal to or less than the threshold Te1 in the process using the mask area before the change, that is, when there is a high possibility that an abnormal area cannot be detected in the abnormality detection process using the mask area before the change, the mask generation unit 102 generates a new mask area with significantly different size, shape, and color from the mask area before the change. Note that the threshold Te1 is appropriately set according to the abnormal state of the inspection target and the like. The mask generation unit 102 generates a mask area obtained by slightly expanding the size of the mask area before the change, as shown in FIGS. 8(C), (E), (G), and (I).

[0060] In this embodiment, the content of the change of the mask area is determined based on the total evaluation value. However, the number, size, and shape of the abnormal area may be estimated from the set of pixels with high evaluation values using the total evaluation value and the evaluation value for each pixel, and based on the estimation, the number, size, and shape of the mask area may be determined and generated.

[0061] Also, when determining the content of the change of the mask area, an optimal mask area may be estimated using Bayesian optimization based on the number, size, shape, and color of the mask areas evaluated so far and the total evaluation value and the evaluation value for each pixel at that time, and the number, size, shape, and color of the mask area may be determined.

[0062] The mask image generation unit 103 assigns the changed mask area generated by the mask generation unit 102 to the inspection image (step S603). The mask image generated by the mask image generation unit 103 is shown in FIG. 8(C).

[0063] The interpolation image generation unit 104 interpolates the mask area of the input mask image to generate an interpolation image and stores it in the memory. The interpolation image generated by the interpolation image generation unit 104 is shown in FIG. 8(D).

[0064] Thereafter, while sequentially changing the position of the mask region until the mask region reaches the final position on the mask image, the interpolation image generation unit 104 generates an interpolation image obtained by interpolating the mask image and stores it in the memory. FIG. 8(G) shows a mask image when the mask image generation unit 103 assigns a mask region to an abnormal region on the inspection target. By performing interpolation on the mask region, the interpolation image generation unit 104 interpolates the region that was abnormal within the mask region and replaces it with a region that is almost normal, generating the image shown in FIG. 8(H). In the case of the mask image in FIG. 8(G), since the mask region is assigned in a form that covers the entire abnormality, the interpolation image generated by the interpolation image generation unit 104 becomes an image that is almost indistinguishable from the normal image as shown in FIG. 8(H).

[0065] Thereafter, when the position of the mask region on the mask image reaches the final position on the mask image (Yes in step S605), the image composition unit 105 composes the plurality of interpolation images stored in the memory to generate the composite image shown in FIG. 8(K) (step S607).

[0066] The smoothing processing unit 106 smooths the composite image generated by the image composition unit 105 and the inspection image (step S608). The difference calculation unit 111 compares the pixels at the same coordinates in the composite image output by the smoothing processing unit 106 and the inspection image, and obtains the difference d for each pixel (step S609). The determination unit 112 determines an evaluation value according to the magnitude of the difference d, obtains the total evaluation value obtained by summing the evaluation values for each pixel, and stores the evaluation value for each pixel and the total evaluation value.

[0067] The determination unit 112 determines whether the number of determinations performed has reached a preset determination count threshold. If it has reached the threshold, the determination unit 112 determines the abnormal region. For the one with the maximum total evaluation value, the determination unit 112 determines the set of pixels as the abnormal region when the number of pixels whose pixel-by-pixel evaluation value exceeds a preset threshold Td1 exceeds a preset threshold Ta1. On the other hand, for the one with the maximum total evaluation value, the determination unit 112 determines that there is no abnormal region when the number of pixels whose pixel-by-pixel evaluation value exceeds a preset threshold Td1 does not exceed a preset threshold Ta1. Note that the method for determining the evaluation value and the thresholds Td1 and Ta1 are appropriately set according to the abnormal state of the inspection target and the like.

[0068] Note that in this embodiment, the determination unit 112 makes the final determination of the abnormal region when the number of determinations using different mask regions reaches a preset determination count threshold. However, when the total evaluation value exceeds a preset threshold, without changing the subsequent mask region, the set of pixels whose pixel-by-pixel evaluation value exceeds a preset threshold Td1 may be determined as the abnormal region.

[0069] In this way, by changing the number, size, shape, and color of the mask regions according to the number, size, shape, and color of the abnormalities on the inspection target, interpolating the mask images with the changed mask regions, and comparing the generated images with the inspection images, the abnormalities of the inspection target can be detected with higher accuracy.

[0070] [3. Effects, etc.] The abnormality detection device 100 in this embodiment is an abnormality detection device 100 that uses a neural network to detect an abnormality in an inspection target based on an inspection image obtained by photographing the inspection target. The abnormality detection device 100 includes a mask generation unit 102 that generates a mask region to be applied to the inspection image, a mask image generation unit 103 that generates a mask image with the mask region applied thereto, and a learned model that is trained to output an interpolated image in which the mask region is interpolated, using, as an input, a mask image in which at least one of the position, number, size, shape, and color of the mask region in the mask image of the mask region is changed, based on the mask image with the mask region applied to a normal image without abnormality among the inspection images. An interpolation image generation unit 104 that outputs an interpolated image obtained by interpolating the mask region of the mask image output by the mask image generation unit 103; an image synthesis unit 105 that synthesizes a plurality of interpolated images generated by the interpolation image generation unit 104 with a plurality of mask images having the mask regions applied thereto at different positions and outputs a synthesized image; and an abnormality detection unit 110 that detects an abnormal region in the inspection image based on the difference between the inspection image and the synthesized image and determines an abnormality in the inspection target. By using a learned model trained to perform mask interpolation using a normal image, for an inspection image including an abnormality, an image generated by applying a mask region and interpolating the mask region has a significant difference from the original inspection image in the region including the abnormality, so that an abnormality in the inspection target can be detected. Furthermore, by using a learned model trained to output an interpolated image in which the mask region is interpolated, using, as an input, a mask image in which at least one of the position, number, size, shape, and color of the mask region in the mask image of the mask region is changed, the abnormality in the inspection target can be detected with higher accuracy regardless of the size and shape of the abnormality on the inspection target and the color of the inspection target.

[0071] The mask generation unit 102 changes at least one of the number, size, shape, and color of the mask areas, and the abnormality detection device 100 may repeat from the generation of the mask areas to the detection of the abnormal areas until a predetermined condition is satisfied. By appropriately setting the size of the mask area, for example, as in the inspection image shown in FIG. 8(A) and the interpolated image shown in FIG. 8(H), the difference between the inspection image and the interpolated image becomes significant. As a result, the difference between the composite image obtained by combining the interpolated image and the inspection image also becomes clear, so that the abnormal area can be detected with higher accuracy. If the set mask area is too large compared to the size of the abnormal area in the inspection target, the proportion of the abnormal area in the interpolated area becomes small, so that the difference between the generated composite image and the abnormal area of the inspection image becomes small, and the detection accuracy of the abnormal area decreases. If the set mask area is too small compared to the size of the abnormal area in the inspection target, the mask area is affected by the surrounding abnormal areas, and the difference between the generated composite image and the abnormal area of the inspection image becomes small, and the detection accuracy of the abnormal area decreases. By appropriately setting the size of the mask area, the abnormal area can be detected with high precision. Also, when the shape of the mask area is similar to the shape of the abnormal area, the abnormal area can be detected with higher precision.

[0072] As a change in the color of the mask area, the abnormality detection device 100 may change at least one of the hue, lightness, and saturation. When the color of the mask area is set to the same color as the color of the inspection target, the difference between the mask area and the surrounding area without the mask becomes small when the interpolation image generation unit 104 performs interpolation, and the accuracy of interpolation decreases. By setting the mask area to a color different from the inspection target, for example, a complementary color, more accurate abnormality detection becomes possible. The mask generation unit 102 may generate stripe patterns, gradations, or grid-like mask areas. For example, by setting a horizontal stripe pattern mask area for a vertically striped inspection target and performing interpolation, the difference between the mask area and the area outside the mask area in the inspection image becomes significant, and the abnormal area can be detected with higher precision. A large area may be set as the mask area, and the inside of the mask area may be composed of various colors.

[0073] The trained model may be configured by an adversarial generation network. By configuring it with an adversarial generation network, it becomes possible to accurately interpolate the mask region.

[0074] The trained model is configured by an adversarial generation network, and the adversarial generation network may include an interpolation network, a global discrimination network, and a local discrimination network. Thereby, it becomes possible to perform the interpolation of the mask region with even higher accuracy.

[0075] The anomaly detection method in the present embodiment is an anomaly detection method that uses a neural network to detect an anomaly of an inspection object based on an inspection image obtained by photographing the inspection object. Based on a mask image obtained by assigning a mask region to a normal image without anomalies among the inspection images, using as input a mask image in which at least one of the position, number, size, shape, and color of the mask region in the mask image of the mask region is changed, a learning step of generating a trained model trained to output an interpolated image in which the mask region is interpolated, a mask generation step of generating a mask region to be assigned to the inspection image, a mask image generation step of generating a mask image with the mask region assigned, an interpolated image generation step of using the trained model to output an interpolated image obtained by interpolating the mask region of the mask image output by the mask image generation step, an image synthesis step of synthesizing a plurality of interpolated images generated in the interpolated image generation step with a plurality of mask images having the mask region assigned at different positions and outputting a synthesized image, and an anomaly detection step of detecting an anomaly region in the inspection image based on the difference between the inspection image and the synthesized image and determining the anomaly of the inspection object. By using a trained model trained to output an interpolated image in which the mask region is interpolated with at least one of the position, number, size, shape, and color of the mask region in the mask image of the mask region changed as input, it is possible to detect the anomaly of the inspection object with higher accuracy regardless of the size and shape of the anomaly on the inspection object and the color of the inspection object.

[0076] The anomaly detection program in this embodiment is an anomaly detection program that causes a computer to execute an anomaly detection method for detecting an anomaly of an inspection target based on an inspection image obtained by photographing the inspection target using a neural network. Based on a mask image obtained by assigning a mask area to a normal image without anomalies among the inspection images, using, as input, a mask image in which at least one of the position, number, size, shape, and color of the mask area in the mask image of the mask area is changed, a learning step of generating a learned model trained to output an interpolated image in which the mask area is interpolated, a mask generation step of generating a mask area to be assigned to the inspection image, a mask image generation step of generating a mask image with the mask area assigned thereto, an interpolated image generation step of outputting an interpolated image obtained by interpolating the mask area of the mask image output by the mask image generation step using the learned model, an image synthesis step of synthesizing a plurality of interpolated images generated in the interpolated image generation step with a plurality of mask images having the mask area assigned thereto at different positions and outputting a synthesized image, and an anomaly detection step of detecting an anomaly area in the inspection image based on the difference between the inspection image and the synthesized image and determining the anomaly of the inspection target. By using a learned model trained to output an interpolated image in which the mask area is interpolated, as input, a mask image in which at least one of the position, number, size, shape, and color of the mask area in the mask image of the mask area is changed, it is possible to detect the anomaly of the inspection target with higher accuracy regardless of the size and shape of the anomaly on the inspection target and the color of the inspection target.

[0077] (Other embodiments) As described above, embodiments have been described as examples of the present invention. However, the present invention is not limited thereto, and can also be applied to embodiments in which changes, replacements, additions, omissions, etc. are made.

[0078] Therefore, other embodiments will be exemplified below.

[0079] The abnormality detection device in another embodiment includes a mask generation unit, a mask image generation unit, an interpolation image generation unit, and an abnormality detection unit, excluding the image composition unit and the smoothing processing unit in the embodiment. The mask generation unit, the mask image generation unit, and the interpolation image generation unit are the same as those described in the embodiment, and the description thereof is omitted. The abnormality detection unit includes a difference calculation unit and a determination unit. The difference calculation unit compares the pixels at the same coordinates in the interpolation image output by the interpolation image generation unit and the inspection image, and obtains the difference d for each pixel. When the number of pixels whose difference between the pixels at the same coordinates exceeds the threshold value Td2 exceeds the threshold value Ta2, the determination unit determines that the region is an abnormal region and outputs it. Note that the threshold values Td2 and Ta2 are appropriately set according to the abnormal state of the inspection target, etc.

[0080] With such a configuration, the processing time for abnormality detection can be shortened, and the accuracy of abnormality detection of the inspection target can be improved.

Explanation of Signs

[0081] 100 Abnormality detection device 101 Input unit 102 Mask generation unit 103 Mask image generation unit 104 Interpolation image generation unit 105 Image composition unit 106 Smoothing processing unit 110 Abnormality detection unit 111 Difference calculation unit 112 Determination unit 300 Training device 301 Input unit 302 Mask generation unit 303 Mask image generation unit 310 Learning unit

Claims

1. An abnormality detection device that uses a neural network to detect an abnormality in an inspection target based on an inspection image obtained by photographing the inspection target, comprising: a mask generation unit that generates a mask region to be applied to the inspection image; a mask image generation unit that generates a mask image to which the mask region is applied; at least one of the position, number, size, shape, and color of the mask region in the mask image of the mask region is changed based on a mask image obtained by applying a mask region to a normal image without abnormality in the inspection image, and an interpolation image in which the mask region is interpolated is output using a learned model that has been trained to output an interpolation image in which the mask region is interpolated, and an interpolation image generation unit that outputs an interpolation image obtained by interpolating the mask region of the mask image output by the mask image generation unit; an image synthesis unit that synthesizes a plurality of the interpolation images generated by the interpolation image generation unit with a plurality of mask images having the mask region applied at different positions and outputs a synthesized image; an abnormality detection unit that detects an abnormal region in the inspection image based on the difference between the inspection image and the synthesized image and determines an abnormality in the inspection target; An abnormality detection device comprising the above.

2. The mask generation unit changes at least one of the number, size, shape, and color of the mask region, and repeats from the generation of the mask region to the detection of the abnormal region until a predetermined condition is satisfied. The abnormality detection device according to Claim 1.

3. The change in color changes at least one of hue, lightness, and saturation. The abnormality detection device according to Claim 1 or 2.

4. The learned model is constituted by an adversarial generation network. The abnormality detection device according to Claim 1.

5. The learned model is constituted by an adversarial generation network, and the adversarial generation network includes an interpolation network, a global discrimination network, and a local discrimination network. The abnormality detection device according to Claim 1.

6. An abnormality detection method that uses a neural network to detect an abnormality in an inspection target based on an inspection image obtained by photographing the inspection target, comprising: A learning step of generating a learned model that is learned to output an interpolated image in which the mask region is interpolated, using, as an input, a mask image in which at least one of the position, number, size, shape, and color of the mask region in the mask image is changed, based on a mask image obtained by adding a mask region to a normal image without abnormality among the inspection images; A mask generation step of generating a mask region to be added to the inspection image; A mask image generation step of generating a mask image to which the mask region is added; An interpolated image generation step of outputting an interpolated image obtained by interpolating the mask region of the mask image output by the mask image generation step, using the learned model; An image synthesis step of synthesizing a plurality of the interpolated images generated in the interpolated image generation step, with a plurality of mask images having the mask regions added at different positions as inputs, and outputting a synthesized image; An abnormality detection step of detecting an abnormal region in the inspection image based on the difference between the inspection image and the synthesized image, and determining the abnormality of the inspection target; An abnormality detection method comprising the above steps.

7. An abnormality detection program for causing a computer to execute an abnormality detection method of detecting an abnormality of an inspection target based on an inspection image obtained by photographing the inspection target, using a neural network, the abnormality detection program comprising: A learning step of generating a learned model that is learned to output an interpolated image in which the mask region is interpolated, using, as an input, a mask image in which at least one of the position, number, size, shape, and color of the mask region in the mask image is changed, based on a mask image obtained by adding a mask region to a normal image without abnormality among the inspection images; A mask generation step of generating a mask region to be added to the inspection image; A mask image generation step of generating a mask image to which the mask region is added; An interpolated image generation step of outputting an interpolated image obtained by interpolating the mask region of the mask image output by the mask image generation step, using the learned model; An image synthesis step of synthesizing a plurality of the interpolated images generated in the interpolated image generation step, with a plurality of mask images having the mask regions added at different positions as inputs, and outputting a synthesized image; An abnormality detection step of detecting an abnormal region in the inspection image based on the difference between the inspection image and the synthesized image, and determining the abnormality of the inspection target; An abnormality detection program that causes [it] to execute.

Images