Image estimation method, evaluation value estimation method, and image estimation device

JP2024000211A5Active Publication Date: 2024-11-29HITACHI LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2022098864
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2022-06-20
Publication Date
2024-11-29
Estimated Expiration
2042-06-20

AI Technical Summary

Benefits of technology

【0011】 本発明によれば、少ない学習画像であっても推定エンジンの推定性能を高めることができる。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To improve estimation performance of an estimation engine even for a less learning image.SOLUTION: An image estimation method includes steps of: acquiring multiple learning images f imaging a target object for a learning; generating multiple conversion images f' by performing a conversion processing on the multiple learning images f; determining an internal parameter of an estimation engine in a machine learning type using the multiple conversion images f'; acquiring an input image h imaging a real target object; generating a converted image h' by performing the conversion processing on the input image h; estimating an estimation image H' by inputting the converted image h' into the trained estimation engine; generating an inverse transformation estimated image H by performing an inverse transformation processing of a part of the conversion processing or all processing of the conversion processing on the estimated image H'; and outputting the inverse conversion estimated image H as an output image. The conversion processing equalizes the pattern of the multiple learning images f related to the conversion item by the image processing to adjust at least one or more conversion items.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image estimation method, an evaluation value estimation method, and an image estimation device. [Background technology]

[0002] In recent years, the performance of machine learning has improved dramatically with the proposal of deep network models such as Convolutional Neural Network (CNN). Many image processing methods using estimation engines based on machine learning have been proposed. For example, as an example of application to visual inspection, Patent Document 1 discloses a method of automatically inspecting defective shapes of welded parts using machine learning. Image processing based on machine learning covers a wide range of applications, including semantic segmentation and recognition, image classification, image conversion, and image quality improvement.

[0003] For example, when the objective is to improve image quality by estimating a high-quality image from an input image, the learning of the estimation engine updates the internal parameters of the estimation engine (such as the network weights or biases) so as to reduce the difference between an estimated image output from the estimation engine using an image of an evaluation object for learning (learning image) as input, and a high-quality image (correct image) that is a previously taught correct estimated image.

[0004] Even when the purpose is image evaluation to estimate an evaluation value of an evaluation object from an input image, the learning of the estimation engine updates the internal parameters of the estimation engine so that the difference between an estimated image output from the estimation engine using a learning image as an input and a correct evaluation value (correct evaluation value) that is taught in advance becomes small, as in the case of image quality improvement. The evaluation value may be the quality level of the evaluation object, the presence or absence of defects, the degree of abnormality, etc. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2020 / 129617 Summary of the Invention [Problem to be solved by the invention]

[0006] In the above-mentioned machine learning-based estimation engine, in order to obtain high estimation performance for input images having a wide range of pattern variations, it is necessary to train the engine with a large number of images that correspond to the pattern variations.

[0007] However, capturing images is generally a laborious task, and as a practical matter, it may be difficult to collect a large number of training images in advance on a production line, etc. In addition, if many pattern variations are taught to an estimation engine, the learning load increases, and there is a risk of overlearning in machine learning. Overlearning is a state in which high performance is obtained for training images, but performance is not obtained for unlearned data, resulting in a decrease in generalization performance.

[0008] The present invention has been made in view of the above circumstances, and has an object to improve the estimation performance of an estimation engine even with a small number of training images. [Means for solving the problem]

[0009] The present application includes a number of means for solving at least some of the above problems, examples of which are as follows.

[0010] In order to solve the above problem, an image estimation method according to one embodiment of the present invention is an image estimation method executed by a computer, and includes: a first learning image acquisition step of acquiring a plurality of learning images f obtained by capturing an object for learning; a first converted image generation step of generating a plurality of converted images f' by applying a conversion process to the plurality of learning images f; a learning step of determining internal parameters of a machine learning estimation engine using the plurality of converted images f'; a second image acquisition step of acquiring an input image h obtained by capturing an actual object; a second converted image generation step of generating a converted image h' by applying the conversion process to the input image h; an estimated image generation step of inputting the converted image h' to the trained estimation engine to estimate an estimated image H'; an inverse transformed estimated image generation step of applying an inverse transformation process of a part or all of the transformation process to the estimated image H' to generate an inverse transformed estimated image H; and an output step of outputting the inverse transformed estimated image H as an output image, wherein the transformation process is an image process that adjusts at least one or more transformation items to uniform the patterns of the plurality of learning images f with respect to the transformation items. Effect of the Invention

[0011] According to the present invention, the estimation performance of an estimation engine can be improved even with a small number of training images.

[0012] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram showing an example of an entire processing sequence performed by the visual inspection system according to the first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing an example of input / output images of the visual inspection system according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example in which the conversion items are position, rotation, and magnification ratio. [Figure 4] FIG. 4 is a diagram showing a schematic example of a case where the transformation items are rotation and distortion. [Figure 5A] FIG. 5A is a diagram showing a schematic example of a case where the conversion items are brightness, contrast, and noise. [Figure 5B] FIG. 5B is a diagram illustrating an example in which the conversion items are shading and shadow. [Figure 6] FIG. 6 is a schematic diagram (part 1) showing an example of a learning method for an estimation engine. [Figure 7] FIG. 7 is a schematic diagram (part 2) showing an example of a learning method for an estimation engine. [Figure 8A] FIG. 8A is a schematic diagram (part 1) showing an example of a GUI (Graphical User Interface). [Figure 8B] FIG. 8B is a schematic diagram (part 2) showing an example of the GUI. [Figure 9] FIG. 9 is a schematic diagram showing an example of an entire processing sequence performed by the visual inspection system according to the second embodiment. [Figure 10] FIG. 10 is a schematic diagram showing an example of a learning method for the estimation engine. [Figure 11] FIG. 11 is a diagram showing the hardware configuration of the appearance inspection system according to each of the first and second embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings for describing the embodiment, the same members are generally given the same reference numerals, and the repeated description thereof will be omitted as appropriate. In the following embodiment, the components (including element steps, etc.) are not necessarily essential, unless otherwise specified or considered to be obviously essential in principle. In addition, when "consisting of A", "consisting of A", "having A", or "including A" is used, it is understood that other elements are not excluded, except when it is specifically specified that only the element is included. Similarly, in the following embodiment, when referring to the shape, positional relationship, etc. of the components, etc., it is understood that the shape, etc. of the components, etc. are substantially approximated or similar to the shape, etc., except when it is specifically specified or considered to be obviously not essential in principle.

[0015] First Embodiment <1. Overall process sequence in visual inspection system> 1 is a schematic diagram showing an example of an entire processing sequence performed by the visual inspection system according to the present embodiment. The visual inspection system according to the present embodiment is an example of an image estimation device, and executes an image estimation method. The image estimation method is a method executed by a computer, and includes a first training image acquisition step of acquiring a plurality of training images f obtained by capturing an object for learning; a first converted image generation step of applying a conversion process to the plurality of training images f to generate a plurality of converted images f'; a learning step of determining internal parameters of a machine learning estimation engine using the plurality of converted images f'; a second image acquisition step of acquiring an input image h obtained by capturing an actual object; a second converted image generation step of applying the conversion process to the input image h to generate a converted image h'; an estimated image generation step of inputting the converted image h' to the trained estimation engine to estimate an estimated image H'; an inverse transformation estimated image generation step of applying an inverse transformation process of a part or all of the conversion process to the estimated image H' to generate an inverse transformation estimated image H; and an output step of outputting the inverse transformation estimated image H as an output image, wherein the conversion process is characterized in that the patterns of the plurality of training images f are made uniform with respect to the conversion items by image processing that adjusts at least one or more conversion items. The estimation engine is also characterized in that it receives an image as input and estimates an enhanced image in which any one of an image restored image, a super-resolution image, and a region of interest is enhanced.

[0016] The above features will be described in detail. One of the purposes of this embodiment is to improve image quality by estimating a high-quality image from an input image. Examples of high-quality images include a restored image in which blur or noise has been removed, a super-resolution image in which image resolution has been improved, and an ROI-enhanced image in which a region of interest (ROI) such as a pattern edge or defect is selectively emphasized. In super-resolution processing, the number of pixels of an input image and an output image may differ. In the learning of the estimation engine, the internal parameters of the estimation engine (network weights, biases, etc.) are updated so that the difference between an estimated image output from the estimation engine using an image of an evaluation object for learning (learning image) as input and a high-quality image (correct image) that is a correct estimated image taught in advance is reduced.

[0017] FIG. 2 is a schematic diagram showing an example of input / output images of the visual inspection system according to the present embodiment. In many industrial products including machines, metals, chemicals, food, and textiles, the structures and appearances of the evaluation objects are diverse, and the types of defects that occur therein are also diverse. FIGS. 2(a1)-(f1) are examples of input images when two gears, one large and one small, are photographed as the evaluation objects, and the defects are, in order, a small foreign body 200, a large foreign body 201, a chip 202, a crack 203, roughness of the gear surface, and a shape deformation of the gear contour. Based on such inspection images, visual inspections are widely performed to evaluate various workmanship such as adhesion of foreign bodies, internal defects and criticality, surface scratches and spots, dirt, shape defects, and assembly defects. Many of these visual inspections are performed by visual judgment by inspectors, and it is required to obtain high-quality images with high defect visibility.

[0018] 2(a2) to (f2) are high-quality images corresponding to Fig. 2(a1) to (f1), respectively, in which defects such as foreign bodies 204, 205, chips 206, and cracks 207 are emphasized, and the texture of the gear surface and the gear contour (edge) are made clear. In the visual inspection system of this embodiment, an image of Fig. 2(a1) to (f1) is input as an input image, and an internal estimation engine is trained to output an image of Fig. 2(a2) to (f2), respectively, as an output image.

[0019] The overall processing sequence in this embodiment will be described with reference to Fig. 1. The processing sequence is roughly divided into a learning phase 100 and an operation phase 101.

[0020] In the learning phase 100, the visual inspection system captures an evaluation object 102 for learning and acquires a learning image f (104) (S103). The image is acquired by capturing an image of the surface or inside of the evaluation object as a digital video using an imaging device such as a CCD (Charge Coupled Device) camera, an optical microscope, a charged particle microscope, an ultrasonic inspection device, or an X-ray inspection device. As another example of "acquiring", an image captured by another system may simply be received and stored in a memory resource of the visual inspection system. The visual inspection system may process still images captured one by one, or may continuously process moving images captured continuously at, for example, 30 fps (frames per second) in real time.

[0021] In general, visual inspection systems use training images f for learning to determine the internal parameters of an estimation engine. To obtain an estimation engine with high estimation performance, it is necessary to prepare many training images f corresponding to pattern variations and use them for learning. However, it may be difficult to collect a large number of training images. To address this issue, data augmentation is known, which processes captured images to generate pseudo images and augments the training images. However, there is a limit to the image variations that can be augmented. In addition, if many pattern variations are taught to the estimation engine, the learning load increases, and there is a risk of overlearning occurring in machine learning.

[0022] In this embodiment, instead of comprehensively preparing learning images for pattern variations, the number of image variations present at the time of learning is reduced, thereby drastically reducing the number of learning images and the learning load. In this embodiment, the information contained in the image is considered to be input to a machine learning estimation engine and other information separately. For example, if the position information on the image of the evaluation object is not related to the estimation in the estimation engine, there is no need to include the position information in the learning image.

[0023] Such information that does not need to be handled by the estimation engine is subjected to a conversion process (called conversion T) in advance by the visual inspection system (S106) to standardize it among all training images (to standardize it to a reference value 108). An image to which conversion T has been applied is called a transformed image. The converted image of training image f in the training phase 100 is f' (109), and the converted image of input image h in the operation phase 101 described later is h' (119).

[0024] Since multiple conversion items may exist in the conversion T, the user selects in advance which conversion items to include in the conversion T (S105). Details regarding the types of conversion items will be described later, but for example, when the position of the evaluation object is included in the conversion items, the conversion T shifts the evaluation object that exists in various positions in each learning image to exist in the same position (reference value) in all learning images.

[0025] The estimation engine receives as input a converted image f' (109) in which all evaluation objects are located at the reference value, and estimates a high-quality image in which the evaluation objects are also located at the reference value as an estimated image F' (111). If conversion T is not performed, it is necessary to prepare and learn a group of learning images that encompass all the different positions of the evaluation objects, but by performing conversion T, it becomes unnecessary to include pattern variations related to the positions of the evaluation objects in the learning images. The visual inspection system trains the converted image f' in the estimation engine and determines the internal parameters 117 of the estimation engine (S110).

[0026] In the operation phase 101, the visual inspection system captures an image of an actual evaluation object 102 (S103) and acquires an input image h (118). Furthermore, the visual inspection system applies a transformation T including the transformation items selected in step S105 of the learning phase 100 to the input image h (S106) and acquires a transformed image h' (119). The visual inspection system then inputs the transformed image h' to an estimation engine that uses the internal parameters 117 determined in the learning phase 100, and estimates an estimated image H' (121), which is a high-quality image of the transformed image h' (S120).

[0027] However, there are cases where some or all of the information homogenized by transformation T is necessary in the final output image output by the visual inspection system. Therefore, the user selects conversion items necessary for the final output from among the conversion items of transformation T (S112), and the visual inspection system applies an inverse transformation U of transformation T for each selected conversion item to the estimated image H' to restore the information before homogenization (S113).

[0028] The image after the inverse transformation is called the inverse transformed estimated image H (122), which is the output image of the visual inspection system. For example, if the estimation engine does not need to handle the position of the evaluation object 102, uniformization is performed using transformation T, but if the position of the evaluation object 102 is information necessary for the user in the output image, the position of the evaluation object that has been uniformized in the estimated image H' is returned to the state in the input image h using transformation U.

[0029] In order to perform the inverse transformation U, when performing transformation T, the appearance inspection system saves information on how the transformation was performed for each transformation item or what the state was before the transformation for each input image h as transformation parameters 123. By performing the inverse transformation U based on the transformation parameters 123, the appearance inspection system can restore the information removed by transformation T for each image by the inverse transformation U.

[0030] Generally, there are many variations in images, but in this embodiment, by performing a transform T and an inverse transform U before and after processing by the estimation engine, respectively, it is possible to partially reduce only the load of machine learning without sacrificing the variation of the output image. Since the image variations handled by the machine learning are reduced, problems related to the cost of collecting training images and the training load can be fundamentally solved. This makes it possible to improve the estimation performance of the estimation engine even with a small number of training images.

[0031] <2. Examples of conversion items in conversion T and inverse conversion U> As described above, in this embodiment, the information contained in an image is considered to be separated into information to be input to a machine learning estimation engine and other information, and the latter information is excluded by transformation T. Furthermore, among the information excluded by transformation T, information necessary for the final output image is restored by inverse transformation U.

[0032] There are multiple conversion items 107, 114 in the conversion T and the inverse conversion U, and what the conversion items 107, 114 should be in the conversion T and the inverse conversion U vary depending on the estimation engine, so any combination is possible. Note that the conversion T and the inverse conversion U may each include multiple conversion items. Specific examples of the conversion items 107, 114 will be described below. Specific examples of the conversion items 107, 114 include position, rotation, inversion, magnification, distortion, image brightness, image contrast, noise, shading, and shadow.

[0033] FIG. 3 is a diagram showing an example in which the conversion items are position, rotation, and magnification ratio.

[0034] First, the learning phase 100 will be described. Figures 3(a1) and (b1) are schematic diagrams showing an example of two learning images f obtained by capturing images of an evaluation object for learning. The evaluation objects in these two images are two gears, one large and one small, and each has the same structure and size, and the foreign matter attached thereto is also the same, but there are differences in the position shift, rotation, and magnification of the evaluation objects on the images. The rectangular dotted frames 300 and 301 surrounding the two gears are auxiliary lines to indicate the differences in position shift, rotation, and magnification of the evaluation objects between the images, and are not drawn in the actual captured images.

[0035] If information on the differences in position shift, rotation, and magnification are necessary for the estimation engine to estimate a high-quality image, this information cannot be removed in advance. On the other hand, if the estimation engine does not need to handle these differences in information, the user selects the position shift, rotation, and magnification as the conversion items of conversion T, and then the appearance inspection system applies conversion T to standardize the evaluation object on the image to the same position, same orientation, and same magnification. Note that the appearance inspection system may automatically select the conversion items of conversion T.

[0036] 3(a2) and (b2) are schematic diagrams showing an example of a converted image f' obtained by standardizing the learning images f shown in FIGS. 3(a1) and 3(b1), respectively, with respect to the above conversion items.

[0037] For example, regarding rotation, in the converted image f', the evaluation object is rotated with the angle at which the centers of the two gears are lined up side by side as the reference value 108. Of course, the reference value 108 for each conversion item is not limited to the example shown in the figure, and can be set arbitrarily. The reference value may be automatically calculated by the visual inspection system based on the learning images. In this case, for example, the average or median value of the above-mentioned angles in the multiple learning images f may be set as the reference value. Furthermore, the visual inspection system may select one of the multiple learning images f and set the above-mentioned angle in the selected learning image f as the reference value. In this case, the state of the other learning images is adjusted to the state of the selected learning image f. Alternatively, the visual inspection system may adopt a reference value input by the user.

[0038] Furthermore, the visual inspection system may set multiple reference values ​​for one conversion item. In this case, for example, a process may be performed in which the state of each learning image is adjusted to the state that is closest among the multiple reference values. The fewer the number of reference values, the more the variation in the learning images is reduced. By having the estimation engine learn using a conversion image f' that is standardized for each conversion item, it is possible to reduce the cost of collecting learning images and the learning load.

[0039] Next, the operation phase 101 will be described. The two input images h used in the operation phase 101 are assumed to be the same as the images in Fig. 3(a1) and (b1). As in the learning phase 100, the visual inspection system generates a converted image h' in which the evaluation object on the image is homogenized to the same position, the same orientation, and the same magnification by the transformation T. The converted image h' becomes the image shown in Fig. 3(a2) and (b2), respectively. Next, the visual inspection system inputs the converted image h' to the estimation engine that has been trained in the learning phase. The estimation engine estimates an estimated image H' (Fig. 3(a3), (b3)), which is a high-quality image of the converted image h'. Fig. 3(a3) and (b3) are schematic diagrams showing an example of the estimated image H'.

[0040] However, estimated image H' loses information relating to the positional shift, rotation, and magnification difference of the object to be evaluated that existed in input image h. Therefore, the visual inspection system performs inverse transformation U on estimated image H' to generate an inverse transformed estimated image H (Figs. 3(a4) and (b4)) in which the information removed by transformation T is restored, and this is used as the output image. Figs. 3(a4) and (b4) are schematic diagrams showing an example of the inverse transformed estimated image H. In inverse transformed estimated image H, rectangular dotted-line frames 309 and 310 surrounding the two gears are the same as the corresponding dotted-line frames 300 and 301 in the input image, and information on position, direction, and magnification, which differ for each image, has been restored.

[0041] By using magnification as a conversion item, even if the actual size of the evaluation object varies, it is possible to standardize the apparent size of the evaluation object on the image by the conversion T. Also, although not shown, by using not only image rotation but also image inversion as a conversion item, it is possible to standardize the appearance.

[0042] FIG. 4 is a diagram showing a schematic example of a case where the transformation items are rotation and distortion.

[0043] In operation phase 101, the visual inspection system applies transformation T to an input image h (Fig. 4(a)), generating a transformed image h' (Fig. 4(b)) in which the orientation and distortion of the object to be evaluated on the image are converted to reference values, and inputs this to the estimation engine. The estimation engine estimates an estimated image H' (Fig. 4(c)), which is a high-quality image of the transformed image h'. At this stage, the estimation engine only improves the quality of the image, so the rectangular dotted-line frames 401, 402 surrounding the two gears drawn as auxiliary lines are the same for both the transformed image h' and the estimated image H'.

[0044] As in the example of Figure 3, the visual inspection system can generate an inverse transformed estimated image H (Figure 4(d)) by performing an inverse transformation U of the transformation T on estimated image H', and use this as the output image. In the inverse transformed estimated image H shown in Figure 4(d), information about the rotation and distortion of the object to be evaluated that existed in the input image h has been restored, and the rectangular dotted-line frames 400, 403 that surround the two gears drawn as auxiliary lines are the same in both images.

[0045] As a variation of the processing method, the inverse transform U does not need to be a complete inverse transform of the transform T. In other words, the transform items of the transform T and the inverse transform U do not need to match. In the example of FIG. 4(b), the transform items of the transform T are rotation and distortion. If you want to include rotation information in the final output image but not distortion information, you can obtain an output image in which only the rotation is restored and distortion is removed by setting the transform item of the inverse transform U to rotation only (FIG. 4(e)). Since distortion is removed, the dotted frame 404, which is an auxiliary line, becomes the same as the dotted frame 402 in the estimated image H' (FIG. 4(c)) by rotating it.

[0046] FIG. 5A is a diagram that shows a schematic example of a case where the conversion items are brightness, contrast, and noise, and FIG. 5B is a diagram that shows a schematic example of a case where the conversion items are shading and shadow.

[0047] The outline of the process other than the conversion items and the meaning of the figures are the same as in Figures 3 and 4.

[0048] 5A(a1)-(a4) are examples in which the conversion items of the conversion T and the inverse conversion U are brightness and contrast. In this case, the appearance inspection system standardizes the apparent brightness and contrast on the learning image or input image (FIG. 5A(a1)) of the evaluation object, which change depending on the imaging conditions, to a reference value by the conversion T to obtain a converted image h' (FIG. 5A(a2)). For example, a process such as matching the brightness histogram of each image to a reference brightness histogram can be considered. A process of converting a color image into a grayscale image can also be considered. The appearance inspection system applies the inverse conversion U to an estimated image H' (5A(a3)), which is estimated by an estimation engine using the converted image h' as ​​input, and generates an inversely converted estimated image H (5A(a4)) as an output image.

[0049] FIG. 5A(b1)-(b5) are examples in which the transformation items of transformation T and inverse transformation U are noise. In this case, the visual inspection system performs transformation T on the learning image or input image (FIG. 5A(b1)) to equalize the noise of these images and obtain a transformed image h' (5A(b2)). The noise may be removed as shown in FIG. 5A(b2), or the noise level of each image may be adjusted to a reference noise amount. If noise is not required in the output image, the estimated image H' from which noise has been removed as shown in FIG. 5A(b3) may be used as the output image as it is. Generally, noise is often unnecessary in high-quality images, but there are cases in which an image looks unnatural if no noise is superimposed on it, or in which it is desired to see the difference in the amount of noise superimposed on each image. In such cases, the visual inspection system can use the inverse transformation estimated image H shown in FIG. 5A(b4) in which noise has been restored by inverse transformation U as the output image.

[0050] As a variation of the processing method, the inverse transformation U does not have to be a complete inverse transformation of the transformation T. In other words, it is possible to change the degree (strength) of applying the inverse transformation U to the transformation T. For example, it would be unnatural if noise were completely removed from the output image as in FIG. 5A(b3), but if it is difficult to observe if noise equivalent to that of the input image is superimposed on the output image as in FIG. 5A(b4), the appearance inspection system can apply a weight to the transformation U to restore a small amount of noise as in FIG. 5A(b5). This is not limited to noise, but the strength can be adjusted for all transformation items.

[0051] FIG. 5B(c1)-(c5) are examples in which the transformation items of the transformation T and the inverse transformation U are shading. In this case, the visual inspection system performs the transformation T on the learning image or the input image (FIG. 5B(c1)) to uniformize the shading of these images and obtain a transformed image h' (FIG. 5B(c2)). As with the noise case of FIG. 5A(b1)-(b5), the visual inspection system can generate as an output image any of the following: an image in which the shading is completely removed (FIG. 5B(c3)), an image in which the shading is completely restored (FIG. 5B(c4)), and an image in which the shading is restored to, for example, half its strength (FIG. 5B(c5)). If the shading appears too strongly in the output image, it is difficult to observe, but there are also cases in which it is unnatural to completely remove the shading. For example, in the real-time display of a high-quality image for a moving image, it is unnatural for the shading not to change at all when the positional relationship between the evaluation object and the lighting is shifted. Therefore, the strength of the restoration by the transformation U can be adjusted according to the user's wishes.

[0052] Fig. 5B (d1) to (d4) are examples in which the transformation items of transformation T and inverse transformation U are shadows. In this case, the visual inspection system applies transformation T to a learning image or an input image (Fig. 5B (d1)) to remove or equalize the shadows in these images to obtain a transformed image h' (Fig. 5B (d2)). If shadows are not necessary in the output image, the visual inspection system may output the estimated image H' from which the shadows have been removed, as shown in Fig. 5B (d3), as it is. Alternatively, the visual inspection system may output the inverse transformed estimated image H, as shown in Fig. 5B (d4), in which noise has been restored by inverse transformation U.

[0053] Although specific examples of conversion items have been shown above, the conversion items are not limited to these, and can be set for any pattern variation included in the image. For example, multiple conversion items may be selected from multiple conversion items 107, 114. The conversion items in conversion T and inverse conversion U vary depending on what pattern variations are handled in the estimation engine and what kind of image the user desires for the main image, so any combination can be set (S105, S112). The conversion items included in inverse conversion U are part or all of the selection items included in conversion T.

[0054] <3. Learning the estimation engine> Figures 6 and 7 are schematic diagrams showing an example of a learning method for an estimation engine. There are two main learning methods, (1) and (2), as shown below. Figures 6(a1)-(a5) and Figures 7(a1)-(a5) correspond to (1), and Figures 6(b1)-(b5) and Figures 7(b1)-(b5) correspond to (2).

[0055] (1) In the learning phase 100, the internal parameters of the image estimation engine are determined based on an estimated image F' estimated by the estimation engine using the transformed image f' as input, and a transformed correct answer image G' obtained by applying a transformation T to the correct answer image G.

[0056] (2) In the learning phase 100, the internal parameters of the estimation engine are determined based on an inverse-transformed estimated image F obtained by applying an inverse transformation U to an estimated image F' estimated by an estimation engine using the transformed image f' as an input, and a ground truth image G for the training image f.

[0057] The learning of the estimation engine is basically performed by updating the internal parameters of the estimation engine, such as the weights and biases of the network, so that the difference between an estimated image output from the estimation engine using a learning image as an input and a previously taught correct image becomes small. Note that as the difference between the estimated image and the correct image, for example, the difference in pixel data for each pixel of each image can be used.

[0058] However, in this embodiment, the estimation engine outputs not an estimated image but a transformed estimated image, which cannot be directly compared with the correct image. Therefore, in (1), a transformed correct image obtained by performing a transformation process on the correct image is compared with the transformed estimated image, and in (2), an inverse transformed estimated image obtained by performing an inverse transformation U on the estimated image is compared with the correct image.

[0059] As a specific example of (1), in the learning method of Figs. 6(a1)-(a5), the visual inspection system applies transformation T to training image f (Fig. 6(a1)) to obtain transformed image f' (Fig. 6(a2)). Next, the visual inspection system uses the transformed image f' as input and estimates an estimated image F' (Fig. 6(a3)) by an estimation engine. The visual inspection system also applies transformation T to ground truth image G (Fig. 6(a4)) to obtain a transformed ground truth image G' (Fig. 6(a5)). The visual inspection system calculates a loss value based on the difference between estimated image F' (Fig. 6(a3)) and transformed ground truth image G' (Fig. 6(a5)), and determines internal parameters of the estimation engine so as to reduce the loss value.

[0060] As a specific example of (2), in the learning method of Figs. 6(b1) to (b5), the visual inspection system applies transformation T to training image f (Fig. 6(b1)) to obtain transformed image f' (Fig. 6(b2)). The visual inspection system uses the transformed image f' as input and estimates estimated image F' (Fig. 6(b3) or reference numeral 111 in Fig. 1) by an estimation engine. The visual inspection system applies inverse transformation U (step S113 in Fig. 1) to estimated image F' to obtain inverse transformed estimated image F (Fig. 6(b4) or reference numeral 115 in Fig. 1). The visual inspection system calculates a loss value based on the difference between the inverse transformed estimated image F (Fig. 6(b4)) and the correct image G (Fig. 6(b5)), and determines internal parameters of the estimation engine so as to reduce the loss value.

[0061] In the above-described embodiment, learning was performed by minimizing the loss value based on the difference between images, but learning may also be performed by a Generative Adversarial Network (GAN). A GAN consists of two competing networks called a Generator and a Discriminator. By using a GAN, learning is possible even in cases where the estimated image and the correct image are not aligned. In addition, as shown in FIG. 7, even if the evaluation target of the estimated image and the correct image is different, the estimated image can be trained to have the image quality of the correct image.

[0062] That is, as a specific example of (1) using GAN, in the learning method of Fig. 7(a1) to (a5), the visual inspection system applies transformation T to training image f (Fig. 7(a1)) to obtain transformed image f' (Fig. 7(a2)). The visual inspection system uses the transformed image f' as input and estimates estimated image F' (Fig. 7(a3) or reference numeral 111 in Fig. 1) by an estimation engine (generator). The visual inspection system also applies transformation T to correct image G (Fig. 7(a4)) to obtain transformed correct image G' (Fig. 7(a5)). The visual inspection system uses a discriminator to determine whether the estimated image F' (Fig. 7(a3)) is the same type as the transformed correct image G' (Fig. 7(a5)), and increases the loss value if it is determined that the two are not the same type. The generator is trained to make the estimated image F' the same type of image as the converted answer image G', and the discriminator is trained to detect that the estimated image F' is not the same type as the converted answer image G'. By repeating this adversarial training, the estimation accuracy of the estimation engine is improved.

[0063] As a specific example of (2) using GAN, in the learning method of Fig. 7(b1) to (b5), the visual inspection system performs transformation T on training image f (Fig. 7(b1)) to obtain transformed image f' (Fig. 7(b2)). The visual inspection system uses the transformed image f' as input and estimates estimated image F' (Fig. 7(b3) or reference numeral 111 in Fig. 1) by an estimation engine (Generator). The visual inspection system performs inverse transformation U (step S113 in Fig. 1) on estimated image F' to obtain inverse transformed estimated image F (Fig. 7(b4) or reference numeral 115 in Fig. 1). The visual inspection system uses a Discriminator to determine whether the inverse transformed estimated image F (Fig. 7(b4)) is the same type as the correct image G (Fig. 7(b5)), and increases the loss value if it is determined that they are not the same type. The generator is trained to make the inverse-transformed estimated image F the same type of image as the ground truth image G, and the discriminator is trained to detect that the inverse-transformed estimated image F is not the same type of image as the ground truth image G. By repeating this adversarial training, the estimation accuracy of the estimation engine is improved.

[0064] <4.GUI (Graphical User Interface)> The visual inspection system according to this embodiment is characterized by having a GUI that accepts designation of a learning method for the estimation engine, including conversion items of the conversion process and conversion items of the inverse conversion process, from a user.

[0065] 8A and 8B are schematic diagrams showing examples of GUI. During learning of the estimation engine, the user can specify with a radio button 800 whether to "compare the estimated image F' with the converted correct image G'" as in FIG. 6(a1)-(a5) or FIG. 7(a1)-(a5) or "compare the inverse-transformed estimated image F with the correct image G" as in FIG. 6(b1)-(b5) or FIG. 7(b1)-(b5). The user can specify with a radio button 801 whether to use "difference between compared images" (FIG. 6) or "GAN" (FIG. 7) as the image comparison method. When the user specifies the ID of the learning data with a pull-down menu 802, some or all of the learning image f, converted image f', estimated image F', inverse-transformed estimated image F, correct image G, and converted correct image G' associated with that ID can be displayed in the image display area 803. In this way, by displaying not only the training image f but also the transformed image f', the estimated image F', the inverse transformed estimated image F, the correct image G, and the transformed correct image G', the user can know what processing has been performed.

[0066] Although the illustration shows a group of images in the learning phase 100, some or all of the input image h, transformed image h', estimated image H', and inverse transformed estimated image (output image) H in the operation phase 101 can also be displayed in a similar manner. The transformation items and transformation methods for transformation T and inverse transformation U can be specified in a transformation specification area 804. The transformation items for transformation T can be specified using check boxes 805, and the method for setting the reference values ​​for the specified transformation items can be specified using radio buttons 806.

[0067] Items that can be specified using radio button 806 include "Automatically calculate reference value," "Match to image of ID XX," and "Specify reference value." When the reference value is a position and "Automatically calculate reference value" is checked, the appearance inspection system calculates, for example, the average position of the center positions of multiple input images f as the reference value. Also, when "Match to image of ID XX" is checked, the appearance inspection system calculates the center position of the input image f associated with the specified ID "XX" as the reference value. When "Specify reference value" is checked, the appearance inspection system displays a screen prompting the user to input a reference value, and adopts the value input by the user on that screen as the reference value.

[0068] Furthermore, the user can specify conversion items of the inverse transformation U using check boxes 807, and can specify the strength of each specified conversion item in an strength specification area 808. The appearance inspection system accepts the specified strengths and executes the inverse transformation U with those strengths.

[0069] <Second embodiment> <5. Estimation of evaluation value by visual inspection system> In the first embodiment, the objective of the visual inspection system is image quality improvement by estimating a high-quality image from an input image, but the present invention is not limited to this. In the second embodiment, the objective of the visual inspection system is image evaluation by estimating an evaluation value from an input image. As examples of the evaluation value, in the case of visual inspection, the evaluation value is any one of the inspection results such as the quality level of the evaluation object, the presence or absence of a defect, the degree of abnormality, and the criticality, in the case of segmentation, the evaluation value is the label of an area, and in the case of defect detection, the evaluation value is a defective area.

[0070] The visual inspection system according to the present embodiment is an example of an evaluation value estimation device, and executes an evaluation value estimation method. The evaluation value estimation method is a method executed by a computer, and includes a first learning image acquisition step of acquiring multiple learning images f obtained by capturing an object for learning, a first converted image generation step of generating multiple converted images f' by applying a conversion process to the multiple learning images f, a learning step of determining internal parameters of a machine learning estimation engine using the multiple converted images f', a second image acquisition step of acquiring an input image h obtained by capturing an actual object, a second converted image generation step of generating a converted image h' by applying the conversion process to the input image h, an estimated evaluation value calculation step of inputting the converted image h' to the trained estimation engine and estimating an estimated evaluation value R', a second estimated evaluation value calculation step of estimating a second estimated evaluation value R based on the value of the input image h and the estimated evaluation value R' related to a part or all of the processing of the conversion process, and an output step of outputting the second estimated evaluation value R as an output value, and the conversion process is characterized in that the pattern of the multiple learning images f is made uniform with respect to the conversion items by image processing that adjusts at least one or more conversion items.

[0071] The above features will be described in detail. FIG. 9 is a schematic diagram showing an example of the overall processing sequence performed by the visual inspection system according to this embodiment. The processing sequence is roughly divided into a learning phase 900 and an operation phase 901. In the learning phase 900, the visual inspection system captures an evaluation object 902 for learning to obtain a learning image f (904) (S903). As in the first embodiment shown in FIG. 1, in this embodiment, information contained in an image is considered to be input to a machine learning estimation engine and other information separately. Information that does not need to be handled by the estimation engine is converted in advance by the visual inspection system (referred to as conversion T) (S906) and made uniform among all learning images (uniform to a reference value 908). An image to which conversion T has been applied is called a converted image. The converted image of the learning image f in the learning phase is f' (909), and the converted image of the input image h in the operation phase described later is h' (919).

[0072] Since multiple conversion items 907 may exist in the conversion T, the user selects in advance which conversion item 907 to include in the conversion T (S905). For example, when the position of the evaluation object is included in the conversion items 907, the evaluation object that exists at various positions in each learning image is shifted so that it exists at the same position (reference value) in all learning images. The estimation engine estimates an estimated evaluation value P' (911) using a converted image f' (909) in which all the evaluation objects are located at the reference value as input. If the conversion T is not performed, it is necessary to prepare and learn a group of learning images that cover the differences in the positions of the evaluation objects, but by performing the conversion T, it is no longer necessary to include pattern variations regarding the position of the evaluation object in the learning images. The appearance inspection system trains the converted image f' in a machine learning estimation engine and determines the internal parameters 917 of the estimation engine (S910).

[0073] In the operation phase 901, the visual inspection system captures an image of an actual evaluation object 902 (S903) and acquires an input image h (918). Furthermore, the visual inspection system applies a transformation T including the transformation item selected in step S905 of the learning phase 900 to the input image h (S906) and acquires a transformed image h' (919). Then, the visual inspection system inputs the transformed image h' to a machine learning estimation engine that uses the internal parameters 917 determined in the learning phase 900, and estimates an estimated evaluation value R' (921) of the transformed image h' (S920).

[0074] However, some or all of the information homogenized by the transformation T may be necessary for estimating the output value that the visual inspection system will ultimately output. Therefore, the visual inspection system saves, for each input image h, information on how the transformation was performed for each transformation item 907 or what the state was before the transformation as transformation parameters 923. The user selects a transformation item 914 that is necessary for the final output from among the transformation items of the transformation T (S912), and a second estimated evaluation value R (922) is estimated by a second estimation engine from the transformation parameters 923 and the estimated evaluation value R' (921) for each selected conversion item (S913). This second estimated evaluation value R becomes the output value of the image processing system.

[0075] For example, consider a case where the conversion item 907 of the conversion T is the position of the evaluation object. If it is not necessary to handle the position of the evaluation object in step S920 of estimating the evaluation value with a machine learning estimation engine, the appearance inspection system performs uniformization using the conversion T. If the estimated evaluation value is the degree of abnormality of the evaluation object and the criterion for determining the degree of abnormality handled by the estimation engine is a change in contrast in the input image of the evaluation object, it is not necessary for the image input to the estimation engine to include positional variations.

[0076] On the other hand, the final anomaly determination may take into account the position information excluded by the conversion T. That is, in addition to the change in contrast, if a large positional deviation of the evaluation object is also considered to be an anomaly, the appearance inspection system estimates (S913) a second estimated evaluation value R (922) by a second estimation engine based on the estimated evaluation value R' (911) related to contrast handled in step S920 for estimating the evaluation value by the estimation engine and the positional information included in the conversion parameter 923, and sets this as the final output value. The second estimated evaluation value R can reflect both anomalies, such as a change in contrast and a positional deviation.

[0077] On the other hand, it is possible to reduce the number of training images and the training load in machine learning by temporarily excluding information related to some of the conversion items 907 using the conversion T. This makes it possible to improve the estimation performance of the estimation engine even with a small number of training images.

[0078] <6. Learning the estimation engine> Fig. 10 is a schematic diagram showing an example of a learning method for an estimation engine. The learning method can be broadly classified into two types, (1) and (2), with Fig. 10(a1) to (a5) corresponding to (1) and Fig. 10(b1) to (b5) corresponding to (2).

[0079] (1) In the learning phase 900, the converted image f' is used as an input to estimate an estimated evaluation value P' in an estimation engine. A correct answer S' of the estimated evaluation value is estimated from the second correct answer evaluation value S and the conversion parameters. Internal parameters of the image estimation engine are determined based on the estimated evaluation value P' and the correct answer S' of the estimated evaluation value.

[0080] (2) In the learning phase 900, an estimation engine estimates an evaluation value P' using the converted image f' as input. A second estimation engine estimates a second estimated evaluation value P (915) using the evaluation value P' and conversion parameters 923 as input. Internal parameters of the estimation engine are determined based on the second estimated evaluation value P (915) and a ground truth value S of the second estimated evaluation value.

[0081] The learning of the estimation engine is basically performed by updating the internal parameters of the estimation engine, such as the network weights and biases, so as to reduce the difference between the estimated evaluation value output from the estimation engine using a training image as input and a correct evaluation value that has been taught in advance (the correct value of the second estimated evaluation value).

[0082] However, in this embodiment, what is output from the estimation engine is not an evaluation value for the learning image f instructed by the user, but an evaluation value for the converted image f', and therefore it cannot be directly compared with the correct value of the second estimated evaluation value provided by the user. Therefore, in (1), the correct value S' of the estimated evaluation value estimated from the correct value S of the second estimated evaluation value is compared with the estimated evaluation value P', and in (2), the second estimated evaluation value P estimated from the estimated evaluation value P' is compared with the correct value S of the second estimated evaluation value.

[0083] As a specific example of (1), in the learning method of Figs. 10(a1) to (a5), the visual inspection system applies transformation T to training image f (Fig. 10(a1)) to obtain transformed image f' (Fig. 10(a2)). Here, as an example, the transformation item of transformation T is the texture of the gear surface, which is uniformed between training images. In Fig. 10(a2), the texture is removed and converted to a plain pattern. The visual inspection system uses this transformed image f' as input and estimates an estimated evaluation value P' (Fig. 10(a3) or reference numeral 911 in Fig. 9) by an estimation engine. The evaluation value is, for example, the degree of abnormality of the evaluation target (gear). Since the texture information has been uniformed, in step S910, the estimation engine estimates the degree of abnormality using a judgment criterion other than texture. For example, since a foreign object 1000 is attached to transformed image f' (Fig. 10(a2)), the degree of abnormality is estimated to be 45%.

[0084] On the other hand, the user gives the appearance inspection system a correct answer value (correct answer value S of the second estimated evaluation value) for the training image f before conversion (Fig. 10(a1)). If the user's criteria for judging the degree of abnormality include not only the presence or absence of foreign matter but also texture (roughness of the gear surface), the abnormality degree will be high at 80% (Fig. 10(a4)) because the training image f has foreign matter attached and the gear surface is rough. The conversion parameter (texture information in this case) is added to the correct answer value S of this second estimated evaluation value (abnormality 80%) to estimate the abnormality degree (correct answer value S' of the estimated evaluation value) when the abnormality degree due to the texture alone is excluded from the abnormality degree. If the increase in the abnormality degree due to the roughness of the gear surface is subtracted and the abnormality degree due to the foreign matter alone is estimated, it will be, for example, 50% (Fig. 10(a5)). The visual inspection device calculates a loss value based on the difference between the estimated evaluation value P' (FIG. 10(a3)) and the correct value S' (FIG. 10(a5)) of the estimated evaluation value, and determines the internal parameters of the estimation engine so as to reduce the loss value.

[0085] As a specific example of (2), in the learning method of Figs. 10(b1) to (b5), the visual inspection system applies transformation T to training image f (Fig. 10(b1)) to obtain transformed image f' (Fig. 10(b2)). The visual inspection system uses this transformed image f' as input and estimates an estimated evaluation value P' (Fig. 10(b3) or reference numeral 911 in Fig. 9) by an estimation engine. As in the case of (1) above, if the transformation item of transformation T is the texture of the gear surface and the evaluation value is the degree of abnormality of the evaluation object (gear), the estimated evaluation value P' does not take into account the degree of abnormality based on the texture.

[0086] Therefore, in step S913, the visual inspection system estimates a second estimated evaluation value P (FIG. 10(b4)) by a second estimation engine based on the estimated evaluation value P' and the conversion parameters 923 (texture information in this case). This enables the user to compare the estimated evaluation value with the correct value (correct value S of the second estimated evaluation value) of FIG. 10(b5) assigned to the training image f (FIG. 10(b1)) before conversion. That is, the visual inspection system calculates a loss value based on the difference between the second estimated evaluation value P (FIG. 10(b4)) and the correct value S of the second estimated evaluation value (FIG. 10(b5)), and determines the internal parameters of the estimation engine so as to reduce the loss value.

[0087] The second embodiment is different from the first embodiment in that what is estimated has changed from an image to an evaluation value, but the conversion items, learning method, GUI, etc. are similar to those of the first embodiment.

[0088] In addition, various machine learning engines can be used as the estimation engine in each of the first and second embodiments, such as a deep neural network represented by a Convolutional Neural Network (CNN), a Support Vector Machine (SVM), a Support Vector Regression (SVR), and a k-nearest neighbor (k-NN), etc. These estimation engines can handle image estimation, region segmentation, classification problems, and regression problems.

[0089] <7. Hardware configuration of image processing system> FIG. 11 is a diagram showing the hardware configuration of the appearance inspection system according to each of the first and second embodiments.

[0090] The visual inspection system 1 includes the above-mentioned imaging device 1106 and computer 1100. An example of the imaging device 1106 has already been described.

[0091] The computer 1100 is hardware that executes the image estimation method of the first embodiment and the evaluation value estimation method of the second embodiment, and includes the following:

[0092] *Processor 1101: Examples of the processor 1101 include a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), and an FPGA (Field Programmable Gate Array), but other hardware may be used as long as it is capable of processing the image processing method.

[0093] *Storage resource 1102: Examples of the storage resource 1102 include non-volatile memories such as RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), and flash memory. The storage resource may store a program that causes the processor 1101 to execute the image estimation method and the evaluation value estimation method described in each of the above embodiments.

[0094] *GUI device 1103: Examples of the GUI device 1103 include a display and a projector, but other hardware may be used as long as it is capable of displaying a GUI.

[0095] * Input device 1104: Examples of the input device 1104 include a keyboard, a mouse, and a touch panel, but other devices that can accept operations from a user may be used. Also, the input device 1104 and the GUI device 1103 may be integrated into one piece of hardware.

[0096] *Communication interface device 1105: Examples of the communication interface device 1105 include USB (Universal Serial Bus), Ethernet, and Wi-Fi. Any other interface device may be used as long as it can directly receive an image from the imaging device 1106 or allows a user to transmit the image to the computer 1100. In addition, a portable non-volatile storage medium (not shown) storing the image may be connected to the communication interface device 1105, and the image may be stored in the computer 1100. Examples of such portable non-volatile storage media include a flash memory, a DVD (Digital Versatile Disc), a CD-ROM (Compact Disc), and a Blu-ray disc.

[0097] The above is the hardware configuration of the computer 1100. Note that the appearance inspection system may include a plurality of computers 1100 and a plurality of image capturing devices 1106.

[0098] The above-mentioned program may be stored in the computer 1100 via the following path: *The program is stored in a portable non-volatile storage medium, and the medium is connected to the communication interface device 1105 to distribute the program to the computer 1100.

[0099] * The program is distributed to the computer 1100 by a program distribution server. The program distribution server has a storage resource that stores the program, a processor that performs a distribution process to distribute the program, and a communication interface device that can communicate with the communication interface device 1105 of the computer 1100.

[0100] This concludes the description of the embodiments. As mentioned above, the embodiments described thus far do not limit the invention according to the claims, and all of the elements and combinations thereof described in the embodiments are not necessarily essential to the solution of the invention.

[0101] In addition, although each embodiment deals with a single input image as input information, each embodiment can be applied to a case where there are multiple input images and multiple types of output images or estimated evaluation values. In this case, the estimation engine becomes another input or another output.

[0102] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0103] The present invention is not limited to the above-described embodiments, and various modifications are included. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the components described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0104] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them as an integrated circuit, for example. In addition, each of the above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, a judgment table, and a file that realizes each function can be placed in a memory, a storage device such as an HDD or SSD, or a recording medium such as an IC (Integrated Circuit) card, an SD (Secure Digital) card, or a DVD. In addition, the control lines and information lines are shown as those considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. In reality, it may be considered that almost all configurations are connected to each other. [Explanation of symbols]

[0105] 100, 900...Learning phase, 101, 901...Operation phase, 102, 902...Evaluation object, 104, 904...Learning image, 107, 114, 907, 914...Conversion item, 108, 908...Reference value, 117, 917...Internal parameter, 109, 909...Transformed image, 111...Estimated image, 115...Inverse transformation estimated image, 118, 918...Input image, 119, 919...Transformed image, 121...Estimated image, 122...Inverse transformation estimated image, 123, 923...Transformation parameter, 200...Foreign object, 201...Foreign object, 203...Crack, 204, 205, 1000... Foreign matter, 207...crack, 300, 301, 309, 310, 400-404...dotted frame, 800, 801...radio button, 802...pull-down menu, 803...image display area, 804...conversion specification area, 805, 807...check box, 806...radio button, 808...intensity specification area, 911, 921...estimated evaluation value, 915, 922...second estimated evaluation value, 1100...computer, 1101...processor, 1102...memory resource, 1103...GUI device, 1104...input device, 1105...communication interface device, 1106...imaging device.

Claims

1. 1. A computer implemented method for image estimation, comprising: A first learning image acquisition step of acquiring a plurality of learning images f obtained by capturing an object for learning; a first converted image generating step of generating a plurality of converted images f′ by performing a conversion process on the plurality of learning images f; a learning step of determining internal parameters of a machine learning estimation engine using the plurality of transformed images f′; a second image acquisition step of acquiring an input image h of a real object; a second converted image generating step of generating a converted image h' by performing the conversion process on the input image h; an estimated image generating step of inputting the converted image h' to the trained estimation engine to estimate an estimated image H'; an inverse transformed estimated image generating step of performing an inverse transform process of a part or all of the transform process on the estimated image H' to generate an inverse transformed estimated image H; and outputting the inverse transformed estimated image H as an output image. The conversion process is image processing for adjusting at least one or more conversion items to uniformize the patterns of the multiple learning images f with respect to the conversion items. Image estimation methods.

2. The image estimation method according to claim 1, a receiving step of receiving, from a user, a conversion item of the conversion processing and a conversion item of the inverse conversion processing, the conversion item being an item not handled by the estimation engine; Image estimation methods.

3. 2. The image estimation method according to claim 1, The estimation engine estimates, using an image as an input, one of an image restored image, a super-resolution image, and an enhanced image in which a region of interest is enhanced. Image estimation methods.

4. 2. The image estimation method according to claim 1, The conversion items include at least one of image brightness, image contrast, image noise, image shading, and shade; Image estimation methods.

5. 2. The image estimation method according to claim 1, The computer determines a reference value for the conversion item from the plurality of learning images f; The equalization is performed by converting the values ​​of the conversion items in each of the learning images f to the reference values. Image estimation methods.

6. 2. The image estimation method according to claim 1, the computer accepts a designation of a degree of the inverse conversion process for each of the conversion items, and performs the inverse conversion process for each of the conversion items at the designated degree; Image estimation methods.

7. 2. The image estimation method according to claim 1, In the learning step, the computer determines internal parameters of the estimation engine based on an estimated image F' estimated by the estimation engine using the transformed image f' as an input and a transformed correct image G' obtained by performing the transformation process on a correct image G. Image estimation methods.

8. 2. The image estimation method according to claim 1, In the learning step, the computer determines internal parameters of the estimation engine based on an inverse-transformed estimated image F obtained by performing an inverse transformation process on an estimated image F' estimated by the estimation engine using the transformed image f' as an input, and a correct image G for the training image f. Image estimation methods.

9. 2. The image estimation method according to claim 1, a display step of displaying a GUI (Graphical User Interface) for receiving designation of the conversion items of the conversion process and the conversion items of the inverse conversion process from a user; Image estimation methods.

10. A computer-implemented rating value estimation method, comprising the steps of: A first learning image acquisition step of acquiring a plurality of learning images f obtained by capturing an object for learning; a first converted image generating step of generating a plurality of converted images f′ by performing a conversion process on the plurality of learning images f; a learning step of determining internal parameters of a machine learning estimation engine using the plurality of transformed images f′; a second image acquisition step of acquiring an input image h of a real object; a second transformed image generating step of generating a transformed image h' by performing the transformation process on the input image h; an estimated evaluation value calculation step of inputting the converted image h' to the trained estimation engine to estimate an estimated evaluation value R'; a second estimated evaluation value calculation step of estimating a second estimated evaluation value R based on the value of the input image h and the estimated evaluation value R′ for a part or all of the conversion processing; an output step of outputting the second estimated evaluation value R as an output value; The conversion process is image processing for adjusting at least one or more conversion items to uniformize the patterns of the multiple learning images f with respect to the conversion items. Evaluation value estimation method.

11. An image estimation apparatus comprising a processor, The processor, A first learning image acquisition step of acquiring a plurality of learning images f obtained by capturing an object for learning; a first converted image generating step of generating a plurality of converted images f′ by performing a conversion process on the plurality of learning images f; a learning step of determining internal parameters of a machine learning estimation engine using the plurality of transformed images f′; a second image acquisition step of acquiring an input image h of a real object; a second converted image generating step of generating a converted image h' by performing the conversion process on the input image h; an estimated image generating step of inputting the converted image h' to the trained estimation engine to estimate an estimated image H'; an inverse transformed estimated image generating step of performing an inverse transform process of a part or all of the transform process on the estimated image H' to generate an inverse transformed estimated image H; and outputting the inverse transformed estimated image H as an output image. The conversion process is image processing for adjusting at least one or more conversion items to uniformize the patterns of the plurality of learning images f with respect to the conversion items. Image estimation device.

12. The image estimation device according to claim 11, The processor receives, from a user, designation of a conversion item of the conversion processing and a conversion item of the inverse conversion processing, the conversion item being an item not handled by the estimation engine. Image estimation device.

13. The image estimation device according to claim 11, The estimation engine estimates an image restored image, a super-resolution image, and an enhanced image in which a region of interest is enhanced, using an image as an input. Image estimation device.

14. The image estimation device according to claim 11, The transformation items include at least one of image brightness, image contrast, image noise, image shading, and shade; Image estimation device.

15. The image estimation device according to claim 11, The processor determines a reference value for the conversion item from the plurality of learning images f; The equalization is performed by converting the values ​​of the conversion items in each of the learning images f to the reference values. Image estimation device.

16. The image estimation device according to claim 11, The processor accepts a designation of a degree of the inverse conversion process for each of the conversion items, and performs the inverse conversion process for each of the conversion items at the designated degree. Image estimation device.

17. The image estimation device according to claim 11, The processor performs control to display a GUI that accepts designation of the conversion items of the conversion processing and the conversion items of the inverse conversion processing from a user. Image estimation device.