Image processing system

Abstract

An objective of the present invention is to make it possible to determine the output of a learning model that includes variations by considering the distribution of correct answer data corresponding to the output of the learning model, and improve the reliability of an image processing system. For example, an image processing system according to the present invention includes: a model application unit that generates an estimated high-quality image by applying a machine learning model to a low-quality image; an evaluation value calculation unit that calculates an evaluation value of the estimated high-quality image; and a determination unit that makes a determination regarding the estimated high-quality image on the basis of the evaluation value of the estimated high-quality image. The determination unit determines whether the estimated high-quality image is uncertain data using the range of variation of the evaluation value of the high-quality image obtained by imaging the low-quality image at high quality and corresponding to the evaluation value of the estimated high-quality image acquired in advance.

Description

Image Processing System

[0001] The present invention relates to an image processing system.

[0002] An image processing system used in inspection and measurement is described in Patent Document 1. Paragraph 0012 of this document states, "The score output unit 30 reads out the inspection images and inspection information extracted by the inspection image extraction unit 10 from the storage unit 70, inputs the read inspection images into a learning model constructed by the learning unit 20, and calculates a judgment score for each inspection image. ... The score output unit 30 assigns the calculated failure score to the inspection information and stores it in the storage unit 70." Paragraph 0013 states, "The judgment unit 40 reads out the inspection information to which the failure score has been assigned by the score output unit 30 from the storage unit 70, compares the failure score with a judgment threshold, which is a threshold for judging the pass / fail of the inspection image, and judges the pass / fail of the inspection image, i.e., whether the inspection image is "pass," "fail," or "unclear."

[0003] Furthermore, paragraph 0031 of the same document states, "The threshold update unit 60 determines whether the threshold update condition, i.e., the predetermined condition for updating the judgment threshold, is met. If the threshold update condition is met, the process proceeds to step S60 to update the judgment threshold, and if the threshold update condition is not met, the pass / fail judgment process is terminated. The threshold update condition in this embodiment is that, of the inspection images that have been judged as "unclear whether the inspection is pass or fail" by the judgment unit 40 and have been assigned predetermined confirmation information by the visual inspection unit 50, the number of inspection images whose failure scores fall within a predetermined range reaches a predetermined number."

[0004] Japanese Patent Application Laid-Open No. 2022-169974

[0005] The technology described in this document calculates a judgment score using a learning model, which may result in variations in the judgment score, such as errors. However, the technology described in this document does not take into account variations in the judgment score, which may result in erroneous judgments.

[0006] Therefore, the present invention aims to improve the reliability of image processing systems by taking into account the distribution of correct data corresponding to the output of the learning model, thereby making it possible to determine the output of the learning model taking into account its variability.

[0007] In order to solve the above problem, the image processing system of the present invention includes, for example, a model application unit that applies a machine learning model to a low-quality image to generate an estimated high-quality image, an evaluation value calculation unit that calculates an evaluation value of the estimated high-quality image, and a judgment unit that makes a judgment regarding the estimated high-quality image based on the evaluation value of the estimated high-quality image, and the judgment unit determines whether the estimated high-quality image is uncertain data using a range of variation in evaluation values ​​of a high-quality image obtained by capturing the low-quality image at a high quality, which corresponds to the evaluation value of the estimated high-quality image, and which has been obtained in advance.

[0008] According to the present invention, by taking into account the distribution of correct data corresponding to the output of the learning model, it is possible to determine the output of the learning model taking into account its variability, thereby improving the reliability of the image processing system.

[0009] 1 is a block diagram of an image processing system according to a first embodiment. FIG. 1 is a block diagram illustrating a configuration for acquiring a variation in evaluation values. FIG. 2 is a diagram illustrating an example of the variation in evaluation values. FIG. 3 is a diagram illustrating an example of a method for determining a variation range. FIG. 4 is a diagram illustrating another example of a method for determining a variation range. FIG. 5 is a diagram illustrating an example of an estimated high-quality image being determined as confirmed data. FIG. 6 is a diagram illustrating another example of an estimated high-quality image being determined as confirmed data. FIG. 7 is a diagram illustrating an example of an estimated high-quality image being determined as uncertain data. FIG. 8 is a diagram illustrating another example of an estimated high-quality image being determined as uncertain data. A flowchart of a process for acquiring a variation in evaluation values. A diagram illustrating an example of a display of the variation in evaluation values. A flowchart of image evaluation processing. A diagram illustrating an example of a display for acquiring a high-quality image. A block diagram of an image processing system according to a second embodiment. A block diagram illustrating a configuration for calculating an input tolerance range. A diagram illustrating a method for calculating an input tolerance range. A diagram illustrating an example of an evaluation value of a low-quality image that falls outside the input tolerance range. A diagram illustrating another example of an evaluation value of a low-quality image that falls outside the input tolerance range. A diagram illustrating an example of an evaluation value of a low-quality image that falls within the input tolerance range. A flowchart from input determination processing to image evaluation processing.

[0010] Hereinafter, an embodiment will be described with reference to the drawings.

[0011] 1 is a block diagram of an image processing system according to a first embodiment of the present invention. In this embodiment, a case will be described in which semiconductor inspection and measurement are performed using images captured by an optical microscope, a scanning electron microscope, or the like.

[0012] The image processing system 1 of this embodiment includes a model application unit 11, an evaluation value calculation unit 12, a determination unit 13, and an image acquisition unit 14.

[0013] <Model Application Unit 11> The model application unit 11 receives as input a low-quality image captured by an optical microscope, a scanning electron microscope, etc. Then, it applies a machine learning model to the low-quality image to generate an estimated high-quality image.

[0014] A low-quality image is an image with a small number of accumulated images and a lot of noise, an image with unsuitable imaging parameters and blurred, an image with low contrast of objects in the image, an image with low resolution, or the like.

[0015] On the other hand, a high-quality image is an image of higher quality than a low-quality image, such as an image with a large number of image accumulations and low noise, an image with appropriate imaging parameters and clear objects in the image, an image with appropriate contrast of objects in the image, or an image with high resolution. While such high-quality images are highly reliable as data, they often require a large number of image accumulations, an extended imaging interval, and an increased imaging time, which reduces throughput from image acquisition to image processing. Therefore, the present invention uses an estimated high-quality image generated by applying a machine learning model to a low-quality image.

[0016] The estimated high-quality image is an image with the same quality as the high-quality image. This ensures the reliability of the data while preventing a decrease in throughput from image acquisition to image processing. Examples of machine learning models include models that improve image quality by removing noise and blur, and models that correct contrast to highlight objects in the image.

[0017] <Evaluation Value Calculation Unit 12> The evaluation value calculation unit 12 calculates an evaluation value of the estimated high-quality image. Here, the evaluation value is an image feature amount obtained by applying a machine learning model such as Deep Learning to the image, an image feature amount obtained using a dimensionality compression method such as t-SNE, an inspection result such as the probability of a defect in an object in the image, or a measurement value such as the width of an object in the image.

[0018] <Determination Unit 13> The determination unit 13 determines the estimated high-quality image based on the evaluation value of the estimated high-quality image. Specifically, the determination unit 13 determines the range of variation in the evaluation value of the estimated high-quality image based on the variation in the evaluation value for each estimated high-quality image. If the entire variation range is within a predetermined allowable range, or if the entire variation range is not within the allowable range, the determination unit 13 determines the estimated high-quality image as confirmed data. If only part of the variation range is not within the allowable range, the determination unit 13 determines the estimated high-quality image as uncertain data. Note that the allowable range is set based on the results of semiconductor inspection and measurement, etc. Furthermore, the variation in the evaluation value is acquired before the determination unit 13 determines the evaluation value of the estimated high-quality image.

[0019] <<Variation in Evaluation Values>> Here, the variation in evaluation values ​​will be described using Fig. 2 and Figs. 3A to 3C. Fig. 2 is a block diagram illustrating a configuration for acquiring the variation in evaluation values. As shown in Fig. 2, the image processing system 1 includes a variation acquisition unit 15 in addition to the model application unit 11 and evaluation value calculation unit 12 shown in Fig. 1. Note that, in this embodiment, the image processing system 1 will be described as including the configurations shown in Figs. 1 and 2, but a device including the configuration shown in Fig. 1 and a device including the configuration shown in Fig. 2 may be separate devices.

[0020] The model application unit 11 applies a machine learning model to each of the plurality of low-quality images to generate a plurality of estimated high-quality images. The low-quality images include low-quality images such as training data for training the machine learning model, verification data for determining hyperparameters of the machine learning model, and test data for verifying the accuracy of the machine learning model. The estimated high-quality images are images estimated by applying the machine learning model to the low-quality images and are equivalent to high-quality images.

[0021] The evaluation value calculation unit 12 calculates an evaluation value of the estimated high-quality image and an evaluation value of a high-quality image that captures an object contained in the low-quality image that was the source of the estimated high-quality image with higher quality.

[0022] 3A is a diagram illustrating an example of a method for determining the variation range. Fig. 3A is a graph in which multiple evaluation values ​​of high-quality images and corresponding evaluation values ​​of high-quality images are obtained, with the horizontal axis representing the evaluation value of the estimated high-quality image and the vertical axis representing the evaluation value of the high-quality image. In Fig. 3A, multiple relationships between the evaluation values ​​of the estimated high-quality images and the evaluation values ​​of the high-quality images are plotted as plot P1.

[0023] The variation acquisition unit 15 calculates a plot distribution range that includes a predetermined number of the multiple plots P1. In FIG. 3A , as an example, the range from the lower limit LL1 to the upper limit UL1 is illustrated as a plot distribution range (90% range) that includes 90% of the plots P1. The determination unit 13 uses the high-quality image evaluation value corresponding to the estimated high-quality image evaluation value as the evaluation value variation. Specifically, in the graph shown in FIG. 3A , the determination unit 13 regards the horizontal axis as the estimated high-quality image evaluation value. For example, when the estimated high-quality image evaluation value is x1, the determination unit 13 uses the estimated high-quality image evaluation value x1 as the value on the horizontal axis of FIG. 3A . Then, the determination unit 13 extracts a lower limit y1 and an upper limit y2 of the high-quality image evaluation value within a 90% range of the estimated high-quality image evaluation value x1, and sets y1 as the lower limit of the variation range of the estimated high-quality image evaluation value x1 and y2 as the upper limit of the variation range. That is, the range DR1 shown in FIG. 3A is the variation range of the estimated high-quality image evaluation value x1.

[0024] FIG. 3B is a diagram illustrating an example of a method for determining the variation range. The plot P1 in FIG. 3B is the same as that in FIG. 3A. In FIG. 3B, the range from the lower limit LL2 to the upper limit UL2 is illustrated as a plot distribution range (99% range) that includes 99% of the plot P1. Comparing FIG. 3A and FIG. 3B, the 99% range is wider in the vertical axis direction than the 90% range. In the example shown in FIG. 3B, the lower limit of the variation range of the estimated high-quality image evaluation value x1 is y3, and the upper limit of the variation range is y4. That is, the range DR2 shown in FIG. 3B is the variation range of the estimated high-quality image evaluation value x1.

[0025] FIG. 3C is a diagram illustrating another example of a method for determining a variation range. The plot P1 in FIG. 3C is the same as that in FIG. 3A . In the first graph shown in FIG. 3C , two plots are extracted whose estimated high-quality image evaluation values ​​are close to the estimated high-quality image evaluation value calculated by the evaluation value calculation unit 12. The lower of the two extracted plots is set as the lower limit of the variation range, and the higher of the two plots is set as the upper limit of the variation range. For example, for the estimated high-quality image evaluation value x2, the evaluation value variation range DR3 is determined with the evaluation value y5 of the high-quality image of the two neighboring plots set as the lower limit and the evaluation value y6 set as the upper limit. Furthermore, for the estimated high-quality image evaluation value x3, the evaluation value variation range DR4 is determined with the evaluation value y7 of the high-quality image of the two neighboring plots (e.g., two adjacent plots) set as the lower limit and the evaluation value y8 set as the upper limit. By determining the variation range of the evaluation values ​​from the evaluation values ​​of the high-quality images of two neighboring plots in this way, it becomes unnecessary to set upper and lower limits across the entire interval as in Figures 3A and 3B, and it is possible to obtain a variation range of the evaluation values ​​that is appropriate for the evaluation value of the estimated high-quality image. Note that the minimum and maximum values ​​may be extracted from the evaluation values ​​of the high-quality images of multiple plots within a predetermined interval, and the variation range of the evaluation values ​​may be determined by setting the maximum value as the upper limit and the minimum value as the lower limit.

[0026] By using the variations in the evaluation values ​​as shown in FIGS. 3A to 3C, the range of variation in the evaluation values ​​can be determined according to the evaluation values ​​of high-quality images calculated by the evaluation value calculation unit 12.

[0027] 4A to 4D, the determination by the determination unit 13 will be specifically described. In Fig. 4A to 4D, the horizontal axis represents the evaluation value, with less than m1 being outside the acceptable range, m1 or more but less than m2 being within the acceptable range, and m2 or more being outside the acceptable range. The boundary values ​​between the acceptable range and the unacceptable range, such as m1 and m2, may be set based on prior knowledge such as the results of ultimately determining which products are good or bad based on the electrical characteristics of semiconductors manufactured to date, or may be set based on the average and standard deviation of the number of inspection and measurement samples.

[0028] 4A is a diagram showing an example in which an estimated high-quality image is determined to be final data. In FIG. 4A, the entire variation range DR5 of the estimated high-quality image evaluation value x4, which has a lower limit of the high-quality image evaluation value y9 corresponding to the estimated high-quality image evaluation value x4 and an upper limit of the high-quality image evaluation value y10, is included in the allowable range. Therefore, in this case, the determination unit 13 determines the estimated high-quality image to be final data.

[0029] 4B is a diagram showing another example in which the estimated high-quality image is determined to be final data. In FIG. 4B, the entire variation range DR6 of the estimated high-quality image evaluation value x5, in which the high-quality image evaluation value y11 corresponding to the estimated high-quality image evaluation value x5 is the lower limit and the high-quality image evaluation value y12 is the upper limit, is not included in the allowable range. Therefore, in this case, the determination unit 13 determines the estimated high-quality image to be final data.

[0030] 4C is a diagram showing an example in which an estimated high-quality image is determined to be uncertain data. In FIG. 4C, the estimated high-quality image evaluation value x6 corresponds to the estimated high-quality image evaluation value y13, which is the lower limit, and the estimated high-quality image evaluation value y14 corresponds to the upper limit. A portion of the variation range DR7 of the estimated high-quality image evaluation value x6 is not included in the allowable range. In this case, although the estimated high-quality image evaluation value x6 is within the allowable range, there is a possibility of variation within the variation range DR7, so the estimated high-quality image evaluation value cannot be determined to be either within the allowable range or outside the allowable range. Therefore, the determination unit 13 determines the estimated high-quality image to be uncertain data.

[0031] 4D shows another example in which an estimated high-quality image is determined to be uncertain data. In FIG. 4D , the estimated high-quality image evaluation value x7 has a lower limit of high-quality image evaluation value y15 and an upper limit of high-quality image evaluation value y16, and a portion of the variation range DR8 of the estimated high-quality image evaluation value x7 is not included in the allowable range. In this case, the estimated high-quality image evaluation value x7 is outside the allowable range, but there is a possibility of variation within the variation range DR8, so the estimated high-quality image evaluation value cannot be determined to be either within the allowable range or outside the allowable range. Therefore, the determination unit 13 determines the estimated high-quality image to be uncertain data.

[0032] In this way, by making a judgment not only based on the evaluation value of the estimated high-quality image but also taking into consideration the range of variation, it is possible to reduce erroneous judgments.

[0033] When the judgment unit 13 judges that the estimated high-quality image is uncertain data, it causes the image acquisition unit 14 (see Figure 1) that acquires images to acquire a high-quality image that captures the object included in the low-quality image with higher quality instead of the estimated high-quality image that has been judged to be uncertain data.

[0034] <Image Acquisition Unit 14> The image acquisition unit 14 acquires a high-quality image by capturing a higher-quality image of an object contained in a low-quality image that is the source of estimation for high-quality learning and that has been determined to be uncertain data. High-quality image capture is achieved by changing the image capture conditions or image processing method. Acquiring a high-quality image involves methods that, compared to generating an estimated high-quality image, provide higher image accuracy and quality but require longer processing time, reduce throughput, or are impossible to execute within a time limit. For example, the image acquisition unit 14 increases the number of integrations to capture a high-quality image with low integration noise. Other examples include cases in which the model application unit 11 and the image acquisition unit 14 apply different functional methods. For example, when a low-quality image with high integration noise is input, the model application unit 11 applies a machine learning model to estimate an image equivalent to a high integration number. In response to this, the image acquisition unit 14 applies a machine learning model that corrects contrast to highlight objects in the image and make defects easier to detect, or a classical image processing method.

[0035] <Process for Obtaining Variation in Evaluation Values> Fig. 5 is a flowchart of a process for obtaining variation in evaluation values. Steps S11 to S13 in the flowchart of Fig. 5 are realized by the configuration shown in Fig. 2.

[0036] In step S11, the model application unit 11 applies a machine learning model to the low-quality image to generate a predicted high-quality image. In step S12, the evaluation value calculation unit 12 calculates an evaluation value of the predicted high-quality image and an evaluation value of a high-quality image that captures an object included in the low-quality image that was the source of the predicted high-quality image with higher quality. In step S13, the variation acquisition unit 15, as described above, uses the evaluation value of the predicted high-quality image and the evaluation value of the high-quality image to acquire the variation in the evaluation value of the high-quality image that corresponds to the evaluation value of the predicted high-quality image.

[0037] Furthermore, in addition to the configuration shown in FIG. 2 , the image processing system 1 may have a display unit (not shown). As step S14 of FIG. 5 , the variation in the evaluation values ​​acquired in step S13 may be displayed on the display unit to present the variation in the evaluation values ​​to the user. FIG. 6 is a diagram illustrating an example of the display of the variation in the evaluation values. As shown in FIG. 6 , the display unit may display the currently used method for determining the variation range along with the variation in the evaluation values. In this case, depending on the display of the method for determining the variation range, as shown in FIGS. 3A and 3B , upper and lower limits are displayed on the first graph representing the variation in the evaluation values. For example, as shown in FIG. 6 , if the method for determining the variation range is "determining the variation range based on the 90% range," an upper limit UL1 and a lower limit LL1 are displayed on the graph. On the other hand, if the method for determining the variation range is "determining the variation range based on the nearest neighbor plot," the upper and lower limits are not displayed on the graph.

[0038] The image processing system 1 may also have an instruction input unit (not shown) that accepts instructions from the user. Then, in step S15 of FIG. 5 , an instruction to change the method for determining the variation range may be accepted from the user who has confirmed the variation in the evaluation values ​​presented in step S14. If there is no instruction from the user, the flow chart shown in FIG. 5 ends. On the other hand, if there is an instruction from the user, the process returns to step S14, and the display of the method for determining the variation range and the display of the upper and lower limits of the graph representing the variation range are changed. The display unit and instruction input unit may be implemented as a single device such as a touch display.

[0039] <Image Evaluation Processing> FIG. 7 is a flowchart of the image evaluation processing. The flowchart of FIG. 7 is realized by the configuration shown in FIG. 1. In step S21, a low-quality image is input to the model application unit 11. In step S22, the model application unit 11 applies a machine learning model to generate a predicted high-quality image. In step S23, the evaluation value calculation unit 12 calculates an evaluation value of the predicted high-quality image generated in step S22. In step S24, the determination unit 13 determines a variation range of a high-quality image corresponding to the evaluation value of the predicted high-quality image based on the variation of the evaluation value calculated in step S13 of FIG. 5. In step S25, the determination unit 13 determines whether the entire variation range determined in step S24 is within an acceptable range. If the entire variation range is within the acceptable range or if the entire variation range is not within the acceptable range, the flowchart ends. On the other hand, if a portion of the variation range is not within the acceptable range, the process proceeds to step S26. In step S26, the user is prompted to acquire a high-quality image instead of the predicted high-quality image. For example, the image processing system 1 uses a display unit not shown in FIG. 1 to notify the user that a high-quality image has been acquired. FIG. 8 is a diagram showing an example of a display indicating that a high-quality image has been acquired. As shown in FIG. 8, the display unit may display the evaluation value, variation range, and tolerance range of the estimated high-quality image along with the indication that a high-quality image has been acquired. By displaying the display shown in FIG. 8, the user can understand that a high-quality image is being acquired and the reason why the data was determined to be uncertain. Then, in step S27, the image acquisition unit 14 changes the image capturing conditions or image processing method and captures a high-quality image.

[0040] According to the present invention, it is possible to provide an image processing system that can ensure the accuracy of image evaluation regardless of variations in image evaluation values.

[0041] 9 is a block diagram of an image processing system 1 of Example 2. The image processing system 1 of this Example verifies low-quality images to be input to the evaluation value calculation unit 12 in Example 1, and excludes low-quality images that are not suitable for input to the evaluation value calculation unit 12. The following will mainly describe the differences from Example 1.

[0042] <Input Verification Unit 16> The image processing system 1 of this embodiment includes the input verification unit 16. The input verification unit 16 calculates an evaluation value of a low-quality image to be input to the model application unit 11, and determines, based on the evaluation value of the low-quality image, whether or not to input the low-quality image to the model application unit 11. Specifically, the input verification unit 16 inputs the low-quality image to the model application unit 11 when the evaluation value of the low-quality image is included in a predetermined input tolerance range, and does not input the low-quality image to the model application unit 11 when the evaluation value of the low-quality image is not included in a predetermined input tolerance range.

[0043] <<Calculation of Acceptable Input Range>> Calculation of the acceptable input range will be described using Figures 10 and 11. Figure 10 is a block diagram illustrating a configuration for calculating the acceptable input range. As shown in Figure 10, the image processing system 1 includes an acceptable input range calculation unit 17 in addition to the model application unit 11 and evaluation value calculation unit 12 shown in Figure 9. Note that in this embodiment, the image processing system 1 will be described as including the configurations of Figures 9 and 10, but it is also possible to have separate devices including the configuration of Figure 9 and the configuration of Figure 10, and for these devices to cooperate to realize the flowchart of Figure 13, which will be described later.

[0044] The model application unit 11 applies a machine learning model to the low-quality image to generate an estimated high-quality image.

[0045] The evaluation value calculation unit 12 calculates an evaluation value of the low-quality image and an evaluation value of the estimated high-quality image.

[0046] FIG. 11 is a diagram illustrating a method for calculating the input tolerance range. The input tolerance range calculation unit 17 acquires multiple evaluation values ​​for low-quality images and corresponding estimated high-quality images, and generates a graph in which the horizontal axis represents the evaluation values ​​for the low-quality images and the vertical axis represents the evaluation values ​​for the estimated high-quality images. In FIG. 11, multiple relationships between the evaluation values ​​for the low-quality images and the estimated high-quality images are plotted as plot P2. The graph is then divided into multiple intervals based on the values ​​on the horizontal axis. In this embodiment, the values ​​on the horizontal axis are divided into four intervals: less than x8, at least x8 but less than x9, at least x9 but less than x10, and at least x10.

[0047] The input tolerance range calculation unit 17 extracts the minimum to maximum evaluation values ​​of the estimated high-quality images of the plots within each of the divided sections, and sets the extracted section as the range of low-quality images as the input tolerance range.

[0048] In Figure 11, there is no plot P2 in the section below x8 or the section above x10. In other words, low-quality images with evaluation values ​​below x8 or above x10 are not included in the training data, verification data, or test data, indicating that the evaluation values ​​of estimated high-quality images estimated from low-quality images are unreliable. Therefore, the section where the evaluation value of low-quality images is below x8 and the section above x10 are outside the input allowable range.

[0049] 11 , plots P2 exist in the section from x8 to less than x9 and the section from x9 to less than x10. In this embodiment, a case will be described in which 12 plots P2 exist in each of the sections from x8 to less than x9 and from x9 to less than x10, but this is not limiting. Furthermore, the maximum value of the estimated high-quality image evaluation value of plots P2 in the section from x8 to less than x9 is set to MAX1, and the minimum value is set to MIN1. The maximum value of the estimated high-quality image evaluation value of plots P2 in the section from x9 to less than x10 is set to MAX2, and the minimum value is set to MIN2.

[0050] In Figure 11, the range from x8 to x9 is larger than the range from the minimum value MIN1 to the maximum value MAX1. This indicates that the variation in the evaluation values ​​calculated from the estimated high-quality image is greater than the variation in the evaluation values ​​calculated from the low-quality image. Since the variation acquisition unit 15 acquires the variation in the evaluation values ​​of the high-quality image from the evaluation values ​​calculated from the estimated high-quality image, if the evaluation value calculated from the input image is in the range from x8 to x9, the variation in the evaluation values ​​calculated from the estimated high-quality image may be large, and the reliability of the variation in the evaluation values ​​may be low. Therefore, if the evaluation value calculated from the input image is in the range from x8 to x9, the variation in the evaluation values ​​of the estimated high-quality image may be large, and the reliability of the evaluation value may be low. Therefore, the range from x8 to x9 is outside the allowable input range. However, the user may specify the allowable range of variation in the evaluation values ​​of the estimated high-quality image in advance.

[0051] In contrast, the range from x9 to x10 is smaller than the range from the minimum value MIN2 to the maximum value MAX2. When the evaluation value of a low-quality image is in the range from x9 to x10, there is a possibility that the variation in the evaluation value of the estimated high-quality image is small, and the reliability of the evaluation value of the estimated high-quality image is high. Therefore, the range from x9 to x10 is set as the input allowable range. Furthermore, the input allowable range may be presented to the user, and the calculation method of the input allowable range may be changed if necessary.

[0052] <<Input Determination>> The determination by the input verification unit 16 will be specifically described using Figures 12A to 12C. In Figures 12A to 12C, the horizontal axis represents the evaluation value of a low-quality image. In Figures 12A to 12C, the input tolerance range in Figure 11 is used. That is, values ​​less than x9 are outside the input tolerance range, values ​​greater than or equal to x9 and less than x10 are within the input tolerance range, and values ​​greater than or equal to x10 are outside the input tolerance range.

[0053] 12A is a diagram showing an example in which the evaluation value of a low-quality image falls outside the input tolerance range. In FIG. 12A , the evaluation value of the low-quality image (plot P3) is less than x8, falling outside the input tolerance range. Therefore, since the evaluation value of the low-quality image (plot P3) does not fall within the input tolerance range, the input verification unit 16 does not input the low-quality image to the model application unit 11, and the image is treated as inappropriate input data, and a high-quality image is acquired by capturing an image at high quality using the image acquisition unit 14.

[0054] 12B is a diagram showing another example in which the evaluation value of a low-quality image falls outside the input tolerance range. In FIG. 12B, the evaluation value of the low-quality image (plot P4) is greater than or equal to x8 and less than x9, which falls outside the input tolerance range. Therefore, since the evaluation value of the low-quality image (plot P4) does not fall within the input tolerance range, the input verification unit 16 does not input the low-quality image to the model application unit 11, and the image is treated as inappropriate input data, so that the image acquisition unit 14 captures the image at a high quality and acquires a high-quality image.

[0055] 12C is a diagram showing an example in which the evaluation value of a low-quality image is within the input tolerance range. In FIG. 12C, the evaluation value of the low-quality image (plot P5) is greater than or equal to x9 and less than x10, which is within the input tolerance range. Therefore, the input verification unit 16 inputs the low-quality image to the model application unit 11 because the evaluation value of the low-quality image (plot P4) is within the input tolerance range.

[0056] <Display Unit> The image processing system 1 of this embodiment may include a display unit (not shown). The display unit of this embodiment displays the graph shown in FIG.

[0057] The display unit also displays the evaluation values ​​of low-quality images that are not input to the model application unit 11 and the input tolerance range. For example, the display may be as shown in FIG. 12A or FIG. 12B . At this time, the reason why the low-quality image was determined to be inappropriate data for input may also be displayed. For example, in the case shown in FIG. 12A , the display unit displays a reason such as the low-quality image is not included in the training data, verification data, or test data, and the evaluation value of the estimated high-quality image based on the low-quality image is unreliable. In the case shown in FIG. 12B , the display unit displays a reason such as the evaluation value of the estimated high-quality image based on the low-quality image may vary greatly, and the evaluation value of the estimated high-quality image based on the low-quality image is unreliable.

[0058] <Input Determination Process> FIG. 13 is a flowchart from the input determination process to the image evaluation process. In step S31, the input verification unit 16 acquires the input tolerance range. In step S32, a low-quality image is input to the input verification unit 16. In step S33, the input verification unit 16 calculates an evaluation value for the low-quality image. In step S34, the input verification unit 16 determines whether the evaluation value for the low-quality image is within the input tolerance range. If it is determined in step S34 that the evaluation value for the low-quality image is within the input tolerance range, the process proceeds to step S21. Note that steps S21 to S27 are the same as in the first embodiment. On the other hand, if it is determined in step S34 that the evaluation value for the low-quality image is not within the input tolerance range, the process proceeds to step S35. In step S35, the display unit displays the evaluation value and the input tolerance range for the low-quality image, as well as the reason why the low-quality image was determined to be inappropriate input data and that a high-quality image will be acquired, thereby presenting these contents to the user.

[0059] When the variation in the evaluation values ​​of the estimated high-quality images is large, the accuracy of estimating the variation in the evaluation values ​​of the high-quality images relative to the evaluation values ​​of the estimated high-quality images decreases, but according to this embodiment, when the variation in the evaluation values ​​of the estimated high-quality images is large, they are not input to the model application unit 11, thereby preventing erroneous judgment in the judgment unit 13. Furthermore, since they are not input to the model application unit 11 and processing by the model application unit 11 is omitted, the processing throughput can be made more efficient.

[0060] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be implemented in hardware, in part or in whole, by designing, for example, an integrated circuit. Furthermore, the above-described configurations, functions, etc. may be implemented in software, by a processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0061] REFERENCE SIGNS LIST 11... model application unit, 12... evaluation value calculation unit, 13... judgment unit, 14... image acquisition unit, 15... variation acquisition unit, 16... input verification unit, 17... input tolerance calculation unit

Claims

1. An image processing system comprising: a model application unit that applies a machine learning model to a low-quality image to generate a predicted high-quality image; an evaluation value calculation unit that calculates an evaluation value of the predicted high-quality image; and a judgment unit that makes a judgment regarding the predicted high-quality image based on the evaluation value of the predicted high-quality image, wherein the judgment unit judges whether the predicted high-quality image is uncertain data using a previously obtained range of variation in evaluation values ​​of a high-quality image obtained by capturing the low-quality image with high quality, which corresponds to the evaluation value of the predicted high-quality image.

2. An image processing system as described in claim 1, characterized in that the judgment unit judges the estimated high-quality image to be confirmed data when the entire variation range is included in a predetermined tolerance range or when the entire variation range is not included in the tolerance range, and judges the estimated high-quality image to be uncertain data when a part of the variation range is not included in the tolerance range.

3. An image processing system as described in claim 2, characterized in that it comprises a variation acquisition unit that acquires the variation range from a distribution of evaluation values ​​of the high-quality image corresponding to the evaluation value of the estimated high-quality image.

4. An image processing system as described in claim 3, characterized in that when the judgment unit judges the estimated high-quality image to be the uncertain data, it causes an image acquisition unit that acquires an image to acquire the high-quality image instead of the estimated high-quality image that has been judged to be the uncertain data.

5. An image processing system as claimed in claim 3, characterized in that the evaluation value includes any one of an image feature calculated from the image, an inspection result of an object in the image, and a measurement value of an object in the image.

6. An image processing system as described in claim 3, comprising: a display unit that displays a method for determining the variation range; and an instruction input unit that accepts an instruction to change the method for determining the variation range.

7. An image processing system as described in claim 3, characterized in that it is provided with a display unit that displays the evaluation value of the estimated high-quality image, the variation range, and the tolerance range for the estimated high-quality image determined to be uncertain data.

8. An image processing system as described in claim 3, further comprising an input verification unit that calculates an evaluation value of the low-quality image input to the model application unit, and determines whether or not to input the low-quality image to the model application unit based on the evaluation value of the low-quality image.

9. An image processing system as described in claim 8, characterized in that the input verification unit inputs the low-quality image to the model application unit when the evaluation value of the low-quality image is included in a previously acquired input tolerance range, and does not input the low-quality image to the model application unit when the evaluation value of the low-quality image is not included in a previously determined input tolerance range.

10. An image processing system as described in claim 9, comprising an input tolerance range calculation unit, characterized in that a distribution of evaluation values ​​of the estimated high quality image corresponding to the evaluation values ​​of the low quality image is obtained, and the width between the minimum and maximum values ​​of the distribution is set as the input tolerance range.

11. An image processing system as claimed in claim 10, further comprising a display unit for displaying the evaluation value of the low-quality image that is not input to the model application unit and the input tolerance range.

Images