Image quality improvement device, image quality improvement method, and image quality improvement program

Abstract

The purpose of the present invention is to provide a technique with which it is possible, when a learning model corresponding to a target image for improving image quality is not created by using machine learning, to re-train a learning model suitable for the target image in a short time. An image quality improvement device according to the present invention creates a new learning model group by implementing re-training using an additional-training image dataset having a feature similar to that of target image data, and acquires image-quality-improved image data in which the image quality of the target image data is improved by inputting the target image data to any learning model included in the learning model group (see fig. 2).

Description

Image quality improvement device, image quality improvement method, and image quality improvement program

[0001] The present invention relates to a technique for converting a low-quality image into a high-quality image.

[0002] There are various techniques for improving image quality. For example, there is a technique for improving image quality by performing processing to reduce noise in the image. In recent years, a technique for converting low-quality images into high-quality images using machine learning has been developed.

[0003] Patent Literature 1 addresses the issue of "providing a medical imaging device that improves image quality by accurately reducing noise while preventing inappropriate image processing by enabling analysis of image processing content in image processing using a machine learning model" and describes the following technology (see abstract): "The device includes a medical image acquisition unit that acquires medical images, a noise evaluation unit 211 that determines whether noise in the medical images exceeds a predetermined reference value, and a noise reduction unit 212 that reduces noise in medical images determined by the noise evaluation unit to have noise exceeding the reference value, wherein the noise reduction unit is constructed by collecting multiple learning datasets consisting of images containing noise as input data and images not containing noise as output data, and has multiple layers that perform convolution on the input images, and wherein any of the multiple layers includes a filter layer incorporating multiple linear or nonlinear filters with predetermined convolution coefficients, thereby reducing noise in the medical images" (see abstract).

[0004] Patent Document 2 describes the technology as follows: "A learning data collection device, a learning data collection method, and a program for collecting learning data that allows efficient re-learning are provided. The learning data collection device (10) includes an inspection image acquisition unit (11) that acquires an inspection image, an area detection result acquisition unit (damage detection result acquisition unit (13)) that acquires area detection results detected by a trained area detector, a correction history acquisition unit (15) that acquires a correction history of the area detection results, a calculation unit (17) that calculates correction quantification information that quantifies the correction history, a database that stores the inspection image, the area detection results, and the correction history in association with each other, an image extraction condition setting unit (19) that sets a threshold value of the correction quantification information as an extraction condition for extracting inspection images to be used for re-learning from the database, and a first learning data extraction unit (21) that extracts inspection images that satisfy the extraction conditions, and the area detection results and correction history associated with the inspection images, as learning data for re-learning the area detector" (see Abstract).

[0005] JP 2020-141908 A

[0006] The technology described in Patent Literature 1 creates a learning model in advance and associates supplementary information, such as the type of image, the part of the image, and the imaging conditions at the time of model creation, with the learning model. When performing image quality improvement processing, a learning model corresponding to the target image is selected based on the supplementary information. Therefore, if there is no learning model among the pre-trained learning models that matches the supplementary information of the target image to be improved (i.e., if it has not been trained), an appropriate learning model cannot be selected, and the image quality improvement processing cannot be performed appropriately.

[0007] The technology described in Patent Document 2 attempts to solve the problem of a pre-trained model not matching the supplementary information through re-training. According to this document, even if the supplementary information does not match the training model, it is possible to create a training model optimized for the target image through re-training. On the other hand, a training model based on machine learning is composed of multiple layers. When re-training is performed, re-training is performed across all of these multiple layers. Therefore, it takes a long time to regenerate the training model. Patent Document 2 does not mention speeding up the process of recreating the training model.

[0008] When implementing image quality improvement using machine learning, it is often difficult to obtain the target image to be improved in advance, making it difficult to prepare a learning model corresponding to the target image in advance (it is not possible to prepare an untrained learning model in advance). Furthermore, in product manufacturing processes and inspection processes, there is a strong demand for minimizing downtime and maintaining constant operation. Therefore, an optimal learning model must be generated in a short time by retraining an untrained learning model. Conventional technologies such as those described in Patent Documents 1 and 2 have difficulty meeting these needs.

[0009] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that enables re-learning a learning model suitable for a target image in a short period of time when a learning model corresponding to the target image whose image quality is to be improved using machine learning has not been created.

[0010] The image quality improvement device of the present invention creates a new group of learning models by performing re-learning using an additional learning image dataset having characteristics similar to those of the target image data, and obtains image quality-improved image data that improves the image quality of the target image data by inputting the target image data into one of the learning models included in the group of learning models.

[0011] According to the image quality improvement device of the present invention, if a learning model corresponding to the target image whose image quality is to be improved using machine learning has not been created, a learning model suitable for the target image can be re-trained in a short period of time.

[0012] 1 is a configuration diagram of an image quality improvement device 2 that improves the image quality of images using machine learning. It is a block diagram explaining the detailed configuration of the improvement processing unit 28 in embodiment 1. It shows the layer configuration of a U-net CNN. It is a diagram showing a mechanism for constructing a new learning model by re-learning in embodiment 1. It shows an example of an accuracy index value calculated by a performance evaluation unit 281b. It shows another method for selecting a learning model based on evaluation index values. It shows another method for selecting a learning model based on evaluation index values. It shows a procedure by which an image quality improvement device 2 according to embodiment 2 creates a general-purpose pre-training model candidate group 281. It shows an example of a pre-training image dataset 25a. It shows an example of blending in a blended training dataset group 25c. It is a diagram explaining the processing performed by a general-purpose pre-training model generation unit 25d. It is a configuration diagram of an image quality improvement device 2 according to embodiment 3. It is a configuration diagram of an image quality improvement device 2 according to embodiment 4. It shows an example of attributes of low-quality image data 287.

[0013] First Embodiment The following describes an example of an apparatus for measuring the shape of a circuit pattern formed on a wafer using an electron microscope or the like. When imaging a circuit pattern formed on a wafer using an electron microscope, an electron beam is irradiated onto the wafer and the reflected electrons are detected by a detector to obtain an image. Irradiating a wafer with an electron beam can sometimes abrade the circuit pattern, damaging it. Therefore, it is desirable to obtain an image with a low electron beam irradiation. However, images obtained with a low electron beam irradiation contain a lot of particulate noise and are not suitable for shape measurement (hereinafter, such images are referred to as low-quality images). In image-based shape measurement, the edge of the circuit pattern is detected using image processing, and the distance between pixels at the target measurement location is measured. Particulate noise can obscure the edge and may be erroneously recognized as the edge of the circuit pattern. To reduce such particulate noise, the same imaging position is photographed multiple times and the integrated average of the images is calculated to reduce the noise (hereinafter, such a noise-reduced image is referred to as a high-quality image). High-quality images are close to images of true circuit patterns (high-quality images without noise). Using high-quality images allows for accurate image measurements (hereafter, measurements taken with high-quality images will be referred to as true values).

[0014] However, to obtain a high-quality image, it is necessary to collect multiple low-quality images. This means that the wafer is irradiated with a large amount of electron beams, which can cause significant damage to the wafer. Therefore, a mechanism is needed to use low-quality images to obtain images of the same quality as high-quality images.

[0015] 1 is a configuration diagram of an image quality improvement device 2 that improves the image quality of an image using machine learning. The image quality improvement device 2 is a device that, when a low-quality image is input, outputs a high-quality image obtained by improving the image quality of the input image. The image quality improvement device 2 includes a learning model creation unit 21 and an image quality improvement unit 22. The learning model creation unit 21 further includes a creation processing unit 25. The image quality improvement unit 22 further includes an improvement processing unit 28. Here, it is assumed that machine learning is performed using an image that includes a shape pattern A.

[0016] The creation processing unit 25 uses a pair of a low-quality image 23 of pattern A and a high-quality image 24 of pattern A to create a learning model for inferring a high-quality image 24 from the low-quality image 23. The creation processing unit 25 can create the learning model using, for example, a U-net (an encoder-decoder convolutional neural network structure having a U-shaped multi-layer structure with skip connections). Creating this learning model generally requires a large number of repeated calculations, which takes a lot of time. The learning model created by the creation processing unit 25 is stored as a learning model 26 for pattern A. The learning model created by the creation processing unit 25 using an image of pattern A is a learning model optimized for pattern A and cannot be used to improve the image quality of images of other patterns.

[0017] The improvement processing unit 28 receives the low-quality image 27 of pattern A as input data, and inputs this into the learning model 26 to output a high-quality image 29. Similar to the creation processing unit 25, the improvement processing unit 28 generates the high-quality image 29 using parameters possessed by each layer of the learning model 26.

[0018] As described above, conventional image quality improvement processes using machine learning have the following problems: (1) it takes time to generate a learning model, and (2) it is not possible to improve the image quality of unlearned images. Therefore, in embodiment 1 of the present invention, a method for speeding up the process by which the improvement processing unit 28 generates a high-quality image 29 will be described. Since the configuration of the image quality improvement device 2 is the same as in FIG. 1, the following description will mainly focus on the process performed by the improvement processing unit 28.

[0019] 2 is a block diagram illustrating the detailed configuration of the improvement processing unit 28 in embodiment 1. The improvement processing unit 28 includes the data and processing units shown in FIG. 2. The overall process performed by the improvement processing unit 28 is roughly as follows: (1) first, general-purpose pre-training model candidates having multiple common feature extraction layers are prepared, (2) from these training models, a model optimized for a new image (untrained image) is created by fine-tuning the training models, (3) a training model that satisfies the accuracy conditions within a limited processing time is selected, and (4) this is used to improve the image quality of an unknown image.

[0020] The general-purpose pre-training model candidate group 281 is a general-purpose training model group that has a common feature extraction layer. For example, multiple training image sets are combined, and a training model is created for each combination. This allows the general-purpose pre-training model candidate group 281 to be generated. Details of the general-purpose pre-training model candidate group 281 will be explained again later using Figures 3 and 4.

[0021] The model candidate selection unit 282 selects a learning model whose performance will be evaluated in the following steps from among the pre-training models in the general-purpose pre-training model candidate group 281. The selected pre-training model 283 will be the evaluation target in the following steps.

[0022] Image dataset 284 is a dataset for additional learning. Image dataset 284 is, for example, an image dataset that was not used when creating general-purpose pre-training model candidate group 281. For example, it is assumed that an image set that was not available when creating general-purpose pre-training model candidate group 281 is used as image dataset 284. Image dataset 284 is an image set that pairs low-quality images and high-quality images.

[0023] The re-learning unit 285 performs re-learning (fine-tuning) on ​​the selected pre-learning model 283 using the image dataset 284. Fine-tuning is a technique for creating a learning model optimized for the image dataset 284 by fine-tuning the selected pre-learning model 283. The new learning model obtained by re-learning is saved as a learning model group 286.

[0024] The new low-quality image data 287 is image data whose image quality is to be improved. The low-quality image data 287 is assumed to have the same characteristics (features) as the image data set 284. In other words, it is assumed that at least one of the learning models in the learning model group 286 obtained by re-learning is suitable for use in improving the image quality of the low-quality image data 287.

[0025] The learning model selection unit 288 arbitrarily selects one of the learning models in the learning model group 286. The improvement processing unit 28 inputs low-quality image data 287 into the learning model selected by the learning model selection unit 288, thereby obtaining an image with improved quality 281a (image quality improvement process 289).

[0026] The performance evaluation unit 281b evaluates the extent to which the image quality of the image quality improved image 281a has been improved relative to the image quality of the low-quality image data 287. The evaluation index may be changed depending on the purpose of using the image quality improved image 281a. For example, if the purpose is image observation, the PSNR (Peak Signal-to-Noise Ratio) for evaluating image quality may be used as the evaluation index, and if the purpose is image measurement, the accuracy evaluation values ​​of the measurement values ​​measured using the image quality improved image 281a and the measurement values ​​obtained from a high-quality image (a higher quality image of the same measurement target as the low-quality image data 287) may be used as the evaluation index.

[0027] FIG. 3 shows the layer configuration of the U-net CNN. The CNN includes an input layer 3a, a convolution layer 3b, a batch normalization layer 3c, a ReLU (Rectified Linear Unit) layer 3d, and an output layer 3e. Conventional machine learning methods learn completely different values ​​for both the convolution layer 3b and the batch normalization layer 3c each time they learn images from various image sets. Therefore, it is not possible to improve the image quality of image patterns other than those used for learning. Furthermore, when fine-tuning is performed, both the convolution layer 3b and the batch normalization layer 3c learn features that are different from unlearned features, so fine-tuning requires a lot of time.

[0028] 4 is a diagram showing a mechanism for constructing a new learning model by re-learning in the first embodiment. In the first embodiment, the batch normalization layer 3c is replaced based on the class label 4a by re-learning. Such a learning model is called a CBN (Convolutive Bottleneck Network). Instead of switching the batch normalization layer 3c, for example, the instance normalization layer may be switched.

[0029] In CBN, the batch normalization layer 3c is switched based on class labels (information for identifying the image patterns and shooting conditions of an image set). Since the convolution layer 3b is not switched, when a set of images of various classes is trained, the convolution layer 3b extracts features common to the various classes. In contrast, the batch normalization layer 3c is switched for each class information, so features of each class are extracted in the batch normalization layer 3c. Furthermore, in the case of an electron microscope for semiconductor inspection, the basic operating principle remains unchanged even when the shooting environment changes, so the convolution layer 3b is more likely to extract common features. Therefore, when one of the training models from which common features have been extracted is selected and fine-tuning (relearning) is performed, it is considered that the difference between the features extracted by the convolution layer 3b before retraining and the features extracted by the convolution layer 3b of the retrained training model for a new image is not very large. Therefore, the optimization process of the convolution layer is completed in a short time.

[0030] Based on the above concept, the relationship between the trained batch normalization layer 3c and the feature amount (i.e., the class label 4a) of the image data used to train the batch normalization layer 3c is stored in advance in the image quality improving device 2. In this way, when a new image is input to the learning device, it can be determined whether or not a learning device corresponding to the image has already been constructed.

[0031] 5 shows an example of accuracy index values ​​calculated by the performance evaluation unit 281b. The horizontal axis indicates the number of fine-tuning iterations, and the vertical axis indicates the evaluation index value. The closer the evaluation index value is to 1.0, the more measurement results can be output that are comparable to high-quality images (high-quality target image data). In other words, a learning model with an evaluation index value close to 1.0 indicates that it estimates an image that is close to a high-quality image.

[0032] In the figure, Bn represents an example of fine-tuning performed using the conventional method described in Figure 3. Model 1 to Model 4 represent the model names of the selected pre-trained models 283. These training models were used as the initial values ​​for fine-tuning to create training models for new images (untrained images), and the accuracy index value was calculated for each iteration of fine-tuning using the image after image quality improvement. Figure 5 shows the results.

[0033] In the case of Bn, it can be seen that 100,000 repeated calculations are required to generate a learning model that can output images that can output measurement results equivalent to high-quality images using fine tuning. In contrast, in the case of Model 1, the accuracy index value reaches 1.0 after about 1,000 iterations. In other words, it can be seen that a learning model with the same level of accuracy can be created with 1 / 100th the number of iterations compared to Bn. Since the number of iterations and the time required to generate the learning model are proportional, it means that a learning model can be generated in 1 / 100th the time.

[0034] The accuracy index values ​​also differ between Model 1 to Model 4. Model 3 reached an accuracy index value of 1.0 after only about 20 iterations, while Model 4 required 20,000 iterations. This is presumably due to the influence of differences in the degree of similarity between the features extracted by the convolution layer 3b of the selected pre-trained model 283 and the features of the image to be newly subjected to image quality improvement. Model 3 had high similarity, and a high-precision learning model was generated with a small number of iterations, but Model 4 had low similarity, and therefore likely required many iterations to generate a high-precision learning model.

[0035] The learning model selection unit 288 selects a learning model suitable for improving the image quality of the new low-image-quality image data 287 based on the evaluation index values ​​as exemplified in Fig. 5. Alternatively, the evaluation index values ​​as shown in Fig. 5 may be displayed on a screen, and the user may select one of the learning models while viewing the screen.

[0036] 6A shows another method for selecting a learning model based on an evaluation index value. The performance evaluation unit 281b derives an allowable number of iterations (e.g., 20 times) from the allowable processing time for creating a learning model, and determines whether the allowable evaluation index value (e.g., 1.0) is satisfied within the range of the allowable number of iterations. As shown in FIG. 6A, the determination result is displayed on the screen (◯: Pass, ×: Fail), and the user may select one of the learning models based on this, or the learning model selection unit 288 may automatically select a learning model that satisfies the evaluation index value.

[0037] FIG. 6B shows another method for selecting a learning model based on evaluation index values. In this example, there are two types of evaluation index values ​​(evaluation index values ​​A and B). For example, when it is desired to achieve both measurement accuracy and visual image reproducibility, multiple types of evaluation index values ​​can be used in this way. In the example of FIG. 6B, the judgment result for evaluation index value A and the judgment result for evaluation index value B do not match. In such a case, the learning model selection unit 288 may automatically select a learning model for which both judgment results match (satisfy the conditions), or the user may select one of the learning models based on their own judgment.

[0038] When improving the image quality of an image that belongs to the same class (having the same attributes such as the image pattern and shooting conditions of the image set) as the low-quality image data 287 used to optimize the learning model through re-learning, the learning model selected by the learning model selection unit 288 during re-learning can be used, so there is no need to re-learn or evaluate the performance again.

[0039] Summary of First Embodiment The image quality improvement device 2 according to the first embodiment creates a new training model group 286 by retraining one of a general-purpose pre-training model candidate group 281, which has previously undergone machine learning to improve the image quality of image data different from the target image data (low-quality image data 287), using an image dataset 284 for additional training that has characteristics similar to the target image data. The image quality improvement device 2 obtains an image quality-improved image 281a by inputting the target image data into a training model in the training model group 286. The image quality improvement device 2 selects a training model suitable for improving the image quality of the target image data by evaluating the evaluation index value of the image quality-improved image 281a. Because the training model group 286 is constructed using an image dataset 284 that has characteristics similar to the target image data, retraining can be completed with a relatively small number of training iterations. Therefore, a training model optimized for the target image data can be constructed in a short retraining time.

[0040] When retraining a learning model, the image quality improving device 2 according to the first embodiment performs retraining in accordance with the CBN model configuration, replacing the batch normalization layer 3c with one that corresponds to the feature of the target image data (i.e., one that corresponds to the class label 4a). This allows retraining to be completed in a shorter time than when all layers in the learning model are reconstructed. CBN is not a model for retraining, but a method in which intermediate layers (in this example, the batch normalization layer 3c) that correspond to various images are prepared in advance. The significance of this embodiment lies in the application of this method to shorten the retraining time.

[0041] <Embodiment 2> Fig. 7 shows a procedure by which an image quality improving device 2 according to embodiment 2 of the present invention creates a general-purpose pre-training model candidate group 281. This procedure is performed by the learning model creation unit 21 (creation processing unit 25). The creation processing unit 25 includes the data and processing unit shown in Fig. 7. The other configurations are the same as those of embodiment 1.

[0042] The pre-learning image dataset 25a is data in which an arbitrarily collected image dataset is associated with the class label of each image. An example of the pre-learning image dataset 25a will be shown again in FIG. 8, which will be described later.

[0043] The dataset blending unit 25b selects and combines (blends) image datasets that have common class elements from among the image datasets included in the pre-training image dataset 25a. The blended datasets are output as a blended training dataset group 25c. A specific example will be shown in FIG. 9, which will be described later. The purpose of combining datasets that have common class elements is to make it easier to extract features common to those datasets in the convolution layer 3b.

[0044] The relationship between the blended learning dataset group 25c, the general-purpose pre-training model generation unit 25d, and the general-purpose pre-training model candidate group 281 will be explained again later with reference to FIG. 10 .

[0045] 8 shows an example of a pre-training image dataset 25a. The class elements include, for example: (a) imaging conditions such as the device ID of the imaging device when each image set was captured, the acceleration voltage, etc.; (b) the shape of the pattern to be captured; etc. In this example, the image set name and the class label are the same, but they do not have to be the same.

[0046] FIG. 9 shows an example of blending in the blended training dataset group 25c. Here, for each combination of datasets used to create a general-purpose pre-training model, the general-purpose pre-training model is distinguished by its training model name. Training model name Model1 is created using a dataset that combines all datasets. Model2 is created using a dataset that combines image sets with an acceleration voltage of AkV. Model3 is created using a dataset that combines image sets with an acceleration voltage of BkV. Model4 is created using a dataset that combines image sets with a beam current of (High). If it is not known in advance what class elements a new image set (untrained images) will contain, the class elements may be randomly selected and blended.

[0047] Figure 10 is a diagram explaining the processing performed by the general-purpose pre-training model generation unit 25d. The general-purpose pre-training model generation unit 25d creates a training model using each blended dataset (each row of the datasets illustrated in Figure 9) included in the blended training dataset group 25c as training data. The training method is the same as the re-training algorithm performed by the re-training unit 285. However, while the re-training unit 285 performs re-training using each pre-training model in the general-purpose pre-training model candidate group 281 as the initial value, the general-purpose pre-training model generation unit 25d creates a training model using a state in which there is no pre-training model as the initial value. The generated training model is saved as one of the pre-training models in the general-purpose pre-training model candidate group 281.

[0048] <Third Embodiment> Fig. 11 is a configuration diagram of an image quality improvement device 2 according to a third embodiment of the present invention. A creation processing unit 25 creates a general-purpose pre-trained model candidate group 281 using the method described in the second embodiment. An improvement processing unit 28 improves the image quality using the method described in the first embodiment. In the third embodiment, these processes are integrated as an image quality improvement device 2, but the processing procedure is the same as that described in the first and second embodiments.

[0049] Data may be exchanged between the creation processing unit 25 and the improvement processing unit 28 using a communication line or a storage medium. Also, in Figure 11, the general-purpose pre-training model candidate group 281 is stored in the creation processing unit 25, but the exact same data may be stored in the improvement processing unit 28. Even if the creation processing unit 25 and the improvement processing unit 28 (i.e., the learning model creation unit 21 and the image quality improvement unit 22) are geographically separated, they can operate as the image quality improvement device 2 by exchanging data between them.

[0050] If creation processing unit 25 and improvement processing unit 28 are physically separated, a division of labor becomes possible, for example, by placing a device including creation processing unit 25 in a company that creates general-purpose pre-training model candidate group 281, and placing a device including improvement processing unit 28 in a company that performs image quality improvement processing (and analysis using the results). With this configuration, a company that analyzes images after image quality improvement can entrust the creation of general-purpose pre-training models to a specialized company.

[0051] <Fourth Embodiment> Fig. 12 is a configuration diagram of an image quality improving device 2 according to a fourth embodiment of the present invention. The image quality improving device 2 according to the fourth embodiment includes an optimal blend estimation unit 1201 in addition to the configuration described in the first to third embodiments. The other configurations are the same as those of the first to third embodiments. For the sake of convenience, the same configurations as those of the first to third embodiments are not shown in the figure, and only the connection relationships of the components are shown.

[0052] Each time it receives new low-quality image data 287, the optimal blend estimation unit 1201 learns the following through machine learning: (a) the learning model selected by the learning model selection unit 288 based on the evaluation results by the performance evaluation unit 281b, (b) the attributes of the new low-quality image data 287 (an example will be shown again in FIG. 13 described later), and (c) the blend combination of the blended training dataset group 25c blended by the dataset blending unit 25b. After completing learning, when the optimal blend estimation unit 1201 receives new low-quality image data 287, it outputs the blend combination of the blended training dataset group 25c that is suitable for the low-quality image data 287 based on the learning results so far. This makes it possible to automatically select the optimal blend combination of the pre-training image dataset 25a.

[0053] 13 shows an example of attributes of the low-quality image data 287. These attributes are similar to the class attributes of the pre-learning image dataset 25a (shown in FIG. 8 as an example).

[0054] <Regarding Modifications of the Present Invention> The present invention is not limited to the above-described embodiment, and various modifications are included. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to an embodiment including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0055] In Figure 10, for the sake of explanation, the learning model creation process is depicted as if there are multiple physically independent processes, but a single computing device may be used to process the blended learning data set in sequence. Also, since the contents of the calculation processes are independent, parallel processing as shown in Figure 10 may be performed by separate processing devices, and the created learning models may be collected via a communication line, etc. Parallel processing makes it possible to generate multiple learning models in a short time.

[0056] In the above embodiment, the improvement processing unit 28 has been described as including the data and functional units described in Fig. 2, but these may be configured as data and functional units separate from the improvement processing unit 28. Similarly, the creation processing unit 25 has been described as including the data and functional units described in Fig. 7, but these may be configured as data and functional units separate from the creation processing unit 25. In either case, the functional units can be arranged as components of the image quality improving device 2.

[0057] In the above embodiments, each functional unit of the image quality improvement device 2 (the creation processing unit 25, the improvement processing unit 28, the optimal combination estimation unit 1201, and the functional units internal to these functional units (as described in Figures 2, 7, 12, etc.)) can be configured by hardware such as a circuit device that implements these functions, or by software that implements these functions being executed by an arithmetic device such as a CPU (Central Processing Unit).

[0058] In the above embodiment, the normalization layer is replaced during re-learning. If the learning device includes a layer other than the normalization layer, the layer may be replaced. That is, in a learning device that includes a feature extraction layer that extracts image features and an intermediate layer that processes the features, the intermediate layer may be replaced during re-learning.

[0059] In the above embodiment, the normalization layer (the hidden layer in the above example) is replaced and re-learning is described, but layers other than the normalization layer may be reconfigured during re-learning. In other words, layers other than the normalization layer may also be reconfigured during re-learning if the learning load is smaller than re-learning all layers in the learning device as in the conventional technology.

[0060] 2: Image quality improvement device 25: Creation processing unit 28: Improvement processing unit 1201: Optimal blend estimation unit

Claims

1. An image quality improvement device that improves the image quality of an image, comprising: an image quality improvement unit that improves the image quality of target image data using a learning model that improves the image quality of images through machine learning; wherein the image quality improvement unit creates a new group of learning models by re-training one of a group of candidate learning models that have previously undergone machine learning to improve the image quality of image data different from the target image data, using an additional learning image dataset that has characteristics similar to the target image data; the image quality improvement unit obtains improved image data in which the image quality of the target image data has been improved by inputting the target image data into one of the learning models included in the group of learning models; and the image quality improvement unit evaluates an evaluation index value of the improved image quality image data to select, from the group of learning models, a learning model that is most suitable for improving the image quality of the target image data, and improves the image quality of the target image data using the selected learning model.

2. The image quality improvement device of claim 1, characterized in that the learning model candidate included in the group of learning model candidates comprises: a feature extraction layer that extracts features of image data; and an intermediate layer that processes the features extracted by the feature extraction layer; and when performing the re-learning, the image quality improvement unit performs the re-learning so as to replace the intermediate layer with one that corresponds to the features of the additional learning image dataset.

3. The image quality improvement device according to claim 1, characterized in that the image quality improvement unit acquires image quality of high-quality target image data, which is image data of the same object as the target image data but has higher image quality than the target image data, and the image quality improvement unit evaluates the image quality of the improved image data by comparing the image quality of the high-quality target image data with the image quality of the improved image data.

4. The image quality improving device according to claim 1, characterized in that: the target image data is observation image data of a sample; the image quality improving unit obtains an evaluation index value as a result of measuring the sample using high-quality target image data having a higher image quality than the target image data; the image quality improving unit obtains an evaluation index value as a result of measuring the sample using the improved image data; and the image quality improving unit evaluates the image quality of the improved image data by comparing the evaluation index value of the high-quality target image data with the evaluation index value of the improved image data.

5. The image quality improvement device described in claim 1, characterized in that the image quality improvement unit outputs the results of evaluating the image quality of the image data to be improved using each of the learning models included in the group of learning models, the image quality improvement unit receives a selection input that selects one of the evaluation results of the learning models, and the image quality improvement unit improves the image quality of the target image data using the learning model selected by the selection input.

6. The image quality improvement device described in claim 1, characterized in that the image quality improvement unit acquires a first evaluation index value of the image quality-improved image data and acquires a second evaluation index value different from the first evaluation index value of the image quality-improved image data, and the image quality improvement unit improves the image quality of the target image data using a learning model included in the learning model group, the learning model having the first evaluation index value that satisfies a standard value and the second evaluation index value that satisfies a standard value.

7. The image quality improvement device according to claim 1, further comprising a creation processing unit that creates the group of candidate learning models, wherein the creation processing unit acquires a pre-learning image dataset in which attribute information indicating the shooting conditions of the images is linked to image data, and the creation processing unit creates the group of candidate learning models by performing machine learning using any two or more of the image data included in the pre-learning image dataset.

8. The image quality improvement device described in claim 7, characterized in that the creation processing unit creates a group of blended learning datasets by combining multiple image data included in the pre-training image datasets that have common attribute information, and the creation processing unit creates the group of learning model candidates by performing machine learning on each of the group of blended learning datasets that have common attribute information.

9. The image quality improvement device according to claim 7, characterized in that the image quality improvement unit and the creation processing unit transmit and receive data to each other via a network, thereby transferring the group of learning model candidates created by the creation processing unit to the image quality improvement unit.

10. The image quality improvement device of claim 7 further comprises an optimal combination estimation unit that estimates a combination of image data contained in the pre-training image dataset that is suitable for creating the group of learning model candidates, and the creation processing unit creates the group of learning model candidates using the combination estimated by the optimal combination estimation unit.

11. The image quality improvement device described in claim 10, characterized in that the optimal combination estimation unit learns through machine learning the relationship between combinations of image data included in the pre-training image dataset and the results of evaluating the image quality of the image quality improved image data using that combination, and the optimal combination estimation unit estimates combinations of image data included in the pre-training image dataset that are suitable for creating the group of candidate learning models based on the results of learning the relationship.

12. A method for improving the image quality of an image, comprising the steps of: improving the image quality of target image data using a learning model that improves the image quality of images through machine learning; wherein in the step of improving the image quality, a new group of learning models is created by re-training one of a group of candidate learning models that have previously undergone machine learning to improve the image quality of image data different from the target image data, using an additional learning image dataset that has characteristics similar to the target image data; and in the step of improving the image quality, the target image data is input into one of the learning models included in the group of learning models to obtain improved image quality image data in which the image quality of the target image data has been improved; and in the step of improving the image quality, an evaluation index value of the improved image quality image data is evaluated to select, from the group of learning models, a learning model that is most suitable for improving the image quality of the target image data, and the selected learning model is used to improve the image quality of the target image data.

13. An image quality improvement program for causing a computer to execute the image quality improvement method according to claim 12.

Images