Image processing system, image processing method, and program

Abstract

To provide a technique for realizing high generalization performance in image processing for improving image quality.SOLUTION: An image processing system includes: a microscope system for acquiring an input image inputted to an image conversion section; a storage section for storing multiple learned models that learn image conversion for converting an input image acquired by the microscope system to an output image having a higher image quality than the input image; a selection section for selecting a learned model from among the multiple learned models stored in the storage section; and an image conversion section for performing image conversion by using a learned model selected by the selection section. Each of the multiple learned models is a learned model that learns by an image at least having an image quality range different from that of other learned models, and the selection section selects the learned model used for the image conversion from the multiple learned models stored in the storage section on the basis of setting information of the microscope system when the input image is acquired.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The disclosure of this specification relates to an image processing system, an image processing method, and a program. [Background technology]

[0002] In recent years, techniques for improving image quality using machine learning, particularly deep learning, have been proposed. Such techniques are described in, for example, Non-Patent Document 1 and Non-Patent Document 2.

[0003] Non-Patent Document 1 describes a technology that uses a generative adversarial network (GAN) model to convert images constrained by the diffraction limit into super-resolution images. Non-Patent Document 2 describes a technology that uses image restoration based on deep learning to expand the observable range to include biological phenomena that were previously difficult to observe. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Hongda Wang, et al., Deep learning enables cross-modality super-resolution in fluorescence microscopy, NATURE METHODS, VOL.16, p.103-110, JANUARY 2019 [Non-patent document 2] Martin Weigert, et al., Content-aware image restoration: pushing the limits of fluorescence microscopy, NATURE METHODS, VOL.15, p.1090-1097, DECEMBER 2018 Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, it is desirable for a trained model in machine learning to have high generalization performance. However, achieving high generalization performance is not easy. For example, in the case of a trained model trained by supervised learning, performance tends to deteriorate for images that are significantly different from those used for training.

[0006] In view of the above circumstances, an object of one aspect of the present invention is to provide a technique that achieves high generalization performance in image processing that improves image quality. [Means for solving the problem]

[0007] An image processing system according to one aspect of the present invention includes a microscope system that acquires an input image to be input to an image conversion unit; a memory unit that stores a plurality of trained models that have learned image conversion to convert the input image acquired by the microscope system into an output image having higher image quality than the input image; a selection unit that selects a trained model from the plurality of trained models stored in the memory unit; and the image conversion unit that performs the image conversion using the trained model selected by the selection unit, wherein each of the plurality of trained models is a trained model that has been trained with an image having at least a different image quality range from other trained models, and the selection unit selects a trained model to be used for the image conversion from the plurality of trained models stored in the memory unit based on setting information of the microscope system at the time the input image was acquired, , times an optical microscope system including a plurality of objective lenses with different refractive indices, and the setting information is information about the objective lenses set in the optical microscope system; twice as many Includes rate. Another aspect of the present invention provides an image processing system comprising: a microscope system that acquires an input image to be input to an image conversion unit; a memory unit that stores a plurality of trained models that have learned image transformation to convert the input image acquired by the microscope system into an output image having higher image quality than the input image; a selection unit that selects a trained model from the plurality of trained models stored in the memory unit; and the image conversion unit that performs the image transformation using the trained model selected by the selection unit, wherein each of the plurality of trained models is a trained model that has been trained with an image having at least a different image quality range from other trained models; the selection unit selects a trained model to be used for the image transformation from the plurality of trained models stored in the memory unit based on setting information of the microscope system at the time the input image was acquired; and the microscope system is an optical microscope system including a plurality of objective lenses with different numerical apertures, and the setting information includes the numerical apertures of the objective lenses set in the optical microscope system.

[0008] An image processing method according to one aspect of the present invention includes: selecting a trained model from among a plurality of trained models stored in a storage unit that stores a plurality of trained models that have learned image transformation to convert an input image acquired by a microscope system into an output image having higher image quality than the input image; performing the image transformation using the selected trained model; each of the plurality of trained models being trained with an image having at least a different image quality range from other trained models; and selecting the trained model based on setting information of the microscope system at the time the input image was acquired, the microscope system , times an optical microscope system including a plurality of objective lenses with different refractive indices, and the setting information is information about the objective lenses set in the optical microscope system; twice as many Includes rate. Another aspect of the present invention provides an image processing method comprising: selecting a trained model from among a plurality of trained models stored in a memory unit that stores a plurality of trained models that have learned image transformation to convert an input image acquired by a microscope system into an output image having higher image quality than the input image; performing the image transformation using the selected trained model; each of the plurality of trained models being a trained model that has been trained with an image having at least a different image quality range from other trained models; selecting the trained model to be used for the image transformation from the plurality of trained models stored in the memory unit based on setting information of the microscope system at the time the input image was acquired; the microscope system being an optical microscope system including a plurality of objective lenses with different numerical apertures; and the setting information including the numerical apertures of the objective lenses set in the optical microscope system.

[0009] A program according to one aspect of the present invention causes a computer to execute a process of selecting a trained model from among a plurality of trained models stored in a storage unit that stores a plurality of trained models that have learned image transformation to convert an input image acquired by a microscope system into an output image having higher image quality than the input image, performing the image transformation using the selected trained model, each of the plurality of trained models being a trained model that has been trained with an image having at least a different image quality range from other trained models, and selecting the trained model to be used for the image transformation from among the plurality of trained models stored in the storage unit based on setting information of the microscope system at the time the input image was acquired, and the microscope system , times an optical microscope system including a plurality of objective lenses with different refractive indices, and the setting information is information about the objective lenses set in the optical microscope system; twice as many Includes rate. Another aspect of the present invention provides a program that causes a computer to execute a process of selecting a trained model from among a plurality of trained models stored in a memory unit that stores a plurality of trained models that have learned image transformation to convert an input image acquired by a microscope system into an output image having higher image quality than the input image, performing the image transformation using the selected trained model, each of the plurality of trained models being a trained model that has been trained with an image having at least a different image quality range from other trained models, and selecting the trained model to be used for the image transformation from the plurality of trained models stored in the memory unit based on setting information of the microscope system at the time the input image was acquired, wherein the microscope system is an optical microscope system including a plurality of objective lenses with different numerical apertures, and the setting information includes the numerical apertures of the objective lenses set in the optical microscope system. [Effects of the Invention]

[0010] According to the above aspect, high generalization performance can be achieved in image processing for improving image quality. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a system 1. [Figure 2] FIG. 2 is a diagram illustrating the physical configuration of an image processing unit 20. [Figure 3] FIG. 2 is a diagram illustrating an example of the functional configuration of an image processing unit 20 according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of multiple trained models. [Figure 5] 10 is a flowchart showing an example of processing performed by the image processing unit 20. [Figure 6] 10 is a diagram showing an example of a screen displayed by the image processing unit 20. FIG. [Figure 7] FIG. 10 is a diagram illustrating another example of multiple trained models. [Figure 8] 10 is a diagram showing another example of a screen displayed by the image processing unit 20. FIG. [Figure 9] 10 is a diagram showing yet another example of a screen displayed by the image processing unit 20. FIG. [Figure 10] 10 is a diagram showing yet another example of a screen displayed by the image processing unit 20. FIG. [Figure 11] FIG. 10 is a diagram illustrating yet another example of multiple trained models. [Figure 12] 10 is a diagram showing yet another example of a screen displayed by the image processing unit 20. FIG. [Figure 13] FIG. 10 is a diagram illustrating an example of the functional configuration of an image processing unit 20 according to a third embodiment. [Figure 14] FIG. 10 is a diagram for explaining selection of a trained model based on similarity. [Figure 15] FIG. 10 is a diagram illustrating an example of the functional configuration of an image processing unit 20 according to a fourth embodiment. [Figure 16] 10 is a diagram showing yet another example of a screen displayed by the image processing unit 20. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] [First embodiment] Fig. 1 is a diagram illustrating an example of the configuration of the system 1. Fig. 2 is a diagram illustrating an example of the physical configuration of the image processing unit 20. The configuration of the system 1 will be described below with reference to Figs. 1 and 2.

[0013] The system 1 shown in FIG. 1 is an example of an image processing system that improves image quality, and includes an image processing unit 20 that performs image processing. The system 1 may further include an image acquisition unit 10 that acquires an image to be input to the image processing unit 20. The image processing unit 20 communicates with the image acquisition unit 10 and a terminal unit 30. The terminal unit 30 includes, for example, a notebook-type terminal device 31, a tablet-type terminal device 32, or the like. The system 1 may include the terminal unit 30.

[0014] The image acquisition unit 10 is a device or system that acquires a digital image of a sample by capturing an image of the sample. The image acquisition unit 10 includes, for example, a digital camera 11, an endoscope system 12, a microscope system 13, etc. The image acquisition unit 10 outputs the acquired image to the image processing unit 20. The image acquired by the image acquisition unit 10 may be sent directly from the image acquisition unit 10 to the image processing unit 20, or may be sent indirectly from the image acquisition unit 10 to the image processing unit 20 via another device.

[0015] The image processing unit 20 is a device or system that performs image processing using machine learning, particularly a trained model of deep learning. The image processing performed by the image processing unit 20 is image conversion that converts an input image into an output image having higher image quality than the input image, and achieves improvements in image quality such as noise reduction, resolution improvement, and aberration correction.

[0016] The image processing unit 20 may be a dedicated or general-purpose computer as long as it includes one or more electric circuits. Specifically, the image processing unit 20 includes, for example, a processor 21 and a memory 22, as shown in Fig. 2. The image processing unit 20 may further include an auxiliary storage device 23, an input device 24, an output device 25, a portable recording medium drive device 26 that drives a portable recording medium 29, a communication module 27, and a bus 28. The auxiliary storage device 23 and the portable recording medium 29 are each an example of a non-transitory computer-readable recording medium on which a program is recorded.

[0017] The processor 21 is an electric circuit including, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 21 loads a program stored in the auxiliary storage device 23 or the portable recording medium 29 into the memory 22, and then executes it to perform programmed processing such as an image processing method described below.

[0018] The memory 22 is, for example, any semiconductor memory such as RAM (Random Access Memory). The memory 22 functions as a work memory for storing programs or data stored in the auxiliary storage device 23 or the portable recording medium 29 when a program is executed. The auxiliary storage device 23 is, for example, a non-volatile memory such as a hard disk or flash memory. The auxiliary storage device 23 is mainly used to store various data and programs.

[0019] The portable recording medium drive device 26 accommodates a portable recording medium 29. The portable recording medium drive device 26 can output data stored in the memory 22 or the auxiliary storage device 23 to the portable recording medium 29, and can also read programs, data, etc. from the portable recording medium 29. The portable recording medium 29 is any recording medium that can be carried around. Examples of the portable recording medium 29 include an SD card, a USB (Universal Serial Bus) flash memory, a CD (Compact Disc), and a DVD (Digital Versatile Disc).

[0020] The input device 24 is, for example, a keyboard, a mouse, etc. The output device 25 is, for example, a display device, a printer, etc. The communication module 27 is, for example, a wired communication module that communicates with the image acquisition unit 10 and the terminal unit 30 connected via an external port. The communication module 27 may also be a wireless communication module. The bus 28 connects the processor 21, the memory 22, the auxiliary storage device 23, etc. so that they can exchange data with each other.

[0021] FIG. 3 is a diagram illustrating an example of the functional configuration of the image processing unit 20 according to this embodiment. FIG. 4 is a diagram for explaining an example of a plurality of trained models. FIG. 5 is a flowchart showing an example of processing performed by the image processing unit 20. FIG. 6 is a diagram showing an example of a screen displayed by the image processing unit 20. Hereinafter, the image processing method performed by the system 1 will be described with reference to FIGS. 3 to 6.

[0022] As shown in Figure 3, the image processing unit 20 has a configuration related to the image processing method performed by the system 1, which includes a selection unit 41 that selects a trained model from a plurality of trained models, an image conversion unit 42 that performs image conversion using the trained model selected by the selection unit 41, and a memory unit 43 that stores a plurality of trained models.

[0023] The selection unit 41, the image conversion unit 42, and the storage unit 43 are functional components realized by one or more electric circuits included in the image processing unit 20. More specifically, the selection unit 41 and the image conversion unit 42 are functional components realized by, for example, the processor 21 and the memory 22 shown in FIG. 2, and may be realized by the processor 21 executing a program stored in the memory 22. The storage unit 43 may be a functional component realized by, for example, the memory 22 and the auxiliary storage device 23 shown in FIG. 2.

[0024] The selection unit 41 selects a trained model in response to the input from among a plurality of trained models stored in the storage unit 43. Furthermore, the selection unit 41 outputs information specifying the selected trained model to the image conversion unit 42.

[0025] The image conversion unit 42 applies the trained model selected by the selection unit 41 to the input image to perform image conversion and generate an output image. The output image may be stored in, for example, the auxiliary storage device 23. The output image may also be displayed on, for example, the output device 25. Furthermore, the output image may be output to, for example, the terminal unit 30 via the communication module 27, or may be displayed on a display device of the terminal unit 30.

[0026] The multiple trained models stored in the storage unit 43 are each trained models that have learned image transformation to convert an input image into an output image having higher image quality than the input image. Hereinafter, when there is no particular distinction between the multiple trained models M1, M2, M3, ..., each or a set of them will be referred to as a trained model M.

[0027] The trained model M stored in the storage unit 43 may be generated in advance by, for example, following the procedure below. If the image quality to be improved by the trained model M is noise, first, multiple pairs of low-noise and high-noise images are prepared. The images constituting the pairs are images of the same object. Specifically, the same sample is captured at the same imaging position to obtain pairs of low-noise and high-noise images. This process is repeated for multiple imaging positions or multiple samples of the same type. Then, the trained model M is generated by training the model to perform image transformation using deep learning to convert noisy images into low-noise images using the multiple image pairs. Note that the amount of noise contained in an image can be adjusted, for example, by changing the length of exposure time or illumination intensity during imaging. Therefore, the model may be trained using image pairs in which an image captured with a relatively long exposure time is used as the low-noise image and an image captured with a relatively short exposure time is used as the noisy image.

[0028] Furthermore, if the image quality to be improved by the trained model M is resolution, first, multiple pairs of low-resolution and high-resolution images are prepared. The images constituting the pairs are images of the same object. Specifically, the same sample is imaged at the same imaging position to obtain pairs of low-resolution and high-resolution images. This process is repeated for multiple imaging positions or for multiple samples of the same type. The trained model M is then generated by training a model to perform image transformation using deep learning to convert low-resolution images into high-resolution images using the multiple image pairs. Note that image resolution can be adjusted by changing the pixel resolution or optical resolution; for example, it may be adjusted by changing the objective lens used during imaging. For this reason, the model may be trained using image pairs in which an image captured using an objective lens with a relatively low numerical aperture is used as the low-resolution image and an image captured using an objective lens with a relatively high numerical aperture is used as the high-resolution image.

[0029] Furthermore, if the image quality to be improved by the trained model M is due to aberration, first, multiple pairs of images, one with insufficiently corrected aberration and the other with sufficiently corrected aberration, are prepared. The images constituting the pairs are images of the same object. Specifically, the same sample is imaged at the same imaging position to obtain pairs of an image with insufficiently corrected aberration and an image with sufficiently corrected aberration. This process is repeated for multiple imaging positions or for multiple samples of the same type. Then, using the multiple image pairs, a model is trained to perform image transformation using deep learning to convert an image with insufficiently corrected aberration into an image with sufficiently corrected aberration, thereby generating the trained model M. Note that the degree of aberration correction can be adjusted, for example, by changing the objective lens used during imaging. Therefore, the model may be trained using image pairs, in which an image captured using an objective lens with relatively large aberration is used as the image with insufficiently corrected aberration, and an image captured using an objective lens with relatively small aberration is used as the image with sufficiently corrected aberration.

[0030] Regardless of the content of the image quality improved by the trained model M, each of the multiple trained models M is a trained model trained using images with at least a different sample type from the other trained models. In this example, as shown in FIG. 4, trained model M1 is a trained model trained using images of cell nuclei. Furthermore, trained model M2 is a trained model trained using images of actin filaments. Furthermore, trained model M3 is a trained model trained using images of mitochondria.

[0031] As shown in Fig. 5, when a user selects an image to be improved using the input device 24 from among the images acquired by the image acquisition unit 10, the image processing unit 20 configured as described above acquires the selected image as an input image 50 (step S1). Here, the image processing unit 20 acquires the image selected by the user as the input image 50, and displays a screen G1 including the input image 50 on the output device 25, as shown in Fig. 6. In this example, the input image 50 shows a cell nucleus.

[0032] Next, when the user referring to screen G1 selects the image quality improvement content to be performed on the input image 50 using the input device 24, the image processing unit 20 displays multiple trained models corresponding to the selected image quality improvement content (step S2). Figure 6 shows an example in which the user selects "noise reduction" using tab 61. Three trained models (M1 to M3) for cell nuclei, actin filaments, and mitochondria are displayed on screen G1 shown in Figure 6.

[0033] Thereafter, when the user uses the input device 24 to select one of the three trained models M according to the type of sample appearing in the input image 50, the image processing unit 20 selects the trained model selected by the user (step S3). Here, the selection unit 41 of the image processing unit 20 selects the trained model M1 for cell nuclei from the multiple trained models M in accordance with the user's selection.

[0034] Furthermore, when the user presses the execute button using the input device 24, the image processing unit 20 converts the input image 50 using the trained model selected in step S3 (step S4). Here, the image conversion unit 42 of the image processing unit 20 applies the trained model M1 selected in step S3 to the input image 50 acquired in step S1, and generates an output image in which noise is reduced compared to the input image 50.

[0035] Finally, the image processing unit 20 displays the converted image obtained in step S4 (step S5). Here, the image processing unit 20 displays the output image generated in step S4 on the output device 25. The output image may be displayed alongside the input image 50, or may be displayed in a pop-up format.

[0036] As described above, in the system 1 according to this embodiment, focusing on the fact that the image quality improvement performance of a single trained model differs depending on the type of sample appearing in the input image, a trained model M is prepared in advance for each type of sample. In other words, a trained model M optimized for each sample is prepared. The user is then prompted to select a trained model corresponding to the type of sample appearing in the input image 50 from among multiple trained models M generated for each sample type. Note that the type of sample corresponding to the trained model selected by the user does not necessarily have to match the type of sample appearing in the input image 50. The user may select a trained model that has been trained using an image of a sample having a shape similar to that of the sample appearing in the input image 50.

[0037] The main reason why the image quality improvement performance of a trained model differs depending on the sample is that the shape of each sample is different. To address this issue, system 1 can apply a trained model that has been trained using an image similar to input image 50, regardless of the sample appearing in input image 50. Therefore, even when images of various samples with different shapes are input as input image 50, stable image quality improvement performance can be achieved regardless of input image 50. Therefore, system 1 as a whole can achieve high generalization performance in image processing that improves image quality.

[0038] Furthermore, system 1 can improve image quality through image processing. When acquiring higher-quality images using the image acquisition unit 10, tradeoffs can arise in other aspects of image quality. For example, increasing the exposure time to reduce noise results in longer image acquisition times. Increasing illumination intensity increases damage to the sample. Increasing the numerical aperture to improve resolution requires the use of an immersion fluid. In particular, a high numerical aperture requires the use of oil as an immersion fluid, which increases the amount of work required by the user, such as cleaning. Increasing the magnification of the objective lens also narrows the observation range. Furthermore, using a higher-performance objective lens for aberration correction increases the cost of the device. System 1 can avoid these disadvantages by improving image quality through image processing.

[0039] However, the image quality improvement performance of a single trained model also differs depending on the image quality range of the input image 50. The main reason why the image quality improvement performance of a trained model differs depending on the image quality range is that the appearance of the image differs depending on the image quality range. As a result, for example, even if an image with a high noise level is input to a trained model that can convert an image with a medium noise level into an image with a low noise level, the noise level may not be sufficiently reduced. Furthermore, if an image with a medium noise level is input to a trained model that can convert an image with a high noise level into an image with a low noise level, effects such as a weakened signal may occur.

[0040] For this reason, although the first embodiment described above shows an example in which a trained model is prepared for each type of sample, a trained model may be prepared for each image quality range before image quality improvement. In this case, each of the multiple trained models stored in the storage unit 43 is a trained model that has been trained using images with at least an image quality range that is different from the other trained models, which is different from the first embodiment described above.

[0041] Fig. 7 is a diagram for explaining another example of a plurality of trained models. Figs. 8 to 10 are diagrams each showing another example of a screen displayed by the image processing unit 20. Below, a modified example of the first embodiment will be described with reference to Figs. 7 to 10.

[0042] In a modified example, each of the multiple trained models for noise reduction stored in the storage unit 43 is a trained model trained using an image having at least a different noise level range from the other trained models. Specifically, the exposure time of the image used to input the model among the images constituting a pair of the multiple trained models is different from that of the other trained models. In the example of FIG. 7, trained model M4 is a model using an image captured with a relatively short exposure time as the image used to input the model among the images constituting a pair. Trained model M5 is a model using an image captured with a medium exposure time as the image used to input the model among the images constituting a pair. Trained model M6 is a model using an image captured with a relatively long exposure time as the image used to input the model among the images constituting a pair.

[0043] Furthermore, each of the multiple trained models for improving resolution stored in the storage unit 43 is a trained model trained using images with at least a different resolution range from the other trained models. Specifically, for example, each of the multiple trained models has a different numerical aperture of the objective lens used when capturing an image used to input the model among images constituting a pair from those of the other trained models. In other words, each of the multiple trained models is a trained model trained using an image acquired using an objective lens with a different numerical aperture from those of the other trained models.

[0044] Furthermore, each of the multiple trained models for improving aberration levels stored in the storage unit 43 is a trained model trained using images with at least a different range of aberration levels from the other trained models. Specifically, for example, in each of the multiple trained models, the aberration performance of the objective lens used when capturing an image used to input the model among images constituting a pair is different from that of the other trained models. In other words, each of the multiple trained models is a trained model trained using an image acquired using an objective lens with aberration performance different from that of the other trained models.

[0045] In this modification, when tab 61 is selected, image processing unit 20 displays screen G2 shown in FIG. 8 on output device 25. By referring to screen G2, the user can understand the noise level that each trained model can handle. For example, the user can understand that trained model M4 can handle images with short exposure times, i.e., images with high noise. Therefore, the user can select a trained model from trained models M4 to M6 depending on the noise level of input image 50. Therefore, system 1 can exhibit stable noise reduction performance regardless of the noise level of input image 50.

[0046] Furthermore, when tab 62 is selected, image processing unit 20 displays screen G3 shown in FIG. 9 on output device 25. By referring to screen G3, the user can understand the resolution supported by each trained model. For example, the user can understand that trained model M7 can convert an image with an NA of approximately 0.8 into an image with an NA of approximately 1.1. Therefore, the user can select a trained model from trained model M7 to trained model M9 depending on the resolution of input image 50, thereby achieving stable resolution improvement performance regardless of the resolution of input image 50.

[0047] Furthermore, when tab 63 is selected, the image processing unit 20 displays screen G4 shown in FIG. 10 on the output device 25. By referring to screen G4, the user can understand the aberration level that each trained model can handle. For example, the user can understand that trained model M10 can further improve the aberration correction state at the achromatic level. Therefore, the user can select a trained model from trained model M10 to trained model M12 depending on the aberration level of the input image 50. Therefore, the system 1 can exhibit stable aberration correction performance regardless of the aberration level of the input image 50.

[0048] As described above, even with this modification, similar to the first embodiment, the system 1 can achieve high generalization performance in image processing that improves image quality throughout the system 1. Furthermore, because image quality improvement is achieved through image processing, it is possible to avoid disadvantages that arise from acquiring images of higher image quality with the image acquisition unit 10.

[0049] While the above describes an example in which exposure time is used to classify the noise level of an image, noise level classification may also use, for example, illumination intensity, or a combination of exposure time and illumination intensity. Furthermore, noise levels and noise output may vary depending on the type of image acquisition device. Specifically, for example, images acquired with a laser scanning microscope and images acquired with a wide-field microscope equipped with a digital camera have different noise levels and noise output. Therefore, noise level classification may also use the type of image acquisition device. A trained model trained on images acquired with a laser scanning microscope and a trained model trained on images acquired with a wide-field microscope may be prepared, and the trained model may be selected based on the image acquisition device that acquired the input image. Furthermore, while the above describes an example in which the numerical aperture of the objective lens is used to classify image resolution, other indicators that affect optical resolution, such as the confocal pinhole diameter, may also be used to classify resolution when images are acquired with a laser scanning microscope. In addition to indicators that affect optical resolution, indicators that affect pixel resolution, such as the magnification of the objective lens, digital zoom, and scan size, may also be used. These indicators may also be used in combination. That is, each of the multiple trained models may be a trained model trained using images acquired with a laser scanning microscope having a confocal pinhole diameter different from that of the other trained models, or may be a trained model trained using images acquired with a pixel resolution different from that of the other trained models.

[0050] In the above, an example has been shown in which the image quality range of the images included in the learning data is classified using the settings of the image acquisition unit 10 at the time of image capture (for example, the device itself, such as the objective lens to be used, and settings for the device, such as exposure time), but the image quality range may also be classified based on factors other than the settings of the image acquisition unit 10. For example, the image may be classified subjectively by a person who observes the image.

[0051] [Second embodiment] Fig. 11 is a diagram for explaining yet another example of a plurality of trained models. Fig. 12 is a diagram showing yet another example of a screen displayed by the image processing unit 20. Hereinafter, the second embodiment will be described with reference to Figs. 11 and 12.

[0052] In this embodiment, each of the multiple trained models for noise reduction stored in the storage unit 43 is a trained model trained using images with at least a different combination of noise level range and sample type from the other trained models. That is, as shown in FIG. 11 , a trained model is prepared for each combination of noise level range and sample type. Specifically, each of the multiple trained models has a different combination of exposure time and sample type for the image used as input to the model among the images constituting a pair. In the example of FIG. 11 , trained models M41 to M43 are models using images captured with a relatively short exposure time for the image used as input to the model, and are models using images of different samples. Trained models M51 to M53 are models using images captured with a medium exposure time for the image used as input to the model among the images constituting a pair, and are models using images of different samples. Trained models M61 to M63 are models using images captured with a relatively long exposure time for the image used as input to the model among the images constituting a pair.

[0053] Furthermore, each of the multiple trained models for improving resolution stored in the storage unit 43 is a trained model trained using images that differ from the other trained models in at least the range of resolution and the combination of sample types. Furthermore, each of the multiple trained models for improving aberration levels stored in the storage unit 43 is a trained model trained using images that differ from the other trained models in at least the range of aberration levels and the combination of sample types.

[0054] In this embodiment, when tab 61 is selected, the image processing unit 20 displays screen G5 shown in FIG. 12 on the output device 25. By referring to screen G5, the user can understand the combination of noise level and sample that each trained model can handle. For example, the user can understand that trained model M41 can handle images in which the sample is a cell nucleus and the exposure time is short, i.e., images with high noise levels. Therefore, the user can select a trained model from trained models M41 to M63 depending on the combination of sample and noise level of the input image 50. Therefore, the system 1 can exhibit stable noise reduction performance regardless of the combination of sample and noise level of the input image 50.

[0055] As described above, according to this embodiment, the system 1 can achieve even higher generalization performance than the first embodiment in image processing that improves the image quality of the entire system 1. Furthermore, since image quality improvement is achieved by image processing, it is possible to avoid disadvantages that arise when the image acquisition unit 10 acquires images of higher image quality.

[0056] [Third embodiment] Fig. 13 is a diagram illustrating the functional configuration of the image processing unit 20 according to this embodiment. Fig. 14 is a diagram for explaining the selection of a trained model based on similarity. Hereinafter, the third embodiment will be described with reference to Figs. 13 and 14.

[0057] 13, the image processing unit 20 according to this embodiment includes a selection unit 71 that selects a trained model from among a plurality of trained models M11 to M13, an image conversion unit 72 that performs image conversion using the trained model selected by the selection unit 71, and a storage unit 73 that stores the plurality of trained models M11 to M13. The image processing unit 20 according to this embodiment differs from the image processing unit 20 according to the first embodiment in the following two points.

[0058] First, the storage unit 73 stores multiple pieces of model supplemental information MS11 to MS13 that indicate the characteristics of the multiple trained models in association with the multiple trained models M11 to M13. When the multiple trained models M11 to M13 are trained models trained using images with at least a different sample type from the other trained models, the multiple pieces of model supplemental information MS11 to MS13 are, for example, multiple representative images corresponding to the sample types of the multiple trained models M11 to M13. It is desirable that the representative images be images that accurately capture the shape characteristics of the samples.

[0059] The second point is that the selection unit 71 selects a trained model to be used for image conversion from among multiple trained models M11 to M13 based on the input image 50 and multiple pieces of model supplement information MS11 to MS13. The selection unit 71 may, for example, compare the input image 50 with each of multiple representative images (model supplement information MS11 to MS13) and select a trained model based on the comparison results. More specifically, as shown in FIG. 14, the selection unit 71 may compare the input image 50 with each of the multiple representative images (model supplement information MS11 to MS13) to calculate similarity and select a trained model corresponding to the representative image with the highest similarity. Note that the similarity may be calculated using any known algorithm, such as an algorithm for calculating local features. Alternatively, the similarity may be calculated using a trained model that has previously trained similar images. In the example shown in FIG. 14, the trained model M11 corresponding to the representative image, which is the model supplement information MS11, is selected as the trained model to be used for image conversion.

[0060] In this embodiment, the image processing unit 20 automatically selects a trained model according to the input image 50. Therefore, the user can obtain an image with improved image quality simply by selecting an image whose image quality is to be improved. Furthermore, because the image processing unit 20 appropriately selects a trained model, the system 1 as a whole can achieve high generalization performance in image processing that improves image quality.

[0061] Although the example in which the plurality of pieces of model supplemental information MS11 to MS13 are images has been shown, the model supplemental information is not limited to images. If the plurality of trained models M11 to M13 are trained models trained with images having at least a different image quality range from the other trained models, the plurality of pieces of model supplemental information MS11 to MS13 may be a plurality of pieces of model image quality information corresponding to the image quality range of the images used to train the plurality of trained models M11 to M13. More specifically, the model image quality information may be, for example, a quantified noise level.

[0062] When the model image quality information is a quantified noise level, the selection unit 71 calculates the noise level of the input image 50 from the input image 50 as the input image quality information. The noise level of the input image 50 is not particularly limited, but may be calculated, for example, as the variance of brightness in the background part of the image. This is because the larger the brightness variance, the greater the noise can be evaluated to be. Furthermore, the selection unit 71 selects a trained model to be used for image conversion from the multiple trained models M11 to M13 based on the results of comparing the input image quality information with the multiple model image quality information. More specifically, the selection unit 71 may select, for example, a trained model corresponding to a noise level closest to the noise level of the input image 50 as the trained model to be used for image conversion. In this case, the user can obtain an image with improved image quality simply by selecting the image to be improved.

[0063] [Fourth embodiment] 15 is a diagram illustrating the functional configuration of the image processing unit 20 according to this embodiment. The fourth embodiment will be described below with reference to FIG.

[0064] 15, the image processing unit 20 according to this embodiment includes a selection unit 81 that selects a trained model from among a plurality of trained models M21 to M23, an image conversion unit 82 that performs image conversion using the trained model selected by the selection unit 81, and a storage unit 83 that stores the plurality of trained models M21 to M23. The image processing unit 20 according to this embodiment differs from the image processing unit 20 according to the first embodiment in the following two points.

[0065] First, the storage unit 83 stores a plurality of pieces of model supplemental information MS21 to MS23 that indicate the features of the plurality of trained models in association with the plurality of trained models M21 to M23. Note that the plurality of trained models M21 to M23 stored in the storage unit 83 are trained models that have been trained using images that are at least in a different image quality range from the other trained models.

[0066] The second point is that the selection unit 81 acquires setting information of the image acquisition unit 10 and selects a trained model to be used for image conversion from the trained models M21 to M23 based on the acquired setting information and the plurality of pieces of model supplement information MS21 to MS23. Specifically, the selection unit 81 acquires, for example, information on the resolution of the microscope system 13 from the microscope system 13 that acquired the input image 50. If the microscope system 13 is a laser scanning microscope, the selection unit 81 acquires, for example, the numerical aperture of the objective lens, the magnification of the objective lens, the scan size of the galvanometer scanner (e.g., 512 × 512, 1024 × 1024), the pinhole diameter, etc. Furthermore, the selection unit 81 calculates the resolution of the input image 50 from this information and compares it with the resolution specified by the plurality of pieces of model supplement information MS21 to MS23 read from the storage unit 83. As a result of the comparison, the selection unit 81 selects, for example, model supplementary information indicating the resolution closest to the resolution of the input image 50, and selects the trained model corresponding to that model supplementary information as the trained model to be used for image conversion.

[0067] In this embodiment, similar to the third embodiment, by acquiring setting information from the image acquisition unit 10, the image processing unit 20 automatically selects a trained model according to the input image 50. Therefore, a user can obtain an image with improved image quality simply by selecting an image whose image quality is to be improved. Also, similar to the third embodiment, the image processing unit 20 selects an appropriate trained model, thereby enabling the system 1 as a whole to achieve high generalization performance in image processing for improving image quality.

[0068] The above-described embodiments are specific examples for facilitating understanding of the invention, and the present invention is not limited to these embodiments. Parts of the above-described embodiments may be applied to other embodiments. The image processing system, image processing method, and program may be modified and changed in various ways without departing from the scope of the claims.

[0069] In the above-described embodiment, an example was shown in which one trained model suitable for an input image was identified. However, the image processing unit 20 may store multiple trained models suitable for the input image, each with a different degree of image quality improvement. In this case, as shown in FIG. 16, a control (in this example, a radio button) for specifying the degree of image quality improvement may be provided on screen G6, and one trained model may be identified based on the degree of image quality improvement desired by the user. This allows the user to select the balance between image quality and side effects associated with image processing. [Explanation of symbols]

[0070] 1 System 10 Image acquisition unit 20 Image processing section 21 processors 22 Memory 41, 71, 81 Selection section 42, 72, 82 Image conversion section 43, 73, 83 Storage section 50 input images M, M1~M6, M11~M13, M21~M23, M41~M43, M51~M53, M61~M63 trained models Additional information for MS11-MS13 and MS21-MS23 models

Claims

1. a microscope system for acquiring an input image to be input to an image conversion unit; a storage unit that stores a plurality of trained models that have learned image transformation to convert the input image acquired by the microscope system into an output image having higher image quality than the input image; a selection unit that selects a trained model from the plurality of trained models stored in the storage unit; and The image conversion unit performs the image conversion using the trained model selected by the selection unit, Each of the plurality of trained models is a trained model trained with an image having at least an image quality range different from that of the other trained models, the selection unit selects a trained model to be used for the image conversion from the plurality of trained models stored in the storage unit based on setting information of the microscope system at the time of acquiring the input image; and the microscope system is an optical microscope system including a plurality of objective lenses with different magnifications; The setting information includes the magnification of an objective lens set in the optical microscope system. An image processing system comprising:

2. A microscope system for acquiring an input image to be input to an image conversion unit; a storage unit that stores a plurality of trained models that have learned image transformation to convert the input image acquired by the microscope system into an output image having higher image quality than the input image; a selection unit that selects a trained model from the plurality of trained models stored in the storage unit; and The image conversion unit performs the image conversion using the trained model selected by the selection unit, Each of the plurality of trained models is a trained model trained with an image having at least an image quality range different from that of the other trained models, the selection unit selects a trained model to be used for the image conversion from the plurality of trained models stored in the storage unit based on setting information of the microscope system at the time of acquiring the input image; and the microscope system is an optical microscope system including a plurality of objective lenses with different numerical apertures; The setting information includes the numerical aperture of an objective lens set in the optical microscope system. An image processing system comprising:

3. 3. The image processing system according to claim 1, the output image has a lower noise level than the input image; The image quality range includes a noise level range. An image processing system comprising:

4. 4. The image processing system according to claim 3, Each of the plurality of trained models is a trained model trained with images acquired by an image acquisition device of a type different from the other trained models. An image processing system comprising:

5. 5. The image processing system according to claim 4, The different types of image acquisition devices are a laser scanning microscope and a wide-field microscope with a digital camera. An image processing system comprising:

6. 3. The image processing system according to claim 1, the output image has a higher resolution than the input image; The image quality range includes a range of resolutions. An image processing system comprising:

7. 3. The image processing system according to claim 2, the output image has a higher resolution than the input image; the image quality range includes a range of resolutions, Each of the plurality of trained models is a trained model trained with an image acquired using an objective lens with a different numerical aperture from the other trained models. An image processing system comprising:

8. 7. The image processing system according to claim 6, Each of the plurality of trained models is a trained model trained with an image acquired at a pixel resolution different from that of the other trained models. An image processing system comprising:

9. 7. The image processing system according to claim 6, Each of the plurality of trained models is a trained model trained using an image acquired by a laser scanning microscope having a different confocal pinhole diameter. An image processing system comprising:

10. 3. The image processing system according to claim 1, the output image has a lower aberration level than the input image; The image quality range includes a range of aberration levels. An image processing system comprising:

11. 11. The image processing system according to claim 10, Each of the plurality of trained models is a trained model trained with an image acquired using an objective lens having aberration performance different from that of the other trained models. An image processing system comprising:

12. 3. The image processing system according to claim 1, the microscope system is a laser scanning microscope; The setting information includes at least one of the scan size and the pinhole diameter of the galvano scanner. An image processing system comprising:

13. Selecting a trained model from among a plurality of trained models stored in a storage unit that stores a plurality of trained models that have learned image transformation for converting an input image acquired by a microscope system into an output image having higher image quality than the input image; Performing the image transformation using the selected trained model; Each of the plurality of trained models is a trained model trained with an image having at least an image quality range different from that of the other trained models, In the selection of the trained model, a trained model to be used for the image conversion is selected from the plurality of trained models stored in the storage unit based on setting information of the microscope system at the time of acquiring the input image; the microscope system is an optical microscope system including a plurality of objective lenses with different magnifications; The setting information includes the magnification of an objective lens set in the optical microscope system. An image processing method comprising:

14. Selecting a trained model from among a plurality of trained models stored in a memory unit that stores a plurality of trained models that have learned image transformation to convert an input image acquired by a microscope system into an output image having higher image quality than the input image, Performing the image transformation using the selected trained model; Each of the plurality of trained models is a trained model trained with an image having at least an image quality range different from that of the other trained models, In the selection of the trained model, a trained model to be used for the image conversion is selected from the plurality of trained models stored in the storage unit based on setting information of the microscope system at the time of acquiring the input image; the microscope system is an optical microscope system including a plurality of objective lenses with different numerical apertures; The setting information includes the numerical aperture of an objective lens set in the optical microscope system. An image processing method comprising:

15. On the computer, Selecting a trained model from among a plurality of trained models stored in a storage unit that stores a plurality of trained models that have learned image transformation for converting an input image acquired by a microscope system into an output image having higher image quality than the input image; Performing the image transformation using the selected trained model; Each of the plurality of trained models is a trained model trained with an image having at least an image quality range different from that of the other trained models, In the selection of the trained model, a trained model to be used for the image conversion is selected from the plurality of trained models stored in the storage unit based on setting information of the microscope system at the time of acquiring the input image. Execute the process, the microscope system is an optical microscope system including a plurality of objective lenses with different magnifications; The setting information includes the magnification of an objective lens set in the optical microscope system. A program characterized by:

16. A computer comprising: Selecting a trained model from among a plurality of trained models stored in a storage unit that stores a plurality of trained models that have learned image transformation for converting an input image acquired by a microscope system into an output image having higher image quality than the input image; Performing the image transformation using the selected trained model; Each of the plurality of trained models is a trained model trained with an image having at least an image quality range different from that of the other trained models, In the selection of the trained model, a trained model to be used for the image conversion is selected from the plurality of trained models stored in the storage unit based on setting information of the microscope system at the time of acquiring the input image. Execute the process, the microscope system is an optical microscope system including a plurality of objective lenses with different numerical apertures; The setting information includes the numerical aperture of an objective lens set in the optical microscope system. A program characterized by:

Images