Optical image processing method, machine learning method, trained model, machine learning preprocessing method, optical image processing module, optical image processing program, and optical image processing system

Abstract

Provided are an optical image processing method, a machine learning method, a trained model, a machine learning pre-processing method, an optical image processing module, an optical image processing program, and an optical image processing system, which make it possible to effectively remove noise in an optical image. This optical image processing module 3 comprises: an image acquisition unit 32 that acquires an optical image in which light L from a target object F is captured; a selection unit 33 that uses condition information including optical detector information indicating the type of an optical detector 21 to select a trained model 36 from among a plurality of trained models 36 respectively constructed in advance by machine learning; and a processing unit 34 that inputs the optical image to the selected trained model 36 and executes image processing to remove noise from the optical image.

Description

[Technical Field]

[0001] The present disclosure relates to an optical image processing method, a machine learning method, a trained model, a machine learning preprocessing method, an optical image processing module, an optical image processing program, and an optical image processing system. [Background technology]

[0002] Conventionally, a technique is known in which light from a sample to be imaged is captured, image data is obtained based on the image capture results, and noise is removed from the image data to output the noise-removed image data (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-21314 Summary of the Invention [Problem to be solved by the invention]

[0004] In the noise removal process described above, a method of removing noise from an optical image obtained by capturing light from an object such as a sample using a trained model based on machine learning may be used. However, in such a case, the relationship between brightness and noise in the image is likely to vary depending on the imaging conditions of the object, such as the type of photodetector used to image the object, and noise tends to be difficult to remove effectively.

[0005] The present disclosure has been made in consideration of such problems, and aims to provide an optical image processing method, a machine learning method, a trained model, a machine learning preprocessing method, an optical image processing module, an optical image processing program, and an optical image processing system that can effectively remove noise in optical images. [Means for solving the problem]

[0006] An optical image processing method according to one aspect of the embodiment includes a first image acquisition step of acquiring an optical image in which light from an object is captured; a selection step of selecting a trained model from a plurality of trained models each constructed in advance by machine learning using imaging information regarding the image of the object; and a processing step of inputting the optical image into the selected trained model and performing image processing to remove noise from the optical image.

[0007] Alternatively, an optical image processing module according to another aspect of the embodiment includes a first image acquisition unit that acquires an optical image in which light from an object is captured, a selection unit that uses imaging information regarding the image of the object to select a trained model from among a plurality of trained models that have each been constructed in advance by machine learning, and a processing unit that inputs the optical image into the selected trained model and performs image processing to remove noise from the optical image.

[0008] Alternatively, an optical image processing program relating to another aspect of the embodiment causes a processor to function as an image acquisition unit that acquires an optical image in which light from an object is captured, a selection unit that uses imaging information regarding the image of the object to select a trained model from among a plurality of trained models that have each been constructed in advance by machine learning, and a processing unit that inputs the optical image into the selected trained model and performs image processing to remove noise from the optical image.

[0009] Alternatively, an optical image processing system according to another aspect of the embodiment includes the optical image processing module and an imaging device that captures an optical image by capturing light from an object.

[0010] According to one or more of the above aspects, imaging information relating to imaging of an object is used to select a trained model to be used for noise removal from among trained models constructed in advance. This allows the trained model selected according to the imaging environment, which changes depending on the imaging conditions of the object, the optical image of the object, etc., to be used for noise removal, thereby realizing noise removal that responds to changes in the imaging conditions of the object. As a result, noise in the optical image can be effectively removed.

[0011] A machine learning method according to another aspect of the embodiment includes a construction step of constructing, by machine learning, a trained model using training images, structure images to which noise based on a predetermined noise distribution model has been added, and the training images and noise-removed image data, which is data obtained by removing noise from the training images, as training data. The optical image processing module may further include a construction unit that constructs, by machine learning, a trained model using training images, structure images to which noise based on a predetermined noise distribution model has been added, and the training images and noise-removed image data, which is data obtained by removing noise from the training images, as training data, and outputs noise-removed image data based on the training images. With the above configuration, by inputting optical images to the trained model, noise removal processing that can effectively remove noise from optical images of an object can be realized.

[0012] According to another aspect of the embodiment, a trained model is constructed by the machine learning method, and causes a processor to perform image processing to remove noise from an optical image of an object. This enables a noise removal process that can effectively remove noise from the optical image of the object using the trained model.

[0013] Furthermore, the machine learning preprocessing method according to the above-described another aspect includes a training image generation step of generating, as training images, structure images to which noise based on a noise distribution model has been added. As a result, when optical images are input to a trained model constructed using the training images generated by the above-described preprocessing method, a noise removal process that can effectively remove noise from the optical images of the target object can be realized. [Effects of the Invention]

[0014] According to one aspect and another aspect of this embodiment, it is possible to provide an optical image processing method, a machine learning method, a trained model, a machine learning preprocessing method, an optical image processing module, an optical image processing program, and an optical image processing system that can effectively remove noise in optical images. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram showing the functional configuration of an optical image processing system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of the optical image processing module of FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of input and output data of the trained model of FIG. 1. [Figure 4] FIG. 4 is a diagram illustrating an example of an optical image acquired by an image acquisition unit. [Figure 5] 10 is a flowchart showing the procedure for creating training images included in teacher data used by the construction unit to construct a trained model. [Figure 6] 10 is a flowchart showing the procedure of observation processing by an optical image processing system including an optical image processing module. [Figure 7] FIG. 10 is a block diagram showing the functional configuration of an optical image processing system according to a second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of input and output data of the trained model of FIG. [Figure 9] 10 is a flowchart showing the procedure of observation processing by an optical image processing system including an optical image processing module. [Figure 10] FIG. 10 is a diagram showing an example of a jig image used for evaluating the brightness-to-noise ratio. [Figure 11] 8 is a diagram showing an example of a luminance-SNR characteristic graph acquired by the determination unit in FIG. 7. FIG. [Figure 12] FIG. 10 is a diagram showing an example of a jig image used for evaluating resolution. [Figure 13] FIG. 10 is a block diagram showing a functional configuration of an optical image processing system according to a modified example. [Figure 14] 10A and 10B are diagrams showing examples of optical images before and after noise removal processing according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and duplicated explanations will be omitted. [First embodiment]

[0017] FIG. 1 is a block diagram showing the functional configuration of an optical image processing system 1 according to a first embodiment. As shown in FIG. 1, the optical image processing system 1 is a system that acquires an optical image of an object F based on light L from the object F. The light L is, for example, light emitted from the object F, light transmitted through the object F, light reflected from the object F, or light scattered by the object F. Examples of the light L include ultraviolet light, visible light, and infrared light. Hereinafter, the light may be referred to as observation light. The optical image processing system 1 includes a camera (imaging device) 2, an optical image processing module 3, a display device 4, and an input device 5.

[0018] The camera 2 captures an optical image by capturing light L from an object F. The camera 2 has a photodetector 21 and an image control unit 22. The photodetector 21 is an imaging element having a plurality of pixels. Examples of the photodetector 21 include a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, a photodiode, an InGaAs sensor, a time delay integration (TDI)-CCD image sensor, a TDI-CMOS image sensor, a pickup tube, an electron multiplying (EM)-CCD image sensor, an electron bombarded (EB)-CMOS image sensor, a single photon avalanche diode (SPAD, single-pixel photon counter (SPPC)), a multi-pixel photon counter (MPPC, silicon photomultiplier (SiPM)), a hybrid photodetector (HPD), an avalanche photodiode (APD), and a photomultiplier tube (PMT). The photodetector 21 may also be a combination of a CCD image sensor, a CMOS image sensor, or the like with an image intensifier (II) or a micro-channel plate (MCP). Examples of the shape of the photodetector 21 include an area sensor, a line sensor that acquires an image by line scanning, a TDI sensor, and a point sensor that acquires an image by two-dimensional scanning. The camera 2 captures an image of light L from an object F that is imaged by an imaging optical system 24 through an objective lens 23, and outputs a digital signal based on the image capture result to the image control unit 22.

[0019] The image control unit 22 executes image processing based on the digital signal from the photodetector 21. The image control unit 22 is configured with, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The image control unit 22 generates image data based on the digital signal received from the photodetector 21, performs predetermined image processing on the generated image data, and outputs the image data to the optical image processing module 3.

[0020] The optical image processing module 3 is a computer such as a PC (Personal Computer). The optical image processing module 3 performs image processing on image data output from the camera 2 to generate an optical image from which noise has been removed. The optical image processing module 3 is connected to the camera 2, the display device 4, and the input device 5 via wired or wireless communication so as to be able to communicate with each other. The generated optical image is subjected to a noise removal process (described later) and then output to the display device 4, where it is displayed. Various input information, such as imaging conditions for the object F, is input to the optical image processing module 3 from the input device 5 based on a user's operation. The optical image processing module 3 also controls the camera 2. Note that the optical image processing module 3 in the first embodiment is an independent device provided outside the camera 2, but may be integrated inside the camera 2. For example, the optical image processing module 3 may be a module equivalent to processing circuits implemented in the camera, such as a CPU and a GPU.

[0021] 2 shows the hardware configuration of the optical image processing module 3. As shown in FIG. 2, the optical image processing module 3 is physically a computer or the like including processors such as a CPU (Central Processing Unit) 101 and a GPU (Graphic Processing Unit) 105, storage media such as a RAM (Random Access Memory) 102 and a ROM (Read Only Memory) 103, a communication module 104, and an input / output module 106, all of which are electrically connected to one another. The optical image processing module 3 may include a display, keyboard, mouse, touch panel display, etc. as the display device 4 and input device 5, or may include a data storage device such as a hard disk drive or semiconductor memory. The optical image processing module 3 may also be composed of multiple computers.

[0022] As shown in FIG. 1, the optical image processing module 3 includes an input unit 31, an image acquisition unit (first image acquisition unit) 32, a selection unit 33, a processing unit 34, and a construction unit 35. The functional units of the optical image processing module 3 shown in FIG. 1 are realized by loading a program (the optical image processing program of the first embodiment) onto hardware such as a CPU 101, a GPU 105, and a RAM 102, which operates a communication module 104, an input / output module 106, and the like under the control of the CPU 101 and the GPU 105, and reads and writes data from and to the RAM 102. The CPU 101 and the GPU 105 of the optical image processing module 3 execute the computer program to cause the optical image processing module 3 to function as the functional units shown in FIG. 1 and sequentially execute processes corresponding to an optical image processing method described below. The CPU 101 and the GPU 105 may be standalone hardware, or either one of them may be used alone. The CPU 101 and GPU 105 may be implemented in programmable logic such as an FPGA, like a software processor. The RAM and ROM may be standalone hardware or may be embedded in programmable logic such as an FPGA. All of the various data required to execute this computer program and the various data generated by the execution of this computer program are stored in an internal memory such as the ROM 103 or RAM 102, or in a storage medium such as a hard disk drive. The internal memory or storage medium of the optical image processing module 3 pre-stores multiple trained models 36 that are read by the CPU 101 and GPU 105 to cause the CPU 101 and GPU 105 to perform noise reduction processing on optical images. Details of the trained models 36 will be described later. Hereinafter, one trained model 36 may be described, but in such cases, the same applies to the other trained models 36.

[0023] Here, an overview of the optical image processing method of the optical image processing module 3 will be described using FIG. 3. FIG. 3 is a diagram showing an example of input / output data of the trained model 36 of FIG. 1. In the optical image processing module 3, in the learning phase by machine learning, multiple trained models 36 are constructed, and in the noise removal phase, the trained models 36 are used to generate an optical image G6 in which noise has been removed from the optical image G1 of the object F. First, in the learning phase, the optical image processing module 3 creates a structure image Gc, which is an image of a structure having a predetermined structure, and generates training images (structure images) Gt, which serve as teacher data, based on the structure image Gc and a noise distribution model (details will be described later). Then, the optical image processing module 3 constructs multiple trained models 36 by machine learning using training data including the training images Gt. In the noise removal phase, the optical image processing module 3 first acquires condition information. The condition information indicates imaging conditions, including the type of photodetector 21, when imaging the object F. The optical image processing module 3 selects a trained model 36 to be used for noise removal processing of the optical image G1 from among a plurality of trained models 36 according to the type of the photodetector 21. Then, the optical image processing module 3 inputs the optical image G1 to the selected trained model 36 and performs image processing to remove noise from the optical image, thereby generating and outputting an optical image G6 from which noise has been removed.

[0024] The function of each functional unit of the optical image processing module 3 will be described in detail below.

[0025] The input unit 31 accepts input of condition information (imaging information). The condition information is information related to imaging of the object F and indicates imaging conditions, etc., when imaging the object F. Specifically, the input unit 31 accepts input of the condition information from a user of the optical image processing system 1. The condition information includes at least one of photodetector information, a gain setting value, a shading correction coefficient, an offset, a noise factor, information indicating dark current noise generated by thermal noise in the photodetector 21, and information indicating a readout noise value in the photodetector 21. The photodetector information is information indicating the type of photodetector 21 used to image the object F. Examples of the photodetector information include information indicating any of a CCD image sensor, a CMOS image sensor, a photodiode, an InGaAs sensor, a TDI-CCD image sensor, a TDI-CMOS image sensor, an image pickup tube, an EM-CCD image sensor, an EB-CMOS image sensor, a SPAD, an MPPC, a HPD, an APD, and a photomultiplier tube. The input unit 31 may accept the input of the condition information as a direct input of information such as numerical values, or as a selective input of information such as numerical values ​​previously set in an internal memory. The input unit 31 accepts the input of the above condition information from the user, but may also acquire some of the condition information (such as the type of photodetector 21) according to the detection result of the control state by the optical image processing module 3.

[0026] The image acquisition unit 32 acquires an optical image obtained by capturing light from the object F. Specifically, the image acquisition unit 32 acquires an optical image output from the camera 2. FIG. 4 is a diagram showing an example of an optical image G1 acquired by the image acquisition unit 32.

[0027] The selection unit 33 uses the condition information (imaging information) to select a trained model 36 to be used for noise removal processing of an optical image from among a plurality of trained models 36 each constructed in advance by machine learning. Specifically, the selection unit 33 refers to the photodetector information included in the condition information and selects the trained model 36 according to the type of photodetector 21 indicated by the photodetector information (condition information). In other words, the selection unit 33 selects the trained model 36 that is most suitable for the photodetector 21 used to image the object F. Details of the selection process of the trained model 36 according to the type of photodetector 21 will be described later.

[0028] The processing unit 34 inputs the optical image into the selected trained model 36 and performs image processing to remove noise from the optical image. That is, as shown in FIG. 3, the processing unit 34 acquires the trained model 36 selected by the selection unit 33 from the built-in memory or storage medium in the optical image processing module 3. Then, the processing unit 34 inputs the optical image G1 acquired by the image acquisition unit 32 into the trained model 36. As a result, the processing unit 34 uses the trained model 36 to perform image processing to remove noise from the optical image G1, thereby generating an optical image G6 from which noise has been removed. The processing unit 34 then outputs the generated optical image G6 to the display device 4 or the like.

[0029] The construction unit 35 uses structural images to which noise based on a predetermined noise distribution model has been added as training images, and uses the training images and noise-removed image data, which is data obtained by removing noise from the training images, as training data to construct a trained model 36 that outputs noise-removed image data based on the training images through machine learning. In this embodiment, the construction unit 35 constructs multiple trained models 36 using a predetermined noise distribution model corresponding to the type of photodetector 21 that can be input as condition information. The type of photodetector 21 that can be input as condition information is the type indicated by the photodetector information that can be input to the input unit 31 (e.g., CCD image sensor, CMOS image sensor, etc.). The construction unit 35 then stores the multiple constructed trained models 36 in an internal memory or a storage medium within the optical image processing module 3. Machine learning includes supervised learning, unsupervised learning, and reinforcement learning, and these learning methods include deep learning and neural network learning. In the first embodiment, a two-dimensional convolutional neural network described in the paper "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising" by Kai Zhang et al. is adopted as an example of a deep learning algorithm. Note that, in addition to being constructed by the construction unit 35, each trained model 36 may be generated by an external computer or the like and downloaded to the optical image processing module 3. Note that the optical images used in machine learning include optical images captured of known structures or images reproduced from such optical images. The training images may be images actually generated for multiple types of known structures, or may be images generated by simulation calculations.

[0030] As a preprocessing step for machine learning, the construction unit 35 generates, as training images, structure images to which noise based on a noise distribution model has been added. The structure images are images obtained by capturing light from a structure having a predetermined structure.

[0031] Specifically, when constructing each trained model 36, the construction unit 35 acquires condition information including photodetector information at the time of simulation calculation from the input unit 31. Then, the construction unit 35 generates a structure image. Then, the construction unit 35 adds noise to the structure image based on a noise distribution model selected based on the photodetector information. In other words, the machine learning preprocessing method includes an input step of accepting input of condition information including photodetector information indicating the type of photodetector 21 used to image the object F, and a training image generation step of generating, as training images, structure images to which noise based on the noise distribution model has been added. Then, in the training image generation step, the noise distribution model to be used is determined from the photodetector information.

[0032] The construction unit 35 constructs each trained model 36 through machine learning using training data prepared for each trained model 36. Specifically, first, the construction unit 35 acquires in advance noise-removed image data in which noise has been removed from training images. The construction unit 35 sets images before noise is added in the process of generating the training images as noise-removed image data. The construction unit 35 performs training through machine learning to construct trained models 36 that output noise-removed image data based on the training images.

[0033] FIG. 5 is a flowchart showing the procedure for creating training images included in teacher data (training data) used by the construction unit 35 to construct the trained model 36.

[0034] A training image (also referred to as a teacher image), which is teacher data, is created in the following procedure. First, the construction unit 35 generates a structure image (step S101). The construction unit 35 may create the structure image by, for example, simulation calculation. Next, for one pixel selected from multiple pixels constituting the structure image, a sigma value, which is the standard deviation of pixel values, is calculated (step S102). The sigma value calculated in step S102 indicates the magnitude of noise. The construction unit 35 selects an appropriate relational expression from the following equations (1), (2), and (3), which will be described later, based on the photodetector information. Then, using the selected relational expression, the construction unit 35 substitutes the pixel value of the pixel of the structure image for the variable Signal to calculate the variable Noise of the pixel, and obtains the calculated variable Noise of the pixel as the magnitude of noise (sigma value) (details will be described later).

[0035] Here, a method for calculating the noise magnitude will be described. The construction unit 35 selects one relational equation from among a plurality of noise relational equations based on the photodetector information included in the condition information. That is, the construction unit 35 selects the relational equation that is most suitable for the photodetector 21 depending on the type of the photodetector 21. Then, the construction unit 35 obtains the noise magnitude of the pixel of the structure image as the noise magnitude based on the relational equation and the structure image.

[0036] In this embodiment, the construction unit 35 selects one relational expression from the following three relational expressions. If the photodetector 21 is not an electron multiplier, the construction unit 35 selects the following formula (1) as the relational expression. As an example, if the photodetector information indicates any of a CCD image sensor, a CMOS image sensor, a photodiode, an InGaAs sensor, a TDI-CCD image sensor, a TDI-CMOS image sensor, and an image pickup tube without a multiplication mechanism, the construction unit 35 selects the following formula (1) as the relational expression.

[0037]

number

[0038] When the above formula (1) is used, the constructing unit 35 assigns the pixel value of each pixel of the optical image acquired by the image acquiring unit 32 to the variable Signal. Then, the constructing unit 35 obtains the variable Noise calculated using the above formula (1) as a numerical value of the standard deviation of the noise value. Note that the other parameters in the above formula (1) may be acquired by receiving input by the input unit 31, or may be set in advance.

[0039] When the photodetector 21 is of an electron multiplying type and not of a photon counting type, the construction unit 35 selects the following formula (2) as the relational expression. As an example, when the photodetector information indicates any of an EM-CCD image sensor, an EB-CMOS image sensor, a SPAD, a HPD, an APD, a photomultiplier tube, and an MPPC, the construction unit 35 selects the following formula (2) as the relational expression.

[0040]

number

[0041] When the photodetector 21 is of an electron multiplying type and a photon counting type, the construction unit 35 selects the following formula (3) as the relational expression. As an example, when the photodetector operates for the purpose of photon counting using a photomultiplier tube, HPD, MPPC, etc., the construction unit 35 selects the following formula (3) as the relational expression.

[0042]

number

[0043] Next, the constructor 35 sets a noise distribution model based on the sigma value calculated in step S102 (step S103). The constructor 35 acquires condition information from the input unit 31 and sets the noise distribution model according to the photodetector information included in the condition information. The noise distribution models include a normal distribution model, a Poisson distribution model, and a Bessel function distribution model. The condition information further includes information indicating the amount of light L. The constructor 35 refers to the photodetector information, and if the photodetector 21 is not an electron multiplier and the amount of light L is not small, sets the normal distribution model as the noise distribution model. If the photodetector 21 is not an electron multiplier and the amount of light L is small, sets the Poisson distribution model as the noise distribution model. For example, the construction unit 35 sets the normal distribution model as the noise distribution model when the photodetector information indicates one of a CCD image sensor, a CMOS image sensor, a photodiode, an InGaAs sensor, a TDI-CCD image sensor, a TDI-CMOS image sensor, and a camera tube without a multiplication mechanism and the amount of light is equal to or greater than a predetermined reference value. Furthermore, for example, the construction unit 35 references the condition information and sets the Poisson distribution model as the noise distribution model when the photodetector information indicates one of a CCD image sensor, a CMOS image sensor, a photodiode, an InGaAs sensor, a TDI-CCD image sensor, a TDI-CMOS image sensor, and a camera tube and the amount of light is less than the reference value. Note that the noise distribution model may include only one of the normal distribution model and the Poisson distribution model. On the other hand, when the photodetector 21 is a multi-stage electron multiplication type with a multiplication factor of 2 per stage, the construction unit 35 sets the Bessel function distribution model as the noise distribution model. As an example, when the photodetector information indicates an EM-CCD image sensor, the constructor 35 sets a Bessel function distribution model as the noise distribution model. In this way, by setting a normal distribution model or a Bessel function distribution model, training data for various noise conditions can be generated. Note that when the photodetector information does not correspond to any of the above photodetectors, the constructor 35 may create a new noise distribution model by calculating a histogram and creating a function representing the noise distribution.The histogram is a histogram of pixel values ​​of an optical image when light having the same light intensity is incident on the photodetector 21. The constructing unit 35 calculates the histogram by, for example, acquiring multiple optical images of a light source whose light intensity does not change over time. As an example, the horizontal axis of the histogram represents the luminance value of the camera 2, and the vertical axis of the histogram represents frequency. Since the noise distribution changes depending on the light intensity, the constructing unit 35 further acquires multiple histograms by changing the light intensity of the light source within the range of light intensity that can be expected when using the optical image processing system 1, and creates a noise distribution model.

[0044] Next, the construction unit 35 calculates a randomly set noise value based on the noise magnitude (sigma value) acquired in step S102 and the noise distribution model set based on the sigma value in step S103 (step S104). Next, the construction unit 35 generates pixel values ​​constituting a training image, which is teacher data, by adding the noise value calculated in step S104 to the pixel value of one pixel (step S105). The construction unit 35 performs the processes of steps S102 to S105 on each of the multiple pixels constituting the structure image (step S106) to generate a training image, which is teacher data (step S107). Furthermore, if more training images are needed, the construction unit 35 determines that the processes of steps S101 to S107 should be performed on another structure image (step S108) to generate another training image, which is teacher data. Note that the other structure image may be an image of a structure having the same structure or may be an image of a structure having a different structure.

[0045] It is necessary to prepare a large number of training images, which are teacher data used to build the trained model 36. Furthermore, structure images should preferably be images with little noise, and ideally, images without noise. Therefore, generating structure images through simulation calculations is effective because it allows the generation of many noise-free images.

[0046] Next, the procedure for observing an optical image of the object F using the optical image processing system 1 according to the first embodiment, i.e., the flow of the optical image acquisition method according to the first embodiment, will be described. Fig. 6 is a flowchart showing the procedure for observing the optical image by the optical image processing system 1 including the optical image processing module 3.

[0047] First, the construction unit 35 constructs a plurality of trained models 36 that output noise-removed image data based on the training images by machine learning using the training images and noise-removed image data as training data (step S200). Next, the input unit 31 accepts input of condition information indicating imaging conditions and the like from an operator (user) of the optical image processing system 1 (step S201).

[0048] Next, the object F is set in the optical image processing system 1, the object F is imaged, and the image acquisition unit 32 acquires an optical image of the object F (step S202). Furthermore, the selection unit 33 uses the photodetector information included in the condition information to select a trained model 36 from among a plurality of trained models 36 each constructed in advance by machine learning (step S203).

[0049] Specifically, when the photodetector 21 indicated by the photodetector information is not an electron multiplier and the amount of light is not small, the selection unit 33 selects a trained model 36 constructed by setting a normal distribution model as the noise distribution model (hereinafter, sometimes referred to as a "trained model 36 of the normal distribution model"). As an example, when the photodetector information indicates any of a CCD image sensor, a CMOS image sensor, a photodiode, an InGaAs sensor, a TDI-CCD image sensor, a TDI-CMOS image sensor, and an image pickup tube, and the information indicating the amount of light is equal to or greater than a reference value, the selection unit 33 selects a trained model 36 of the normal distribution model. Furthermore, when the photodetector 21 indicated by the photodetector information is not an electron multiplier and the amount of light is small, the selection unit 33 selects a trained model 36 constructed by setting a Poisson distribution model as the noise distribution model (hereinafter, sometimes referred to as a "trained model 36 of the Poisson distribution model"). As an example, when the photodetector information indicates any of the above photodetectors and the information indicating the amount of light is less than a reference value, the selection unit 33 selects a trained model 36 of the Poisson distribution model. On the other hand, when the photodetector 21 indicated by the photodetector information is an electron multiplying type, the selection unit 33 selects a trained model 36 constructed with a Bessel function distribution model set as the noise distribution model (hereinafter, sometimes referred to as a "trained model 36 of the Bessel function distribution model"). As an example, when the photodetector information is an EM-CCD image sensor, the selection unit 33 selects the trained model 36 of the Bessel function distribution model. That is, when the photodetector 21 of the camera 2 is not an electron multiplying type, the selection unit 33 selects the trained model 36 constructed to be suitable for the photodetector 21 that is not an electron multiplying type. Furthermore, when the photodetector 21 of the camera 2 is an electron multiplying type, the selection unit 33 selects the trained model 36 constructed to be suitable for the photodetector 21 that is an electron multiplying type.

[0050] Next, the processing unit 34 inputs the optical image of the object F into the trained model 36 selected by the selection unit 33, and performs noise reduction processing on the optical image (step S204). Furthermore, the processing unit 34 outputs the optical image that has been subjected to noise reduction processing to the display device 4 (step S205).

[0051] According to the optical image processing module 3 described above, condition information indicating the imaging conditions when imaging the object F is used to select a trained model 36 to be used for noise removal from among the trained models 36 constructed in advance. As a result, the trained model 36 selected according to the imaging environment that changes depending on the imaging conditions of the object F is used for noise removal, so that noise removal that corresponds to changes in the imaging conditions of the object F can be realized. As a result, noise in the optical image can be effectively removed.

[0052] In particular, in the optical image processing module 3 of the first embodiment, the input unit 31 accepts input of condition information indicating the imaging conditions when imaging the object F, and the selection unit 33 uses the condition information to select a trained model 36 to be used in the noise removal process, where each trained model 36 is a trained model constructed in advance by machine learning using a noise distribution model, and the selection unit 33 selects a trained model 36 constructed in advance in accordance with the condition information. This allows the imaging conditions of the object F to be taken into consideration and the most appropriate trained model 36 to be selected from among multiple trained models 36 that appropriately remove noise, thereby enabling noise removal in the optical image of the object F in response to changes in the imaging conditions. Note that if the condition information is the same, the same trained model 36 can be used even if the imaging environment of the object F is different.

[0053] Here, an example of the effect of the noise removal process by the optical image processing module 3 of the first embodiment will be described. For example, in an example where a CMOS image sensor (C13440-20 ORCA (trademark)-Flash 4.0 V3 manufactured by Hamamatsu Photonics KK) is used as the photodetector 21 and visible light is irradiated onto the object F as the observation light, the standard deviation of noise in the optical image G1 (see FIG. 3) is 3.31, and the standard deviation of noise in the optical image G6 is 0.32. In an example where a CMOS image sensor (C14440-20 ORCA (trademark)-Fusion manufactured by Hamamatsu Photonics KK) different from the above CMOS image sensor is used as the photodetector 21 and visible light is irradiated onto the object F as the observation light, the standard deviation of noise in the optical image G1 is 6.91, and the standard deviation of noise in the optical image G6 is 0.66. In an example in which a different CMOS image sensor (C15440-20 ORCA (trademark)-FusionBT manufactured by Hamamatsu Photonics K.K.) from the two CMOS image sensors described above was used as the photodetector 21 and visible light was irradiated onto the object F as the observation light, the standard deviation of noise in the optical image G1 was 6.91, and the standard deviation of noise in the optical image G6 was 0.65. In an example in which an InGaAs sensor (C12741-03 InGaAs camera manufactured by Hamamatsu Photonics K.K.) was used as the photodetector 21 and infrared light was irradiated onto the object F as the observation light, the standard deviation of noise in the optical image G1 was 7.54, and the standard deviation of noise in the optical image G6 was 2.69. Note that in each of the above examples, a photodetector 21 that is not an electron multiplier is used, and therefore a normal distribution model is set as the noise distribution model.

[0054] In an example where an EM-CCD image sensor (Hamamatsu Photonics C9100-23B ImagEM (registered trademark) X2 EM-CCD camera) was used as the photodetector 21, the amplification factor was 300, and visible light was irradiated as the observation light onto the object F, the following results were obtained. Specifically, when the digital value was 2200 (counts), the standard deviation of the noise in the optical image G1 was 41.5, and the standard deviation of the noise in the optical image G6 was 5.66. Furthermore, when the digital value was 2500 (counts), the standard deviation of the noise in the optical image G1 was 44.1, and the standard deviation of the noise in the optical image G6 was 7.73. Furthermore, in an example where the amplification factor was 1200 under the above conditions, the following results were obtained. Specifically, when the digital value was 2200 (counts), the standard deviation of the noise in the optical image G1 was 86.9, and the standard deviation of the noise in the optical image G6 was 13.2. Furthermore, when the digital value is 2500 (count), the standard deviation of noise in optical image G1 is 91.5, and the standard deviation of noise in optical image G6 is 15.5. Note that in each of the above examples, since an electron multiplying photodetector 21 is used, a Bessel function distribution model is set as the noise distribution model.

[0055] As shown in the above examples, according to the optical image processing module 3 of the first embodiment, the most suitable trained model 36 is selected from multiple trained models 36 depending on the type of photodetector 21, so that an optical image G6 can be obtained in which noise in the optical image G1 has been effectively removed.

[0056] In the optical image processing module 3 of the first embodiment, the condition information includes photodetector information (information) indicating the type of photodetector 21 used to capture an image of the object F, and each trained model 36 is a trained model constructed according to the type of photodetector 21 that may be input as condition information. The selection unit 33 selects a trained model 36 constructed in advance using a noise distribution model corresponding to the type of photodetector 21 indicated by the photodetector information. In general, it can be said that the characteristics of noise relative to luminance differ depending on the type of photodetector 21. According to the above configuration, a trained model 36 corresponding to the photodetector 21 used to capture an image of the object F is selected from among multiple trained models 36 constructed according to the type of photodetector 21, thereby enabling noise removal in the optical image of the object F corresponding to the photodetector 21 used for the object F.

[0057] In the optical image processing module 3 of the first embodiment, the noise distribution model used to construct the trained model 36 includes a normal distribution model. As a result, for example, when a general photodetector 21 that is not an electron multiplying type is used to capture an image of the object F, noise in the optical image can be effectively removed by selecting the trained model 36 constructed by machine learning using the normal distribution model.

[0058] In the optical image processing module 3 of the first embodiment, the noise distribution model used to construct the trained model 36 includes a Bessel function distribution model. As a result, for example, when an electron multiplying photodetector 21 is used to capture an image of the object F, noise in the optical image can be effectively removed by selecting the trained model 36 constructed by machine learning using the Bessel function distribution model.

[0059] The optical image processing module 3 of the first embodiment includes a construction unit 35 that constructs, by machine learning, a trained model 36 that outputs noise-removed image data based on the training image, using a structure image to which noise based on a predetermined noise distribution model has been added as training image and the training image and noise-removed image data, which is data obtained by removing noise from the training image. According to the above configuration, when an optical image is input to the trained model 36, a noise removal process that can effectively remove noise in the optical image of the object F can be realized.

[0060] The optical image processing module 3 of the first embodiment has a preprocessing function of generating, as training images, structure images to which noise based on a noise distribution model has been added. As a result, when an optical image is input to the trained model 36, which is generated by the above preprocessing method and constructed using the training images for the above machine learning method, a noise removal process that can effectively remove noise from the optical image of the object F can be realized.

[0061] The optical image processing module 3 of the first embodiment accepts input of condition information including photodetector information indicating the type of photodetector 21 used to image the object F, and in the training image generation process, has a function of determining the noise distribution model to be used from the photodetector information. The relationship between pixel values ​​and noise in an optical image varies depending on the type of photodetector 21 used to image the object F. With the above configuration, it is possible to obtain training images in which noise is appropriately added to optical images, taking into account the type of photodetector 21 used to image the object F. As a result, a trained model 36 can be constructed that performs noise removal in optical images of the object F corresponding to the photodetector 21.

[0062] In the preprocessing of the optical image processing module 3 of the first embodiment, the noise distribution model used to generate training images includes a normal distribution model and a Poisson distribution model. This makes it possible to obtain training images in which noise is appropriately added to structural images, for example, when a general photodetector 21 that is not an electron multiplier is used to capture an image of the object F. As a result, it is possible to construct a trained model 36 that performs noise removal in the optical image of the object F corresponding to the photodetector 21. In particular, because the noise distribution model further includes a Poisson distribution model in addition to the normal distribution model, it is possible to obtain training images in which noise is appropriately added to structural images, even when the amount of light L is low.

[0063] In the preprocessing of the optical image processing module 3 of the first embodiment, the noise distribution model includes a Bessel function distribution model. This makes it possible to obtain training images in which noise is appropriately added to structure images, for example, when an electron multiplying photodetector 21 is used to capture an image of the object F. As a result, a trained model 36 can be constructed that performs noise removal in the optical image of the object F corresponding to the photodetector 21. [Second embodiment]

[0064] FIG. 7 is a block diagram showing the functional configuration of an optical image processing system 1A according to a second embodiment. FIG. 8 is a diagram showing an example of input / output data of a trained model 36A shown in FIG. 7. The optical image processing module 3A according to the second embodiment differs from the first embodiment in that it includes an image acquisition unit (second image acquisition unit) 30A and an identification unit 37A. Furthermore, the optical image processing module 3A differs from the first embodiment in that each trained model 36A is a trained model constructed in advance by machine learning using image data, and that the selection unit 33A has a function of selecting a trained model 36A from among multiple trained models 36A using image characteristics (described in detail below) of a jig image (optical image of a structure), which is an optical image of a jig (structure). Specifically, as shown in FIG. 8, in the noise removal phase, the optical image processing module 3A first acquires a jig image G27. The optical image processing module 3A then identifies image characteristics of the jig image G27 based on the jig image G27, and selects a trained model 36A to be used for noise removal processing based on the identified image characteristics. Each trained model 36A is a trained model constructed in advance by machine learning, with image data used as training data. As training data, image data captured with various types of photodetectors 21, gain settings, read modes, etc. may be used. The image data may be optical images actually generated using the optical image processing module 3A for multiple types of objects F, or image data generated by simulation calculations.

[0065] Fig. 9 is a flowchart showing the procedure of observation processing by the optical image processing system 1A including the optical image processing module 3A of Fig. 7. As shown in Fig. 9, in the optical image processing module 3A according to the second embodiment, the processing shown in steps S301 and S303 is executed by replacing the processing of steps S201 and S203 by the optical image processing module 3 of the first embodiment shown in Fig. 6.

[0066] The image acquisition unit 30A acquires a jig image by capturing light from the jig (step S301). Specifically, the image acquisition unit 30A acquires an optical image capturing the light from the jig using the camera 2. The light from the jig includes fluorescence from the jig, reflected light from the jig, transmitted light from the jig, and scattered light from the jig. The jig may be a grayscale chart capable of evaluating gradation performance using gradually changing density steps, or a jig having charts with various resolutions. That is, the image acquisition unit 30A acquires a jig image G27 captured using the camera 2 prior to the observation process of the object F. Then, the image acquisition unit (first image acquisition unit) 32A acquires an optical image of the object F captured using the camera 2. However, the timing of acquiring the optical images of the jig and the object F is not limited to the above, and they may be acquired simultaneously or at opposite times.

[0067] Next, the specifying unit 37A specifies the image characteristics of the jig image (step S302). Specifically, the specifying unit 37A specifies the brightness-noise characteristics, the resolution characteristics, or the like as the image characteristics of the jig image.

[0068] When a jig having a grayscale chart capable of evaluating gradation performance using gradually changing density steps is used, the determination unit 37A can analyze the luminance values ​​and noise for each of multiple measurement areas as noise characteristics of the optical image of the jig and obtain a luminance-to-noise ratio characteristic graph as the noise characteristics. The luminance-to-noise ratio of the optical image can be measured using a grayscale chart. That is, the determination unit 37A analyzes the luminance values ​​and noise for each of multiple measurement areas with different densities and obtains a luminance-to-noise ratio characteristic graph. Specifically, the determination unit 37A selects multiple measurement areas with different densities, analyzes the standard deviation and average luminance values ​​of the multiple measurement areas, and obtains a luminance-SNR (Signal-to-Noise Ratio) characteristic graph as the noise characteristics. At this time, the determination unit 37A calculates the SNR for each measurement area using the formula SNR = (average luminance value) ÷ (standard deviation of luminance value). Figure 10 shows a jig image G27 of a jig, which is a grayscale chart. 11 shows an example of a luminance-SNR characteristic graph acquired by the determination unit 37A. Here, instead of the above-described luminance-SNR characteristic graph, the determination unit 37A may acquire a characteristic graph in which the vertical axis represents noise calculated from the standard deviation of luminance values ​​as the noise characteristic.

[0069] When a jig having a resolution chart in which the resolution changes stepwise in one direction is used, the specifying unit 37A can also acquire the resolution distribution in the optical image of the jig as the resolution characteristics. Furthermore, the specifying unit 37A has a function of acquiring the resolution characteristics for an image after applying multiple trained models 36A to the optical image of the jig and performing noise reduction processing. FIG. 12 shows a jig image G26 of the jig, which is a resolution chart. The resolution of the jig image can be measured using MTF (Modulation Transfer Function) or CTF (Contrast Transfer Function).

[0070] Next, the specifying unit 37A applies the plurality of trained models 36A to the jig image. When the jig is a grayscale chart, the specifying unit 37A acquires, as the image characteristic, the brightness-to-noise ratio characteristic generated as a result of applying the plurality of trained models 36A. When the jig is a resolution chart, the specifying unit 37A acquires, as the image characteristic, the resolution characteristic, which is the distribution of the resolution of the image after noise removal generated as a result of applying the plurality of trained models 36A.

[0071] Next, the selection unit 33A selects a trained model 36A to be used for the noise removal process from among the multiple trained models 36A stored in the optical image processing module 3A based on the image characteristics (imaging information) acquired by the identification unit 37A (step S303). Specifically, the selection unit 33A selects a trained model 36A used to generate an image with relatively superior characteristics based on the characteristics of the image after noise removal process is performed using the multiple trained models 36A for the jig image. For example, if the jig is a grayscale chart, the selection unit 33A selects, as the final trained model 36A, a trained model 36A constructed using image data having brightness-to-noise ratio characteristics closest to the brightness-to-noise ratio characteristics acquired by the identification unit 37A. Furthermore, for example, if the jig is a resolution chart, the selection unit 33A uses the resolution characteristics identified by the identification unit 37A to select the trained model 36A used for the image in which the change in resolution of each distribution before and after the noise removal process is smallest. However, the resolution chart can also be used as an aid when using a grayscale chart. For example, a condition that the resolution obtained by the resolution chart does not change may be added, and a trained model 36 that meets this condition may be selected using the grayscale chart. Furthermore, the image characteristics of the image data used to construct the trained model 36A may be acquired from the image data by the selector 33A, or may be calculated in advance outside the optical image processing module 3A. Here, the selector 33A may select the trained model 36A using brightness-noise characteristics instead of brightness-noise ratio characteristics as noise characteristics. By using such brightness-noise characteristics, the dominant noise factor (shot noise, readout noise value, etc.) can be identified for each signal amount detected by the photodetector 21 from the slope of the graph of each signal amount region, and the trained model 36A can be selected based on the identification results.

[0072] The optical image processing module 3A of the second embodiment described above includes an image acquisition unit 30A that acquires a jig image and an identification unit 37A that identifies image characteristics of the jig image. The selection unit 33A uses the image characteristics identified by the identification unit 37A to select a trained model 36A to be used for noise removal processing from among multiple trained models 36A. Each trained model 36A is a trained model constructed in advance by machine learning using image data, and the selection unit 33 selects a trained model 36A constructed in advance based on the image characteristics. This identifies the image characteristics of the optical image of the structure, and based on the image characteristics, a trained model 36A to be used for noise removal is selected from the trained models 36A constructed in advance. This makes it possible to achieve optimal noise reduction for each image characteristic. [Variations]

[0073] Although various embodiments of the present disclosure have been described above, the embodiments of the present disclosure are not limited to the above embodiments. For example, the construction unit 35 may generate training images by actually capturing images when constructing each trained model 36. In other words, the training images may be optical images actually generated using the camera 2 of multiple types of known structures.

[0074] The optical image processing system 1 may be a scanning type. The example shown in FIG. 13 differs from the above-described embodiments in that the optical image processing system 1 includes a confocal microscope 2B. The confocal microscope 2B acquires images that enable the construction of an optical tomographic image of an object F. The confocal microscope 2B is configured by connecting a confocal microscope unit 6 to a connection port P1 of a microscope 7 for connecting an external unit. The confocal microscope unit 6 is a device that irradiates an object F placed on a stage of the microscope 7 with excitation light via a microscope optical system, such as an imaging lens 71 and an objective lens 72, within the microscope 7, and receives (detects) fluorescence (light) from the object F in response to the excitation light via the microscope optical system of the microscope 7 to generate and output an optical tomographic image.

[0075] In detail, the confocal microscope unit 6 includes a main housing 61, a lens barrel 62, a scan mirror 63 fixed inside the main housing 61, a fixed mirror 64, a subunit 65, and a scan lens 66 fixed inside the lens barrel 62. The lens barrel 62 forms part of the main housing 61, and is detachably connected to a connection port P1 of the microscope 7. Each component of the confocal microscope unit 6 will be described in detail below.

[0076] The scan lens 66 in the lens barrel 62 is an optical element that relays the reflecting surface of the scan mirror 63 to the pupil position of the objective lens 72, and at the same time focuses the excitation light (observation light) on a primary image plane of the microscope optical system of the microscope 7. The scan lens 66 guides the excitation light scanned by the scan mirror 63 to the microscope optical system, thereby irradiating the object F, and guides the fluorescence (observation light) generated from the object F in response to this to the scan mirror 63. In detail, the scan lens 66 is configured to image the pupil of the objective lens 72 on the scan mirror 63, and guides the fluorescence imaged by the objective lens 72 and imaging lens 71 of the microscope 7 to the scan mirror 63.

[0077] The scan mirror 63 in the main housing 61 is an optical scanning element such as a MEMS (Micro Electro Mechanical System) mirror configured so that a reflector can be tilted on two axes. The scan mirror 63 has the role of scanning the excitation light output from the subunit 65 onto the object F by continuously changing the reflection angle, and guiding the fluorescence generated in response to the excitation light toward the subunit 65.

[0078] The fixed mirror 64 is a light reflecting element fixed within the main housing 61, which reflects the excitation light output from the subunit 65 toward the scan mirror 63 and reflects the fluorescence reflected by the scan mirror 63 toward the subunit 65 coaxially with the excitation light.

[0079] Subunit 65 includes a base plate 651, a total reflection mirror 652 arranged on base plate 651, a light source 653, a dichroic mirror 654, a pinhole plate 655, and a photodetector 656. Total reflection mirror 652 reflects the first excitation light having a wavelength λ1 emitted by subunit 65 and the first fluorescence having a wavelength range Δλ1 generated from object F in response to the first excitation light. Dichroic mirror 654 is a beam splitter arranged in the reflection direction of the first fluorescence from total reflection mirror 652, and has the property of transmitting the first fluorescence having a wavelength range Δλ1 and reflecting the first excitation light having a wavelength λ1 shorter than the wavelength range Δλ1. Light source 653 is a light-emitting element (e.g., a laser diode) that outputs first excitation light (e.g., laser light) having a wavelength λ1, and is arranged so that the first excitation light is reflected by dichroic mirror 654 toward total reflection mirror 652 coaxially with the first fluorescence. The pinhole plate 655 is an aperture member that limits the light flux of the first fluorescence and is arranged so that the pinhole position coincides with the conjugate position of the spot of the first excitation light on the object F, and constitutes a confocal optical system together with the light source 653, etc. The pinhole diameter of the pinhole plate 655 can be adjusted externally, making it possible to change the resolution and signal intensity of the image detected by the photodetector 656.

[0080] The photodetector 656 has its detection surface facing the pinhole plate 655 and receives and detects the first fluorescence that has passed through the pinhole plate 655. Examples of the photodetector 656 include photodetectors similar to those in the above-described embodiments (e.g., a CCD image sensor, a CMOS image sensor, etc.). The subunit 65 captures light from the object F using the photodetector 656 and outputs a digital signal based on the image capture result to the optical image processing module 3. The image acquisition unit 32 of the optical image processing module 3 generates image data based on the digital signal received from the subunit 65, thereby acquiring an optical image. As in the above-described embodiments, the configuration of this modified example also uses a trained model 36 selected according to the imaging environment, which changes depending on the imaging conditions of the object F, for noise removal. This enables noise removal that corresponds to changes in the imaging conditions of the object F. FIG. 14 shows optical images G1 and G6 obtained when a PMT is used as the photodetector 656 and visible light is irradiated on the object F as observation light.

[0081] The condition information input from the input unit 31 is not limited to photodetector information. For example, the condition information may include information indicating the gain setting value of the photodetector 21 used to capture the object F. Each trained model 36 is a trained model constructed according to the gain setting value of the photodetector 21 that may be input as condition information. The selection unit 33 may select a trained model 36 constructed in advance corresponding to the gain setting value of the photodetector 21 indicated by the photodetector information. In this case, the construction unit 35 constructs multiple trained models 36 as follows. First, differences between the above embodiment and this modified example in the training image creation procedure will be mainly described. In step S102, the construction unit 35 acquires the magnitude of noise according to gain setting values ​​that are expected to be input. The construction unit 35 first selects an appropriate relational expression based on the photodetector information. Then, the construction unit 35 sets a constant G (information representing the gain setting value) included in the selected relational expression for each gain setting value that is expected to be input, and calculates the magnitude of noise for each gain setting value. In this modification, an appropriate relational expression is selected from the above-mentioned expressions (2) and (3) including the constant G. Then, the constructing unit 35 generates training images for each gain setting value, in the same manner as in the above-mentioned embodiment, based on the noise distribution model set using the sigma value (steps S103 to S107).

[0082] Next, differences between the above embodiment and this modification in the procedure for processing the observation of the optical image of the object F will be mainly described. In step S201, the input unit 31 accepts input of condition information including photodetector information and a gain setting value from an operator (user) of the optical image processing system 1. In step S203, the selection unit 33 uses the photodetector information and the gain setting value included in the condition information to select a trained model 36 from among a plurality of trained models 36 each constructed in advance by machine learning. That is, in this modification, a trained model 36 of a normal distribution model, a trained model 36 of a Poisson distribution model, and a trained model 36 of a Bessel function distribution model are constructed for each gain setting value that is expected to be input, and an appropriate trained model 36 corresponding to the gain setting value is selected from the plurality of trained models 36 constructed in this manner.

[0083] In general, it can be said that the characteristics of noise relative to brightness differ depending on the gain setting value of the photodetector 21. According to this modification, the trained model 36 corresponding to the gain setting value is selected from among a plurality of trained models 36 constructed according to the gain setting value of the photodetector 21, and therefore noise can be removed from the optical image of the object F corresponding to the gain setting value.

[0084] Alternatively, for example, the condition information may include information indicating the readout mode of the photodetector 21 used to capture the object F, and each trained model 36 may be a trained model constructed according to the readout mode of the photodetector 21 that may be input as the condition information. The selection unit 33 may select a trained model 36 constructed in advance corresponding to the readout mode of the photodetector 21 indicated by the photodetector information. In this case, the construction unit 35 constructs multiple trained models 36 as follows. First, differences between the above embodiment and this modified example in the training image creation procedure will be mainly described. In step S102, the construction unit 35 acquires the magnitude of noise according to the readout mode that is expected to be input. The construction unit 35 first selects an appropriate relational expression based on the photodetector information. Then, the construction unit 35 sets the constant R (information representing the readout noise value corresponding to the readout mode) included in the selected relational expression for each readout noise value that is expected to be input, and calculates the magnitude of noise for each readout noise value. In this modification, an appropriate relational expression is selected from the above-mentioned expressions (1) and (2) including the constant R. Then, the constructing unit 35 generates training images for each read noise value based on the noise distribution model set using the sigma value, in the same manner as in the above-mentioned embodiment (steps S103 to S107).

[0085] Next, differences between the above embodiment and this modification in the procedure for processing the observation of the optical image of the object F will be mainly described. In step S201, the input unit 31 accepts input of condition information including photodetector information and readout noise value information from an operator (user) of the optical image processing system 1. In step S203, the selection unit 33 selects a trained model 36 from among a plurality of trained models 36 each constructed in advance by machine learning, using the photodetector information and readout noise value included in the condition information. That is, in this modification, a trained model 36 of a normal distribution model, a trained model 36 of a Poisson distribution model, and a trained model 36 of a Bessel function distribution model are constructed for each readout noise value that is expected to be input, and an appropriate trained model 36 corresponding to the readout noise value is selected from the plurality of trained models 36 constructed in this manner.

[0086] In general, it can be said that the characteristics of noise relative to brightness differ depending on the readout mode of the photodetector 21. According to this modification, a trained model 36 corresponding to the readout mode of the photodetector 21 is selected from among a plurality of trained models 36 constructed according to the readout mode (specifically, the readout noise value corresponding to the readout mode), thereby making it possible to remove noise in the optical image of the object F corresponding to the readout mode.

[0087] The optical image processing method may further include an input step of accepting input of condition information indicating imaging conditions when imaging the object, wherein the imaging information includes the condition information, and each of the plurality of trained models is a trained model constructed in advance by machine learning using a predetermined noise distribution model, and the selection step may select a trained model constructed in advance in response to the condition information. The optical image processing module may further include an input unit that accepts input of condition information indicating imaging conditions when imaging the object, wherein the imaging information includes the condition information, and each of the plurality of trained models is a trained model constructed in advance by machine learning using a predetermined noise distribution model, and the selection unit may select a trained model constructed in advance in response to the condition information. This allows the imaging conditions of the object to be taken into consideration and the most appropriate trained model that appropriately removes noise to be selected from the plurality of trained models, thereby enabling noise removal in optical images of the object in response to changes in imaging conditions.

[0088] In the optical image processing method, the condition information may include information indicating the type of photodetector used to image the object, and each of the multiple trained models may be constructed according to the type of photodetector that may be input as the condition information. The selection step may select a trained model that has been pre-constructed using a predetermined noise distribution model corresponding to the type of photodetector indicated by the condition information. Also, in the optical image processing module, the condition information may include information indicating the type of photodetector used to image the object, and each of the multiple trained models may be constructed according to the type of photodetector that may be input as the condition information. The selection unit may select a trained model that has been pre-constructed using a predetermined noise distribution model corresponding to the type of photodetector indicated by the condition information. Generally, noise characteristics relative to luminance can be said to differ depending on the type of photodetector. According to the above configuration, a trained model corresponding to the photodetector used to image the object is selected from among multiple trained models constructed according to the type of photodetector, thereby enabling noise removal from an optical image of the object that corresponds to the photodetector used for the object.

[0089] In the optical image processing method, the condition information may include information indicating a gain setting value of a photodetector used to image the object, and each of the multiple trained models may be a trained model constructed according to the gain setting value of the photodetector that may be input as the condition information. The selection step may select a trained model constructed in advance corresponding to the gain setting value of the photodetector indicated by the condition information. Furthermore, in the optical image processing module, the condition information may include information indicating a gain setting value of a photodetector used to image the object, and each of the multiple trained models may be a trained model constructed according to the gain setting value of the photodetector that may be input as the condition information. The selection unit may select a trained model constructed in advance corresponding to the gain setting value of the photodetector indicated by the condition information. Generally, it can be said that noise characteristics relative to luminance differ depending on the gain setting value of the photodetector. According to the above configuration, a trained model corresponding to the gain setting value is selected from multiple trained models constructed according to the gain setting value of the photodetector, thereby enabling noise removal from an optical image of the object corresponding to the gain setting value.

[0090] In the optical image processing method, the condition information may include information indicating a readout mode of a photodetector used to image the object, and each of the multiple trained models may be a trained model constructed according to a readout mode of the photodetector that may be input as the condition information. The selection step may select a trained model constructed in advance corresponding to the readout mode of the photodetector indicated by the condition information. Also, in the optical image processing module, the condition information may include information indicating a readout mode of the photodetector used to image the object, and each of the multiple trained models may be a trained model constructed according to a readout mode of the photodetector that may be input as the condition information. The selection unit may select a trained model constructed in advance corresponding to the readout mode of the photodetector indicated by the condition information. Generally, it can be said that noise characteristics relative to luminance differ depending on the readout mode of the photodetector. With this configuration, a trained model corresponding to the readout mode is selected from among multiple trained models constructed according to the readout modes of the photodetector, thereby enabling noise removal from an optical image of the object corresponding to the readout mode.

[0091] In the above embodiment, the noise distribution model may include at least one of a normal distribution model and a Poisson distribution model. As a result, for example, when a general photodetector that is not an electron multiplying type is used to capture an image of an object, noise in an optical image can be effectively removed by selecting a trained model constructed by machine learning using a normal distribution model or a trained model constructed by machine learning using a Poisson distribution model.

[0092] In the above embodiment, the noise distribution model may include a Bessel function distribution model. As a result, for example, when an electron multiplying photodetector is used to capture an image of an object, noise in an optical image can be effectively removed by selecting a trained model constructed by machine learning using the Bessel function distribution model.

[0093] The optical image processing method may further include a second image acquisition step of acquiring an optical image of the structure by capturing light from the structure having a predetermined structure, and an identification step of identifying image characteristics of the optical image of the structure, where the captured image information includes the image characteristics, and each of the plurality of trained models is a trained model previously constructed by machine learning using the image data, and the selection step may select a pre-constructed trained model based on the image characteristics. The optical image processing module may further include a second image acquisition unit of acquiring an optical image of the structure by capturing light from the structure having a predetermined structure, and an identification unit of identifying image characteristics of the optical image of the structure, where the captured image information includes the image characteristics, and each of the plurality of trained models is a trained model previously constructed by machine learning using the image data, and the selection unit may select a pre-constructed trained model based on the image characteristics. This identifies the image characteristics of the optical image of the structure, and a trained model to be used for noise removal is selected from the pre-constructed trained models based on the image characteristics. This allows optimal noise reduction to be achieved for each image characteristic.

[0094] The machine learning preprocessing method further includes an input step of accepting input of condition information including photodetector information indicating the type of photodetector used to image the object, and in the training image generation step, a noise distribution model to be used is determined from the photodetector information. While the relationship between pixel values ​​and noise in an optical image varies depending on the type of photodetector used to image the object, the above configuration allows for the type of photodetector used to image the object to be taken into consideration, making it possible to obtain training images in which noise is appropriately added to the optical image. As a result, a trained model can be constructed that performs noise removal in an optical image of the object corresponding to the photodetector.

[0095] In the machine learning preprocessing method, the noise distribution model may include at least one of a normal distribution model and a Poisson distribution model. This makes it possible to obtain training images in which noise is appropriately added to optical images, for example, when a general photodetector that is not an electron multiplier is used to capture an image of the target. As a result, a trained model that performs noise removal in optical images of the target corresponding to the type of photodetector can be constructed.

[0096] In the machine learning preprocessing method, the noise distribution model may include a Bessel function distribution model. This makes it possible to obtain training images in which noise is appropriately added to optical images, for example, when an electron-multiplying photodetector is used to capture an image of the target. As a result, a trained model that performs noise removal in optical images of the target corresponding to the type of photodetector can be constructed. [Explanation of symbols]

[0097] 1,1A...optical image processing system, 2...camera (imaging device), 3,3A...optical image processing module, 21,656...photodetector, 30A...image acquisition unit (second image acquisition unit), 31...input unit, 32,32A...image acquisition unit (first image acquisition unit), 33,33A...selection unit, 34...processing unit, 35...construction unit, 36,36A...trained model, 37A...identification unit, F...object, G1,G6...optical image, G26,G27...jig image (optical image of structure), Gc...structure image, Gt...training image, L...light.

Claims

1. a first image acquisition step of acquiring an optical image obtained by capturing light from an object; an input step of receiving input of condition information indicating imaging conditions when imaging the object; a selection step of selecting a trained model from a plurality of trained models each constructed in advance by machine learning using imaging information including the condition information related to imaging of the object; A processing step of inputting the optical image into the selected trained model and performing image processing to remove noise from the optical image; Equipped with Each of the plurality of trained models is a trained model constructed in advance by machine learning using a predetermined noise distribution model, In the selection step, the trained model constructed in advance in response to the condition information is selected. Optical image processing methods.

2. the condition information includes information indicating a type of photodetector used to capture an image of the object, Each of the plurality of trained models is a trained model constructed according to a type of photodetector that can be input as the condition information, In the selection step, the trained model constructed in advance is selected using the predetermined noise distribution model corresponding to the type of the photodetector indicated by the condition information. The optical image processing method of claim 1 .

3. the condition information includes information indicating a gain setting value of a photodetector used to capture an image of the object, Each of the plurality of trained models is a trained model constructed according to a gain setting value of a photodetector that may be input as the condition information, In the selection step, the trained model constructed in advance in accordance with the gain setting value of the photodetector indicated by the condition information is selected. The optical image processing method of claim 1 .

4. the condition information includes information indicating a readout mode of a photodetector used to capture an image of the object; Each of the plurality of trained models is a trained model constructed according to a readout mode of a photodetector that can be input as the condition information, In the selection step, the trained model constructed in advance in accordance with the readout mode of the photodetector indicated by the condition information is selected. The optical image processing method of claim 1 .

5. the noise distribution model includes at least one of a normal distribution model and a Poisson distribution model; The optical image processing method according to any one of claims 1 to 4.

6. the noise distribution model includes a Bessel function distribution model; The optical image processing method according to any one of claims 1 to 4.

7. a construction step of constructing a trained model by machine learning, using training images of structures to which noise based on a predetermined noise distribution model has been added, and using the training images and noise-removed image data, which is data obtained by removing noise from the training images, as training data, and outputting the noise-removed image data based on the training images; a training image generation step of generating, as the training image, a structure image to which noise based on the noise distribution model has been added; A machine learning method comprising:

8. further comprising an input step of accepting input of condition information including photodetector information indicating the type of photodetector used to image the object; In the training image generation step, the noise distribution model to be used is determined from the photodetector information. The machine learning method of claim 7.

9. The machine learning method of claim 8 , wherein the noise distribution model includes at least one of a normal distribution model and a Poisson distribution model.

10. The machine learning method of claim 8 , wherein the noise distribution model comprises a Bessel function distribution model.

11. a first image acquisition unit that acquires an optical image obtained by capturing light from an object; an input unit that receives input of condition information indicating imaging conditions when imaging the object; a selection unit that selects a trained model from a plurality of trained models that have been constructed in advance by machine learning, using imaging information including the condition information related to imaging of the object; A processing unit that inputs the optical image into the selected trained model and performs image processing to remove noise from the optical image; Equipped with Each of the plurality of trained models is a trained model constructed in advance by machine learning using a predetermined noise distribution model, The selection unit selects the trained model constructed in advance in response to the condition information. Optical image processing module.

12. the condition information includes information indicating a type of photodetector used to capture an image of the object, Each of the plurality of trained models is a trained model constructed according to a type of photodetector that can be input as the condition information, the selection unit selects the trained model constructed in advance using the predetermined noise distribution model corresponding to the type of the photodetector indicated by the condition information. The optical image processing module of claim 11 .

13. the condition information includes information indicating a gain setting value of a photodetector used to capture an image of the object, Each of the plurality of trained models is a trained model constructed according to a gain setting value of a photodetector that may be input as the condition information, the selection unit selects the trained model constructed in advance in accordance with the gain setting value of the photodetector indicated by the condition information. The optical image processing module of claim 11 .

14. the condition information includes information indicating a readout mode of a photodetector used to capture an image of the object; Each of the plurality of trained models is a trained model constructed according to a readout mode of a photodetector that can be input as the condition information, the selection unit selects the trained model constructed in advance in accordance with the readout mode of the photodetector indicated by the condition information. The optical image processing module of claim 11 .

15. the noise distribution model includes at least one of a normal distribution model and a Poisson distribution model; The optical image processing module according to any one of claims 11 to 14.

16. the noise distribution model includes a Bessel function distribution model; The optical image processing module according to any one of claims 11 to 14.

17. The system further includes a construction unit that constructs a trained model by machine learning, using training images of structures to which noise based on a predetermined noise distribution model has been added and training data obtained by removing noise from the training images, and outputs the noise-removed image data based on the training images. The optical image processing module according to any one of claims 11 to 16.

18. The processor, an image acquisition unit that acquires an optical image obtained by capturing light from an object; an input unit that accepts input of condition information indicating imaging conditions when imaging the object; a selection unit that selects a trained model from a plurality of trained models that have been constructed in advance by machine learning, using imaging information including the condition information related to imaging of the object; and The optical image is input to the selected trained model, and the optical image is processed by a processing unit that performs image processing to remove noise from the trained model. Each of the plurality of trained models is a trained model constructed in advance by machine learning using a predetermined noise distribution model, The selection unit selects the trained model constructed in advance in response to the condition information. Optical image processing program.

19. An optical image processing module according to any one of claims 11 to 17; an imaging device that captures light from the object to obtain the optical image; An optical image processing system comprising:

Images