Optical image processing method, machine learning method, trained model, machine learning preprocessing method, optical image processing module, optical image processing program, and optical image processing system

Abstract

Provided are an optical image processing method, a machine learning method, a trained model, a machine learning preprocessing method, an optical image processing module, an optical image processing program, and an optical image processing system which are capable of effectively removing noise in an optical image. The optical image processing module 3 comprises: an image acquisition unit 32 that acquires an optical image in which light L from a target object F is captured; a noise map generation unit 33 that, on the basis of relationship data representing a relationship between a pixel value and a standard deviation of a noise value that evaluates the spread of the noise value, derives the standard deviation of the noise value from the pixel value of each pixel in the optical image, and generates a noise map, which is data in which the derived standard deviation of the noise value is associated with each pixel of the optical image; and a processing unit 34 that executes an image process for inputting the optical image and the noise map into a trained model 36 constructed in advance by machine learning, and removing noise from the optical image.

Description

[Technical Field]

[0001] The present disclosure relates to an optical image processing method, a machine learning method, a trained model, a machine learning preprocessing method, an optical image processing module, an optical image processing program, and an optical image processing system. [Background technology]

[0002] Conventionally, a technique is known in which light from a sample to be imaged is captured, image data is obtained based on the image capture results, and noise is removed from the image data to output the noise-removed image data (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-21314 Summary of the Invention [Problem to be solved by the invention]

[0004] In the noise removal process described above, a method of removing noise from an optical image obtained by capturing light from an object such as a sample using a trained model based on machine learning may be used. In this case, the nature of the noise varies depending on conditions such as the type of photodetector used for capturing the image, and therefore, depending on the conditions and the trained model, the noise may not be effectively removed.

[0005] The present disclosure has been made in consideration of such problems, and aims to provide an optical image processing method, a machine learning method, a trained model, a machine learning preprocessing method, an optical image processing module, an optical image processing program, and an optical image processing system that can effectively remove noise in optical images. [Means for solving the problem]

[0006] An optical image processing method according to one aspect of the embodiment includes an image acquisition step of acquiring an optical image in which light from an object is captured; a noise map generation step of deriving an evaluation value from the pixel values ​​of each pixel of the optical image based on relational data representing the relationship between pixel values ​​and an evaluation value that evaluates the spread of noise values, and generating a noise map that is data in which the derived evaluation value is associated with each pixel of the optical image; and a processing step of inputting the optical image and the noise map into a trained model that has been constructed in advance by machine learning, and performing image processing to remove noise from the optical image.

[0007] Alternatively, an optical image processing module according to another aspect of the embodiment includes an image acquisition unit that acquires an optical image in which light from an object is captured; a noise map generation unit that derives an evaluation value from the pixel values ​​of each pixel of the optical image based on relational data that represents the relationship between pixel values ​​and an evaluation value that evaluates the spread of noise values, and generates a noise map that is data in which the derived evaluation value is associated with each pixel of the optical image; and a processing unit that inputs the optical image and the noise map into a trained model that has been constructed in advance by machine learning, and performs image processing to remove noise from the optical image.

[0008] Alternatively, an optical image processing program according to another aspect of the embodiment causes a processor to function as an image acquisition unit that acquires an optical image in which light from an object is captured by irradiating the object with observation light; a noise map generation unit that derives an evaluation value from the pixel values ​​of each pixel of the optical image based on relational data that represents the relationship between pixel values ​​and evaluation values ​​that evaluate the spread of noise values, and generates a noise map that is data in which the derived evaluation values ​​are associated with each pixel of the optical image; and a processing unit that inputs the optical image and the noise map into a trained model that has been constructed in advance by machine learning, and performs image processing to remove noise from the optical image.

[0009] Alternatively, an optical image processing system according to another aspect of the embodiment includes the optical image processing module and an imaging device that captures an optical image by capturing light from an object.

[0010] According to one or more of the above aspects, an evaluation value is derived from the pixel values ​​of each image in the optical image based on relationship data representing the relationship between the pixel value and an evaluation value evaluating the spread of noise values, and a noise map is generated as data associating the derived evaluation value with each pixel of the optical image. The optical image and the noise map are then input into a trained model previously constructed by machine learning, and image processing is performed to remove noise from the optical image. With this configuration, noise in each pixel of the optical image is removed by machine learning, taking into account the spread of noise values ​​evaluated from the pixel values ​​of each pixel of the optical image. This makes it possible to achieve noise removal corresponding to the relationship between pixel values ​​and noise spread in the optical image using the trained model. As a result, noise in the optical image can be effectively removed.

[0011] A machine learning method according to another aspect of the embodiment includes a construction step of constructing, by machine learning, a trained model using training images of structures to which noise based on a predetermined noise distribution model has been added, the training images, a noise map generated from the training images based on relational data representing the relationship between pixel values ​​and an evaluation value evaluating the spread of noise values, and noise-removed image data, which is data obtained by removing noise from the training images, as training data. The trained model outputs noise-removed image data based on the training images and the noise map. The optical image processing module may also include a construction unit that constructs, by machine learning, a trained model using training images of structures to which noise based on a predetermined noise distribution model has been added, the training images, the noise map generated from the training images based on the relational data, and the noise-removed image data, which is data obtained by removing noise from the training images, as training data. According to the above configuration, a trained model that achieves noise removal corresponding to the relationship between pixel values ​​and the spread of noise can be constructed using optical images, which are training images, the noise map generated from the training images, and the noise-removed image data. As a result, noise in optical images of an object can be more effectively removed using the trained model.

[0012] A trained model according to another aspect of the embodiment is a trained model constructed by a machine learning method, and causes a processor to perform image processing to remove noise from an optical image of an object. This allows noise removal to be achieved using the trained model in accordance with the relationship between pixel values ​​and the extent of noise in the optical image. As a result, noise in the optical image can be effectively removed.

[0013] Furthermore, a preprocessing method for the machine learning method according to another aspect of the present invention includes a training image generation step of generating, as training images, structure images to which noise based on a noise distribution model has been added, and a noise map generation step of deriving an evaluation value from the pixel values ​​of each pixel of the structure image based on the relationship data and generating a noise map, which is data associating each pixel of the structure image with the derived evaluation value. According to this configuration, the noise map, which is training data for the machine learning method, corresponds to the relationship between pixel values ​​and an evaluation value evaluating the spread of pixel values ​​and noise values. Thus, when an optical image and a noise map generated from the optical image are input to a trained model constructed using the training images and noise map generated by the preprocessing method, noise removal corresponding to the relationship between pixel values ​​and the spread of noise can be achieved. As a result, noise in the optical image of the object can be more effectively removed. [Effects of the Invention]

[0014] According to one aspect and another aspect of this embodiment, it is possible to provide an optical image processing method, a machine learning method, a trained model, a machine learning preprocessing method, an optical image processing module, an optical image processing program, and an optical image processing system that can effectively remove noise in optical images. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram showing the functional configuration of an optical image processing system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of the optical image processing module of FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of input and output data of the trained model of FIG. 1. [Figure 4] FIG. 4 is a diagram illustrating an example of an optical image acquired by an image acquisition unit. [Figure 5] 10A and 10B are diagrams illustrating an example of a noise standard deviation map generated by a noise map generating unit. [Figure 6] 10 is a flowchart showing the procedure for creating training images included in teacher data used by the construction unit to construct a trained model. [Figure 7] 10 is a flowchart showing the procedure of observation processing by an optical image processing system including an optical image processing module. [Figure 8] 10A and 10B are diagrams illustrating examples of optical images acquired by an image acquisition unit before and after noise removal processing. [Figure 9] FIG. 10 is a block diagram showing the functional configuration of an optical image processing system according to a second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of input and output data of the trained model of FIG. [Figure 11] 10 is a flowchart showing the procedure of observation processing by an optical image processing system including an optical image processing module. [Figure 12] FIG. 10 is a diagram showing an example of a jig image used for evaluating the brightness-to-noise ratio. [Figure 13] 10A and 10B are diagrams illustrating an example of a noise standard deviation map generated by a noise map generating unit. [Figure 14] FIG. 10 is a block diagram showing a functional configuration of an optical image processing system according to a modified example. [Figure 15] 10A and 10B are diagrams showing examples of optical images before and after noise removal processing according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and duplicated explanations will be omitted. [First embodiment]

[0017] FIG. 1 is a block diagram showing the functional configuration of an optical image processing system 1 according to a first embodiment. As shown in FIG. 1, the optical image processing system 1 is a system that acquires an optical image of an object F based on light L from the object F. The light L includes light emitted from the object F, light transmitted through the object F, light reflected from the object F, and light scattered by the object F. Examples of the light L include ultraviolet light, visible light, and infrared light. Hereinafter, the light may be referred to as observation light. The optical image processing system 1 includes a camera (imaging device) 2, an optical image processing module 3, a display device 4, and an input device 5.

[0018] The camera 2 captures an optical image by capturing light L from an object F. The camera 2 has a photodetector 21 and an image control unit 22. The photodetector 21 is an imaging element having a plurality of pixels. Examples of the photodetector 21 include a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide Semiconductor) image sensor, a photodiode, an InGaAs sensor, a TDI (Time Delay Integration)-CCD image sensor, a TDI-CMOS image sensor, a pickup tube, an EM (Electron Multiplying)-CCD image sensor, an EB (Electron Bombarded)-CMOS image sensor, a SPAD (Single Photon Avalanche Diode, SPPC (Single-Pixel Photon Counter)), an MPPC (Multi-Pixel Photon Counter, SiPM (Silicon Photomultiplier)), an HPD (Hybrid Photo Detector), an APD (Avalanche Photodiode), and a photomultiplier tube (PMT (Photomultiplier Tube). The photodetector 21 may be a CCD image sensor, a CMOS image sensor, or the like, with an image intensifier (II) or an MCP (micro-channel chuck). The photodetector 21 may be a combination of a photodetector plate (a photodetector plate, a ...

[0019] The image control unit 22 executes image processing based on the digital signal from the photodetector 21. The image control unit 22 is configured with, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The image control unit 22 generates image data based on the digital signal received from the photodetector 21, performs predetermined image processing on the generated image data, and outputs the image data to the optical image processing module 3.

[0020] The optical image processing module 3 is a computer such as a PC (Personal Computer). The optical image processing module 3 performs image processing on image data output from the camera 2 to generate an optical image from which noise has been removed. The optical image processing module 3 is connected to the camera 2, the display device 4, and the input device 5 via wired or wireless communication so as to be able to communicate with each other. The generated optical image is subjected to a noise removal process (described later) and then output to the display device 4, where it is displayed. Various input information, such as imaging conditions for the object F, is input to the optical image processing module 3 from the input device 5 based on a user's operation. The optical image processing module 3 also controls the camera 2. Note that the optical image processing module 3 in the first embodiment is an independent device provided outside the camera 2, but may be integrated inside the camera 2. For example, the optical image processing module 3 may be a module equivalent to processing circuits implemented in the camera, such as a CPU and a GPU.

[0021] 2 shows the hardware configuration of the optical image processing module 3. As shown in FIG. 2, the optical image processing module 3 is physically a computer or the like including processors such as a CPU (Central Processing Unit) 101 and a GPU (Graphic Processing Unit) 105, storage media such as a RAM (Random Access Memory) 102 and a ROM (Read Only Memory) 103, a communication module 104, and an input / output module 106, all of which are electrically connected to one another. The optical image processing module 3 may include a display, keyboard, mouse, touch panel display, etc. as the display device 4 and input device 5, or may include a data storage device such as a hard disk drive or semiconductor memory. The optical image processing module 3 may also be composed of multiple computers.

[0022] As shown in FIG. 1, the optical image processing module 3 includes an input unit 31, an image acquisition unit 32, a noise map generation unit 33, a processing unit 34, and a construction unit 35. The functional units of the optical image processing module 3 shown in FIG. 1 are realized by loading a program (the optical image processing program of the first embodiment) onto hardware such as a CPU 101, a GPU 105, and a RAM 102. Under the control of the CPU 101 and the GPU 105, the communication module 104, the input / output module 106, and the like are operated, and data is read and written to the RAM 102. The CPU 101 and the GPU 105 of the optical image processing module 3 execute the computer program to cause the optical image processing module 3 to function as the functional units shown in FIG. 1 and sequentially execute processes corresponding to the optical image processing method described below. The CPU 101 and the GPU 105 may be standalone hardware, or either one of them may be used. The CPU 101 and the GPU 105 may also be implemented in programmable logic such as an FPGA, like a software processor. The RAM and ROM may be standalone hardware or may be embedded in programmable logic such as an FPGA. All of the various data required for executing this computer program and the various data generated by the execution of this computer program are stored in built-in memory such as the ROM 103 and RAM 102, or in a storage medium such as a hard disk drive. Furthermore, the built-in memory or storage medium in the optical image processing module 3 pre-stores multiple trained models 36 that are read by the CPU 101 and GPU 105 to cause the CPU 101 and GPU 105 to perform noise reduction processing on optical images. Details of the trained models 36 will be described later. Hereinafter, one trained model 36 may be described, but in such cases, the same applies to the other trained models 36.

[0023] Here, an overview of the optical image processing method of the optical image processing module 3 will be described using FIG. 3. FIG. 3 is a diagram showing an example of input / output data of the trained model 36 of FIG. 1. In the optical image processing module 3, a plurality of trained models 36 are constructed in the learning phase by machine learning, and in the noise removal phase, the trained models 36 are used to generate an optical image G6 in which noise has been removed from the optical image G1 of the object F. First, in the learning phase, the optical image processing module 3 creates a structure image (optical image) Gc, which is an image of a structure having a predetermined structure, and generates training images Gt, which serve as teacher data, based on the structure image Gc and a noise distribution model (described in detail below). Then, the optical image processing module 3 constructs the trained model 36 by machine learning using training data including the training images Gt. In the noise removal phase, the optical image processing module 3 first acquires condition information. The condition information indicates imaging conditions, including the type of photodetector 21, when imaging the object F. The optical image processing module 3 derives a relation graph G3 or the like showing a relational expression (relational data) between pixel values ​​and the standard deviation of noise values ​​(an evaluation value that evaluates the spread of noise values) based on the optical image G1 and the imaging conditions, etc., and generates a noise standard deviation map (noise map) G5. Then, the optical image processing module 3 inputs the optical image G1 and the noise standard deviation map G5 to a trained model 36, and performs image processing to remove noise from the optical image, thereby generating and outputting an optical image G6 from which noise has been removed.

[0024] The function of each functional unit of the optical image processing module 3 will be described in detail below.

[0025] The input unit 31 accepts input of condition information. Specifically, the input unit 31 accepts input of condition information indicating imaging conditions, etc., used by the camera 2 when capturing an optical image of the object F from a user of the optical image processing system 1. The condition information includes at least one of photodetector information, a gain setting value, a shading correction coefficient, an offset, a noise factor, information indicating dark current noise generated by thermal noise in the photodetector 21, and information indicating a readout noise value in the photodetector 21. The photodetector information is information indicating the type of photodetector 21 used to capture an image of the object F. Examples of the photodetector information include information indicating any of a CCD image sensor, a CMOS image sensor, a photodiode, an InGaAs sensor, a TDI-CCD image sensor, a TDI-CMOS image sensor, an image pickup tube, an EM-CCD image sensor, an EB-CMOS image sensor, a SPAD, an MPPC, a HPD, an APD, and a photomultiplier tube. The input unit 31 may accept the input of the condition information as a direct input of information such as numerical values, or as a selective input of information such as numerical values ​​previously set in an internal memory. The input unit 31 accepts the input of the above condition information from the user, but may also acquire some of the condition information (such as the type of photodetector 21) according to the detection result of the control state by the optical image processing module 3.

[0026] The image acquisition unit 32 acquires an optical image obtained by capturing light from the object F. Specifically, the image acquisition unit 32 acquires an optical image output from the camera 2. FIG. 4 is a diagram showing an example of an optical image G1 acquired by the image acquisition unit 32.

[0027] The noise map generation unit 33 derives an evaluation value from the pixel value of each pixel of the optical image based on relational data representing the relationship between the pixel value and an evaluation value that evaluates the spread of noise values, and generates a noise map. The noise map is data that associates the derived evaluation value with each pixel of the optical image. At this time, the noise map generation unit 33 derives the evaluation value from the imaging conditions and the pixel value of each pixel of the optical image. In this embodiment, first, the noise map generation unit 33 selects one relational equation (relational data) from multiple relational equations based on the imaging conditions included in the condition information acquired by the input unit 31. Then, the noise map generation unit 33 uses the selected relational equation to derive the standard deviation of noise values ​​from the pixel values ​​of each pixel of the optical image acquired by the image acquisition unit 32. Then, the noise map generation unit 33 generates a noise standard deviation map by associating the derived standard deviation of noise values ​​with each pixel of the optical image.

[0028] Here, the process of selecting a relational equation by the noise map generator 33 will be described. The noise map generator 33 selects one relational equation from among a plurality of relational equations based on the photodetector information included in the condition information. That is, the noise map generator 33 selects the relational equation that is most suitable for the photodetector 21 depending on the type of the photodetector 21. In this embodiment, the noise map generator 33 selects one relational equation from the following three relational equations:

[0029] If the photodetector 21 is not an electron multiplying type, the noise map generator 33 selects the following formula (1) as the relational expression. As an example, if the photodetector information indicates any of a CCD image sensor, a CMOS image sensor, a photodiode, an InGaAs sensor, a TDI-CCD image sensor, a TDI-CMOS image sensor, and an image pickup tube without a multiplication mechanism, the noise map generator 33 selects the following formula (1) as the relational expression.

[0030]

number

[0031] When the above formula (1) is used, the noise map generating unit 33 substitutes the pixel value of each pixel of the optical image acquired by the image acquiring unit 32 into the variable Signal. Then, the noise map generating unit 33 obtains the variable Noise calculated using the above formula (1) as a numerical value of the standard deviation of the noise values. Note that the other parameters in the above formula (1) may be acquired by receiving input from the input unit 31, or may be set in advance.

[0032] FIG. 5 is a diagram illustrating an example of a noise standard deviation map generated by the noise map generator 33. The noise map generator 33 uses the relational expression (1) between pixel values ​​and the standard deviation of noise values ​​to substitute various pixel values ​​for the variable Signal to obtain the correspondence between pixel values ​​and the variable Noise, thereby deriving a relationship graph G3 representing the correspondence between pixel values ​​and the standard deviation of noise values. The noise map generator 33 then derives relationship data G2 representing the correspondence between each pixel position and pixel value from the optical image G1 acquired by the image acquisition unit 32. Furthermore, the noise map generator 33 applies the correspondence indicated by the relationship graph G3 to each pixel value in the relationship data G2 to derive the standard deviation of noise values ​​corresponding to each pixel position in the optical image. As a result, the noise map generator 33 associates the derived noise standard deviation with each pixel position and derives relationship data G4 indicating the correspondence between each pixel position and the noise standard deviation. Then, the noise map generating unit 33 generates a noise standard deviation map G5 based on the derived relational data G4.

[0033] When the photodetector 21 is of an electron multiplying type and not of a photon counting type, the noise map generator 33 selects the following formula (2) as the relational expression. As an example, when the photodetector information indicates any of an EM-CCD image sensor, an EB-CMOS image sensor, a SPAD, a HPD, an APD, a photomultiplier tube, and an MPPC, the noise map generator 33 selects the following formula (2) as the relational expression.

[0034]

number

[0035] When the photodetector 21 is of an electron multiplying type and a photon counting type, the noise map generator 33 selects the following formula (3) as the relational expression. As an example, when the photodetector operates for the purpose of photon counting using a photomultiplier tube, HPD, MPPC, etc., the noise map generator 33 selects the following formula (3) as the relational expression.

[0036]

number

[0037] The processing unit 34 inputs the optical image and the noise map into a trained model 36 previously constructed by machine learning, and performs image processing to remove noise from the optical image. That is, as shown in FIG. 3 , the processing unit 34 acquires the trained model 36 constructed by the construction unit 35 from an internal memory or a storage medium in the optical image processing module 3. In this embodiment, the processing unit 34 acquires a trained model 36 corresponding to the type of the photodetector 21 from among the multiple trained models 36. Then, the processing unit 34 inputs the optical image G1 acquired by the image acquisition unit 32 and the noise standard deviation map G5 generated by the noise map generation unit 33 into the trained model 36. As a result, the processing unit 34 performs image processing to remove noise from the optical image G1 using the trained model 36, thereby generating an optical image G6 from which noise has been removed. The processing unit 34 then outputs the generated optical image G6 to the display device 4 or the like.

[0038] The construction unit 35 uses, as training images, structural images to which noise based on a predetermined noise distribution model has been added. The construction unit 35 uses, as training data, the training images, a noise map generated from the training images based on a relationship between pixel values ​​and the standard deviation of noise values, and noise-removed image data, which is data obtained by removing noise from the training images, to construct a trained model 36 by machine learning. The trained model 36 outputs noise-removed image data based on the training images and the noise map. In this embodiment, the construction unit 35 constructs the trained model 36 according to the type of photodetector 21. The construction unit 35 then stores each constructed trained model 36 in an internal memory or a storage medium within the optical image processing module 3. Machine learning includes supervised learning, unsupervised learning, and reinforcement learning, and includes deep learning, neural network learning, and the like. In the first embodiment, a two-dimensional convolutional neural network described in the paper “Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising” by Kai Zhang et al. is used as an example of a deep learning algorithm. Each trained model 36 may be constructed by the construction unit 35, or may be generated by an external computer or the like and downloaded to the optical image processing module 3. The optical images used for machine learning include optical images of known structures or images that reproduce the optical images. The training images may be images actually generated for multiple types of known structures, or may be images generated by simulation calculations.

[0039] As a preprocessing step for machine learning, the construction unit 35 generates, as training images, structure images to which noise based on a noise distribution model has been added. Then, the construction unit 35 derives an evaluation value from the pixel value of each pixel of the optical image based on relational data representing the relationship between the pixel value and an evaluation value that evaluates the spread of the noise value, and generates a noise map, which is data that associates the derived evaluation value with each pixel of the optical image.

[0040] Specifically, when constructing each trained model 36, the construction unit 35 acquires condition information, including photodetector information used during simulation calculation, from the input unit 31. The construction unit 35 then generates a structure image. The construction unit 35 then adds noise to the structure image based on a noise distribution model selected based on the photodetector information. The construction unit 35 then generates a noise standard deviation map based on training images using a method similar to that used by the noise map generation unit 33 shown in FIG. 5 . That is, the machine learning preprocessing method includes an input step of accepting input of condition information, including photodetector information indicating the type of photodetector 21 used to capture the object F; a training image generation step of generating, as training images, structure images to which noise based on the noise distribution model has been added; and a noise map generation step of deriving an evaluation value from the pixel values ​​of each pixel of the optical image based on relational data indicating the relationship between the pixel value and an evaluation value that evaluates the spread of the noise value, and generating a noise map, which is data associating each pixel of the optical image with the derived evaluation value. In the training image generation step, the noise distribution model to be used is determined from the photodetector information.

[0041] The construction unit 35 constructs each trained model 36 through machine learning using training data prepared for each trained model 36. Specifically, first, the construction unit 35 acquires in advance noise-removed image data in which noise has been removed from training images. The construction unit 35 sets images before noise is added in the process of generating the training images as noise-removed image data. The construction unit 35 performs training through machine learning to construct a trained model 36 that outputs noise-removed image data based on the training images and the noise standard deviation map.

[0042] FIG. 6 is a flowchart showing the procedure for creating training images included in teacher data (training data) used by the construction unit 35 to construct the trained model 36.

[0043] A training image (also referred to as a teacher image), which is teacher data, is created in the following procedure. First, the construction unit 35 generates a structure image (step S101). The construction unit 35 may create the structure image, for example, by simulation calculation. Next, for one pixel selected from multiple pixels constituting the structure image, a sigma value, which is the standard deviation of pixel values, is calculated (step S102). The sigma value calculated in step S102 indicates the magnitude of noise. Similar to the method for generating a noise map described above, the construction unit 35 selects an appropriate relational expression from among the above formulas (1), (2), and (3) based on the photodetector information. Then, using the selected relational expression, the construction unit 35 substitutes the pixel value of the pixel of the structure image for the variable Signal to calculate the variable Noise of the pixel, and obtains the calculated variable Noise of the pixel as the magnitude of noise (sigma value).

[0044] Then, the constructing unit 35 sets a noise distribution model based on the sigma value calculated in step S102 (step S103). The constructing unit 35 acquires condition information from the input unit 31, and sets a noise distribution model according to the photodetector information included in the condition information.

[0045] The noise distribution model includes a normal distribution model, a Poisson distribution model, and a Bessel function distribution model. The condition information further includes information indicating the amount of light L. The construction unit 35 references the photodetector information, and if the photodetector 21 is not an electron multiplier and the amount of light L is not small, the construction unit 35 sets the normal distribution model as the noise distribution model. If the photodetector 21 is not an electron multiplier and the amount of light L is small, the construction unit 35 sets the Poisson distribution model as the noise distribution model. For example, if the photodetector information indicates a CCD image sensor, a CMOS image sensor, a photodiode, an InGaAs sensor, a TDI-CCD image sensor, a TDI-CMOS image sensor, or a pickup tube without a multiplication mechanism and the amount of light is equal to or greater than a predetermined reference value, the construction unit 35 sets the normal distribution model as the noise distribution model. As an example, the construction unit 35 references the condition information and, if the photodetector information indicates a CCD image sensor, a CMOS image sensor, a photodiode, an InGaAs sensor, a TDI-CCD image sensor, a TDI-CMOS image sensor, or an image pickup tube and the light intensity is less than a reference value, sets the Poisson distribution model as the noise distribution model. The noise distribution model may include only one of the normal distribution model and the Poisson distribution model. On the other hand, if the photodetector 21 has a multiplication factor of 2 per stage and is a multi-stage electron multiplication type, the construction unit 35 sets the Bessel function distribution model as the noise distribution model. As an example, if the photodetector information indicates an EM-CCD image sensor, the construction unit 35 sets the Bessel function distribution model as the noise distribution model. In this way, by setting the normal distribution model or the Bessel function distribution model, training data for various noise conditions can be generated. If the photodetector information does not correspond to any of the above photodetectors, the construction unit 35 may create a new noise distribution model by calculating a histogram and creating a function representing the noise distribution. The histogram is a histogram of pixel values ​​of an optical image when light having the same light amount is incident on the photodetector 21. The constructing unit 35 calculates the histogram by, for example, acquiring a plurality of optical images in which a light source whose light amount does not change over time is captured.As an example, the horizontal axis of the histogram represents the luminance value of the camera 2, and the vertical axis of the histogram represents the frequency. Since the noise distribution changes depending on the amount of light, the constructor 35 further obtains multiple histograms by changing the amount of light from the light source within the range of light amounts that can be assumed when the optical image processing system 1 is in use, and creates a noise distribution model.

[0046] Next, the construction unit 35 calculates a randomly set noise value based on the noise magnitude (sigma value) acquired in step S102 and the noise distribution model set based on the sigma value in step S103 (step S104). Next, the construction unit 35 generates pixel values ​​constituting a training image, which is teacher data, by adding the noise value calculated in step S104 to the pixel value of one pixel (step S105). The construction unit 35 performs the processes of steps S102 to S105 on each of the multiple pixels constituting the structure image (step S106) to generate a training image, which is teacher data (step S107). Furthermore, if more training images are needed, the construction unit 35 determines that the processes of steps S101 to S107 should be performed on another structure image (step S108) to generate another training image, which is teacher data. Note that the other structure image may be an image of a structure having the same structure or may be an image of a structure having a different structure.

[0047] It is necessary to prepare a large number of training images, which are teacher data used to build the trained model 36. Furthermore, structure images should preferably be images with little noise, and ideally, images without noise. Therefore, generating structure images through simulation calculations is effective because it allows the generation of many noise-free images.

[0048] Next, the procedure for observing an optical image of the object F using the optical image processing system 1 according to the first embodiment, i.e., the flow of the optical image acquisition method according to the first embodiment, will be described. Fig. 7 is a flowchart showing the procedure for observing the optical image processing system 1 including the optical image processing module 3.

[0049] First, the construction unit 35 constructs a trained model 36 by machine learning using training images, a noise standard deviation map generated from the training images based on the relational expression, and noise-removed image data as training data, and outputs noise-removed image data based on the training images and the noise standard deviation map (step S200). In this embodiment, a plurality of trained models 36 are constructed. Next, the input unit 31 accepts input of condition information indicating imaging conditions and the like from an operator (user) of the optical image processing system 1 (step S201).

[0050] Next, the object F is set in the optical image processing system 1, the object F is imaged, and the optical image processing module 3 acquires an optical image of the object F (step S202). Furthermore, the optical image processing module 3 derives the standard deviation of the noise values ​​from the pixel values ​​of each pixel in the optical image based on the relational expression between the pixel values ​​and the standard deviation of the noise values, and associates the derived noise standard deviation with each pixel value to generate a noise standard deviation map (step S203).

[0051] Next, the processing unit 34 inputs the optical image of the object F and the noise standard deviation map into the trained model 36 that has been constructed and stored in advance, and executes a noise removal process on the optical image (step S204). Furthermore, the processing unit 34 outputs the optical image that has been subjected to the noise removal process to the display device 4 (step S205).

[0052] According to the optical image processing module 3 described above, the standard deviation of noise values ​​is derived from the pixel values ​​of each image in the optical image using a relational expression (relational data) that expresses the relationship between pixel values ​​and the standard deviation of noise values ​​that evaluate the spread of noise values, and a noise standard deviation map is generated, which is data that associates the standard deviation of the derived noise values ​​with each pixel of the optical image. The optical image and the noise standard deviation map are then input to a trained model 36 that has been constructed in advance by machine learning, and image processing is performed to remove noise from the optical image. This makes it possible to use the trained model 36 to achieve noise removal that corresponds to the relationship between pixel values ​​and noise spread in the optical image. As a result, noise in the optical image can be effectively removed.

[0053] In particular, the noise characteristics of optical images vary depending on the type of photodetector 21, the gain setting value, the readout mode, and the like. Therefore, when attempting to achieve noise removal through machine learning, it is conceivable to prepare learning models trained under various conditions. In such a case, a learning model must be constructed for each noise condition, such as the type of photodetector 21, the gain setting value, and the readout mode. This requires the generation of a huge number of learning models, which may require a significant amount of time. In this regard, according to the present embodiment, by generating a noise map from an optical image and using the noise map as input data for machine learning, the noise conditions required to generate a trained model 36 can be reduced, and the learning time required to construct the trained model 36 can be significantly reduced.

[0054] Here, an example of the effect of the noise removal process by the optical image processing module 3 of the first embodiment will be described. For example, in an example where a CMOS image sensor (C13440-20 ORCA (trademark)-Flash 4.0 V3 manufactured by Hamamatsu Photonics KK) is used as the photodetector 21 and visible light is irradiated onto the object F as the observation light, the standard deviation of noise in the optical image G1 (see FIG. 3) is 3.31, and the standard deviation of noise in the optical image G6 is 0.48. In an example where a CMOS image sensor (C14440-20 ORCA (trademark)-Fusion manufactured by Hamamatsu Photonics KK) different from the above CMOS image sensor is used as the photodetector 21 and visible light is irradiated onto the object F as the observation light, the standard deviation of noise in the optical image G1 is 6.91, and the standard deviation of noise in the optical image G6 is 0.79. In an example in which a CMOS image sensor (C15440-20 ORCA (trademark)-FusionBT manufactured by Hamamatsu Photonics K.K.) different from the above two CMOS image sensors was used as the photodetector 21 and visible light was irradiated onto the object F as the observation light, the standard deviation of noise in the optical image G1 was 6.91 and the standard deviation of noise in the optical image G6 was 0.69. In an example in which an InGaAs sensor (C12741-03 InGaAs camera manufactured by Hamamatsu Photonics K.K.) was used as the photodetector 21 and infrared light was irradiated onto the object F as the observation light, the standard deviation of noise in the optical image G1 was 7.54 and the standard deviation of noise in the optical image G6 was 1.53. Note that in each of the above examples, a photodetector 21 that is not an electron multiplier is used, and therefore a normal distribution model is set as the noise distribution model.

[0055] In an example where an EM-CCD image sensor (Hamamatsu Photonics C9100-23B ImagEM (registered trademark) X2 EM-CCD camera) was used as the photodetector 21, the amplification factor was 300, and visible light was irradiated as the observation light onto the object F, the following results were obtained. Specifically, when the digital output value was 2200 (counts), the standard deviation of noise in optical image G1 was 41.5, and the standard deviation of noise in optical image G6 was 5.66. Furthermore, when the digital output value was 2500 (counts), the standard deviation of noise in optical image G1 was 44.1, and the standard deviation of noise in optical image G6 was 7.74. Furthermore, in an example where the amplification factor was 1200 under the above conditions, the following results were obtained. Specifically, when the digital output value was 2200 (counts), the standard deviation of noise in optical image G1 was 86.9, and the standard deviation of noise in optical image G6 was 13.5. Furthermore, when the digital output value was 2500 (counts), the standard deviation of noise in optical image G1 was 91.5, and the standard deviation of noise in optical image G6 was 15.7. In each of the above examples, an electron multiplying photodetector 21 was used, and therefore a Bessel function distribution model was set as the noise distribution model. FIG. 8 shows optical images G1 and G6 obtained when a SPAD sensor was used as the photodetector 21 and visible light was irradiated onto object F as observation light. The standard deviation of noise in optical image G1 was 30, and the standard deviation of noise in optical image G6 was 5.8.

[0056] As shown in the above examples, according to the optical image processing module 3 of the first embodiment, by creating a noise map corresponding to the actually measured noise value, it is possible to obtain an optical image G6 in which the noise in the optical image G1 has been effectively removed.

[0057] The optical image processing module 3 of the first embodiment includes an input unit 31 that accepts input of condition information indicating the imaging conditions when imaging the object F, and a noise map generation unit 33 derives the standard deviation of noise values ​​from the imaging conditions and the pixel values ​​of each pixel in the optical image, and the condition information includes information indicating the type of photodetector 21 used to image the object F. The relationship between pixel values ​​and noise in the optical image varies depending on the type of photodetector 21 used to image the object F. However, with the above configuration, the type of photodetector 21 used to image the object F is taken into consideration to evaluate the spread of noise values ​​in the pixel values ​​of each pixel in the optical image, thereby achieving noise removal that corresponds to the relationship between pixel values ​​and the spread of noise in the optical image. As a result, noise in the optical image can be removed more effectively.

[0058] In the optical image processing module 3 of the first embodiment, the spread of noise values ​​is evaluated as the standard deviation of the noise values. This allows the spread of noise values ​​in the pixel values ​​of each pixel of the optical image to be evaluated more accurately, making it possible to achieve noise removal that corresponds to the relationship between pixel values ​​and noise. As a result, noise in the optical image can be removed more effectively.

[0059] The optical image processing module 3 of the first embodiment includes a construction unit 35 that constructs, by machine learning, a trained model 36 that outputs noise-removed image data based on the training image and the noise standard deviation map, using training data in which structure images to which noise based on a predetermined noise distribution model has been added are used as training images, a noise standard deviation map generated from the training images based on relational data, and noise-removed image data, which is data in which noise has been removed from the training images. According to the above configuration, a trained model 36 that achieves noise removal corresponding to the relationship between pixel values ​​and noise spread can be constructed using optical images that are training images, the noise map generated from the images, and the noise-removed image data. As a result, noise in the optical image of the object F can be more effectively removed using the trained model 36.

[0060] The optical image processing module 3 of the first embodiment has a machine learning preprocessing function that generates training images of structures to which noise based on a noise distribution model has been added, derives the standard deviation of noise values ​​from the pixel values ​​of each pixel in the structure images based on relational data, and generates a noise standard deviation map, which is data associating the derived standard deviation of noise values ​​with each pixel in the optical image. With this configuration, the noise standard deviation map, which is training data for the machine learning method, corresponds to the relationship between pixel values ​​and the standard deviation of noise values ​​that evaluate the spread of pixel values ​​and noise values. Thus, when the optical images and the noise standard deviation map generated from the optical images are input to the trained model 36 constructed using the training images and noise map generated by the preprocessing method, noise removal corresponding to the relationship between pixel values ​​and the spread of noise can be achieved. As a result, noise in the optical image of the object F can be more effectively removed.

[0061] The optical image processing module 3 of the first embodiment has a function of accepting input of condition information including photodetector information indicating the type of photodetector 21 used to image the object F, and a function of determining a noise distribution model to be used from the photodetector information in the process of generating training images. The relationship between pixel values ​​and noise in an optical image varies depending on the type of photodetector 21 used to image the object F, but with the above configuration, it is possible to obtain training images in which noise is appropriately added to structure images, taking into account the type of photodetector 21 used to image the object F.

[0062] In the optical image processing module 3 of the first embodiment, the noise distribution model includes a normal distribution model and a Poisson distribution model. This makes it possible to obtain training images in which noise is appropriately added to structural images, for example, when a general photodetector 21 that is not an electron multiplying type is used to capture an image of the object F. In particular, because the noise distribution model further includes a Poisson distribution model in addition to the normal distribution model, it is possible to obtain training images in which noise is appropriately added to structural images, even when the amount of light L is low.

[0063] In the optical image processing module 3 of the first embodiment, the noise distribution model includes a Bessel function distribution model. This makes it possible to obtain a training image in which noise is appropriately added to a structure image, for example, when an electron multiplying photodetector 21 is used to capture an image of the object F.

[0064] When an electron multiplying photodetector 21 is used, the noise distribution changes according to the multiplication fluctuation that occurs during multiplication. In the first embodiment, a Bessel function distribution model is applied when the photodetector 21 is an electron multiplying type, so that training images with appropriate noise added can be generated. [Second embodiment]

[0065] FIG. 9 is a block diagram showing the functional configuration of an optical image processing system 1A according to a second embodiment. FIG. 10 is a diagram showing an example of input / output data of the trained model 36 shown in FIG. 9. The optical image processing module 3A according to the second embodiment differs from the first embodiment in that the image acquisition unit 32A has a function of acquiring an optical image of the jig, and the noise map generation unit 33A has a function of deriving a graph showing the relationship between pixel values ​​and the standard deviation of noise values ​​from the optical image of the jig. Specifically, as shown in FIG. 10, in the noise removal phase, the optical image processing module 3A acquires a jig image G26, which is an optical image of the jig. Based on the jig image G26, the optical image processing module 3A plots the relationship between the true pixel values ​​and the SNR of each of the multiple pixels included in the jig image G26 on a graph G28 and draws an approximate curve to derive a relationship graph showing the relationship between pixel values ​​and the standard deviation of noise values, thereby generating a noise standard deviation map G5.

[0066] Fig. 11 is a flowchart showing the procedure of observation processing by the optical image processing system 1A including the optical image processing module 3A of Fig. 10. As shown in Fig. 11, in the optical image processing module 3A according to the second embodiment, the processing shown in steps S301 and S302 is executed by replacing the processing of steps S201 and S203 by the optical image processing module 3 of the first embodiment shown in Fig. 7.

[0067] The image acquisition unit 32A acquires an optical image of the jig by capturing light from the jig (step S301). Specifically, the image acquisition unit 32A acquires an optical image capturing light from the jig using the camera 2. The light from the jig includes light emitted from the jig, light transmitted through the jig, light reflected from the jig, and light scattered from the jig. As shown in FIG. 12, a jig having a grayscale chart capable of evaluating gradation performance using gradually changing density steps is used as the jig. That is, the image acquisition unit 32A acquires a jig image G26 captured using the camera 2 prior to the observation process of the object F. Then, the image acquisition unit 32A acquires an optical image of the object F captured using the camera 2. However, the timing of acquiring the optical images of the jig and the object F is not limited to the above, and they may be acquired simultaneously or at opposite times.

[0068] Based on the optical image of the jig obtained as a result of capturing an image of the jig, the noise map generating unit 33A derives relational data representing the relationship between pixel values ​​and evaluation values ​​that evaluate the spread of noise values ​​(step S302). Specifically, the noise map generating unit 33A derives a noise standard deviation map representing the relationship between pixel values ​​and the standard deviation of noise values ​​from the optical image of the jig.

[0069] FIG. 13 is a diagram illustrating an example of a noise standard deviation map generated by the noise map generation unit 33A. The noise map generation unit 33A derives a relationship graph G27 representing the relationship between pixel values ​​and the standard deviation of noise values ​​by plotting the relationship between true pixel values ​​and SNR for each of multiple measurement regions with different densities included in the jig image G26 on graph G28 (see FIG. 10) and drawing an approximate curve. Specifically, the noise map generation unit 33A selects multiple measurement regions with different densities, analyzes the standard deviation and average luminance values ​​of the multiple measurement regions, and obtains a luminance-SNR (signal-to-noise ratio) characteristic graph as graph G28. At this time, the noise map generation unit 33A calculates the SNR for each measurement region by SNR = (average luminance value) ÷ (standard deviation of luminance value). Then, similar to the first embodiment, the noise map generation unit 33A derives relationship data G2 representing the correspondence between each pixel position and pixel value from the optical image G1 acquired by the image acquisition unit 32A. Furthermore, the noise map generation unit 33A derives the standard deviation of the noise value corresponding to each pixel position in the optical image by applying the correspondence indicated by the relationship graph G27 to each pixel in the relationship data G2. As a result, the noise map generation unit 33A associates the derived noise standard deviation with each pixel position and derives relationship data G4 indicating the correspondence between each pixel position and the noise standard deviation. Then, the noise map generation unit 33A generates a noise standard deviation map G5 based on the derived relationship data G4.

[0070] In the optical image processing module 3A of the second embodiment, the image acquisition unit 32 acquires an optical image of the jig in which light from the jig (e.g., light emitted from the jig, light transmitted through the jig, light reflected from the jig, or light scattered from the jig) is captured, and the noise map generation unit 33A derives relational data from the optical image of the jig. According to the above configuration, relational data is generated based on an optical image obtained by actually capturing an image of the jig, and a noise standard deviation map is generated, thereby realizing noise removal corresponding to the relationship between pixel values ​​and noise spread. As a result, noise in the optical image can be removed more effectively.

[0071] The image acquisition unit 32A acquires a plurality of optical images captured in the absence of the object F, and the noise map generation unit 33A derives the relational data from the plurality of optical images, which may be images captured under different imaging conditions. According to the above configuration, the relational data is generated based on optical images actually captured, and a noise standard deviation map is generated, thereby achieving noise removal corresponding to the relationship between pixel values ​​and noise spread. As a result, noise in optical images can be removed more effectively.

[0072] Specifically, in step S301 described above, the image acquisition unit 32A may acquire multiple optical images captured in the absence of the object F, and in step S302 described above, the noise map generation unit 33A may derive the relationship between pixel values ​​and the standard deviation of noise values ​​from the optical images acquired by the image acquisition unit 32A. The multiple optical images are images captured under different imaging conditions. As an example, the image acquisition unit 32A acquires multiple optical images captured using the camera 2 in the absence of the object F prior to the observation process of the object F, while changing the light intensity of the observation light source or the exposure time of the camera 2. The noise map generation unit 33A then derives true pixel values ​​for each optical image and, similar to the second embodiment, derives the standard deviation of noise based on the true pixel values. Furthermore, similar to the second embodiment, the noise map generation unit 33A derives a relationship graph representing the relationship between pixel values ​​and the standard deviation of noise values ​​by plotting the relationship between the true pixel values ​​and the standard deviation of noise on a graph and drawing an approximation curve. Finally, the noise map generating unit 33A generates a noise standard deviation map from the optical image acquired by the image acquiring unit 32A based on the derived relationship graph, in the same manner as in the second embodiment. [Variations]

[0073] Although various embodiments of the present disclosure have been described above, the embodiments of the present disclosure are not limited to the above embodiments. For example, the construction unit 35 may generate training images by actually capturing images when constructing each trained model 36. In other words, the training images may be optical images actually generated using the camera 2 of multiple types of known structures.

[0074] The optical image processing system 1 may be a scanning type. The example shown in FIG. 14 differs from the above-described embodiments in that the optical image processing system 1 includes a confocal microscope 2B. The confocal microscope 2B acquires images that enable the construction of an optical tomographic image of an object F. The confocal microscope 2B is configured by connecting a confocal microscope unit 6 to a connection port P1 of a microscope 7 for connecting an external unit. The confocal microscope unit 6 is a device that irradiates an object F placed on a stage of the microscope 7 with excitation light via a microscope optical system, such as an imaging lens 71 and an objective lens 72, within the microscope 7, and receives (detects) fluorescence (light), which is emitted from the object F in response to the excitation light, via the microscope optical system of the microscope 7, to generate and output an optical tomographic image.

[0075] In detail, the confocal microscope unit 6 includes a main housing 61, a lens barrel 62, a scan mirror 63 fixed inside the main housing 61, a fixed mirror 64, a subunit 65, and a scan lens 66 fixed inside the lens barrel 62. The lens barrel 62 forms part of the main housing 61, and is detachably connected to a connection port P1 of the microscope 7. Each component of the confocal microscope unit 6 will be described in detail below.

[0076] The scan lens 66 in the lens barrel 62 is an optical element that relays the reflecting surface of the scan mirror 63 to the pupil position of the objective lens 72, and at the same time focuses the excitation light (observation light) on a primary image plane of the microscope optical system of the microscope 7. The scan lens 66 guides the excitation light scanned by the scan mirror 63 to the microscope optical system, thereby irradiating the object F, and guides the fluorescence (observation light) generated from the object F in response to this to the scan mirror 63. In detail, the scan lens 66 is configured to image the pupil of the objective lens 72 on the scan mirror 63, and guides the fluorescence imaged by the objective lens 72 and imaging lens 71 of the microscope 7 to the scan mirror 63.

[0077] The scan mirror 63 in the main housing 61 is an optical scanning element such as a MEMS (Micro Electro Mechanical System) mirror configured so that a reflector can be tilted on two axes. The scan mirror 63 has the role of scanning the excitation light output from the subunit 65 onto the object F by continuously changing the reflection angle, and guiding the fluorescence generated in response to the excitation light toward the subunit 65.

[0078] The fixed mirror 64 is a light reflecting element fixed within the main housing 61, which reflects the excitation light output from the subunit 65 toward the scan mirror 63 and reflects the fluorescence reflected by the scan mirror 63 toward the subunit 65 coaxially with the excitation light.

[0079] Subunit 65 includes a base plate 651, a total reflection mirror 652 arranged on base plate 651, a light source 653, a dichroic mirror 654, a pinhole plate 655, and a photodetector 656. Total reflection mirror 652 reflects the first excitation light having a wavelength λ1 emitted by subunit 65 and the first fluorescence having a wavelength range Δλ1 generated from object F in response to the first excitation light. Dichroic mirror 654 is a beam splitter that is disposed in the reflection direction of the first fluorescence from total reflection mirror 652 and has the property of transmitting the first fluorescence having a wavelength range Δλ1 and reflecting the first excitation light having a wavelength λ1 shorter than the wavelength range Δλ1. Light source 653 is a light-emitting element (e.g., a laser diode) that outputs first excitation light (e.g., laser light) having a wavelength λ1, and is arranged so that the first excitation light is reflected by dichroic mirror 654 toward total reflection mirror 652 coaxially with the first fluorescence. The pinhole plate 655 is an aperture member that limits the light flux of the first fluorescence and is arranged so that the pinhole position coincides with the conjugate position of the spot of the first excitation light on the object F, and constitutes a confocal optical system together with the light source 653, etc. The pinhole diameter of the pinhole plate 655 can be adjusted externally, making it possible to change the resolution and signal intensity of the image detected by the photodetector 656.

[0080] The photodetector 656 has its detection surface facing the pinhole plate 655 and receives and detects the first fluorescence that has passed through the pinhole plate 655. Examples of the photodetector 656 include photodetectors similar to those in the above-described embodiments (e.g., a CCD image sensor, a CMOS image sensor, etc.). The subunit 65 captures light from the object F using the photodetector 656 and outputs a digital signal based on the image capture result to the optical image processing module 3. The image acquisition unit 32 of the optical image processing module 3 acquires an optical image by generating image data based on the digital signal received from the subunit 65. As in the above-described embodiments, the configuration of this modified example also uses the trained model 36 to achieve noise removal corresponding to the relationship between pixel values ​​and noise spread in the optical image acquired from the confocal microscope 2B, thereby effectively removing noise from the optical image. FIG. 15 shows optical images G1 and G6 obtained when a PMT is used as the photodetector 21 and visible light is irradiated onto the object F as observation light. In this way, even with the optical image processing module 3 of this modified example, by creating a noise map corresponding to the actually measured noise value, it is possible to obtain an optical image G6 in which the noise in the optical image G1 has been effectively removed.

[0081] In the above embodiment, the construction unit 35 constructs the trained model 36 depending on the type of photodetector 21. However, the trained model 36 may be a common trained model regardless of the type of photodetector 21. When constructing a common trained model 36, for example, the construction unit 35 may set a normal distribution model as the noise distribution model, in which case, it is not necessary to acquire condition information. Furthermore, for example, when the amount of light L is small, the construction unit 35 may set a Poisson distribution model as the noise distribution model, in which case, it may acquire information indicating the amount of light L as the condition information.

[0082] The optical image processing method may further include an input step of accepting input of condition information indicating imaging conditions when imaging the object. In the noise map generation step, an evaluation value may be derived from the imaging conditions and the pixel values ​​of each pixel in the optical image. The condition information may include information indicating the type of photodetector used to image the object. The optical image processing module may further include an input unit that accepts input of condition information indicating imaging conditions when imaging the object. The noise map generation unit may derive an evaluation value from the imaging conditions and the pixel values ​​of each pixel in the optical image. The condition information may include information indicating the type of photodetector used to image the object. The relationship between pixel values ​​and noise in the optical image varies depending on the type of photodetector used to image the object. According to the above configuration, the type of photodetector used to image the object is taken into consideration when evaluating the spread of noise values ​​in the pixel values ​​of each pixel in the optical image, thereby achieving noise removal that corresponds to the relationship between pixel values ​​and noise spread in the optical image. As a result, noise in the optical image can be more effectively removed.

[0083] The image acquisition step may acquire an optical image of the jig in which light from the jig is captured, and the noise map generation step may derive the relational data from the optical image of the jig. Alternatively, the image acquisition unit may acquire an optical image of the jig in which light from the jig is captured, and the noise map generation unit may derive the relational data from the optical image of the jig. According to the above configuration, the relational data is generated based on an optical image obtained by actually capturing an image of the jig, and a noise map is generated, thereby realizing noise removal corresponding to the relationship between pixel values ​​and noise spread. As a result, noise in the optical image can be removed more effectively.

[0084] The image acquisition step may acquire a plurality of optical images captured in a state where there is no object, and the noise map generation step may derive relational data from the plurality of optical images, where the plurality of optical images are images captured under different imaging conditions. The image acquisition unit may also acquire a plurality of optical images captured in a state where there is no object, and the noise map generation unit may derive relational data from the plurality of optical images, where the plurality of optical images are images captured under different imaging conditions. According to the above configuration, the relational data is generated based on optical images actually captured, and the noise map is generated, thereby realizing noise removal corresponding to the relationship between pixel values ​​and noise spread. As a result, noise in optical images can be more effectively removed.

[0085] In the above embodiment, the evaluation value may be the standard deviation of the noise values. This allows the spread of noise values ​​in the pixel values ​​of each pixel of the optical image to be more accurately evaluated, thereby realizing noise removal that corresponds to the relationship between pixel values ​​and noise. As a result, noise in the optical image can be more effectively removed.

[0086] The machine learning preprocessing method may further include an input step of accepting input of condition information including photodetector information indicating the type of photodetector used to image the object, and in the training image generation step, a noise distribution model to be used may be determined from the photodetector information. While the relationship between pixel values ​​and noise in an optical image varies depending on the type of photodetector used to image the object, the above configuration makes it possible to obtain training images in which noise is appropriately added to structural images, taking into account the type of photodetector used to image the object.

[0087] In the machine learning preprocessing method, the noise distribution model may include at least one of a normal distribution model and a Poisson distribution model, which makes it possible to obtain training images in which noise is appropriately added to the structure image, for example, when a general photodetector that is not an electron multiplier is used to capture an image of the target object.

[0088] In the machine learning preprocessing method, the noise distribution model may include a Bessel function distribution model, which makes it possible to obtain training images in which noise is appropriately added to the structure image, for example, when an electron multiplying photodetector is used to image the target object. [Explanation of symbols]

[0089] 1,1A...optical image processing system, 2...camera (imaging device), 3,3A...optical image processing module, 21,656...photodetector, 31...input section, 32,32A...image acquisition section, 33,33A...noise map generation section, 34...processing section, 35...construction section, 36...trained model, F...object, G1...optical image, G3,G27...relationship graph (relationship data), G5...noise standard deviation map (noise map), G6...optical image, G26...jig image (optical image of jig), Gc...structure image, Gt...training image, L...light.

Claims

1. an image acquisition step of acquiring an optical image obtained by capturing light from an object; a noise map generating step of deriving an evaluation value from the pixel value of each pixel of the optical image based on relational data representing the relationship between the pixel value and an evaluation value that evaluates the spread of noise values, and generating a noise map that is data associating the derived evaluation value with each pixel of the optical image; a processing step of inputting the optical image and the noise map into a trained model constructed in advance by machine learning, and performing image processing to remove noise from the optical image; An optical image processing method comprising:

2. further comprising an input step of accepting input of condition information indicating imaging conditions when imaging the object; In the noise map generation step, the evaluation value is derived from the imaging conditions and pixel values ​​of each pixel of the optical image; The condition information includes information indicating the type of photodetector used to image the object. The optical image processing method of claim 1 .

3. In the image acquisition step, an optical image of the jig is acquired by capturing light from the jig; In the noise map generation step, the relationship data is derived from an optical image of the jig. The optical image processing method of claim 1 .

4. In the image acquisition step, a plurality of optical images are acquired in a state where the object is not present, The noise map generating step includes deriving the relationship data from the plurality of optical images; The plurality of optical images are images captured under different imaging conditions. The optical image processing method of claim 1 .

5. 5. The optical image processing method according to claim 1, wherein the evaluation value is a standard deviation of noise values.

6. a construction step of constructing, by machine learning, a trained model that outputs the noise-removed image data based on the training image and the noise map, using as training data a structural image to which noise based on a predetermined noise distribution model has been added, the training image, a noise map generated from the training image based on relational data representing the relationship between pixel values ​​and an evaluation value that evaluates the spread of noise values, and noise-removed image data, which is data obtained by removing noise from the training image.

7. A preprocessing method for the machine learning method according to claim 6, comprising: a training image generation step of generating, as the training image, the structure image to which noise based on the noise distribution model has been added; a noise map generation step of deriving the evaluation value from the pixel value of each pixel of the structure image based on the relationship data, and generating a noise map that is data associating the derived evaluation value with each pixel of the structure image.

8. further comprising an input step of accepting input of condition information including photodetector information indicating the type of photodetector used to image the object; In the training image generation step, the noise distribution model to be used is determined from the photodetector information. The machine learning preprocessing method according to claim 7 .

9. The machine learning preprocessing method according to claim 8 , wherein the noise distribution model includes at least one of a normal distribution model and a Poisson distribution model.

10. The machine learning preprocessing method of claim 8 , wherein the noise distribution model includes a Bessel function distribution model.

11. an image acquisition unit that acquires an optical image obtained by capturing light from an object; a noise map generating unit that derives an evaluation value from the pixel value of each pixel of the optical image based on relational data that indicates a relationship between the pixel value and an evaluation value that evaluates the spread of noise values, and generates a noise map that is data that associates the derived evaluation value with each pixel of the optical image; a processing unit that inputs the optical image and the noise map into a trained model that has been constructed in advance by machine learning, and performs image processing to remove noise from the optical image; An optical image processing module comprising:

12. further comprising an input unit that accepts input of condition information indicating an imaging condition when imaging the object; the noise map generation unit derives the evaluation value from the imaging conditions and pixel values ​​of each pixel of the optical image; The condition information includes information indicating the type of photodetector used to image the object. The optical image processing module of claim 11 .

13. the image acquisition unit acquires an optical image of the jig obtained by capturing light from the jig; The noise map generator derives the relationship data from an optical image of the jig. The optical image processing module of claim 11 .

14. the image acquisition unit acquires a plurality of optical images captured in a state in which the object is not present, the noise map generator derives the relationship data from the plurality of optical images; The plurality of optical images are images captured under different imaging conditions. The optical image processing module of claim 11 .

15. 15. The optical image processing module according to claim 11, wherein the evaluation value is a standard deviation of noise values.

16. The optical image processing module according to any one of claims 11 to 15, further comprising a construction unit that constructs, by machine learning, a trained model that outputs the noise-removed image data based on the training image and the noise map, using, as training data, structural images to which noise based on a predetermined noise distribution model has been added, the training images, the noise map generated from the training images based on the relational data, and noise-removed image data, which is data obtained by removing noise from the training images.

17. The processor, an image acquisition unit that acquires an optical image obtained by capturing light from an object; a noise map generator that derives an evaluation value from the pixel value of each pixel of the optical image based on relational data that indicates the relationship between the pixel value and an evaluation value that evaluates the spread of noise values, and generates a noise map that is data that associates the derived evaluation value with each pixel of the optical image; and An optical image processing program that functions as a processing unit that inputs the optical image and the noise map into a trained model that has been constructed in advance by machine learning, and performs image processing to remove noise from the optical image.

18. An optical image processing module according to any one of claims 11 to 16; an imaging device that captures light from the object to obtain the optical image; An optical image processing system comprising:

Images