Laser scanning microscope, image processing device, method of operating a laser scanning microscope, and program

Abstract

To provide a novel technique of removing noise from an image acquired by using a SiPM.SOLUTION: A laser scan-type microscope includes: a scanner for scanning a sample by a laser beam; a detector having a silicon photomultiplier (SiPM); and a controller 6 for executing image processing to remove dark count noise on the basis of the frequency of appearance of dark count noise in the scanned image by SiPM.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The disclosure of this specification relates to a laser scanning microscope, an image processing apparatus, an operation method of a laser scanning microscope, and a program.

Background Art

[0002] In recent years, as a detection element of a laser scanning microscope, a Silicon Photomultiplier (hereinafter referred to as SiPM) has attracted attention. The luminance value of an image acquired using SiPM is a value obtained by multiplying the number of photons incident on SiPM by a certain coefficient. Therefore, based on the proportional relationship between the number of photons and the luminance value, quantitative measurement of the number of photons from an image is possible, and the use of SiPM as a photon counting device for counting the number of photons is expected to be widespread.

[0003] On the other hand, when using SiPM, there is a problem that dark count noise occurs. The dark count noise generated in SiPM occurs randomly and is output as a signal corresponding to an integer number of photons. Due to this characteristic, it is not easy to distinguish the dark count noise from the signal output by detecting photons, and it may cause incorrect counting of the number of photons. Technologies related to such technical problems are described in, for example, Patent Document 1.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technology described in Patent Document 1, the effects of dark count noise can be reduced by activating only the pixels to which light is incident. However, the effects of dark count noise generated in the activated pixels cannot be avoided.

[0006] Based on the circumstances described above, one aspect of the present invention is to provide a new technology for removing noise from images acquired using SiPM. [Means for solving the problem]

[0007] A laser scanning microscope according to one aspect of the present invention comprises a scanner that scans a sample with laser light, a detector having a silicon photomultiplier (SiPM), and a control device that performs image processing to remove dark count noise from a scanned image of the sample scanned by the scanner, which is generated based on a signal output from the detector, based on the frequency of dark count noise appearance in the SiPM. The control device includes an estimation unit that estimates the amount of dark count noise included in the scanned image as a noise amount based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, and a removal unit that removes the dark count noise from the scanned image according to the noise amount estimated by the estimation unit, wherein the settings of the laser scanning microscope include the settings of the SiPM and the scan settings of the laser scanning microscope, and the estimation unit includes an acquisition unit that acquires information regarding the frequency of occurrence of the dark count noise in the SiPM based on the settings of the SiPM, and a calculation unit that statistically calculates the noise amount based on the frequency of occurrence of the dark count noise identified from the information acquired by the acquisition unit and the scan settings. A laser scanning microscope according to another aspect of the present invention comprises a scanner that scans a sample with laser light, a detector having a Silicon Photomultiplier (SiPM), and a control device that performs image processing to remove dark count noise from a scanned image of the sample scanned by the scanner, which is generated based on a signal output from the detector, based on the frequency of occurrence of dark count noise in the SiPM. The control device includes an estimation unit that estimates the amount of dark count noise included in the scanned image as a noise amount based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, and a removal unit that removes the dark count noise from the scanned image according to the noise amount estimated by the estimation unit, wherein the estimation unit estimates the location of the dark count noise on the scanned image based on the scanned image and the estimated noise amount, and the removal unit removes the dark count noise from the location of occurrence estimated by the estimation unit on the scanned image by the amount of noise estimated by the estimation unit.

[0008] An image processing apparatus according to one aspect of the present invention includes a processor, which performs image processing on a scanned image of a sample scanned by a scanner included in a laser scanning microscope, the scanned image being generated based on a signal output from a detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, to remove dark count noise based on the frequency of occurrence of dark count noise in the SiPM. Performing the image processing includes estimating the amount of dark count noise included in the scanned image as a noise amount based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, and removing the dark count noise from the scanned image according to the estimated noise amount, wherein the settings of the laser scanning microscope include the settings of the SiPM and the scan settings of the laser scanning microscope, and the estimation includes obtaining information on the frequency of occurrence of the dark count noise in the SiPM based on the settings of the SiPM, and statistically calculating the noise amount based on the frequency of occurrence of the dark count noise identified from the obtained information and the scan settings. An image processing apparatus according to another aspect of the present invention comprises a processor, the processor performs image processing on a scanned image of a sample scanned by a scanner included in a laser scanning microscope, the scanned image generated based on a signal output from a detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, to remove dark count noise based on the frequency of occurrence of dark count noise in the SiPM. Performing the image processing includes estimating the amount of dark count noise included in the scanned image as a noise amount based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, and removing the dark count noise from the scanned image according to the estimated noise amount, the estimation includes estimating the location of the dark count noise on the scanned image based on the scanned image and the estimated noise amount, and the removal includes removing the dark count noise from the estimated location on the scanned image by the estimated noise amount.

[0009] An operating method according to one aspect of the present invention is an operating method for a laser scanning microscope, wherein a scanner included in the laser scanning microscope generates a scanned image of a sample scanned by the scanner included in the laser scanning microscope based on a signal output from a detector having a silicon photomultiplier (SiPM) included in the laser scanning microscope, and performs image processing on the scanned image to remove dark count noise based on the frequency of dark count noise occurrence in the SiPM. Performing the image processing includes estimating the amount of dark count noise included in the scanned image as a noise amount based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, and removing the dark count noise from the scanned image according to the estimated noise amount, wherein the settings of the laser scanning microscope include the settings of the SiPM and the scan settings of the laser scanning microscope, and the estimation includes obtaining information on the frequency of occurrence of the dark count noise in the SiPM based on the settings of the SiPM, and statistically calculating the noise amount based on the frequency of occurrence of the dark count noise identified from the obtained information and the scan settings. Another aspect of the present invention relates to an operating method for a laser scanning microscope, comprising: generating a scanned image of a sample scanned by a scanner included in the laser scanning microscope based on a signal output from a detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope; and performing image processing on the scanned image to remove dark count noise based on the frequency of occurrence of dark count noise in the SiPM. Performing the image processing includes estimating the amount of dark count noise included in the scanned image as a noise amount based on the frequency of occurrence of dark count noise in the SiPM and the settings of the laser scanning microscope; and removing the dark count noise from the scanned image according to the estimated noise amount. The estimation includes estimating the location of the dark count noise on the scanned image based on the scanned image and the estimated noise amount; and the removal includes removing the dark count noise from the estimated location on the scanned image by the estimated noise amount.

[0010] A program according to one aspect of the present invention causes the computer of a laser scanning microscope to generate a scanned image of a sample scanned by a scanner included in the laser scanning microscope based on a signal output from a detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, and to perform image processing on the scanned image to remove dark count noise based on the frequency of occurrence of dark count noise in the SiPM. Performing the image processing includes estimating the amount of dark count noise included in the scanned image as a noise amount based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, and removing the dark count noise from the scanned image according to the estimated noise amount, wherein the settings of the laser scanning microscope include the settings of the SiPM and the scan settings of the laser scanning microscope, and the estimation includes obtaining information on the frequency of occurrence of the dark count noise in the SiPM based on the settings of the SiPM, and statistically calculating the noise amount based on the frequency of occurrence of the dark count noise identified from the obtained information and the scan settings. A program according to another aspect of the present invention causes the computer of a laser scanning microscope to generate a scanned image of a sample scanned by a scanner included in the laser scanning microscope based on a signal output from a detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, and to perform image processing on the scanned image to remove the dark count noise based on the frequency of occurrence of the dark count noise in the SiPM. Performing the image processing includes estimating the amount of the dark count noise included in the scanned image as a noise amount based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, and removing the dark count noise from the scanned image according to the estimated noise amount, wherein the estimation includes estimating the location of the dark count noise on the scanned image based on the scanned image and the estimated noise amount, and the removal includes removing the dark count noise from the estimated location on the scanned image by the estimated noise amount. [Effects of the Invention]

[0011] According to the above embodiment, a new technique for removing noise from images acquired using SiPM can be provided. [Brief explanation of the drawing]

[0012] [Figure 1] This figure illustrates the configuration of a laser scanning microscope according to the first embodiment. [Figure 2] Figure 1 illustrates the functional configuration of the control device shown. [Figure 3] This figure illustrates a more detailed configuration of the estimation unit shown in Figure 2. [Figure 4] This is an example of a flowchart of the processing performed by the laser scanning microscope according to the first embodiment. [Figure 5]This is an example of a flowchart of dark count noise estimation processing. [Figure 6] This is an example of a flowchart of occurrence frequency acquisition processing. [Figure 7] This is an example of a screen related to SiPM settings. [Figure 8] This is an example of a table including information on the occurrence frequency of dark count noise. [Figure 9] This is an example of another table including information on the occurrence frequency of dark count noise. [Figure 10] This is an example of a flowchart of occurrence amount estimation processing. [Figure 11] This is an example of a screen related to scan settings. [Figure 12] This is an example of a flowchart of occurrence position estimation processing. [Figure 13] This is an example of a flowchart of dark count noise removal processing. [Figure 14] This is a figure comparing the histograms of a scanned image and a corrected image. [Figure 15] This is a figure comparing the histograms of a scanned image and a smoothed image. [Figure 16] This is an example of a flowchart of the processing performed by the laser scanning microscope according to the second embodiment. [Figure 17] This is a figure for explaining a modified example of the laser scanning microscope.

Mode for Carrying Out the Invention

[0013] (First Embodiment) FIG. 1 is a diagram illustrating the configuration of the laser scanning microscope according to the present embodiment. The fluorescence microscope 100 is an example of a laser scanning microscope, and includes a scanner that scans a sample with laser light, a detector having SiPM, and a control device that generates an image of the sample scanned by the scanner (hereinafter referred to as a scanned image) based on a signal output from the detector. Hereinafter, the configuration of the fluorescence microscope 100 will be described in detail with reference to FIG. 1.

[0014] As illustrated in Figure 1, the fluorescence microscope 100 includes a scan unit 1 that scans sample A two-dimensionally with laser light, and a first detection unit 2 and a second detection unit 3 that detect fluorescence generated in sample A when irradiated with laser light and incident via the scan unit 1.

[0015] The scanning unit 1 includes a laser light source 11 that emits laser light and an illumination optical system 12 that guides the laser light from the laser light source 11 to the sample A. The laser light source 11 is equipped with multiple types of lasers with different oscillation wavelengths, such as 405 nm, 488 nm, and 543 nm, and is equipped with an AOTF (Acousto-Optics tunable Filter) that can control the emission of laser light at each oscillation wavelength. The illumination optical system 12 includes an optical fiber 13 that guides the laser light from the laser light source 11 and a collimating lens 14.

[0016] Furthermore, the scan unit 1 includes an objective lens 15 that focuses fluorescence from sample A, an imaging lens 16 that images the fluorescence focused by the objective lens 15, a scanner 17 that scans sample A with laser light, a pupil projection lens 18 that converts the fluorescence imaged by the imaging lens 16 into approximately parallel light, an excitation dichroic mirror 19 that branches the approximately parallel light fluorescence from the optical path of the laser light, a confocal lens 20 that focuses the branched fluorescence, and a confocal pinhole 21 that allows only the fluorescence generated from the focal position of the objective lens 15 to pass through.

[0017] The scanner 17 includes, for example, two galvanometer mirrors that scan sample A in mutually orthogonal directions. The scanner may also include other scanning devices, such as resonant mirrors, instead of or in addition to the galvanometer mirrors.

[0018] Multiple excitation dichroic mirrors 19 with different spectral transmittances or reflectances are fixed to a rotatable excitation turret 22. By rotating the excitation turret 22, the excitation dichroic mirrors 19 inserted into the optical path can be changed.

[0019] The first detection unit 2 includes two photometric dichroic mirrors 31A and 31B that decompose (spectrometrically) the fluorescence incident from the scan unit 1 (fluorescence that has passed through the confocal pinhole 21) into two optical paths according to the wavelength range, a wavelength selection mechanism 32A that selects the wavelength of light to be detected from the fluorescence of one optical path decomposed by the photometric dichroic mirror 31B, a photodetector 33A that detects light of the wavelength selected by the wavelength selection mechanism 32A, a wavelength selection mechanism 32B that selects the wavelength of light to be detected from the fluorescence of the other optical path decomposed by the photometric dichroic mirror 31B, and a photodetector 33B that detects light of the wavelength selected by the wavelength selection mechanism 32B.

[0020] Multiple dichroic photometering mirrors 31A and 31B with different spectral transmittances or reflectances are fixed to rotatable photometering turrets 34A and 34B, respectively. By rotating the photometering turrets 34A and 34B, the dichroic photometering mirrors 31A and 31B inserted into the optical path can be changed.

[0021] The photometric dichroic mirror 31A transmits the fluorescence from the scan unit 1 towards the second detection unit 3 or reflects it towards the photometric dichroic mirror 31B, depending on the wavelength range. The photometric dichroic mirror 31B transmits the fluorescence from the photometric dichroic mirror 31A towards the wavelength selection mechanism 32A or reflects it towards the wavelength selection mechanism 32B, depending on the wavelength range.

[0022] The wavelength selection mechanism 32A includes a diffraction grating (VPH (Volume Phase Holographic)) 35A that separates fluorescence into spectral components, a oscillating mirror 36A that reflects the fluorescence separated by the diffraction grating 35A, an imaging lens 37A that focuses the fluorescence reflected by the oscillating mirror 36A onto the light-receiving surface of the photodetector 33A, and a slit (light-shielding slit) 38A that partially blocks the fluorescence focused by the imaging lens 37A.

[0023] The diffraction grating 35A is configured to spectrally separate the fluorescence spectral components transmitted through the photometric dichroic mirror 31B in one direction. The oscillating mirror 36A is provided so as to be able to oscillate around an oscillating axis perpendicular to the direction of the arrangement of the spectral series spectrally separated by the diffraction grating 35A. This oscillating mirror 36A can change the spectral components that pass through the slit 38A depending on the oscillating angle.

[0024] The slit 38A comprises a fixed member 39A and a movable member 40A positioned with a gap between it and the fixed member 39A in the direction of the spectral sequence. The movable member 40A is provided to be movable relative to the fixed member 39A in the direction of the spectral sequence, and can widen or narrow the gap between it and the fixed member 39A, i.e., the aperture through which fluorescence passes. The photodetector 33A has different wavelength sensitivity characteristics from the photodetectors 33C and 33D described later, and for example, has higher wavelength sensitivity on the short wavelength side than the photodetectors 33C and 33D.

[0025] The wavelength selection mechanism 32B has the same configuration as the wavelength selection mechanism 32A. That is, the wavelength selection mechanism 32B comprises a diffraction grating (VPH) 35B, a oscillating mirror 36B, an imaging lens 37B, and a slit (light-shielding slit) 38B.

[0026] The diffraction grating 35B is configured to spectrally separate the fluorescence spectral components reflected by the photometric dichroic mirror 31B in one direction. The oscillating mirror 36B is provided so as to be able to oscillate around an oscillating axis perpendicular to the arrangement direction of the spectral series spectrally separated by the diffraction grating 35B, and the spectral components that pass through the slit 38B can be changed according to the oscillating angle. The slit 38B comprises a fixed member 39B and a movable member 40B. The photodetector 33B has the same wavelength sensitivity characteristics as the photodetector 33A.

[0027] The second detection unit 3 has the same configuration as the first detection unit 2. Specifically, the second detection unit 3 comprises two photometric dichroic mirrors 31C and 31D, a wavelength selection mechanism 32C, a photodetector 33C, a wavelength selection mechanism 32D, and a photodetector 33D.

[0028] Similar to the photometric dichroic mirrors 31A and 31B, multiple dichroic mirrors 31C and 31D with different spectral transmittances or reflectances are fixed to rotatable photometric turrets 34C and 34D, respectively. By rotating the photometric turrets 34C and 34D, the photometric dichroic mirrors 31C and 31D inserted into the optical path can be changed.

[0029] The photometric dichroic mirror 31C is configured to transmit or reflect fluorescence from the photometric dichroic mirror 31A of the first detection unit 2 towards the photometric dichroic mirror 31D, depending on the wavelength range. The photometric dichroic mirror 31D is configured to transmit or reflect fluorescence from the photometric dichroic mirror 31C towards the wavelength selection mechanism 32C, depending on the wavelength range, towards the wavelength selection mechanism 32D.

[0030] The wavelength selection mechanism 32C is designed to select the wavelength of light to be detected from the fluorescence that has passed through the photometric dichroic mirror 31D. This wavelength selection mechanism 32C comprises a diffraction grating (VPH) 35C, an oscillating mirror 36C, an imaging lens 37C, and a slit (light-shielding slit) 38C.

[0031] The diffraction grating 35C is configured to spectrally separate the fluorescence spectral components from the photometric dichroic mirror 31D in one direction. The oscillating mirror 36C is provided so as to be able to oscillate around an oscillating axis perpendicular to the direction of the spectral sequence spectrally separated by the diffraction grating 35C, and the spectral components that pass through the slit 38C can be changed according to the oscillating angle.

[0032] The slit 38C comprises a fixed member 39C and a movable member 40C. The photodetector 33C is configured to detect light of a wavelength selected by the wavelength selection mechanism 32C. This photodetector 33C has different wavelength sensitivity characteristics than photodetectors 33A and 33B, and for example, it has higher sensitivity at longer wavelengths than photodetectors 33A and 33B.

[0033] The wavelength selection mechanism 32D is configured to select the wavelength of light to be detected from the fluorescence reflected by the photometric dichroic mirror 31D. This wavelength selection mechanism 32D comprises a diffraction grating (VPH) 35D, an oscillating mirror 36D, an imaging lens 37D, and a slit (light-shielding slit) 38D.

[0034] The diffraction grating 35D is configured to spectrally separate the fluorescence spectral components from the photometric dichroic mirror 31D in one direction. The oscillating mirror 36D is provided so as to be able to oscillate around an oscillating axis perpendicular to the direction of the spectral sequence spectrally separated by the diffraction grating 35D, and the spectral components that pass through the slit 38D can be changed according to the oscillating angle.

[0035] The slit 38D comprises a fixed member 39D and a movable member 40D. The photodetector 33D is configured to detect light of a wavelength selected by the wavelength selection mechanism 32D. This photodetector 33D has the same wavelength sensitivity characteristics as the photodetector 33C.

[0036] The detection elements of photodetectors 33A, 33B, 33C, and 33D use SiPMs, which are kept at a constant temperature by a Peltier element. That is, photodetectors 33A, 33B, 33C, and 33D each have a SiPM and a Peltier element as a cooling mechanism. The SiPM includes a multi-pixel avalanche photodiode (APD), operates in Geiger mode to be sensitive to single photons, and outputs a signal with an intensity proportional to the number of photons.

[0037] The SiPM contained in photodetectors 33A and 33B are of a different type than the SiPM contained in photodetectors 33C and 33D. These differences in type result in differences in the sensitivity characteristics of photodetectors 33A and 33B compared to photodetectors 33C and 33D. Hereafter, unless otherwise specified, each of these, or collectively referred to as photodetector 33, will be used to describe photodetector 33A, 33B, 33C, and 33D.

[0038] The fluorescence microscope 100 further includes an input device 4, a display device 5, and a control device 6. The input device 4 receives input for various settings of the fluorescence microscope 100 in response to user input operations, such as inputting the fluorescent dye or wavelength range to be observed, inputting the settings for the photodetector 33 (SiPM) used for image acquisition, and inputting scan settings. The input device 4 is, for example, a keyboard, mouse, or touch panel.

[0039] The display device 5 displays various information, such as a setting input screen that allows input of settings for the fluorescence microscope 100, and images. The display device 5 is, for example, an LCD (Liquid Crystal Display).

[0040] The control device 6 controls various parts of the fluorescence microscope 100. For example, the control device 6 controls the fluorescence microscope 100 so that the scanner 17 scans sample A with laser light and detects fluorescence from sample A, according to the settings of the fluorescence microscope 100. Furthermore, the control device 6 generates a scanned image of sample A based on the signal output from the photodetector 33 of the fluorescence microscope 100. More specifically, the scanned image is constructed based on the fluorescence intensity signal output from the photodetector 33 and the scanning position signal of the scanner 17.

[0041] The control device 6 is, for example, a PC (Personal Computer) and includes a processor 6a and memory 6b. The various controls performed by the control device 6 may be implemented, for example, by the processor 6a executing a program stored in memory 6b (so-called software processing), by hardware processing, or by a combination of software processing and hardware processing. The processor 6a includes, for example, one or more integrated circuits. The integrated circuit may be, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).

[0042] Memory 6b stores the program executed by processor 6a. Memory 6b includes a non-temporary computer-readable medium that stores the program executed by processor 6a. Memory 6b may include, for example, one or more arbitrary semiconductor memories, one or more other storage devices. Semiconductor memories include, for example, volatile memories such as RAM (Random Access Memory), non-volatile memories such as ROM (Read Only Memory), programmable ROM, and flash memory. RAM may include, for example, DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory). Other storage devices may include, for example, magnetic storage devices including, for example, magnetic disks as computer-readable media, and optical storage devices including, for example, optical disks as computer-readable media.

[0043] In the fluorescence microscope 100 configured as described above, since SiPM is used as the photodetector, the scanned image generated by the control device 6 contains randomly occurring noise called dark count noise. Therefore, in the fluorescence microscope 100, the control device 6 performs image processing on the generated scanned image to remove the dark count noise generated by the SiPM. More specifically, the control device 6 performs image processing on the scanned image to remove the dark count noise based on the frequency of occurrence of dark count noise in the SiPM. In other words, the control device 6 is an example of an image processing device that performs image processing to remove dark count noise. By performing image processing using statistical information such as the frequency of occurrence, dark count noise can be removed more effectively than with typical image processing for noise reduction that has been performed conventionally.

[0044] Figure 2 illustrates the functional configuration of the control device shown in Figure 1, and mainly shows the configuration related to the image processing described above. Figure 3 illustrates a more detailed configuration of the estimation unit shown in Figure 2. Below, the configuration related to the image processing performed by the control device 6 as a measure against dark count noise will be described with reference to Figures 2 and 3.

[0045] As shown in Figure 2, the control device 6 comprises a scanning image generation unit 50, an estimation unit 60, a storage unit 70, a removal unit 80, and an output unit 90. Furthermore, as shown in Figure 3, the estimation unit 60 comprises an acquisition unit 61, a calculation unit 62, a noise image generation unit 63, and a determination unit 64. Of these functional configurations, the scanning image generation unit 50, the estimation unit 60, the removal unit 80, and the output unit 90 are implemented by the processor 6a by executing a program stored in memory 6b, for example, while the storage unit 70 is implemented by memory 6b, for example.

[0046] The scanning image generation unit 50 generates a scanned image of sample A. The estimation unit 60 estimates the amount of dark count noise (hereinafter simply referred to as noise amount) contained in the scanned image generated by the scanning image generation unit 50. The removal unit 80 removes the dark count noise from the scanned image according to the noise amount estimated by the estimation unit 60. The output unit 90 outputs the corrected image generated by the above image processing to the display device 5.

[0047] In detail, the estimation unit 60 estimates the amount of noise based on the frequency of dark count noise occurrence in SiPM and the settings of the fluorescence microscope 100. The estimation unit 60 calculates the photometry time per scan image from the settings of the fluorescence microscope 100 and estimates the amount of noise in combination with the frequency of occurrence and the calculated photometry time. The photometry time per scan image refers to the time during which photometry is actually performed, i.e., the period during which dark count noise may occur, out of the scanning time from the start to the end of scanning to acquire an image. The scanning time includes the time during which the scanner overswings relative to the scanning range in which the image is acquired or moves to the next line, but during these periods, no photometry is performed and no signal is detected, so dark count noise is not detected. The photometry time per scan image corresponds to the time obtained by subtracting the time during which dark count noise is not detected from the scanning time, and is the total photometry time for all pixels that make up the image. Furthermore, the settings for the fluorescence microscope 100 include SiPM settings (e.g., settings related to applied voltage, constant temperature) and scan settings for the fluorescence microscope 100 (e.g., scan speed, scan size, number of scans per image).

[0048] More specifically, in the estimation unit 60, the acquisition unit 61 and the calculation unit 62 estimate the amount of noise. The acquisition unit 61 acquires information on the frequency of occurrence of dark count noise in the SiPM (hereinafter simply referred to as frequency information) based on the SiPM settings included in the settings of the fluorescence microscope 100. This is because the frequency of occurrence in the SiPM may differ depending on the SiPM settings.

[0049] The memory unit 70 stores frequency information, and the acquisition unit 61 may acquire frequency information from the memory unit 70. The memory unit 70 stores frequency information for each SiPM setting. For example, the memory unit 70 may store frequency information for each setting related to the voltage applied to the SiPM, or it may store frequency information for each setting related to the temperature at which the SiPM is kept constant.

[0050] Furthermore, the memory unit 70 may store frequency information for each individual SiPM contained in the photodetector 33, or it may store frequency information for each type of SiPM contained in the photodetector 33. This is because, similar to the SiPM settings, the frequency of appearance of SiPMs may differ for each individual or type. Therefore, the acquisition unit 61 may acquire frequency information corresponding to the individual or type of SiPM used to generate the scanning image.

[0051] When the acquisition unit 61 acquires frequency information, the calculation unit 62 statistically calculates the noise amount based on the frequency of dark count noise occurrences identified from the frequency information and the scan settings included in the settings of the fluorescence microscope 100. Specifically, the calculation unit 62 calculates the number of times dark count noise occurs per scan image (i.e., the number of pixels containing dark count noise) as the noise amount from the total photometric time for all pixels constituting the image, which is identified from the frequency of occurrence and the photometric time identified from the scan settings.

[0052] The calculation unit 62 may also identify the frequency of dark count noise occurrences for each intensity of dark count noise from the frequency information acquired by the acquisition unit 61. In other words, the calculation unit 62 may identify the frequency of occurrences of dark count noise that was mistakenly detected as 1 photon, 2 photons, ..., and N photons. In that case, the calculation unit 62 may calculate the amount of noise for each intensity of dark count noise.

[0053] When the estimation unit 60 calculates the noise amount as the number of pixels, the removal unit 80 subtracts a predetermined value from the pixel values ​​of the number of pixels calculated by the calculation unit 62 as the noise amount. Here, the predetermined value should correspond to the signal intensity output from the SiPM by detecting photons.

[0054] Furthermore, if the calculation unit 62 calculates the noise amount for each intensity, the removal unit 80 subtracts a value corresponding to the intensity from the pixel value of the number of pixels for which the calculation unit 62 calculated the noise amount for each intensity. Here, the value corresponding to the intensity only needs to be proportional to the number of photons.

[0055] As described above, in the fluorescence microscope 100, the control device 6 estimates the amount of noise using the occurrence frequency and the settings of the fluorescence microscope 100, making it possible to accurately estimate the amount of dark count noise and remove it without excess or deficiency. Furthermore, the control device 6 calculates the amount of noise as the number of pixels and adjusts the pixel value for the calculated number of pixels by a value corresponding to the detection of photons. This makes it possible to remove dark count noise from the scanned image without losing the characteristic of images obtained using SiPM, where the pixel value is proportional to the number of incident photons. Therefore, with the fluorescence microscope 100, a good corrected image with suppressed dark count noise can be obtained, and more accurate photon counting becomes possible.

[0056] The estimation unit 60 may estimate the location of dark count noise in addition to the amount of noise. More specifically, the estimation unit 60 may estimate the location of dark count noise based on the estimated amount of noise, and the removal unit 80 may remove the dark count noise from the location on the scanned image estimated by the estimation unit 60 by the amount of noise estimated by the estimation unit 60.

[0057] More specifically, in the estimation unit 60, the noise image generation unit 63 and the determination unit 64 estimate the occurrence locations. The noise image generation unit 63 calculates a score for the dark count noise contained in each pixel of the scanned image based on the scanned image, and generates a noise image consisting of these scores. The noise image is used to identify locations (candidate locations) where the occurrence of dark count noise is suspected. The score for dark count noise only needs to correlate with the dark count noise. Specifically, the score may correlate with the intensity of the dark count noise, or it may correlate with the probability of the occurrence of dark count noise. This is because candidate locations can be identified in either case.

[0058] When the noise image generation unit 63 generates a noise image, the determination unit 64 determines the location of the dark count noise based on the noise image and the estimated noise amount. Specifically, based on the score of the noise image, the determination unit 64 determines candidate locations as occurrence locations, starting with the most likely candidate locations, in order of the noise amount.

[0059] As described above, in the fluorescence microscope 100, the control device 6 identifies the location of dark count noise on a pixel-by-pixel basis and adjusts the pixel value from each identified pixel by a value corresponding to the detection of photons. This effectively removes dark count noise that occurs randomly in both the spatial and temporal directions. Therefore, the fluorescence microscope 100 provides a good corrected image with suppressed dark count noise and enables more accurate photon counting.

[0060] Figure 4 is an example of a flowchart of the processing performed by the laser scanning microscope according to this embodiment. Figure 5 is an example of a flowchart of the dark count noise estimation process. Figure 6 is an example of a flowchart of the occurrence frequency acquisition process. Figure 7 is an example of a screen related to SiPM settings. Figure 8 is an example of a table containing information on the occurrence frequency of dark count noise. Figure 9 is an example of another table containing information on the occurrence frequency of dark count noise. Figure 10 is an example of a flowchart of the occurrence amount estimation process. Figure 11 is an example of a screen related to scan settings. Figure 12 is an example of a flowchart of the occurrence location estimation process. Figure 13 is an example of a flowchart of the dark count noise removal process. Figure 14 is a comparison of the histograms of the scanned image and the corrected image. Figure 15 is a comparison of the histograms of the scanned image and the smoothed image.

[0061] The following describes in detail specific examples of image processing performed by the fluorescence microscope 100 to remove dark count noise during scan image generation, with reference to Figures 4 to 15. Note that the processing shown in Figure 4 is initiated, for example, by the processor 6a of the control device 6 executing a program stored in memory 6b.

[0062] First, the fluorescence microscope 100, for example, in response to user input using the input device 4, scans sample A with laser light to generate a scanned image of sample A (step S1). Here, the control device 6 constructs the scanned image based on the output signal from the photodetector 33 and the signal regarding the scanning position of the scanner 17.

[0063] Subsequently, the fluorescence microscope 100 performs dark count noise estimation and dark count noise removal processing (steps S2 and S3). These processes are performed by the control device 6. The control device 6 generates a corrected image from the scanned image with suppressed dark count noise by performing image processing, including dark count noise estimation and dark count noise removal processing, on the scanned image generated in step S1.

[0064] In the dark count noise estimation process, as shown in Figure 5, the control device 6 obtains the frequency of occurrence of dark count noise in the frequency acquisition process (step S11), estimates the amount of occurrence of dark count noise in the amount estimation process (step S12), and estimates the location of occurrence of dark count noise in the location estimation process (step S13).

[0065] In the frequency acquisition process, as shown in Figure 6, the control device 6 first identifies the individual SiPM used to generate the scan image in step S1 (step S21). The individual SiPM used to generate the scan image may be identified, for example, by referring to the channel setting area R1 of the setting screen as shown in Figure 7. In this example, the channel "CH1" used to generate the scan image indicates the photodetector "SiPM(A)". Therefore, the control device 6 identifies the photodetector "SiPM(A)" as the individual SiPM used to generate the scan image. Note that "SiPM(A)" is the SiPM included in the photodetector 33A shown in Figure 1.

[0066] Next, the control device 6 first identifies the voltage applied to the SiPM used when generating the scanned image in step S1 (step S22). The voltage applied to the SiPM may be identified, for example, by referring to the channel setting area R1 of the setting screen as shown in Figure 7. In this example, the voltage applied to generate the scanned image is "5V", so the control device 6 identifies "5V" as the voltage applied to the SiPM.

[0067] Once the SiPM unit and the applied voltage are identified, the control device 6 obtains the frequency of dark count noise occurrence in the SiPM (step S23). Here, the control device 6 refers to Table T1, shown in Figure 8, which is stored in memory 6b and contains the frequency of dark count noise occurrence for each SiPM unit and each applied voltage, and obtains the frequency of occurrence corresponding to the combination of unit "SiPM(A)" and applied voltage "5V" from Table T1. In this example, the control device 6 obtains "4.00" (times / msec) as the frequency of occurrence.

[0068] The information stored in table T1 shown in Figure 8 is information measured in advance through experiments, and can be generated, for example, by the following procedure. First, light is not incident on each SiPM of the fluorescence microscope 100. Then, images are acquired using these SiPMs, which have been kept at a constant temperature of 0°C, with applied voltages of 5V and 8V, respectively. The number of pixels in these images in which dark count noise appears is then counted. Finally, the frequency of occurrence is calculated by dividing the counted number of pixels by the total photometric time for all pixels that make up the image.

[0069] Once the frequency of dark count noise occurrences is obtained, the control device 6 determines the SiPM's constant temperature (step S24). If the SiPM's constant temperature is fixed at a specific temperature in the fluorescence microscope 100, that fixed value is obtained. Alternatively, if the user can set the constant temperature, the constant temperature may be determined by referring to the information on the setting screen, similar to when the individual or applied voltage was identified. In this example, the control device 6 obtains, for example, 0°C as the constant temperature.

[0070] Once the constant temperature is determined, the control device 6 obtains a correction coefficient for the frequency of occurrence (step S25). Here, the control device 6 refers to table T2, shown in Figure 9, which is stored in memory 6b and contains correction coefficients for the frequency of occurrence for each constant temperature, and obtains the correction coefficient corresponding to the constant temperature "0°C" from table T2. In this example, the control device 6 obtains "1.00" as the correction coefficient.

[0071] The information stored in Table T2 shown in Figure 9 is information measured in advance through experiments, etc., and can be generated, for example, by the following procedure. The occurrence frequency for each SiPM constant temperature is calculated using the same procedure as when generating the information stored in Table T1 shown in Figure 8. The calculated occurrence frequencies are grouped by constant temperature. Then, a correction coefficient corresponding to each group is calculated, using the group with a constant temperature of 0°C as the reference. Specifically, the correction coefficient is calculated by dividing the representative value (mean, median, etc.) of each group by the representative value of the group with a constant temperature of 0°C.

[0072] The control device 6 corrects the occurrence frequency acquired in step S23 using the correction coefficient acquired in step S25 (step S26). Here, the control device 6 calculates the occurrence frequency of dark count noise in the SiPM used to generate the scanned image by multiplying the occurrence frequency "4.00" (times / msec) acquired in step S23 by the correction coefficient "1.00" acquired in step S25. This determines an occurrence frequency that is independent of the intensity of the dark count noise.

[0073] Finally, the control device 6 determines the occurrence frequency of dark count noise for each intensity (step S27) and terminates the occurrence frequency acquisition process. Here, the control device 6 uses a Poisson distribution to determine the occurrence frequency of dark count noise corresponding to the number of photons. In this example, the control device 6 determines the occurrence frequency of dark count noise corresponding to 1 photon to be "3.96" (times / msec) and the occurrence frequency of dark count noise corresponding to 2 photons to be "0.00797" (times / msec).

[0074] In the occurrence amount estimation process that follows the occurrence frequency acquisition process, as shown in Figure 10, the control device 6 first identifies the scan settings used to generate the scanned image (step S31). The scan settings used to generate the scanned image may be identified, for example, by referring to the scan setting area R2 of the setting screen as shown in Figure 11. The scan settings only need to provide information on the total photometric time for all pixels constituting the image, and may include the scan speed (1 μsec / pixel) and scan size (512 × 512 pixels), as shown in Figure 11.

[0075] Once the scan settings are identified, the control device 6 calculates the amount of dark count noise in the scanned image in terms of pixels based on the scan settings and the occurrence frequency acquired in the occurrence frequency acquisition process (step S32), and then terminates the occurrence amount estimation process. Here, the control device 6 calculates the amount of dark count noise in terms of pixels for each intensity of dark count noise corresponding to the number of photons. In this example, the control device 6 calculates the amount of dark count noise corresponding to one photon using an occurrence frequency of "3.96" (times / msec) and the total photometric time for all pixels constituting the image, "1 × 512 × 512" (μsec), that is, as 1038 pixels in the scanned image. Furthermore, the control device 6 calculates the amount of dark count noise corresponding to two photons using an occurrence frequency of "0.00797" (times / msec) and the total photometric time for all pixels constituting the image, "1 × 512 × 512" (μsec), that is, as 2 pixels in the scanned image.

[0076] In the occurrence location estimation process, which follows the occurrence quantity estimation process, the control device 6 first applies a smoothing filter to the scanned image (step S41), as shown in Figure 12, to generate a smoothed image. The smoothing process is equivalent to a low-pass filter process, and the smoothing filter is an example of a noise reduction filter commonly used in the field of image processing. Specifically, smoothing filters include, for example, median filters, averaging filters, Gaussian filters, and recursive filters.

[0077] Next, the control device 6 generates a noise image by taking the difference between the scanned image and the smoothed image (step S42). Here, the control device 6 generates a noise image by subtracting the pixel value of the corresponding pixel in the smoothed image from the pixel value of each pixel in the scanned image. The smoothed image is an image consisting mainly of signal components, from which dark count noise has been removed. Therefore, by subtracting the smoothed image from the scanned image, a noise image consisting mainly of noise components (dark count noise) can be generated.

[0078] Finally, the control device 6 identifies the pixel locations where noise appears based on the noise image and the noise amount estimated in the occurrence amount estimation process (step S43), and terminates the occurrence location estimation process. Here, the control device 6 selects pixels with large pixel values ​​from the noise image, up to the number of pixels calculated as the noise amount, and identifies the locations of the selected pixels as the pixel locations where dark count noise appears. In this example, the control device 6 identifies the top 2 pixels in the noise image with the highest pixel values ​​as the pixel locations where 2 photons of dark count noise appear, and identifies the top 1038 pixels in the remaining pixels in the noise image with the highest pixel values ​​as the pixel locations where 1 photon of dark count noise appears.

[0079] When the dark count noise estimation process, which includes the frequency acquisition process, the quantity estimation process, and the location estimation process, is completed, the control device 6 performs the dark count noise removal process (step S3).

[0080] In the dark count noise removal process, the control device 6 repeatedly removes dark count noise from the scanned image for each number of photons, as shown in Figure 13 (steps S51 to S53). In this example, the control device 6 first subtracts the pixel value corresponding to 2-photon dark count noise from the pixel value of the pixel where 2-photon dark count noise, identified in the occurrence location estimation process, appears in the scanned image. For example, if the pixel value corresponding to 1-photon dark count noise is 16, then the pixel value corresponding to 2-photon dark count noise is 32. Subsequently, the control device 6 subtracts the pixel value corresponding to 1-photon dark count noise from the pixel value of the pixel where 1-photon dark count noise, identified in the occurrence location estimation process, appears in the scanned image. This generates a corrected image from which dark count noise has been removed from the scanned image.

[0081] Finally, the control device 6 outputs the corrected image (step S54), and the process shown in Figure 4 is completed. Here, the control device 6 may output the corrected image to memory 6b for storage in memory 6b, or it may output the corrected image to the display device 5 for display on the display device 5.

[0082] As described above, by performing the process shown in Figure 3, the fluorescence microscope 100 can remove dark count noise from the scanned image by a statistically calculated amount. This avoids excessive removal of dark count noise or excessive noise remaining due to insufficient removal. Furthermore, it is possible to remove dark count noise while maintaining the image characteristics suitable for photon counting that the scanned image possesses. This point will be explained with reference to Figures 14 and 15.

[0083] Figure 14 shows the histogram of pixel values ​​(intensity) in the scanned image as a dashed line, and the histogram of pixel values ​​(intensity) in the corrected image as a solid line. Similarly, Figure 15 shows the histogram of pixel values ​​(intensity) in the scanned image as a dashed line, and the histogram of pixel values ​​(intensity) in the smoothed image as a solid line.

[0084] Due to the characteristic of SiPM to output a signal with intensity proportional to the number of incident photons, the histogram of the scanned image can clearly distinguish and detect peaks containing background, peaks containing one-photon, and peaks containing two-photon, as shown by the dashed lines in Figures 14 and 15. Therefore, the number of photons can be counted quantitatively.

[0085] The image processing described above for scanned images removes dark count noise by subtracting a value corresponding to the distance between peaks in the histogram (e.g., value for 1 photon, value for 2 photons) from the pixel value of the pixel in the scanned image where dark count noise appears. On the histogram, this process corresponds to a shift from the peaks for 1-photons or 2-photons to the peak for the background. Therefore, as shown by the solid line in Figure 14, even in the histogram of the corrected image obtained by the image processing described above, the peaks containing the background, the peaks containing the 1-photon peak, and the peaks containing the 2-photon peak are maintained and can be clearly distinguished and detected.

[0086] In contrast, the smoothing process compresses the peaks of the histogram and broadens the bases. As shown by the solid line in Figure 15, in the histogram of the smoothed image, the boundaries between the peaks containing the background, the peaks containing one-photons, and the peaks containing two-photons become blurred, making it impossible to distinguish between pixels corresponding to the background, pixels corresponding to one-photons, and pixels corresponding to two-photons.

[0087] Thus, the fluorescence microscope 100 makes it possible to maintain image characteristics suitable for photon counting, which was previously difficult to achieve simultaneously with noise reduction. Therefore, with the fluorescence microscope 100, accurate photon counting is possible using a corrected image with suppressed dark count noise.

[0088] (Second embodiment) The laser scanning microscope according to this embodiment has the same configuration as the fluorescence microscope 100 shown in Figure 1. Therefore, the components of the laser scanning microscope according to this embodiment are referred to by the same reference numerals as the components of the fluorescence microscope 100.

[0089] Figure 16 is an example of a flowchart of the processing performed by the laser scanning microscope according to this embodiment. The laser scanning microscope according to this embodiment differs from the fluorescence microscope 100 in that it performs the processing shown in Figure 16 instead of the processing shown in Figure 4. Hereinafter, with reference to Figure 16, a specific example of the image processing performed by the laser scanning microscope according to this embodiment to remove dark count noise when generating a scanned image will be described. Note that the processing in Figure 16 is also started, for example, by the processor 6a of the control device 6 executing a program stored in memory 6b, similar to the processing in Figure 4.

[0090] The process shown in Figure 16 differs from the process shown in Figure 4 in that it generates a scanned image, estimates the dark count noise, and removes the dark count noise for each number of photons corresponding to the dark count noise. The laser scanning microscope focuses on the lowest number of photons first. Specifically, the laser scanning microscope first initializes N, which represents the number of photons to focus on, to N=1 (step S61). Then, the laser scanning microscope repeats the scanned image generation process, the dark count noise estimation process, and the dark count noise removal process, incrementing the number of photons to focus on by one each time, for a number of photons corresponding to the maximum intensity of the dark count noise contained in the scanned image (steps S62 to S66).

[0091] The process in step S62 is the same as the process in step S1 in Figure 4. The dark count noise estimation process in step S63 differs from the dark count noise estimation process shown in Figure 4 in that it generates a noise image using a trained model built by machine learning instead of a smoothing filter.

[0092] Specifically, the control device 6 generates a noise image representing the spatial distribution of the probability of dark count noise occurring within a scanned image, based on a trained model that has learned the relationship between an image with dark count noise added and the spatial distribution of the dark count noise added to that image, and the scanned image. In other words, the noise image is a probabilistic image, and the pixel values ​​of the noise image indicate the probability that the pixel contains dark count noise. Furthermore, by representing the pixel values ​​of the noise image as real numbers, it becomes possible to continuously represent the probabilities, thereby improving the accuracy of identifying the pixel locations where dark count noise appears, as described later.

[0093] The trained model used to generate the noisy images can be constructed, for example, by following these steps: First, generate images with artificially added dark count noise to the original image without dark count noise, and images containing only the added dark count noise. Generate multiple sets of these images by changing the original image, the pattern (spatial intensity distribution) of the added dark count noise, and the frequency of appearance of the added dark count noise. Then, using all the image sets, perform machine learning on the relationship between the image with added dark count noise and the spatial distribution of the added dark count noise, thereby constructing a trained model that can generate images containing only dark count noise from images with added dark count noise.

[0094] The method for identifying the pixel locations where dark count noise appears from a noisy image and the method for removing noise from a scanned image are basically the same as in the first embodiment. Specifically, the control device 6 selects pixels in the noisy image in order from those with large pixel values ​​(probability of appearance) based on the number of pixels calculated as the noise amount, and identifies the positions of the selected pixels as the pixel locations where dark count noise appears. Then, it subtracts a value corresponding to the dark count noise from the pixel value of the identified pixel location in the image containing the dark count noise (for example, a scanned image).

[0095] However, in this embodiment, in the first iteration, one photon of dark count noise is removed, and in the second iteration, two photons of dark count noise are removed. This is repeated for the number of photons corresponding to the maximum intensity of the dark count noise. More specifically, in the first iteration, the pixel locations where dark count noise appears are identified from the noise image for the total amount of noise contained in the scanned image (number of pixels containing dark count noise), and a value corresponding to one photon of dark count noise is subtracted from the pixel value of the identified pixel location in the scanned image. As a result, the pixel value of all pixels containing dark count noise in the scanned image is adjusted by a value equivalent to one photon.

[0096] Subsequently, the image generated by removing one photon of dark count noise from the scanned image (referred to as the updated image) is input into the trained model described above to generate a new noise image. In the second iteration, the pixel locations where dark count noise appears are identified in the newly generated noise image by the amount of noise (number of pixels) obtained by subtracting the noise amount (number of pixels) of one photon of dark count noise from the total amount of noise (number of pixels containing dark count noise) in the scanned image. A value corresponding to one photon of dark count noise is then subtracted from the pixel value of the identified pixel location in the updated image. As a result, for all pixels in the scanned image that contain two or more photons of dark count noise, the pixel value is adjusted by a value equivalent to one additional photon, for a total of two photons of adjustment.

[0097] If the maximum intensity of dark count noise in the scanned image is equivalent to 2 photons, the process described above should be repeated twice. If the maximum intensity of dark count noise in the scanned image is equivalent to 3 photons, the process should be repeated three times, and if it is equivalent to 4 photons, the process should be repeated four times.

[0098] Similar to the fluorescence microscope 100 according to the first embodiment, the laser scanning microscope according to this embodiment can also remove dark count noise, and accurate photon counting can be performed using a corrected image with suppressed dark count noise.

[0099] The embodiments described above are specific examples provided to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modified forms of the embodiments described above and alternative forms that replace the embodiments described above may be included. In other words, each embodiment can be modified in terms of its components without departing from its spirit and scope. Furthermore, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. In addition, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in an embodiment. Moreover, the processing procedures shown in each embodiment may be performed in a different order, as long as they do not contradict each other. That is, the laser scanning microscope, image processing apparatus, method of operating the laser scanning microscope, and program of the present invention can be modified in various ways without departing from the scope of the claims.

[0100] In the embodiments described above, as shown in Figures 8 and 9, memory 6b stores the frequency of occurrence for each combination of SiPM individual and applied voltage, and further stores a correction coefficient for each constant temperature. However, this information regarding the frequency of occurrence may be stored in memory 6b in a different format. For example, the frequency of occurrence may be stored for each combination of SiPM individual, applied voltage, and constant temperature. That is, the table structure is not limited to a two-dimensional structure, but may have a three-dimensional or higher structure.

[0101] In the embodiment described above, an example was shown in which the occurrence frequency is stored for each combination of SiPM and applied voltage. However, if the change in occurrence frequency dependent on the applied voltage is almost constant regardless of the SiPM or electrical circuit, the change in occurrence frequency dependent on the applied voltage may be managed as a correction coefficient. In that case, the occurrence frequency may be managed only for each SiPM.

[0102] Furthermore, if the difference in occurrence frequency due to individual differences in the same type of SiPM is negligibly small, memory 6b may store the occurrence frequency by type rather than by individual SiPM. Also, if there is almost no difference in occurrence frequency between individuals or types, memory 6b may store the occurrence frequency as a single fixed value.

[0103] Furthermore, if the frequency of occurrence corresponding to the setting used is not stored in memory 6b, the frequency of occurrence corresponding to the setting used may be calculated by interpolating the frequency of occurrence of a different setting. For example, if the frequency of occurrence corresponding to the applied voltage to be used is not stored in the table stored in memory 6b, the frequency of occurrence corresponding to the applied voltage to be used may be calculated by interpolation from the frequency of occurrence corresponding to a different applied voltage.

[0104] Furthermore, instead of storing the occurrence frequency itself in a table format as shown in Figures 8 and 9, memory 6b may store functions or parameters for calculating the occurrence frequency as information related to the occurrence frequency. For example, the number of APDs constituting the SiPM and the occurrence frequency of dark count noise per APD may be stored as parameters, and the occurrence frequency of dark count noise in the SiPM may be calculated using these.

[0105] In the embodiments described above, an example was shown in which SiPM settings and scan settings are obtained from the current settings of the fluorescence microscope 100. However, this information may also be stored together with the scanned image, and image processing to remove dark count noise may be performed using this information stored together with the scanned image. In this case, image processing does not necessarily have to be performed when the scanned image is generated, but may be performed at any time on the scanned image.

[0106] Furthermore, the entity that performs the image processing may be a different device from the control device that generates the scanned image. Image processing to remove dark count noise from the scanned image may be performed by an image processing device separate from the control device, and this image processing device may be an external device not included in the laser scanning microscope.

[0107] In the embodiment described above, an example was shown in which information regarding the frequency of occurrence is read from the memory 6b of the control device 6. However, the information regarding the frequency of occurrence may be stored in a device other than the control device included in the laser scanning microscope, such as a server device located in the cloud. The laser scanning microscope may obtain the information regarding the frequency of occurrence from a server device located in the cloud and remove dark count noise from the scanned image.

[0108] In the embodiment described above, an example was shown in which pre-stored information on occurrence frequency is read and used for image processing. However, information on occurrence frequency may be created at the necessary timing. For example, as shown in Figure 17, the first detection unit 2 and the second detection unit 3 may be provided with blocking sections (light-shielding sections 41A, 41B, 41C, and 41D) that block the incidence of light to the SiPM of the photodetector 33. The control device 6 may generate information on occurrence frequency based on another scanned image generated with the same SiPM settings as when generating the scanned image to be processed, while the blocking sections are blocking the incidence of light to the SiPM. By removing offset components, etc., from the scanned image acquired under such conditions, an image containing only the dark count noise component can be obtained. By converting the brightness values ​​of the image containing only the dark count noise component into the number of photons, the number of pixels containing dark count noise and the number of photons of dark count noise contained in each pixel can be identified. From this information, the occurrence frequency used for image processing may be identified.

[0109] In this specification, the expression "based on A" does not mean "based solely on A," but rather "based on at least A," and furthermore, "based at least partially on A." That is, "based on A" may also mean based on B in addition to A, or based on a part of A. [Explanation of symbols]

[0110] 4 Input devices 5 Display device 6 Control device 6a processor 6b Memory 11 Laser light source 17 Scanners 33A, 33B, 33C, 33D Photodetectors 44A, 44B, 44C, 44D Light shielding part 50 Scanning Image Generation Unit 60 Estimation part 61 Acquisition Department 62 Calculation Section 63 Noise Image Generation Unit 64 Decision Section 70 Memory section 80 Removal part 90 Output section 100 Fluorescence Microscopes

Claims

1. A laser scanning microscope, A scanner that scans a sample with laser light, A detector having a Silicon Photomultiplier (SiPM), The system includes a control device that performs image processing to remove dark count noise from a scanned image of the sample scanned by the scanner, which is generated based on a signal output from the detector, based on the frequency of dark count noise occurrence in the SiPM. The control device is An estimation unit estimates the amount of dark count noise included in the scanned image as noise based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope. The system includes a removal unit that removes the dark count noise from the scanned image according to the amount of noise estimated by the estimation unit, The settings of the laser scanning microscope include the SiPM settings and the scan settings of the laser scanning microscope. The estimation unit, An acquisition unit that acquires information regarding the frequency of occurrence of the dark count noise in the SiPM based on the settings of the SiPM, The system includes a calculation unit that statistically calculates the noise amount based on the frequency of occurrence of the dark count noise identified from the information acquired by the acquisition unit and the scan settings. A laser scanning microscope characterized by the following features.

2. In the laser scanning microscope according to claim 1, The calculation unit calculates the number of pixels containing the dark count noise included in the scanned image as the noise amount, The removal unit subtracts a predetermined value from the pixel values ​​of the aforementioned number of pixels in the scanned image. A laser scanning microscope characterized by the following features.

3. In the laser scanning microscope according to claim 1, The calculation unit statistically calculates the amount of noise for each intensity based on the frequency of occurrence of the dark count noise for each intensity, which is identified from the information acquired by the acquisition unit, and the scan settings. A laser scanning microscope characterized by the following features.

4. In the laser scanning microscope according to claim 3, The calculation unit calculates the number of pixels containing the dark count noise included in the scanned image as the noise amount for each intensity, The removal unit subtracts a value corresponding to the intensity from the pixel values ​​of the number of pixels calculated for each intensity of the scanned image. A laser scanning microscope characterized by the following features.

5. A laser scanning microscope, A scanner that scans a sample with laser light, A detector having a Silicon Photomultiplier (SiPM), The system includes a control device that performs image processing to remove dark count noise from a scanned image of the sample scanned by the scanner, which is generated based on a signal output from the detector, based on the frequency of dark count noise occurrence in the SiPM. The control device is An estimation unit estimates the amount of dark count noise included in the scanned image as noise based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope. The system includes a removal unit that removes the dark count noise from the scanned image according to the amount of noise estimated by the estimation unit, The estimation unit estimates the location of the dark count noise on the scanned image based on the scanned image and the estimated noise amount. The removal unit removes the dark count noise from the appearance position on the scanned image estimated by the estimation unit, by the amount of noise estimated by the estimation unit. A laser scanning microscope characterized by the following features.

6. In the laser scanning microscope according to claim 5, The estimation unit further, A noise image generation unit generates a noise image consisting of a score relating to the dark count noise contained in each pixel of the scanned image based on the scanned image, The system includes a determination unit that determines the location of occurrence based on the noise image and the estimated amount of noise. A laser scanning microscope characterized by the following features.

7. In the laser scanning microscope according to claim 6, The noise image generation unit generates the noise image based on the difference between the scanned image and the scanned image to which a noise reduction filter has been applied. A laser scanning microscope characterized by the following features.

8. In the laser scanning microscope according to claim 6, The noise image generation unit generates the noise image based on the scanned image, the spatial distribution of the occurrence probability of the dark count noise included in the scanned image, a trained model that has learned the relationship between the image with dark count noise added and the spatial intensity distribution of the dark count noise added to the image, and the scanned image. A laser scanning microscope characterized by the following features.

9. In the laser scanning microscope according to claim 1 or claim 5, further, The control device is The SiPM includes a memory for storing information regarding the frequency of occurrence of the dark count noise, The frequency of occurrence is determined from the information obtained from the memory. A laser scanning microscope characterized by the following features.

10. In the laser scanning microscope according to claim 9, The memory stores the information regarding the frequency of occurrence of the dark count noise for each individual SiPM included in the detector. The control device identifies the occurrence frequency from the information corresponding to the individual SiPM used to generate the scanned image, which is obtained from the memory. A laser scanning microscope characterized by the following features.

11. In the laser scanning microscope according to claim 9, The memory stores the information regarding the frequency of occurrence of the dark count noise for each setting of the SiPM. The control device identifies the occurrence frequency from the information obtained from the memory, which corresponds to the settings of the SiPM used to generate the scanned image. A laser scanning microscope characterized by the following features.

12. In the laser scanning microscope according to claim 11, The aforementioned settings for the SiPM include settings relating to the voltage applied to the SiPM. A laser scanning microscope characterized by the following features.

13. In the laser scanning microscope according to claim 11, The aforementioned settings for the SiPM include settings relating to the temperature at which the SiPM is kept constant. A laser scanning microscope characterized by the following features.

14. In the laser scanning microscope according to claim 1 or claim 5, further, The SiPM is equipped with a blocking section that blocks the incidence of light onto the SiPM, The control device determines the occurrence frequency based on an image generated with the same SiPM settings as when generating the scanning image, while the blocking unit is blocking the incidence of light to the SiPM. A laser scanning microscope characterized by the following features.

15. In the laser scanning microscope according to claim 1 or claim 5, The control device further includes an output unit that outputs a corrected image generated by performing the image processing on the scanned image to a display device. A laser scanning microscope characterized by the following features.

16. Equipped with a processor, The processor performs image processing on the scanned image of a sample scanned by a scanner included in a laser scanning microscope, which is generated based on a signal output from a detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, to remove the dark count noise based on the frequency of occurrence of dark count noise in the SiPM. Performing the aforementioned image processing means Based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, the amount of the dark count noise included in the scanned image is estimated as the noise amount. This includes removing the dark count noise from the scanned image according to the estimated amount of noise, The settings of the laser scanning microscope include the SiPM settings and the scan settings of the laser scanning microscope. The above estimation means that Based on the settings of the SiPM, information regarding the frequency of occurrence of the dark count noise in the SiPM is obtained, This includes statistically calculating the amount of noise based on the frequency of occurrence of the dark count noise identified from the acquired information and the scan settings. An image processing apparatus characterized by the following:

17. comprising a processor, The processor performs image processing on the scanned image of a sample scanned by a scanner included in a laser scanning microscope, which is generated based on a signal output from a detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, to remove the dark count noise based on the frequency of occurrence of dark count noise in the SiPM. Performing the aforementioned image processing means Based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, the amount of the dark count noise included in the scanned image is estimated as the noise amount. This includes removing the dark count noise from the scanned image according to the estimated amount of noise, The estimation described above includes estimating the location of the dark count noise on the scanned image based on the scanned image and the estimated noise amount. The removal includes removing the dark count noise from the estimated appearance location on the scanned image by the estimated amount of noise. An image processing apparatus characterized by the following:

18. A method for operating a laser scanning microscope, Based on the signal output from the detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, a scanned image of the sample scanned by the scanner included in the laser scanning microscope is generated. Image processing is performed on the scanned image to remove the dark count noise based on the frequency of occurrence of dark count noise in the SiPM. Performing the aforementioned image processing means Based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, the amount of the dark count noise included in the scanned image is estimated as the noise amount. This includes removing the dark count noise from the scanned image according to the estimated amount of noise, The settings of the laser scanning microscope include the SiPM settings and the scan settings of the laser scanning microscope. The above estimation means that Based on the settings of the SiPM, information regarding the frequency of occurrence of the dark count noise in the SiPM is obtained, This includes statistically calculating the amount of noise based on the frequency of occurrence of the dark count noise identified from the acquired information and the scan settings. A method of operation characterized by the following features.

19. A method for operating a laser scanning microscope, Based on the signal output from the detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, a scanned image of the sample scanned by the scanner included in the laser scanning microscope is generated. Image processing is performed on the scanned image to remove the dark count noise based on the frequency of occurrence of dark count noise in the SiPM. Performing the aforementioned image processing means Based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, the amount of the dark count noise included in the scanned image is estimated as the noise amount. This includes removing the dark count noise from the scanned image according to the estimated amount of noise, The estimation described above includes estimating the location of the dark count noise on the scanned image based on the scanned image and the estimated noise amount. The removal includes removing the dark count noise from the estimated appearance location on the scanned image by the estimated amount of noise. A method of operation characterized by the following features.

20. In the computer of the laser scanning microscope, Based on the signal output from the detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, a scanned image of the sample scanned by the scanner included in the laser scanning microscope is generated. Image processing is performed on the scanned image to remove the dark count noise based on the frequency of dark count noise occurrence in the SiPM. Execute the process, Performing the aforementioned image processing means Based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, the amount of the dark count noise included in the scanned image is estimated as the noise amount. This includes removing the dark count noise from the scanned image according to the estimated amount of noise, The settings of the laser scanning microscope include the SiPM settings and the scan settings of the laser scanning microscope. The above estimation means that Based on the settings of the SiPM, information regarding the frequency of occurrence of the dark count noise in the SiPM is obtained, This includes statistically calculating the amount of noise based on the frequency of occurrence of the dark count noise identified from the acquired information and the scan settings. A program characterized by the following features.

21. A computer for a laser scanning microscope, Based on the signal output from the detector having a Silicon Photomultiplier (SiPM) included in the laser scanning microscope, a scanned image of the sample scanned by the scanner included in the laser scanning microscope is generated. Image processing is performed on the scanned image to remove the dark count noise based on the frequency of dark count noise occurrence in the SiPM. Execute the process, Performing the aforementioned image processing means Based on the frequency of occurrence of the dark count noise in the SiPM and the settings of the laser scanning microscope, the amount of the dark count noise included in the scanned image is estimated as the noise amount. This includes removing the dark count noise from the scanned image according to the estimated amount of noise, The estimation described above includes estimating the location of the dark count noise on the scanned image based on the scanned image and the estimated noise amount. The removal includes removing the dark count noise from the estimated appearance location on the scanned image by the estimated amount of noise. A program characterized by the following features.

Images