Laser scanning microscope system

Abstract

To suppress a distortion of an image in in-vivo imaging.SOLUTION: A laser scan type microscope 1 comprises: a scanner 104 that scans a bio-sample in a main scan direction and in a sub scan direction; a controller 200 that controls the main scan direction; and a processing device 20. The processing device 20 is configured to estimate an oscillation direction of a sample S on the basis of a preliminary image, and next, the processing device 20 is configured to cause the controller 200 to get the main scan direction close to the oscillation direction. Also, the processing device 20 is configured to estimate waveform information on the oscillation on the basis of an object image created by scanning performed in a state where the main scan direction gets close to the oscillation direction. Further, the processing device 20 is configured to perform a pixel shift to the object image in accordance with an amount of displacement occurring in each area of the sample S to be calculated based on the waveform information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a laser scanning microscope system. [Background technology]

[0002] Laser scanning microscope systems, typified by confocal microscopes, are known as observation devices used in in vivo imaging, which observe living biological samples. Laser scanning microscope systems have high resolution in the optical axis direction, and therefore can observe the interior of biological samples, as described in Patent Document 1, for example. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4694139 Summary of the Invention [Problem to be solved by the invention]

[0004] In a laser scanning microscope system that scans a sample two-dimensionally, the information for each pixel that makes up the image reflects the state of the sample at a different time. This results in a time difference of up to one frame between pixels, and when scanning a biological sample, the image can be distorted by vibrations caused by life-sustaining activities such as pulsation and breathing.

[0005] The technology described in Patent Document 1 can prevent focus shift in the optical axis direction caused by pulsation. However, it cannot suppress image distortion. Therefore, a new technology that suppresses image distortion caused by pulsation, breathing, etc. is desired.

[0006] In view of the above circumstances, an object of one aspect of the present invention is to provide a technique for suppressing image distortion in in vivo imaging. [Means for solving the problem]

[0007] A laser scanning microscope system according to one aspect of the present invention includes: a scanner that scans a biological sample in a main scanning direction and a sub-scanning direction; a controller for controlling the direction of the main scanning direction relative to the biological sample; and a device. process The apparatus estimates a vibration direction of the biological sample based on a preliminary image of the biological sample generated by a first scan of the biological sample by the scanner; The controller The scanner is configured to: bring the main scanning direction closer to the vibration direction; estimate waveform information of the vibration of the biological sample based on a target image, which is an image of the biological sample generated by a second scan of the biological sample by the scanner performed with the main scanning direction brought closer to the vibration direction; and perform pixel shifts on the target image according to the amount of displacement that has occurred in each region of the biological sample from the start of the second scan to the scanning time of each of the multiple regions of the biological sample, calculated based on the waveform information. [Effects of the Invention]

[0008] According to the above aspect, image distortion can be suppressed in in vivo imaging. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the configuration of a laser scanning microscope system 1 according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating the configuration of a confocal microscope 100. [Figure 3] FIG. 10 is a diagram for explaining a scanning method. [Figure 4] FIG. 10 is a diagram illustrating an example of distortion occurring in an image region. [Figure 5] FIG. 5 is a diagram for explaining cell deformation due to the distortion occurring in FIG. 4. [Figure 6] FIG. 10 is a diagram illustrating another example of distortion occurring in an image region. [Figure 7]FIG. 7 is a diagram for explaining cell deformation due to strain occurring in FIG. 6. [Figure 8] FIG. 10 is a diagram illustrating yet another example of distortion occurring in an image region. [Figure 9] FIG. 9 is a diagram for explaining cell deformation due to strain occurring in FIG. 8. [Figure 10] FIG. 10 is a diagram for explaining distortion occurring in a confocal image. [Figure 11] 1 is a flowchart of a process performed by the laser scanning microscope system 1. [Figure 12] 10A and 10B are diagrams for explaining the relationship between a vibration waveform and a displacement amount. [Figure 13] FIG. 1 is a diagram for explaining the input and output of a trained model M1 according to the first embodiment. [Figure 14] 4 is a flowchart of a learning process according to the first embodiment. [Figure 15] FIG. 10 is a diagram for explaining the input and output of a trained model M2 according to the second embodiment. [Figure 16] FIG. 10 is a diagram for explaining the input and output of a trained model M3 according to the second embodiment. [Figure 17] 10 is a flowchart of a vibration direction learning process according to the second embodiment. [Figure 18] 10 is a flowchart of a waveform information learning process according to the second embodiment. [Figure 19] 10 is a flowchart of a vibration direction estimation process according to the third embodiment. [Figure 20] FIG. 10 is a diagram illustrating the configuration of a laser scanning microscope system according to a fourth embodiment. [Figure 21] FIG. 10 is a diagram for explaining the input and output of a trained model M4 according to the fourth embodiment. [Figure 22] 10 is a flowchart of a learning process according to a fourth embodiment. [Figure 23] FIG. 1 is a diagram illustrating a hardware configuration of a computer for realizing the processing device according to the embodiment described above. [Figure 24]FIG. 1 is a diagram illustrating the configuration of a two-photon excitation microscope 400. [Figure 25] FIG. 1 is a diagram illustrating the configuration of a confocal microscope 500. [Figure 26] FIG. 6 is a diagram illustrating the configuration of a confocal microscope 600. DETAILED DESCRIPTION OF THE INVENTION

[0010] FIG. 1 is a diagram illustrating the configuration of a laser scanning microscope system 1 according to an embodiment of the present invention. The laser scanning microscope system 1 is a laser confocal microscope equipped with a microscope device 10 and a processing device 20, and generates a confocal image of a sample S. The sample S is not particularly limited, but may be, for example, a biological specimen such as a mouse. The microscope device 10 includes a confocal microscope 100 and a controller 200 that controls the confocal microscope 100.

[0011] Fig. 2 is a diagram illustrating the configuration of the confocal microscope 100. Fig. 3 is a diagram for explaining a scanning method. Hereinafter, a method for generating a confocal image will be explained with reference to Figs. 2 and 3.

[0012] As shown in FIG. 2, the confocal microscope 100 includes a laser 101, a beam expander 102, a dichroic mirror 103, a scanner 104, a pupil projection optical system 105, and an objective lens 106 on the illumination light path.

[0013] The laser 101 emits laser light in, for example, the visible range, ultraviolet range, or infrared range. The output of the laser light emitted from the laser 101 is controlled, for example, by a controller 200. The beam expander 102 is an optical system that adjusts the beam of laser light (collimated light) from the laser 101 in accordance with the pupil diameter of the objective lens 106. The dichroic mirror 103 is a light separation means that separates the excitation light (laser light) from the detection light (fluorescence) from the sample S, and separates the laser light and the fluorescence based on the difference in wavelength.

[0014] The scanner 104 uses laser light to two-dimensionally scan the sample S on the stage 107. The scanner 104 includes, for example, two galvanometer mirrors that scan the sample S in directions perpendicular to each other, and performs a raster scan as shown in FIG. 3. That is, the scanner 104 scans the sample S in the main scanning direction with one galvanometer mirror and shifts the scan line SL in the sub-scanning direction perpendicular to the main scanning direction with the other galvanometer mirror, thereby scanning the sample S in both the main scanning direction and the sub-scanning direction. The main scanning direction and the sub-scanning direction are controlled by the controller 200 by changing the drive timing and drive speed of the two galvanometer mirrors that scan in directions perpendicular to each other. The galvanometer mirror is an example of a scanner mirror included in the scanner 104, and a MEMS mirror or the like can also be used.

[0015] The pupil projection optical system 105 is an optical system that projects an image of the scanner 104 onto the pupil position of the objective lens 106. The objective lens 106 may be provided with a correction collar for correcting spherical aberration that changes depending on the observation depth.

[0016] The confocal microscope 100 further includes, on the detection optical path (the optical path transmitted through the dichroic mirror 103), a mirror 108, a confocal lens 109, a confocal diaphragm 110, a condenser lens 111, and a photodetector 112. The signal output from the photodetector 112 is output to an A / D converter 113.

[0017] The confocal lens 109 is a lens that collects the fluorescence onto the confocal diaphragm 110. The confocal diaphragm 110 is an diaphragm that is arranged at a position optically conjugate with the focal plane of the objective lens 106. The confocal diaphragm 110 has a pinhole that transmits the fluorescence generated from the focal position of the objective lens 106. The condenser lens 111 is a lens that guides the fluorescence that has passed through the confocal diaphragm 110 to a photodetector 112.

[0018] The photodetector 112 is, for example, a photomultiplier tube (PMT), and outputs an analog signal corresponding to the amount of incident fluorescent light. The A / D converter 113 converts the analog signal from the photodetector 112 into a digital signal (luminance signal), and outputs it to the processing device 20.

[0019] In the laser scanning microscope system 1 configured as described above, the confocal microscope 100 scans the sample S with laser light using the scanner 104, and detects fluorescence from each position on the sample S with the photodetector 112 via the confocal optical system (objective lens 106, pupil projection optical system 105, and confocal lens 109). Furthermore, the confocal microscope 100 samples the detected fluorescence signal with the A / D converter 113, thereby outputting a digital signal indicating the light intensity at each spot Sp (see FIG. 3 ) on the biological sample. Thereafter, the processing device 20 generates a confocal image based on the digital signal from the confocal microscope 100 and scanning position information of the scanner 104. Furthermore, the processing device 20 performs pixel shifting, which will be described later, on the confocal image to convert it into an image of the sample S with reduced distortion.

[0020] FIG. 4 is a diagram for explaining an example of distortion occurring in an image region. FIG. 5 is a diagram for explaining cell deformation due to the distortion occurring in FIG. 4. FIG. 6 is a diagram for explaining another example of distortion occurring in an image region. FIG. 7 is a diagram for explaining cell deformation due to the distortion occurring in FIG. 6. FIG. 8 is a diagram for explaining yet another example of distortion occurring in an image region. FIG. 9 is a diagram for explaining cell deformation due to the distortion occurring in FIG. 8. FIG. 10 is a diagram for explaining distortion occurring in a confocal image. Hereinafter, image distortion occurring in a confocal image will be described with reference to FIGS. 4 to 10.

[0021] As shown in Figure 3, a confocal image is generated by sequentially scanning each position of the sample S with a laser beam. Therefore, the information of each pixel in the image is information about the sample S at different times. Furthermore, the sample S is constantly vibrating due to life-sustaining activities such as pulsation and breathing. Therefore, the information about the sample S at different times is added with different displacements corresponding to the time difference caused by the vibrations compared to the information about the sample S at a reference time. As a result, the image is distorted due to the difference in the amount of displacement between the pixels.

[0022] The above-mentioned distortions occurring in the confocal image can be roughly divided into two patterns depending on the relationship between the vibration direction and the main scanning direction. (First pattern) When the main scanning direction is perpendicular to the vibration direction (Second pattern) When the main scanning direction is parallel to the vibration direction

[0023] In the first pattern, in which the main scanning direction is perpendicular to the vibration direction, elements within the scanning range SR expand and contract, resulting in distortion of the image. Specifically, for example, if scanning is performed from the upper left to the lower right of the page, as shown in FIG. 4, the regions (regions R1 and R3) scanned during the period in which vibration causes downward displacement of the sample S are visualized in the image IM11 as regions (regions R11 and R13) that are elongated vertically compared to their original lengths. Furthermore, the region (region R2) scanned during the period in which vibration causes upward displacement of the sample S is visualized in the image IM11 as a region (region R12) that is vertically contracted compared to their original lengths. As a result, as shown in FIG. 5, if the scanning range SR includes multiple cells (cells C1, C2, and C3), cells C1, C2, and C3 expand and contract along the vibration direction in the image, respectively, and are transformed into cells C11, C12, and C13.

[0024] In the second pattern, in which the main scanning direction is parallel to the vibration direction, elements within the scanning range SR move in the vibration direction, causing distortion in the image. Specifically, for example, if scanning is performed from the upper left to the lower right of the page, lines scanned during a period in which the sample S is displaced to the right due to vibration are visualized as shifted to the right. Similarly, lines scanned during a period in which the sample S is displaced to the left due to vibration are visualized as shifted to the left. Therefore, as shown in Figure 6, elongated regions parallel to the sub-scanning direction (regions R4, R5, and R6) are visualized as regions curved left and right along the sub-scanning direction in image IM12 (regions R14, R15, and R16). As a result, as shown in Figure 7, if the scanning range SR includes multiple cells (cells C1, C2, and C3), cells C1, C2, and C3 move left and right in image IM12, respectively, and are visualized as cells C21, C22, and C23, which are positioned differently from their original positions.

[0025] The vibration direction usually includes both main-scanning and sub-scanning components. Therefore, the general distortion that occurs in a confocal image is a combination of distortions caused by these two patterns. In other words, as shown in Figure 8, each region (region R1, region R2, region R3) within the scanning range SR is visualized as being deformed into a complex shape (region R21, region R22, region R23) by stretching and shifting in image IM13. As a result, as shown in Figure 9, if the scanning range SR includes multiple cells (cell C1, cell C2, cell C3), the images of the cells (cell C31, cell C32, cell C33) also undergo complex deformation combining stretching and shifting.

[0026] Furthermore, the shape of the actual sample S is not as simplified as those shown in Figures 5, 7, and 9. In particular, when the sample S is a biological specimen, it almost never has a fixed shape like industrial samples. This makes it even more difficult to estimate the original shape. Therefore, when a confocal image IM100 of a mouse brain, such as that shown in Figure 10, is acquired, it is not easy to determine whether this confocal image correctly visualizes the mouse brain. Furthermore, even if it is possible to recognize that the confocal image is distorted, it is not easy to correctly correct the distortion.

[0027] Note that regions R101 to R103 shown in FIG. 10 show the actual shapes of the respective image regions (image region IM101 to image region IM103) of the confocal image IM100. Image regions IM101 and IM102 show the appearance of specific regions in the brain, and image region IM103 shows how nerve fibers in the brain vary in thickness and thinning. However, as shown in regions R101 and R102, in reality, each of the specific regions has a size that differs from the size on the image. Furthermore, as shown in region R103, the nerve fibers have an almost uniform thickness.

[0028] Fig. 11 is a flowchart of the processing performed by the laser scanning microscope system 1. Fig. 12 is a diagram for explaining the relationship between the vibration waveform and the displacement amount. Below, a method for suppressing the above-mentioned distortion occurring in the confocal image will be described with reference to Figs. 11 and 12.

[0029] The laser scanning microscope system 1 first performs a first scan (step S1). Here, the microscope device 10 scans the sample S with the scanner 104 in accordance with instructions from the processing device 20, and the processing device 20 generates an image of the sample S based on a signal from the microscope device 10. The image generated in step S1 will hereinafter be referred to as a preliminary image.

[0030] Next, the laser scanning microscope system 1 estimates the vibration direction (step S2). Here, the processing device 20 estimates the vibration direction of the sample S based on the preliminary image generated in step S1. The image distortion caused by vibration is a combination of distortions caused by two patterns as described above, and there is a pattern of regularity. By using this pattern, the processing device 20 estimates the vibration direction from the preliminary image.

[0031] There are no particular limitations on the specific method for estimating the vibration direction from the preliminary image. For example, the vibration direction may be estimated using a trained model generated by machine learning. Alternatively, the vibration direction may be estimated by comparing multiple preliminary images. Alternatively, the vibration direction may be estimated using any image processing.

[0032] After estimating the vibration direction, the laser scanning microscope system 1 adjusts the main scanning direction (step S3). Here, the processing device 20 controls the controller 200 so that the main scanning direction approaches the vibration direction estimated in step S1. In other words, the processing device 20 controls the controller 200 to make the controller 200 adjust the main scanning direction to approach the vibration direction estimated in step S1.

[0033] The method for adjusting the main scanning direction is not particularly limited, but for example, the controller 200 may change the drive timing and drive speed of two galvanometer mirrors included in the scanner 104 that scan in directions perpendicular to each other so that the main scanning direction approaches the vibration direction.

[0034] Thereafter, the laser scanning microscope system 1 performs a second scan (step S4). Here, the microscope device 10 scans the sample S with the scanner 104 in a state where the main scanning direction is brought closer to the vibration direction in accordance with an instruction from the processing device 20, and the processing device 20 generates an image of the sample S based on a signal from the microscope device 10. The image generated in step S4 will hereinafter be referred to as the target image.

[0035] By moving the vibration direction closer to the main scanning direction in step S3, the component of the vibration that is perpendicular to the main scanning direction is reduced. This makes it possible to suppress the distortion (expansion / shrinkage) of the first pattern described above, which is included in the target image generated in step S4. Note that suppressing distortion of the first pattern is more effective than suppressing distortion of the second pattern. This is because correcting the distortion of the first pattern requires canceling the expansion / shrinkage that occurs in the image, but such correction results in information loss and degradation of resolution. In other words, suppressing distortion of the first pattern makes it possible to reduce the information loss that occurs during image correction, enabling image correction with minimal degradation of resolution.

[0036] In step S3, it is more preferable that the processing device 20 causes the controller 200 to align the main scanning direction with the vibration direction. By aligning the main scanning direction with the vibration direction, distortion of the first pattern included in the target image can be minimized.

[0037] Next, the laser scanning microscope system 1 estimates vibration waveform information (step S5). Here, the processing device 20 estimates the vibration waveform information of the sample S based on the target image generated in step S4. The waveform information may be output as a parameter set of a predetermined function representing the vibration waveform. For example, assuming that the vibration waveform is a sine wave, the waveform may be expressed as a function defined by a parameter set consisting of amplitude A, period T, phase B, and angle θ (vibration direction). In this case, the processing device 20 may output the amplitude A, period T, phase B, and angle θ as the vibration waveform information. Alternatively, assuming that a vibration waveform with a known vibration direction (angle θ) is a sine wave, the waveform may be output as a parameter set of a predetermined function representing the vibration waveform. In this case, the processing device 20 may output the amplitude A, period T, and phase B as the vibration waveform information.

[0038] The specific method for estimating the vibration waveform information from the target image is not particularly limited. For example, the vibration waveform information may be estimated using a trained model generated by machine learning. Alternatively, the vibration waveform information may be estimated by comparing multiple target images.

[0039] Finally, the laser scanning microscope system 1 performs pixel shifting on the target image (step S6). Here, the processing device 20 calculates the amount of displacement for each of the multiple regions of the sample S from the waveform information estimated in step S5, and performs pixel shifting on the target image according to the calculated amount of displacement. In the second scan of step S4, an image (target image) in which the distortion of the first pattern is suppressed is generated in advance. Therefore, by correcting the distortion of the second pattern by pixel shifting according to the amount of displacement, a corrected image in which distortion caused by vibration is suppressed can be obtained.

[0040] The displacement amount of each region refers to the amount of displacement that occurs in each region from the start of the second scan to the time of scanning each of the multiple regions of the sample S. As shown in Figure 12, once the vibration waveform W(p) is determined, it is possible to calculate the displacement amount D of the region corresponding to each pixel p.

[0041] Furthermore, each area of ​​the sample S is an area on the sample S corresponding to one line. The displacement amount of an area may be the displacement amount of a pixel that represents the area (for example, the pixel scanned first), or may be the average displacement amount of the pixels included in the area.

[0042] The pixel shift is performed in units of regions, so it is desirable that the difference in the amount of displacement (magnitude of distortion) between pixels within a region is sufficiently small.

[0043] Step S6 will be described in more detail. To correct the distortion of the second pattern, it is sufficient to reduce the difference in the amount of displacement between the regions. Therefore, in step S6, the processing device 20 determines the amount of shift for each region (each line) so as to compensate for the difference in the amount of displacement calculated for each region.

[0044] Specifically, the processing device 20 may determine the shift amount so that the displacement amounts of the multiple regions are aligned to the maximum amplitude of vibration (amplitude Amax shown in FIG. 12). Alternatively, the processing device 20 may determine the shift amount so that the displacement amounts are aligned to zero. In other words, the shift amount may be an amount that cancels out the displacement occurring in each region, that is, the displacement amount itself. Furthermore, the displacement amounts of the multiple regions after pixel shifting do not necessarily need to be the same, as long as the difference in displacement amount is reduced by compensating for the difference in displacement amount.

[0045] Once the shift amount is determined, the processing device 20 shifts each pixel of the target image in the main scanning direction by the shift amount determined for the line to which the pixel belongs. In other words, the processing device 20 shifts pixels by a predetermined amount for each line. This reduces the difference in displacement between lines, thereby correcting the distortion of the second pattern.

[0046] As described above, by executing the process shown in FIG. 11 , the laser scanning microscope system 1 can obtain an image in which image distortion caused by vibrations resulting from the life-sustaining activities of a biological sample has been corrected. Therefore, the laser scanning microscope system 1 can suppress image distortion in in vivo imaging. Furthermore, the laser scanning microscope system 1 acquires a target image by setting the main scanning direction so that the expansion / contraction deformation contained in the preliminary image is converted into pixel shift. Therefore, correction can be performed simply by pixel shift processing without performing scaling processing on the target image. This reduces the computational load and avoids degradation of resolution. Therefore, a high-resolution image with suppressed distortion can be obtained in a short time by correcting the target image (confocal image).

[0047] Specific examples of the above-described laser scanning microscope system 1 will be described below in the form of respective embodiments. First Embodiment FIG. 13 is a diagram for explaining input and output of a trained model M1 according to this embodiment. FIG. 14 is a flowchart of a learning process according to this embodiment. Hereinafter, this embodiment will be described with reference to FIGS. 13 and 14. Note that the configuration of a laser scanning microscope system according to this embodiment (hereinafter simply referred to as a laser scanning microscope system) is the same as that of the laser scanning microscope system 1 shown in FIGS. 1 and 2. Furthermore, the flow of processing executed by the laser scanning microscope system according to this embodiment is as shown in FIG. 11.

[0048] In the laser scanning microscope system of this embodiment, the vibration direction estimation process (step S2) and vibration waveform information estimation process (step S5) shown in Figure 11 are performed using a trained model M1 that has learned the relationship between the image, scanning time, and waveform information (amplitude A, angle θ, period T, phase B) of the vibration of the sample captured in the image, as shown in Figure 13.

[0049] Specifically, in step S2, the processing device 20 inputs the preliminary image and the scanning time of the scanner 104 when acquiring the preliminary image into the trained model M1, and estimates the vibration direction by acquiring the vibration direction (angle θ) contained in the beat information (waveform information) output from the trained model M1.

[0050] In step S5, the processing device 20 inputs the target image and the scanning time of the scanner 104 when acquiring the target image into the trained model M1, and estimates the vibration waveform by acquiring the amplitude A, period T, and phase B contained in the pulsation information output from the trained model M1. The scanning time is input because the heart rate differs depending on the scanning time even if the deformation in the image is the same.

[0051] The trained model M1 shown in Fig. 13 can be generated by previously executing the learning process shown in Fig. 14. In the following, we will explain an example in which the trained model M1 is trained using the same laser scanning microscope system as used during observation, but training may also be performed using a different laser scanning microscope system.

[0052] The laser scanning microscope system first scans a sample of the same type as sample S (e.g., a mouse) in a state where the vibration of the sample is stopped (step S11). Here, the state where the vibration is stopped is, for example, a state where the mouse is dead.

[0053] Next, the laser scanning microscope system converts the image obtained in step S11 (image without vibration) into an image with vibration using pulsation information (step S12). The pulsation information is, for example, information that represents the vibration waveform of the pulsation of the sample. Like the vibration waveform information described above, the pulsation information is a parameter set of a predetermined function that represents the vibration waveform caused by pulsation. Here, the processing device 20 converts the image without vibration into an image with vibration by adding the displacement that occurs in each pixel up to the time of scanning, which is calculated using the pulsation information, to the image without vibration. Note that in step S12, parameter values ​​of the pulsation information are changed to generate multiple images with vibration, and a sufficient number of images with vibration are generated for learning.

[0054] There are no particular limitations on the method for calculating the amount of displacement to be added to each pixel, but it can be calculated, for example, as follows: If the pixel position in the main scanning direction is x, the pixel position in the sub-scanning direction is y, the number of pixels in the main scanning direction is W, the scanning time per pixel is tp, and the value obtained by dividing the retrace time by the scanning time per pixel is R, the time t(x, y) when each pixel is scanned can be approximated by the following formula. t(x, y)=(y×(W+R)+x)×tp

[0055] When pulsation is approximated by a sine wave, the pixel value of the image without vibration is li(x, y), the component of the amplitude of the pulsation in the main scanning direction is Ax, and the component of the amplitude of the pulsation in the sub-scanning direction is Ay, then the pixel value of the image with vibration, lo(x, y), can be expressed by the following equation. lo(x, y)=li(x´, y´)

[0056] where x', y', Ax, and Ay are as follows: x´=x+Ax×sin((2π / T)×(t(x,y)+B)) y´=y+Ay×sin((2π / T)×(t(x,y)+B)) Ax=Acosθ Ay=A sinθ

[0057] Although the example in which the displacement amount is calculated for each pixel and added up has been shown, if the scanning time per line is sufficiently short compared to the vibration period, it is also possible to calculate the displacement amount for each line using, for example, the pixel at the beginning of the line and assign the same displacement amount to each line. This can shorten the time required to generate a large amount of training data.

[0058] Finally, the laser scanning microscope system learns a combination of the image with vibration, the pulsation information, and the scanning time (step S13). Here, the processing device 20 trains the trained model M1 so that it outputs the pulsation information used in step S12 in response to an input of the combination of the image with vibration generated in step S12 and the scanning time in step S11.

[0059] The laser scanning microscope system according to this embodiment can correct image distortion caused by pulsation, thereby suppressing image distortion in in vivo imaging.

[0060] <Second embodiment> FIG. 15 is a diagram for explaining input and output of a trained model M2 according to this embodiment. FIG. 16 is a diagram for explaining input and output of a trained model M3 according to this embodiment. FIG. 17 is a flowchart of a vibration direction learning process according to this embodiment. FIG. 18 is a flowchart of a waveform information learning process according to this embodiment. Hereinafter, this embodiment will be described with reference to FIGS. 15 to 18. Note that the configuration of a laser scanning microscope system according to this embodiment (hereinafter simply referred to as a laser scanning microscope system) is the same as the laser scanning microscope system 1 shown in FIGS. 1 and 2. Also, the flow of processing executed by the laser scanning microscope system according to this embodiment is as shown in FIG. 11.

[0061] In the laser scanning microscope system according to this embodiment, the vibration direction estimation process (step S2) shown in Fig. 11 is performed using a trained model M2 that has learned the relationship between an image and the vibration direction of a sample captured in the image, as shown in Fig. 15, and the vibration waveform information estimation process (step S5) shown in Fig. 11 is performed using a trained model M3 that has learned the relationship between an image, scanning time, and the waveform of vibration of a sample captured in the image, as shown in Fig. 16. Note that the trained model M3 differs from the trained model M1 in that training is performed using an image obtained by converting an image (image without vibration) into an image with vibration using pulsation information related to the main scanning direction.

[0062] Specifically, in step S2, the processing device 20 inputs the preliminary image to the trained model M2 and acquires the vibration direction (angle θ) output from the trained model M2, thereby estimating the vibration direction.

[0063] Also, in step S5, the processing device 20 inputs the target image and the scanning time of the scanner 104 when acquiring the target image into the trained model M3, and estimates the vibration waveform by acquiring the amplitude A, period T, and phase B contained in the pulsation information regarding the main scanning direction output from the trained model M3.

[0064] The trained model M2 shown in Fig. 15 may be generated by previously executing the training process shown in Fig. 17. Note that the processes of steps S21 and S22 shown in Fig. 17 are similar to the processes of steps S11 and S12 shown in Fig. 14.

[0065] Finally, the laser scanning microscope system learns the combination of the image with vibration and the vibration direction (step S23). Here, the processing device 20 trains the trained model M2 so that it outputs the vibration direction (angle θ) included in the pulsation information used in step S22 in response to the input of the image with vibration generated in step S22.

[0066] The trained model M3 shown in Fig. 16 may be generated by previously executing the training process shown in Fig. 18. Note that the process of step S31 shown in Fig. 18 is the same as the process of step S11 shown in Fig. 14.

[0067] Thereafter, the image obtained in step S31 (image without vibration) is converted into an image with vibration using pulsation information related to the main scanning direction (step S32). The pulsation information related to the main scanning direction is, for example, information that represents a vibration waveform related to the main scanning direction component of the pulsation of the sample. The pulsation information related to the main scanning direction does not need to include the angle θ, or the angle θ may be given as a fixed value (for example, θ=0).

[0068] Finally, the laser scanning microscope system learns a combination of the image with vibration and pulsation information related to the main scanning direction (step S33). Here, the processing device 20 trains the trained model M3 so that it outputs the pulsation information (amplitude A, period T, phase B) related to the main scanning direction used in step S32 in response to an input of the combination of the image with vibration generated in step S32 and the scanning time in step S31.

[0069] The laser scanning microscope system according to this embodiment can also correct image distortion caused by pulsation, thereby suppressing image distortion in in vivo imaging.

[0070] <Third embodiment> Fig. 19 is a flowchart of vibration direction estimation processing according to this embodiment. Hereinafter, this embodiment will be described with reference to Fig. 19. Note that the configuration of a laser scanning microscope system according to this embodiment (hereinafter simply referred to as a laser scanning microscope system) is the same as the laser scanning microscope system 1 shown in Figs. 1 and 2. Also, the flow of processing executed by the laser scanning microscope system according to this embodiment is as shown in Fig. 11.

[0071] In the laser scanning microscope system according to this embodiment, the vibration direction estimation process shown in Fig. 19 is performed in step S2 shown in Fig. 11. Specifically, the laser scanning microscope system first changes the main scanning direction and performs multiple scans (step S41). That is, multiple first scans are performed by the scanner 104 in multiple states with different main scanning directions, and multiple preliminary images are generated. For example, the main scanning direction may be changed 18 times by 10°, and multiple scans may be performed each time the main scanning direction is changed.

[0072] Thereafter, the laser scanning microscope system compares the multiple preliminary images generated in step S41 (step S42) and estimates the vibration direction (step S43). Here, in step S42, the processing device 20 may extract the same feature point from multiple preliminary images acquired in the same main scanning direction and obtain the maximum amount of variation in the main scanning direction coordinates of the feature point for each main scanning direction. In step S43, the processing device 20 may identify the main scanning direction with the largest amount of maximum variation and estimate that main scanning direction as the vibration direction.

[0073] The laser scanning microscope system according to this embodiment can also correct image distortion caused by pulsation, thereby suppressing image distortion in in vivo imaging.

[0074] <Fourth embodiment> FIG. 20 is a diagram illustrating the configuration of a laser scanning microscope system according to this embodiment. FIG. 21 is a diagram for explaining the input and output of a trained model M4 according to this embodiment. FIG. 22 is a flowchart of the learning process according to this embodiment. The laser scanning microscope system shown in FIG. 20 differs from the laser scanning microscope system 1 in that it includes a sensor 300 attached to a sample S. In other respects, it is the same as the laser scanning microscope system 1. The sensor 300 is a sensor that measures the waveform of a biological sample, such as an electrocardiogram sensor. The sensor waveform (electrocardiogram waveform) measured by the sensor 300 is output to the controller 200. The flow of processing executed by the laser scanning microscope system according to this embodiment is as shown in FIG. 11.

[0075] In the laser scanning microscope system of this embodiment, the vibration waveform information estimation process (step S5) shown in Figure 11 is performed using a trained model M4 shown in Figure 21, which has learned the relationship between the vibration waveform of the sample shown in the image and the sensor waveform obtained by the sensor 300 measuring the vibration of the sample.

[0076] Specifically, in step S5, the processing device 20 inputs the sensor waveform obtained by measuring the vibration of the sample S with the sensor 300 while acquiring the target image (during the second scan) into the trained model M4, and estimates the vibration waveform by acquiring the amplitude A, period T, and phase B contained in the pulsation information regarding the main scanning direction output from the trained model M4.

[0077] The trained model M4 shown in Fig. 21 can be generated by previously executing the training process shown in Fig. 22. The laser scanning microscope system first scans a sample of the same type as the sample S (e.g., a mouse) while measuring an electrocardiogram, with the vibration direction and the main scanning direction aligned (step S51). This allows for a combination of an image in which the vibration direction and the main scanning direction are aligned and the sensor waveform during image acquisition.

[0078] Next, the laser scanning microscope system calculates pulsation information in the main scanning direction from the image obtained in step S51 using the trained model (step S52). Here, for example, the trained model M3 described above may be used. Note that steps S51 and S52 are repeated until sufficient sensor waveforms and pulsation information necessary for training are obtained.

[0079] Finally, the laser scanning microscope system learns a combination of the sensor waveform acquired in step S51 and the pulsation information calculated in step S52 using the image corresponding to the sensor waveform (step S53). Here, the processing device 20 trains the trained model M4 so as to output the vibration direction (angle θ) included in the pulsation information calculated in step S52 in response to the input of the sensor waveform acquired in step S52.

[0080] The laser scanning microscope system according to this embodiment can also correct image distortion caused by pulsation, thereby suppressing image distortion in in vivo imaging.

[0081] Fig. 23 is a diagram illustrating a hardware configuration of a computer for realizing the processing device according to the above-described embodiment. The hardware configuration shown in Fig. 23 includes, for example, a processor 21, a memory 22, a storage device 23, a reading device 24, a communication interface 26, and an input / output interface 27. The processor 21, the memory 22, the storage device 23, the reading device 24, the communication interface 26, and the input / output interface 27 are connected to one another via, for example, a bus 28.

[0082] The processor 21 may be, for example, a single processor, a multiprocessor, or a multi-core processor. The processor 21 reads and executes a program stored in the storage device 23, thereby executing the control processing exemplified in FIG. 11 and the like.

[0083] The memory 22 is, for example, a semiconductor memory and may include a RAM area and a ROM area. The storage device 23 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device.

[0084] The reader 24 accesses the removable storage medium 25 in accordance with, for example, an instruction from the processor 21. The removable storage medium 25 is realized by, for example, a semiconductor device, a medium for inputting and outputting information by magnetic action, or a medium for inputting and outputting information by optical action. An example of a semiconductor device is a USB (Universal Serial Bus) memory. An example of a medium for inputting and outputting information by magnetic action is a magnetic disk. An example of a medium for inputting and outputting information by optical action is a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc), a Blu-ray Disc, etc. (Blu-ray is a registered trademark).

[0085] The communication interface 26 communicates with other devices, for example, in accordance with instructions from the processor 21. The input / output interface 27 is, for example, an interface between an input device and an output device. The input device is, for example, a device such as a keyboard, mouse, or touch panel that accepts instructions from a user. The output device is, for example, a display device such as a monitor, and an audio device such as a speaker.

[0086] The program executed by the processor 21 is provided to the computer in the following form, for example. (1) It is pre-installed in the storage device 23. (2) Provided by a removable storage medium 25. (3) Provided from a server such as a program server.

[0087] 23 is an example, and embodiments are not limited thereto. For example, part of the above-described configuration may be deleted, or new configuration may be added. In another embodiment, for example, part or all of the functions of the above-described processing device may be implemented as hardware using an FPGA (Field Programmable Gate Array), an SoC (System-on-a-Chip), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or the like.

[0088] The above-described embodiments are illustrative examples provided to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modifications and alternatives to the above-described embodiments may be included. In other words, the components of each embodiment may be modified without departing from the spirit and scope of the invention. Furthermore, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. Furthermore, some components may be deleted from or added to the components shown in each embodiment. Furthermore, the order of the processing steps shown in each embodiment may be reversed as long as no contradictions are present. In other words, the laser scanning microscope system of the present invention can be modified and altered in various ways without departing from the scope of the claims.

[0089] Fig. 24 is a diagram illustrating the configuration of a two-photon excitation microscope 400. In the above-described embodiment, an example was shown in which the laser scanning microscope system included the confocal microscope 100, but the laser scanning microscope system may include the two-photon excitation microscope 400 shown in Fig. 24 instead of the confocal microscope 100.

[0090] As shown in FIG. 24, the two-photon excitation microscope 400 includes a laser 401, a scanner 402, a pupil projection optical system 403, a mirror 404, a dichroic mirror 405, and an objective lens 406 on the illumination light path.

[0091] The laser 401 is, for example, an ultrashort pulse laser that emits laser light in the near-infrared region. The scanner 402 is, for example, two galvanometer mirrors that scan in the main scanning direction and the sub-scanning direction. The pupil projection optical system 403 projects an image of the scanner 402 onto the pupil position of the objective lens 406. The dichroic mirror 405 is a light separation means that separates the excitation light (laser light) from the detection light (fluorescence) from the sample S, and separates the laser light and fluorescence by wavelength.

[0092] The two-photon excitation microscope 400 further includes a pupil projection optical system 407 and a photodetector 408 on the detection optical path (the reflected optical path of the dichroic mirror 405). The signal output from the photodetector 408 is output to an A / D converter 409.

[0093] The pupil projection optical system 407 is an optical system that projects an image of the pupil of the objective lens 406 onto a photodetector 408. The photodetector 408 is, for example, a photomultiplier tube (PMT), and outputs an analog signal corresponding to the amount of incident fluorescent light. The A / D converter 409 converts the analog signal from the photodetector 408 into a digital signal (luminance signal), and outputs it to the processing device 20.

[0094] Even with a multiphoton excitation microscope such as the two-photon excitation microscope shown in Fig. 24, distortion occurs in the image for the same reasons as with a confocal microscope. For this reason, the image distortion may be corrected by performing the processing shown in Fig. 11.

[0095] In the above-described embodiment, an example has been shown in which the controller 200 changes the drive timing and drive speed of two galvanometer mirrors that scan in directions perpendicular to each other, thereby bringing the main scanning direction closer to the vibration direction, but the main scanning direction may also be adjusted by other methods.

[0096] For example, the laser scanning microscope system may include a confocal microscope 500 shown in FIG. 25, which includes an image rotator 501 between the scanner 104 and the dichroic mirror 103, instead of the confocal microscope 100. In this case, the controller 201 that controls the image rotator 501 may change the orientation of the image rotator 501 so that the main scanning direction approaches the vibration direction. Even when the main scanning direction is adjusted in this manner, image distortion can be corrected by performing the processing shown in FIG. 11. Note that by using the image rotator 501, a resonant scanner with restrictions on changing the rotation direction may be used as the scanner 104.

[0097] Furthermore, the laser scanning microscope system may include a confocal microscope 600 shown in FIG. 26, which includes a rotation stage 601 on which the sample S is placed, instead of the confocal microscope 100. In this case, the controller 202 that controls the rotation stage 601 may change the orientation of the rotation stage 601 so that the main scanning direction approaches the vibration direction. Even when the main scanning direction is adjusted in this manner, image distortion can be corrected by performing the processing shown in FIG. 11. Note that by using the rotation stage 601, a resonant scanner with restrictions on changing the rotation direction may be used as the scanner 104.

[0098] Furthermore, in the above-described embodiment, the fluorescence observation method for detecting fluorescence has been described as an example, but the observation method performed by the laser scanning microscope system is not limited to the fluorescence observation method, and the above-described laser scanning microscope system may be applied to other observation methods.

[0099] Although the above-described embodiment shows an example of observing a biological sample, the sample is not limited to a biological sample. Even in the case of an industrial sample or the like, if regular vibrations occur during scanning for some reason, the distortion caused by the vibrations can be corrected by applying the above-described laser scanning microscope system.

[0100] In addition, in the above-described embodiment, vibrations caused by pulsation were used as an example, but image distortion may be corrected by estimating vibrations caused by breathing instead of or in addition to pulsation. Similar to pulsation, vibrations caused by breathing can be considered periodic, so a vibration resulting from a combination of pulsation and breathing may be treated as waveform information. Furthermore, in the above-described embodiment, an example was shown in which waveform information was estimated assuming the vibration to be a sinusoidal vibration, but the vibration waveform is not limited to a sinusoidal waveform. The vibration waveform may be approximated by a function that more closely resembles the actual vibration.

[0101] In the above-described embodiment, an example was shown in which the shift amount was determined for each region, which is the calculation unit of the displacement amount, but the calculation unit of the displacement amount and the shift amount may be different. For example, the displacement amount may be calculated for each region consisting of multiple pixels, and the shift amount may be determined for each pixel from the displacement amount calculated for each region.

[0102] In this specification, the expression "based on A" does not mean "based only on A," but also means "based at least on A," and further means "based at least partially on A." That is, "based on A" may be based on B in addition to A, or may be based on a part of A. [Explanation of symbols]

[0103] 1. Laser scanning microscope system 10 Microscope equipment 20 Processing equipment 21 processors 100, 500, 600 Confocal Microscope 101, 401 laser 104, 402 scanner 200, 201, 202 Controllers 300 sensors 400 Two-Photon Excitation Microscope 501 Image Rotator 601 Rotating Stage M1, M2, M3 pre-trained models

Claims

1. a scanner that scans the biological sample in a main scanning direction and a sub-scanning direction; a controller that controls the direction of the main scanning direction relative to the biological sample; a processing device; The processing device includes: estimating a vibration direction of the biological sample based on a preliminary image of the biological sample generated by a first scan of the biological sample by the scanner; causing the controller to move the main scanning direction with respect to the biological sample closer to the vibration direction; determining a shift amount in the main scanning direction for each of a plurality of lines of a target image, which is an image of the biological sample generated by the second scan, the shift amount being calculated based on waveform information of the vibration of the biological sample, and which corresponds to the plurality of regions, so as to compensate for a difference in the amount of displacement that has occurred in each region from the start of scanning in the second scan of the biological sample by the scanner, which is performed with the main scanning direction brought close to the vibration direction, to the time of scanning each of the plurality of regions of the biological sample; shifting the plurality of lines of the target image in the main scanning direction by the shift amount of the plurality of lines; configured to run A laser scanning microscope system characterized by:

2. 2. The laser scanning microscope system according to claim 1, The processing device is further configured to estimate waveform information of vibrations of the biological specimen based on the target image. A laser scanning microscope system characterized by:

3. 3. The laser scanning microscope system according to claim 1, the scanner includes two scanner mirrors that scan in directions orthogonal to each other; The controller controls the two scanner mirrors so that the main scanning direction approaches the vibration direction. A laser scanning microscope system characterized by:

4. 3. The laser scanning microscope system according to claim 1, further comprising: Includes an image rotator The controller changes the orientation of the image rotator so that the main scanning direction approaches the vibration direction. A laser scanning microscope system characterized by:

5. 3. The laser scanning microscope system according to claim 1, further comprising: a rotating stage on which the biological sample is placed, The controller changes the orientation of the rotary stage so that the main scanning direction approaches the vibration direction. A laser scanning microscope system characterized by:

6. 6. The laser scanning microscope system according to claim 1, The step of estimating the vibration direction includes inputting the preliminary image into a trained model that has trained a relationship between an image and a vibration direction of a sample shown in the image. A laser scanning microscope system characterized by:

7. 6. The laser scanning microscope system according to claim 1, The step of estimating the vibration direction includes comparing a plurality of preliminary images generated by a plurality of first scans by the scanner performed in a plurality of states in which the main scanning direction is different from one another. A laser scanning microscope system characterized by:

8. 3. The laser scanning microscope system according to claim 2, The estimation of the vibration waveform information includes inputting the target image and the scanning time of the scanner required to acquire the target image into a trained model that has trained the relationship between the image, the scanning time, and the waveform of the vibration of the sample captured in the image. A laser scanning microscope system characterized by:

9. The laser scanning microscope system according to claim 1, further comprising: a sensor for measuring vibrations of the biological sample; The processing device is further configured to estimate the waveform information based on a waveform of vibration of the biological sample measured by the sensor; Inferring the waveform information includes inputting a sensor waveform obtained by the sensor measuring the vibration of the biological sample during the second scan into a trained model that has trained a relationship between the waveform of the vibration of the sample shown in the image and a sensor waveform obtained by the sensor measuring the vibration of the sample. A laser scanning microscope system characterized by:

10. 3. The laser scanning microscope system according to claim 2, estimating the vibration direction includes inputting the preliminary image and the scanning time of the scanner required to acquire the preliminary image into a trained model that has trained a relationship between an image, a scanning time, and a waveform of a sample captured in the image; The step of estimating the waveform information of the vibration includes inputting the target image and the scanning time of the scanner required to acquire the target image into the trained model. A laser scanning microscope system characterized by:

Images