Sample observation device
The sample observation device addresses the low contrast issue in phase contrast microscopy by using an image processing unit to adjust contrast and reduce noise, enhancing the visibility of transparent samples by optimizing parameters related to sample distance, brightness, and density, and adding margin images, resulting in clearer sample observations.
Patent Information
- Application Number
- PCT/JP2024/016845
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-30
- Filing Date
- 2024-05-02
- Publication Date
- 2024-12-05
AI Technical Summary
Phase contrast microscopes struggle to clearly observe colorless and transparent samples due to low image contrast, especially when out of focus, making it difficult to distinguish boundaries and shapes.
A sample observation device with an illumination optical system, imaging optical system, image sensor, and image processing unit that enhances image contrast by adjusting parameters based on the distance between the objective lens and sample, brightness values, sample density, and magnification, while also reducing noise and adding margin images to improve visibility.
The device significantly enhances image contrast and clarity, allowing for better observation of samples by correcting contrast changes based on these factors and reducing noise, thereby improving the visibility of transparent samples.
Smart Images

Figure JP2024016845_05122024_PF_FP_ABST
Abstract
Description
Sample observation device
[0001] The present invention relates to a sample observation device.
[0002] A phase-contrast microscope is known as a device for observing living cells.
[0003] Patent Publication No. 2017-072700
[0004] A sample observation device according to a first aspect of the present invention includes an illumination optical system that irradiates illumination light onto a sample through an objective lens, a reflecting member that is installed opposite the objective lens and reflects light, an imaging optical system that forms an image from the light received by the objective lens, an image sensor that captures the image formed by the imaging optical system, and an image processing unit that performs image processing to enhance the contrast of the image captured by the image sensor.
[0005] A sample observation device according to a second aspect of the present invention comprises an illumination optical system that irradiates illumination light onto a sample via an objective lens, an imaging optical system that forms an image from the light received by the objective lens, an image sensor that captures the image formed by the imaging optical system, and an image processing unit that performs image processing to enhance the contrast of the image captured by the image sensor, wherein the image processing unit has a noise reduction processing unit that performs processing to reduce noise, and the noise reduction processing unit performs the noise reduction processing with different intensities for a foreground that is the portion of the image in which the sample is shown, and a background that is the portion of the image in which other than the sample is shown.
[0006] FIG. 1 is a schematic diagram showing the schematic configuration of a time-lapse observation device including a sample observation device according to an embodiment; FIG. 2 is a side cross-sectional view showing the schematic configuration of a microscope device that is a part of the sample observation device according to an embodiment; FIG. 3 is a flowchart showing the flow of image processing according to an embodiment; FIG. 4 is a schematic diagram showing an example of a margin image added to an image in margin addition processing;
[0007] Preferred embodiments of the present invention will be described below. Note that the following embodiments do not limit the scope of the invention. Although multiple features are described in the embodiments, not all of these are necessarily essential to the invention. Furthermore, multiple features may be combined in any desired manner.
[0008] First, a time-lapse observation device 100 equipped with a sample observation device according to this embodiment will be described with reference to FIG. 1 . FIG. 1 is a schematic diagram showing the general configuration of the time-lapse observation device 100 equipped with a sample observation device according to this embodiment. The time-lapse observation device 100 is an observation device for observing living cells over time. It is composed of a time-lapse observation chamber 101 capable of maintaining a constant internal environment, a microscope MS, and a control device 103. The microscope MS and the control device 103 include the components of the sample observation device according to this embodiment. When observing a sample, if the sample is colorless and transparent, the image obtained with the microscope MS has low contrast at the in-focus position. At a defocused position slightly off the in-focus position, the contrast is restored, but the boundaries between colors and shapes are unclear, making it difficult to observe the sample. The sample observation device according to this embodiment aims to enhance the contrast of the image obtained with the microscope MS, and the time-lapse observation device 100 observes the sample using this sample observation device. It should be noted that FIG. 1 shows the main components described below, but does not show all of the components of the time-lapse observation device 100 .
[0009] The time-lapse observation chamber 101 is equipped with a door 102 for inserting and removing samples, and when the door is closed, a constant environment is maintained inside, allowing samples such as living cells to be observed over time. The time-lapse observation chamber 101 may also have an artificial climate generator (not shown). A microscope device MS is placed inside the time-lapse observation chamber 101, and a control device 103 is installed outside the time-lapse observation chamber 101. The time-lapse observation chamber 101 can be, for example, an incubator.
[0010] The control device 103 has an image processing unit P that performs image processing to enhance the contrast of images obtained by the microscope device MS, and an operation unit 104 that remotely controls the microscope device MS. The image processing unit P performs image processing on images obtained by the microscope device MS using computer programs and data stored in a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory). The results of the image processing in the image processing unit P can be observed on a display device such as a monitor (not shown). The operation unit 104 remotely controls the microscope device MS using various buttons, a user interface, etc.
[0011] FIG. 2 is a side cross-sectional view showing a schematic configuration of a microscope MS including a sample observation device 1 according to this embodiment. The microscope MS includes a light source 10 via an objective lens OBL, an epi-illumination light source unit 11 that directs light from the light source 10 to a sample SA mounted on a sample stage 12 above the optical system, a reflecting member 14 such as a mirror that faces the objective lens OBL and is mounted above the sample SA, the objective lens OBL, an imaging lens IL, an image sensor 15 positioned at the image plane I formed by the imaging lens IL, and a movable stage 16. The sample stage 12 is transmissive to the light from the light source 10, and an observation container 13 is placed on the sample stage. The sample SA is placed in the observation container 13. The sample observation device 1 according to this embodiment includes the objective lens OBL, the light source 10, the light source unit 11, the imaging lens IL, the reflecting member 14, the image sensor 15, and the image processing unit P. The sample observation device 1 may be configured without the reflecting member 14. The sample observation device 1 will be described in detail later. A well plate, for example, can be used as the observation container 13. The areas surrounded by dashed lines indicate the approximate range of each component.
[0012] A method for obtaining a reflected bright-field image using a microscope MS using a sample observation device 1 will be described. In the microscope MS, light from a light source 10 is emitted upward in FIG. 2 by a half mirror 11a in a light source unit 11, then passes through a sample stage 12 via an objective lens OBL, and is irradiated perpendicularly onto a sample SA. The light irradiated onto the sample SA passes through the sample, is reflected by the culture medium in an observation container 13 and the observation container 13, and a portion of the light returns as light that has passed through the sample SA. In a configuration including a reflecting member 14, the light further passes through the sample SA, is reflected by the reflecting member 14, and returns as light that has passed through the sample SA. This light is imaged on an image plane I by an objective optical system composed of an objective lens OBL and an imaging lens IL, which are arranged below the sample SA, to obtain a reflected bright-field image.
[0013] Next, a method for generating and observing an image from the obtained reflected bright-field image will be described. The objective optical system is composed of an objective lens OBL and an imaging lens IL. Light from the light source 10 is converted into parallel light by the objective lens OBL and focused on image plane I by the imaging lens IL. An image sensor 15 is installed at the position of image plane I of the imaging lens IL. The image sensor 15 is a solid-state image sensor and has multiple pixels. The image focused on image plane I by the imaging lens IL is photoelectrically converted by the image sensor 15 and can be observed as an image on a display device such as a monitor (not shown). The microscope MS can be remotely operated using the operation unit 104 of the control device 103. The objective optical system can change the field of view of the obtained image by moving the movable stage 16 in the X direction, which is the left-right direction in FIG. 2, and in the Y direction, which is perpendicular to the X direction. The objective optical system can also be moved in the Z direction, which is perpendicular to both the X direction and the Y direction, which is the up-down direction in FIG. 2, to adjust the focus position. At this time, the movable stage 16 may be configured to be movable in the Z direction together with the objective optical system, so that the focusing position can be adjusted. The objective optical system, the image sensor 15, the light source 10, the light source unit 11, and the movable stage 16 are disposed in a sealed container 17.
[0014] The image processing device P performs image processing to enhance the contrast of the image obtained by the image pickup device 15, and displays an image for observation (hereinafter referred to as "observation image") on a display device (not shown).
[0015] Next, the sample observation device according to this embodiment will be described.
[0016] A sample observation device 1 (1) as an example of a sample observation device according to the first embodiment includes an illumination optical system that irradiates illumination light onto a sample via an objective lens, a reflective member that is installed opposite the objective lens and reflects light, an imaging optical system that forms an image from the light received by the objective lens, an image sensor that captures the image formed by the imaging optical system, and an image processing unit that performs image processing to enhance the contrast of the image captured by the image sensor.
[0017] According to the first embodiment, light from a light source is irradiated onto a sample via an objective lens, and the light that has passed through the sample is reflected by a reflecting member and irradiates the sample again, thereby increasing the intensity of the light that passes through the sample. This increases the contrast of the image obtained by imaging. Furthermore, by including an image processing unit, the contrast of the image obtained by image processing can be increased.
[0018] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may have an intensity setting unit that determines parameters used in image processing to enhance contrast, depending on the distance between the objective lens and the sample.
[0019] As a result, by correcting the image contrast, which changes depending on the distance between the objective lens and the sample, with the parameter value, the contrast can be enhanced, making it easier to observe the image. Note that the closer the distance between the objective lens and the sample is to the focal length of the objective lens, the lower the image contrast, so it is preferable to set the parameter value so that the effect of enhancing the contrast is greater. Also, the closer the sample is to the in-focus position, the lower the image contrast, so the parameter value may be set so that the effect of enhancing the contrast is greater. Note that the in-focus position may be a position within the depth of field.
[0020] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may have an intensity setting unit that determines parameters used in image processing to enhance contrast according to the brightness value of the image.
[0021] By correcting the contrast of an image, which varies depending on the brightness value, with the parameter value, the contrast can be strengthened, making it possible to make the image easier to observe.
[0022] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may calculate the brightness value based on the difference between the maximum and minimum brightness values in the image.
[0023] In this way, the difference between the brightest and darkest luminance values in the image can be quantified, and the luminance value can be calculated based on the width of the luminance expression that constitutes the image. Alternatively, the image can be divided into small regions, and the luminance value can be calculated based on the difference between the maximum and minimum luminance values for each small region, and then the luminance values can be added up for the entire image.
[0024] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may calculate the brightness value based on the variance of each pixel in the entire image.
[0025] This allows the luminance value to be calculated based on the number of pixels whose values deviate from the average luminance value by quantifying the variance of the luminance values across the entire image. Alternatively, the image may be divided into small regions, the luminance value calculated based on the variance of each small region, and the luminance values calculated across the entire image may be added together.
[0026] In the sample observation device 1 (1) according to the first embodiment, the intensity setting unit may determine parameters to be used in image processing for enhancing contrast according to the magnitude relationship between the brightness value and a preset reference value.
[0027] This allows the parameters to be applied according to the brightness value when determining the intensity. There is not necessarily one reference value. For example, in an 8-bit (0 to 255) brightness range, if the difference between the maximum and minimum brightness values is less than half (brightness value 128), the contrast is low and observation becomes difficult. Such a value can be used as a reference value that determines whether to switch the value of the parameter to be applied.
[0028] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may determine parameters used in image processing to enhance contrast, depending on the sample density, which is the ratio of the area of the sample shown in the image to the total area of the image.
[0029] By correcting the contrast of the image, which varies depending on the specimen density, with the parameter value, the contrast can be enhanced, making the image easier to observe. The specimen density may be the number of cells or the like that constitute the specimen present in the image. Note that it is preferable to set the parameter value so that the effect of enhancing the contrast increases as the specimen density decreases.
[0030] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may calculate the area of the sample shown in the image by extracting the outline of the sample in the image.
[0031] This allows the length of the entire contour to be calculated from the contour of the sample. This allows the density of the sample to be determined by roughly estimating the area of the sample from the contour length. For example, the area of the sample can be determined by multiplying the contour length by A (A is an arbitrary value), and then calculating the ratio of the area of the sample to the area of the entire image to determine the density of the sample.
[0032] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may divide the image into a plurality of regions and calculate the area of the sample shown in the image by image-recognizing the sample for each region.
[0033] This makes it possible to improve the accuracy in calculating the area of the sample.
[0034] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may have an intensity setting unit that determines parameters used in image processing to enhance contrast according to the magnification of the objective lens.
[0035] This allows the contrast of the image, which changes depending on the magnification of the objective lens, to be corrected using the parameter value, making it possible to make the image easier to observe. Note that it is preferable to set the parameter value so that the effect of strengthening the contrast increases as the magnification of the objective lens increases.
[0036] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may have a noise reduction processing unit that performs processing to reduce noise as image processing.
[0037] This removes unwanted signals and noise from the image, improving contrast. Unwanted signals and noise include random noise caused by random changes in brightness and color information due to fluctuations in the light reaching the image sensor, and electrical noise generated within electronic circuits during image generation, such as analog-to-digital conversion and amplification.
[0038] In the sample observation device 1 (1) according to the first embodiment, the noise reduction processing unit may have parameters of different magnitudes to be applied to the foreground, which is the part of the image where the sample is shown, and the background, which is the part other than the sample, and may perform processing to reduce noise using the parameters.
[0039] This allows for weighting using different parameters for the foreground and background, making it possible to more effectively suppress noise in the background area where objects other than the sample are captured. In this case, a difference image obtained by calculating the difference between the brightness value of each pixel and the average brightness value of the entire image may be used to distinguish between the foreground and background.
[0040] In the sample observation device according to the first embodiment, the image processing unit may have an averaging processing unit that performs processing to average a brightness histogram that represents the distribution of brightness values of an image, as image processing.
[0041] As a result, the brightness is balanced across the entire image, making the specimen being observed stand out more clearly. The processing performed in the averaging processor may include, for example, a process of distributing an image biased toward bright brightness regions in a histogram across the entire brightness range, or a process using a technique such as CLAHE (Contrast Limited Adaptive Histogram Equalization).
[0042] In the sample observation device 1 (1) according to the first embodiment, the image processing unit may have a determination unit that determines whether or not to perform image processing depending on the conditions when the image was captured.
[0043] This makes it possible to determine whether image processing should be performed when the imaging conditions result in low contrast. The conditions used when the image was captured preferably include values such as the magnification of the objective lens and the density of the sample. The sample's shape, such as its thickness, may also be used.
[0044] Next, a sample observation device 1 (2) according to the second embodiment will be described. The sample observation device 1 (2) according to the second embodiment includes an illumination optical system that irradiates illumination light onto the sample via an objective lens, an imaging optical system that forms an image from the light received by the objective lens, an image sensor that captures the image formed by the imaging optical system, and an image processing unit that performs image processing to enhance the contrast of the image captured by the image sensor. The image processing unit includes a noise reduction processing unit that performs processing to reduce noise as part of the image processing, and the noise reduction processing unit performs the noise reduction processing with different intensities for the foreground, which is the portion of the image in which the sample is shown, and the background, which is the portion in which everything other than the sample is shown.
[0045] According to the second embodiment, the sample is illuminated by the illumination optical system via an objective lens, and the light that passes through the sample and returns is collimated by the objective lens and focused by the imaging optical system to form an image on the imaging element. This image is converted into a digital data image by photoelectric conversion, and the image processing unit can perform image processing to enhance the contrast of the image. Furthermore, the image processing unit can remove unnecessary signals and noise contained in the image and further suppress noise in areas other than the sample, making it easier to observe the sample.
[0046] In the sample observation devices 1(1) and 1(2) according to the first and second embodiments, the image processing unit may have, as image processing, a margin addition processing unit that performs processing to add a margin image to at least a part of the outer periphery of the image.
[0047] This allows the distribution of brightness values across the entire image to be changed by the margin image, making it possible to adjust the effect of strengthening the contrast. It is preferable to add a margin image to a position that does not affect the portion of the image that contains the sample. For example, a method can be considered in which an image composed of two or more brightness values is added to the periphery of the image, or at least a portion of the periphery.
[0048] In the sample observation devices 1(1) and 1(2) according to the first and second embodiments, the image processing unit may have an intensity setting unit that sets the brightness value constituting the margin image as a parameter used in image processing to enhance contrast.
[0049] This allows the effect of increasing contrast to be changed depending on the brightness values that make up the margin image.
[0050] In the sample observation devices 1(1) and 1(2) according to the first and second embodiments, the image processing unit may have a determination unit that determines whether to perform processing to add a margin image when the magnification of the objective lens is greater than a preset threshold value.
[0051] This allows the design to add a blank image and strengthen the contrast when the magnification of the objective lens is greater than a threshold value. Note that it is preferable to determine the threshold value in advance based on the results of image processing after image conversion.
[0052] In the sample observation devices 1(1) and 1(2) according to the first and second embodiments, the image processing unit may have a determination unit that determines whether to perform processing to add a margin image when the sample density, which is the ratio of the area of the sample shown in the image to the total area of the image, is smaller than a preset threshold value.
[0053] This allows for a design that adds a margin image to enhance contrast when the specimen density is lower than a threshold value. Note that it is preferable to determine the threshold value in advance based on the results of image processing.
[0054] In the sample observation devices 1(1) and 1(2) according to the first and second embodiments, the image processing unit may have an image style conversion processing unit that performs image style conversion processing on an image that has undergone image processing.
[0055] As a result, an image whose contrast has been enhanced by image processing can be converted into an image that is easier to observe and has an appearance suitable for observation. The style transfer process preferably uses an image that has undergone image processing as a source and a phase-contrast image captured by a phase-contrast microscope as a target, constructs a learning model that uses machine learning to map from the source to the target, and performs the conversion using this learning model. The style transfer process can be performed using, but is not limited to, CycleGAN or the like. Other techniques include convolutional neural networks (CNN), generative adversarial networks (GAN), conditional GAN (CGAN), deep conventional GAN (DCGAN), Pix2Pix, etc. By performing the conversion using such a learning model, it is possible to obtain an image that looks like it is being observed using a phase-contrast microscope. A phase-contrast microscope is a microscope that observes a sample by utilizing the difference in phase between light that has passed through the sample and the background area in the image, and can obtain images that are easier to observe for transparent samples.
[0056] In the sample observation devices according to the first and second embodiments described above, the image processing unit P includes an image acquisition unit that acquires an image captured by the image sensor, an intensity setting unit that sets the intensity of the effect of enhancing contrast in image processing, a noise reduction processing unit that reduces noise in the image, a determination unit that determines whether or not to perform subsequent image processing, a margin addition processing unit that performs processing to add margins to the image, and an image style conversion processing unit. The flow of image processing performed by the image processing unit P in this embodiment will be described below with reference to FIG.
[0057] 3, in step ST1, the image acquisition unit acquires an image captured by an image sensor. At this time, the magnification of the objective lens, the distance between the objective lens and the sample, and other settings of the device when the image was captured, may be acquired. The numerical aperture of the objective lens may also be acquired. After step ST1, the process proceeds to step ST2.
[0058] In step ST2, the averaging processing unit averages (flattens) the brightness histogram of the image. For example, contrast-limited adaptive histogram equalization (CLAHE) is applied as the averaging processing. After step ST2, step ST3 is executed.
[0059] In step ST3, the intensity setting unit determines a first intensity setting condition to be used when determining parameters to be used in the noise reduction process to be performed after this process, based on the magnification of the objective lens, the distance between the objective lens and the sample, the brightness value calculated from the image, etc. acquired in step ST1. In this embodiment, it is preferable to use the brightness value or the sample density as the first intensity setting condition. After step ST3, the process proceeds to step ST4.
[0060] In step ST4, the magnitude relationship between the first intensity setting condition and a reference value is compared, and parameters are determined based on the magnitude relationship. If a greater contrast enhancement effect is desired, parameters are set to enhance the contrast enhancement effect. For example, if the sample density is used as the first intensity setting condition, parameters are set so that the smaller the sample density, the greater the contrast enhancement effect. Furthermore, if a luminance value calculated from the image is used, parameters are set so that the smaller the luminance value, the greater the contrast enhancement effect. Furthermore, if the distance between the objective lens and the sample is used, parameters are set so that the closer the distance is to the focal length, the greater the contrast enhancement effect. When the parameters are multiplied by the luminance values of the pixels constituting the image, the greater the parameter value becomes, and when the parameters are divided by the parameter value, the greater the parameter value becomes, the greater the contrast enhancement effect becomes. The parameters determined in step ST4 correspond to the foreground smoothing parameter FG and the background smoothing parameter BG in the noise reduction process performed in step ST5. After step ST4, the process proceeds to step ST5.
[0061] In step ST5, noise reduction processing is performed using the parameters determined in step ST4, which are a foreground smoothing parameter FG and a background smoothing parameter BG, which will be described below.
[0062] The foreground smoothing parameter FG and the background smoothing parameter BG are parameters that perform weighting in the calculation for converting the brightness values of the foreground and background in the noise reduction process. The type of calculation will be explained below together with the details of the noise reduction process.
[0063] The details of the noise reduction process will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the noise reduction process.
[0064] First, the image brightness values are normalized between 0 and 1 (step ST50). Next, an image is generated by smoothing the brightness values of the image using a process such as blurring (step ST51). A difference image is then generated by taking the difference between the average brightness value of the image and the brightness value of each pixel, and the foreground and background of the image are distinguished (step ST52). Next, using the difference image generated in step ST52, a foreground smoothing parameter FG, and a background smoothing parameter BG, noise is reduced with different intensities for the foreground and background (step ST53). For foreground brightness values closer to the average, the average brightness value is increased, while for foreground brightness values farther from the average, the deviation in brightness is reduced. The background is composed of brightness values close to the average, thereby suppressing background noise. When determining the values of the foreground smoothing parameter FG and the background smoothing parameter BG, it is preferable to determine in advance whether the structure of the sample remains intact in the foreground and whether noise is suppressed in the background other than the sample by looking at the results of image processing, such as the image converted, the image with the enlarged range of interest, the brightness histograms of those images, the calculated value of the density of the sample reflected from the image, and the number of samples reflected from the image. Since strong processing is applied to the background, if the larger the parameter value, the greater the noise reduction effect, it is preferable that FG≦BG. For example, it is preferable that FG≦BG≦10×FG. The greater the difference between the foreground smoothing parameter FG and the background smoothing parameter BG, the stronger the contrast as a result of image processing. The above parameters are determined taking this into consideration.
[0065] This method is just an example, and it is not necessary to perform all of the steps.
[0066] In this way, noise reduction processing is performed. After step ST5, the process proceeds to step ST6.
[0067] In step ST6, the determination unit determines whether to perform the margin addition process in step ST9, which is performed subsequently, depending on the conditions under which the image was captured. It is preferable to use the objective lens magnification acquired in step ST1 as the condition under which the image was captured. Alternatively, the sample density may be calculated from the image acquired in step ST1 and used as the sample density. If it is determined that margin addition process should be performed, the process proceeds to step ST7. If it is determined that margin addition process should not be performed, the process proceeds to step ST10. In this determination, if the objective lens magnification is used as the condition under which the image was captured, it is determined that margin addition process should be performed if the magnification is greater than a threshold value. In the case of the sample density as the condition under which the image was captured, it is determined that margin addition process should be performed if the density is less than the threshold value. The effect of this embodiment can be achieved by setting the threshold value for the objective lens magnification to, for example, 10x or 20x. The effect of this embodiment can be achieved by setting the threshold value for the sample density to, for example, 10%.
[0068] In step ST7, the intensity setting unit determines second intensity setting conditions to be used when determining parameters for enhancing contrast that are used in the margin addition process in step ST9, based on the magnification of the objective lens, the distance between the objective lens and the sample, a predetermined brightness value calculated from the brightness value of the image, and the like, acquired in step ST1. It is preferable to use the magnification of the objective lens as the second intensity setting condition for determining the parameters to be used in the margin addition process. After step ST7, the process proceeds to step ST8.
[0069] In step ST8, the second intensity setting condition is compared with a reference value to determine whether it is larger or smaller. Parameters are set depending on the magnitude relationship. The parameters here are the brightness values constituting the margin image (described later) and the type of margin image to be added, and changing these parameters adjusts the effect of strengthening the contrast. Step ST8 is followed by step ST9.
[0070] In step ST9, the margin-adding processor adds a margin image to the image according to the parameters determined in step ST8. Adding a margin image can change the contrast of the image obtained as a result of the style conversion process in the subsequent step ST10. As shown in FIG. 5, a margin image composed of two or more brightness values may be added to the periphery of the image. FIG. 5(a) shows a margin image in which a border M1 with a brightness value N1 is added to the periphery of image Q, and a border M2 with a brightness value N2 is further added to the periphery of that border. For example, the parameters may be determined such that the brightness value N1 is set to 0 (8 bits), the maximum value of the brightness value N2 is set to 180 (8 bits), and the brightness value N2 is changed depending on the effect of increasing contrast.
[0071] In addition to the brightness value, the border width may also be set as a parameter determined in step ST8. Depending on the objective lens magnification, cell type, etc., for example, when the objective lens magnification is high or for cell types with wide contour areas, the border width may be set wider, as shown in Figure 5(b). The margin image may also have a striped pattern consisting of three or more borders, alternating between high and low brightness values. In this case, increasing the number of stripes increases the range of large brightness differences within the image, thereby enhancing the effect of enhancing contrast. Figure 5(c) shows an example in which a border M2 with a brightness value N2 is added outside the periphery of a border M1 with a brightness value N1, and a border M3 with a brightness value N3 is further added outside the periphery of the border M2. In Figure 5(c), the brightness values N1, N2, and N3 do not change continuously, resulting in a striped pattern. Furthermore, the margin image is not limited to an image with a single brightness, but may also have a gradation in which the brightness value changes gradually and finely. Using a margin image with a gradation results in a more gradual contrast. A gradation may be achieved by continuously changing the brightness values N1, N2, and N3. These may be combined, for example, to add a gradational blank image when the specimen density is high, and to add a striped blank image when the specimen density is low. Alternatively, instead of adding a blank image, style transfer processing may be performed using the image before adding the blank image and an image of an assumed ideal cell as input images for the style transfer processing in step ST10. Step ST9 is followed by step ST10.
[0072] In step ST10, the style conversion processing unit performs style conversion processing to convert the style of the image. The style conversion processing generates a learning model for converting a source bright-field image into a target phase-contrast image, and performs conversion using this learned model. After step ST10, step ST11 is executed.
[0073] In step ST11, the image after image processing is subjected to post-processing to obtain a more stable image that is easier to observe. Step ST11 may be performed as needed. The post-processing is as follows.
[0074] Image processing may be performed after capturing an image divided into multiple images, and then a stitching process is performed to stitch the multiple images together as post-processing. Furthermore, as post-processing, a process is performed to adjust the average brightness value of each of the multiple images to a specified value. This allows the average brightness of the entire image, which varies with each capture, to be adjusted, allowing the image to be viewed with stable brightness. The specified value is a value determined by the observer.
[0075] The above example shows a specific example of this embodiment, but this embodiment is not limited to these. Some of the processes may not be executed, and the order of execution is not limited to the above as long as the purpose is to strengthen contrast.
[0076] REFERENCE SIGNS LIST 100 Time-lapse observation device 101 Time-lapse observation chamber 102 Door 103 Control device 104 Operation unit 1 Sample observation device MS Microscope device 10 Light source 11 Light source unit 11a Half mirror 12 Sample stage 13 Observation container 14 Reflecting member 15 Image pickup element 16 Movable stage 17 Sealed container SA Sample OBL Objective lens IL Imaging lens P Image processing unit
Claims
1. A sample observation device having: an illumination optical system that irradiates illumination light onto a sample through an objective lens; a reflecting member that is placed opposite the objective lens and reflects light; an imaging optical system that forms an image from the light received by the objective lens; an image sensor that captures the image formed by the imaging optical system; and an image processing unit that performs image processing to enhance the contrast of the image captured by the image sensor.
2. A sample observation device as described in claim 1, wherein the image processing section has an intensity setting section which determines parameters used in the image processing for enhancing the contrast according to the distance between the objective lens and the sample.
3. A sample observation device as described in claim 1, wherein the image processing section has an intensity setting section which determines parameters used in the image processing for enhancing the contrast according to the luminance value of the image.
4. A sample observation device according to claim 3, wherein the image processing section calculates the brightness value based on the difference between a maximum brightness value and a minimum brightness value in the image.
5. A sample observation device according to claim 3, wherein the image processing section calculates the brightness value based on the variance of each pixel in the entire image.
6. A sample observation device as described in any one of claims 3 to 5, wherein the intensity setting unit determines parameters used in the image processing to enhance the contrast depending on the magnitude relationship between the brightness value and a preset reference value.
7. A sample observation device as claimed in any one of claims 1 to 6, wherein the image processing unit determines parameters used in the image processing to enhance the contrast in accordance with the sample density, which is the ratio of the area of the sample shown in the image to the total area of the image.
8. A sample observation device according to claim 7, wherein the image processing section calculates the area of the sample shown in the image by extracting the contour of the sample in the image.
9. A sample observation device according to claim 7, wherein the image processing unit divides the image into a plurality of regions and calculates the area of the sample shown in the image by performing image recognition of the sample for each of the regions.
10. A sample observation device as described in any one of claims 1 to 9, wherein the image processing unit has an intensity setting unit that determines parameters used in the image processing to enhance the contrast according to the magnification of the objective lens.
11. A sample observation device according to any one of claims 1 to 10, wherein the image processing section has a noise reduction processing section that performs processing to reduce noise as the image processing.
12. A sample observation device as described in claim 11, wherein the noise reduction processing unit has parameters of different magnitudes to be applied to the foreground, which is the part of the image in which the sample is shown, and the background, which is the part in which other than the sample is shown, and performs processing to reduce the noise using the parameters.
13. A sample observation device according to any one of claims 1 to 12, wherein the image processing section has an averaging processing section that performs processing to average a brightness histogram representing a distribution of brightness values of the image as the image processing.
14. A sample observation device according to any one of claims 1 to 13, wherein the image processing section has a determination section that determines whether or not to perform the image processing depending on the conditions under which the image was captured.
15. A sample observation device comprising: an illumination optical system which irradiates illumination light onto a sample through an objective lens; an imaging optical system which forms an image from the light received by the objective lens; an image sensor which captures the image formed by the imaging optical system; and an image processing unit which performs image processing to enhance the contrast of the image captured by the image sensor, wherein the image processing unit has a noise reduction processing unit which performs processing to reduce noise as part of the image processing, and the noise reduction processing unit performs the noise reduction processing with different intensities for a foreground which is the part of the image in which the sample is shown, and a background which is the part other than the sample.
16. A sample observation device according to any one of claims 1 to 15, wherein the image processing unit has a margin addition processing unit that performs processing to add a margin image to at least a part of the outer periphery of the image as the image processing.
17. A sample observation device according to claim 16, wherein the image processing section has an intensity setting section that sets the brightness value constituting the margin image as a parameter used in the image processing for enhancing the contrast.
18. A sample observation device as described in claim 16 or 17, wherein the image processing unit has a judgment unit that judges whether to perform processing to add the margin image when the magnification of the objective lens is greater than a preset threshold value.
19. A sample observation device as described in any one of claims 16 to 18, wherein the image processing unit has a judgment unit that judges whether to perform processing to add the margin image when the sample density, which is the ratio of the area of the sample shown in the image to the total area of the image, is smaller than a preset threshold value.
20. A sample observation device according to any one of claims 1 to 19, wherein the image processing unit has an image style conversion processing unit that performs an image style conversion process on the image that has been subjected to the image processing.
Citation Information
Patent Citations
Microscope apparatus and image acquisition method
JP2012003116A
Microscope control device, image display device, image control server, method for generating focus position information, image display method, image control method, and microscopic image control system
JP2012037861A
Observation device
JP2018146602A
Image conversion method, program, and image processing device
WO2022113367A1
