Microscope system

Abstract

To appropriately and quickly find an object portion in a visual field of a microscope.SOLUTION: A microscope system 1 comprises: a first optical system that includes a transmitted light source 121 and enables observation of an object from an eyepiece 111 by a transmitted bright-field microscopy; a second optical system that shares at least part of the optical system with the first optical system and enables observation different from the observation of the object from the eyepiece 111; a camera 300 that acquires a digital image at least the different observation through the second optical system; a processor 500 that performs object detection on the digital image acquired by the camera 300 through the second optical system and outputs auxiliary information including information on a result of the object detection; and an AR display 400 that superimposes the auxiliary information on an optical image by the transmitted bright-field microscopy, which is formed on an optical axis of the first optical system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a microscope system. [Background technology]

[0002] For example, Patent Document 1 describes a technology relating to an AR (Augmented Reality) microscope that analyzes an image of a specimen to detect a target site and displays the detection result superimposed on an image observed through an eyepiece.

[0003] Furthermore, for example, Patent Document 2 describes a technique relating to a microscope suitable for intracytoplasmic sperm injection, which is a type of ART (Assisted Reproductive Technology). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 231204 [Patent Document 2] International Publication No. 2012 / 150689 Summary of the Invention [Problem to be solved by the invention]

[0005] In conventional AR microscopes, target regions are detected only by analyzing images that are substantially identical to the image observed through the eyepiece, i.e., images with substantially the same field of view as the observed image and acquired using the same observation method (e.g., transmitted light bright-field observation images). However, in cases where it is difficult to identify the target region using transmitted light bright-field observation, or when a foreign object is present in the specimen in a position that blocks the target region in the observation direction, it can be difficult to detect the target region using such analysis. Therefore, detecting the target region using only the above-described analysis can take a long time and can result in reduced detection accuracy.

[0006] In view of the above circumstances, an object of one aspect of the present invention is to provide a technique that enables a target site to be found appropriately and quickly within the field of view of a microscope. [Means for solving the problem]

[0007] A microscope system according to one embodiment of the present invention comprises a first optical system including a transmitted illumination light source and capable of observing an object through an eyepiece using transmitted light bright-field observation; a second optical system that shares at least a portion of its optical system with the first optical system and capable of observing the object differently from the observation through the eyepiece; an image sensor that acquires digital images obtained through at least the different observation through the second optical system; a processing device that performs object detection on the digital images acquired through the second optical system by the image sensor and outputs auxiliary information including information on the results of the object detection; and a superimposition device that superimposes the auxiliary information on an optical image obtained through transmitted light bright-field observation formed on the optical axis of the first optical system. [Effects of the Invention]

[0008] According to the above aspect, it is possible to appropriately and quickly find a target site within the field of view of the microscope. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a microscope system according to a first embodiment. [Figure 2] FIG. 1 shows a first example of a sample in a container. [Figure 3] 3 is a diagram showing an example of a display of an auxiliary image during the observation of the sample shown in FIG. 2. FIG. [Figure 4] FIG. 10 shows a second example of a sample in a container. [Figure 5] 5 is a diagram showing an example of a display of an auxiliary image during the observation of the sample shown in FIG. 4. FIG. [Figure 6] FIG. 10 is a diagram showing a third example of a sample in a container. [Figure 7]7A and 7B are diagrams showing examples of auxiliary images displayed during sample observation shown in FIG. 6. [Figure 8] FIG. 10 is a diagram showing an example of changing the AR support setting. [Figure 9] FIG. 10 is a diagram illustrating a microscope system according to a second embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a timing chart relating to illumination and exposure in the microscope system shown in FIG. [Figure 11] FIG. 10 is a diagram for explaining the wavelength of illumination light of the microscope system according to the third embodiment. [Figure 12] FIG. 10 is a diagram illustrating a microscope system according to a fourth embodiment. [Figure 13] FIG. 10 is a diagram showing a fourth example of a sample in a container. [Figure 14] 14 is a diagram showing an example of a display of an auxiliary image during sample observation shown in FIG. 13 by the microscope system according to the fifth embodiment. FIG. [Figure 15] FIG. 10 is a diagram illustrating a microscope system according to a sixth embodiment. [Figure 16] 10A and 10B are diagrams illustrating an example of superimposed display of an auxiliary image. [Figure 17] FIG. 13 is a diagram illustrating a microscope system according to a seventh embodiment. [Figure 18] FIG. 1 is a diagram illustrating an example of a hardware configuration of a computer for realizing a processing device. [Figure 19] FIG. 10 is a diagram illustrating teacher data. DETAILED DESCRIPTION OF THE INVENTION

[0010] Each embodiment will be described in detail below with reference to the drawings.

[0011] (First embodiment) In this embodiment, an AR microscope suitable for use in applications such as intracytoplasmic sperm injection, which is a type of ART, will be described.

[0012] As people are getting married and giving birth later in life, the number of patients undergoing infertility treatment is increasing, and the demand for ART is also on the rise. ART is a general term for techniques that involve fertilizing eggs and sperm extracted from a human outside the body, such as in vitro fertilization (IVF) and intracytoplasmic sperm injection (ICSI). It is distinct from conventional artificial insemination, in which collected sperm are injected into the uterus and fertilized with an egg inside the body.

[0013] ART, which involves fertilizing eggs and sperm outside the body, requires that eggs be extracted from the mother beforehand. For this, a doctor collects follicular fluid from the follicles in the ovaries (a process known as egg retrieval), and an embryologist then finds and retrieves eggs from the collected follicular fluid (a process known as egg candidacy).

[0014] It is not easy to find and retrieve eggs from the follicular fluid collected during egg retrieval without missing any. This is because the follicular fluid collected from the follicles contains various cells, such as blood and other tissue fragments, in addition to eggs. For this reason, correctly identifying and retrieving eggs is not necessarily easy even for an experienced person, and it is a difficult task for inexperienced embryologists.

[0015] Furthermore, because a stereomicroscope is used for egg candling, the follicular fluid containing the eggs is removed from the incubator and exposed to room air during the candling process. The shorter the exposure period, the better. Therefore, embryologists are required to perform egg candling as quickly and reliably as possible.

[0016] FIG. 1 is a diagram illustrating a microscope system 1 according to this embodiment. The microscope system 1 is a system for observing a sample by looking through an eyepiece 111, and as shown in FIG. 1, includes a microscope 100, two cameras (camera 200 and camera 300), an AR display 400, and a processing device 500. Below, the microscope system 1 will be described using an example of its use for egg candling. In this case, the sample is follicular fluid contained in a container C. The microscope system 1 assists the user in the task of finding an egg from the follicular fluid by displaying AR within the field of view of the eyepiece 111.

[0017] The microscope 100 is a stereomicroscope equipped with a binocular tube 110, and has two parallel observation optical paths (an optical path for the right eye and an optical path for the left eye) created by the objective lens 102, making it possible to observe follicular fluid containing ova from vertically above the sample. The microscope 100 makes it possible to observe the sample in three dimensions by looking through the eyepiece lens 111, making it suitable for picking up ova during egg candling.

[0018] Camera 200 and camera 300 are each digital cameras equipped with an imaging element. The imaging element is, for example, a two-dimensional image sensor such as a CCD image sensor or a CMOS image sensor. Note that "CCD" is an abbreviation for Charge Coupled Device, and "CMOS" is an abbreviation for Complementary Metal Oxide Semiconductor.

[0019] Camera 200 is a first imaging device that captures an image of the sample from vertically above the sample. Camera 200 captures an image of the sample from the same direction as microscope 100 by using beam splitter 201 arranged on the optical path between objective lens 102 and eyepiece lens 111. On the other hand, camera 300 is a second imaging device that captures an image of the sample from vertically below the sample. Images captured by camera 200 and camera 300 are output to processing device 500.

[0020] The processing device 500 generates an auxiliary image based on at least one of a first image, which is a digital image acquired by the camera 200, and a second image, which is a digital image acquired by the camera 300, and outputs the generated auxiliary image to the AR display 400. The auxiliary image includes information specifying an area where the presence of an object is estimated. Specifically, the processing device 500 performs object detection on at least one of the first image and the second image, and outputs an auxiliary image including information specifying an area where the presence of an object (an egg in this example) is estimated, as auxiliary information including information on the result of the object detection.

[0021] The object detection performed by the processing device 500 is performed using a trained model that uses a neural network that has learned about the target object through deep learning. The trained model may be a neural network such as CNN, and may also be an SVM. "CNN" is an abbreviation for Convolutional Neural Network, and "SVM" is an abbreviation for Support Vector Machine. In an example of performing egg candling, the processing device 500 may perform object detection using a trained model that has learned about eggs. The processing device 500 may perform object detection on at least one of the first image and the second image, and output an auxiliary image containing information identifying the presence area where the presence of an egg is estimated. However, as will be described later, in order to perform highly accurate estimation in various environments, it is desirable to perform object detection on each of the first image and the second image, and output an auxiliary image containing information identifying the presence area. The object detection algorithm is not particularly limited. Any known algorithm may be employed.

[0022] The AR display 400 is a superimposition device that superimposes the auxiliary image, which is auxiliary information output by the processing device 500, onto an optical image of the sample obtained by a transmitted light bright-field observation method, which is formed on the optical path of the microscope 100, more specifically, on the optical axis of the first optical system described below. More specifically, during the egg candling procedure of finding an egg from follicular fluid, the AR display 400 superimposes an auxiliary image including information identifying the area where the egg exists onto the optical image of the follicular fluid. This assists the embryologist in the egg candling procedure in his or her search for the egg.

[0023] The AR display 400 is, for example, a liquid crystal display, but may also be, for example, a MEMS such as a digital mirror device. The AR display 400, together with a projection optical system (not shown), may constitute a projection device that projects an auxiliary image displayed on the AR display 400 onto an image plane on the observation optical path. This image plane on the observation optical path is a plane on which an optical image of the sample is formed, and is located, for example, between the imaging lens 112 and the eyepiece 111. Note that "MEMS" is an abbreviation for Micro Electro Mechanical Systems.

[0024] 1 shows an example in which the AR display 400 is disposed on an optical path branched from the observation optical path, and light emitted from the AR display 400 is converged into the observation optical path by a beam splitter 401 disposed on the observation optical path. However, the AR display 400 is not limited to a projection device as long as it superimposes an auxiliary image on an image plane. For example, if the AR display 400 is a transmissive liquid crystal display, it may be disposed on the image plane. In this case, the auxiliary image directly displayed on the image plane by the AR display 400 is superimposed on the optical image.

[0025] The configuration of the microscope 100 will be described in more detail below. The microscope 100 includes a stage 101, an objective lens 102, a condenser lens 103, an imaging lens 104, a binocular tube 110, an illumination device 120, an illumination device 130, and a zoom lens 140. The zoom magnification of the zoom lens 140 can be changed by the user operating a zoom coupler 141.

[0026] The illumination device 120 is a first illumination device that irradiates illumination light onto the sample from vertically below the sample. The illumination device 120 includes a light source 121 and a beam splitter 122. The illumination light emitted from the light source 121 passes through the beam splitter 122 and the condenser lens 103 and illuminates the sample from below.

[0027] In the microscope 100, a sample is observed using light from an illumination device 120 as a transmitted illumination light source. This is because the target, an ovum, is a transparent phase object. Specifically, the light from the illumination device 120 passes through a condenser lens 103, transmits through the sample, and enters the objective lens 102. Then, it passes through a zoom lens 140 and enters a binocular tube 110, where an image is formed by an imaging lens 112 provided in the binocular tube 110. As a result, a user of the microscope system 1 can look through the eyepiece 111 and magnify and observe the optical image of the sample formed on the image plane by the imaging lens 112 with the eyepiece 111. In this way, in the microscope system 1, the illumination device 120, the condenser lens 103, the objective lens 102, the zoom lens 140, and the imaging lens 112 form a first optical system, and the microscope system 1 enables upright observation of the sample (observation from vertically above the sample) through the eyepiece 111 using transmitted light bright-field observation.

[0028] The camera 200 also uses transmitted illumination light from the illumination device 120 to capture an image of the sample. The reason for using transmitted light is the same as that of the microscope 100. Specifically, a portion of the light that has passed through the sample and entered the objective lens 102 enters the camera 200 via the zoom lens 140 and the beam splitter 201 and is used for capturing an image with the camera 200. In other words, the image sensor (first image sensor) included in the camera 200 acquires a digital image of the optical image of the sample formed on the optical axis of the first optical system by the transmitted light bright-field observation method.

[0029] The illumination device 130 is a second illumination device that irradiates illumination light onto the sample from vertically above the sample. The illumination device 130 includes a light source 131 and a beam splitter 132. The illumination light emitted from the light source 131 passes through the beam splitter 132, the zoom lens 140, and the objective lens 102 and illuminates the sample from above.

[0030] The camera 300 photographs the sample using transmitted illumination light from the illumination device 130. The reason for using transmitted light is the same as that for the microscope 100. Specifically, light that has passed through the sample and entered the condenser lens 103 is imaged on the camera 300 by the imaging lens 104. The camera 300 photographs the sample using the light that has entered via the imaging lens 104. In this way, in the microscope system 1, the illumination device 130, the condenser lens 103, the objective lens 102, and the imaging lens 104 constitute a second optical system, and an image of the sample by inverted observation is obtained by imaging with the camera 300. In other words, the image sensor (second image sensor) included in the camera 300 acquires a digital image by inverted observation (observation of the sample from vertically below), which is different from observation of the sample through the eyepiece 111 (erect observation), via the second optical system.

[0031] The microscope system 1 configured as described above has a camera 200 that photographs the sample from vertically above and a camera 300 that photographs the sample from vertically below. By having cameras in two directions, particularly vertically above and below, the user of the microscope system 1 can quickly find an object (egg) that exists within the field of view of the microscope 100 without missing it. This will be explained below with reference to FIGS. 2 to 7.

[0032] FIG. 2 is a diagram showing a first example of a sample in a container. FIG. 2 illustrates a case where an ovum S1 and a tissue fragment S3 do not overlap in the observation direction (vertical direction) within follicular fluid S2. In this case, as shown in FIG. 3, both the digital image D1 acquired by camera 200 and the digital image D2 acquired by camera 300 can capture the ovum S1 with relatively high contrast. Therefore, the processing device 500 can detect the ovum S1 with high accuracy when performing object detection on either image, and can generate an auxiliary image D3 including a bounding box indicating the position and size of the ovum S1, as shown in FIG. 3.

[0033] The AR display 400 superimposes the auxiliary image D3 output from the processing device 500 onto the optical image of the sample S, and the optical image and the bounding box B are displayed superimposed within the field of view F of the microscope 100 shown in FIG. 3 when the user looks through the eyepiece 111. This allows a user, such as an embryologist, to quickly find the ovum S1 present within the field of view of the microscope 100 without missing it. Here, the color of the bounding box B is determined by the color of light emitted from the AR display 400, and may be green, for example.

[0034] As shown in Figure 2, if the tissue fragment S3 is not positioned to block the ovum S1, the user can find the ovum S1 using only the optical image. However, the presence of the bounding box B can help the user to notice the presence of the ovum S1 more quickly.

[0035] Even if the tissue piece S3 is not overlapping, a situation may occur in which the ovum S1 is difficult to see due to blood or other factors being mixed in the follicular fluid S2. By using a model that has thoroughly learned the characteristics of an ovum using deep learning or other methods for object detection performed by the processing device 500, the presence of an ovum can be estimated with high accuracy even from images that would be difficult for an unexperienced person to identify as an ovum. Therefore, even if the tissue piece S3 is not in a position that blocks the ovum S1, the user's egg candling process can be supported by using an auxiliary image.

[0036] FIG. 4 shows a second example of a sample in a container. FIG. 4 illustrates a case where an ovum S1 and a tissue fragment S3 overlap in the follicular fluid S2 in the observation direction, particularly a case where the tissue fragment S3 is positioned vertically above the ovum S1. The ovum S1 often sinks to the bottom of the collected follicular fluid S2. In contrast, during egg candling, a stereomicroscope capable of stereoscopically viewing the ovum S1 is used to pick up the ovum S1, so observation is performed vertically from above. In such an upright observation environment, if the tissue fragment S3 floating on the surface of the follicular fluid S2 is positioned above the ovum S1 that has sunk near the bottom of the container C, the ovum S1 will be hidden in the shadow of the tissue fragment S3 in the optical image and will be completely invisible or will not be able to be observed with good contrast.

[0037] This is also true for the digital image D11 acquired by the camera 200, as shown in Fig. 5. However, as shown in Fig. 5, in the digital image D12 acquired by the camera 300 through inverted observation, the positional relationship in the depth direction between the ovum S1 and the tissue fragment S3 is reversed compared to the digital image D11, and therefore the ovum S1 can be captured with relatively high contrast. Therefore, the processing device 500 can detect the ovum S1 with high accuracy by performing object detection on at least the digital image D12 through inverted observation, and can generate an auxiliary image D13 including a bounding box indicating the position and size of the ovum S1, as shown in Fig. 5.

[0038] The AR display 400 superimposes the auxiliary image D13 output from the processing device 500 onto the optical image of the sample S, and the optical image and the bounding box B are displayed superimposed within the field of view F of the microscope 100 shown in FIG. 5 when the user looks through the eyepiece 111. This allows a user such as an embryologist to easily notice the presence of the ovum S1 even when the presence of the tissue fragment S3 makes it difficult to see the ovum S1 in the optical image. This makes it possible to quickly collect the ovum S1 using a pipette or the like, avoiding the tissue fragment S3 and the like.

[0039] Figure 6 is a diagram showing a third example of a sample in a container. Figure 6 illustrates a case in which tissue fragments (tissue fragment S3, tissue fragment S4) overlap vertically above and below the ovum S1 in the follicular fluid S2. Since the tissue fragment S4 may sink to the bottom like the ovum S1, a situation may arise in which tissue fragments are located above and below the ovum S1, as shown in Figure 6. In such an observation environment, as shown in Figure 7, the ovum S1 is hidden in the shadow of the tissue fragment S3 not only in the optical image but also in the images captured by the camera (digital image D21, digital image D22), making it difficult to observe with good contrast.

[0040] However, the processing device 500 can estimate the presence of the ovum S1 if even a part of the ovum S1 is visible. Furthermore, even if the probability of the presence area obtained by object detection for the digital images D21 and D22 is low, if the presence areas obtained from the two images substantially match, it may be possible to determine that an ovum exists with a higher probability than the probability calculated for each. In this way, the processing device 500 may identify the presence area (the position and size of the ovum S1) by combining the object detection results for the images captured from both above and below. The processing device 500 may generate an auxiliary image D23 including a bounding box indicating the position and size of the identified ovum S1.

[0041] By having the AR display 400 superimpose the auxiliary image D23 output from the processing device 500 onto the optical image of the sample S, the optical image and the bounding box B are displayed superimposed within the field of view F of the microscope 100 shown in FIG. 7 when the user looks through the eyepiece 111. This allows a user such as an embryologist to easily notice the presence of the egg S1 even if the presence of tissue fragments S3 and S4 makes it difficult to see the egg S1 in the optical image. This makes it possible to quickly collect the egg S1 using a pipette or the like, avoiding the tissue fragments S3 and S4.

[0042] As shown in the above examples, the direction from which an ovum can be photographed with good contrast may vary depending on the state of the sample. In this regard, the microscope system 1 can significantly increase the likelihood of photographing an ovum with good contrast by providing both the camera 200 that photographs from vertically above and the camera 300 that photographs from vertically below. Furthermore, the microscope system 1 may perform object detection on both the images acquired by the camera 200 and the images acquired by the camera 300 to identify the presence area. In Figures 2, 4, and 6, an example was described in which an ovum is sunk to the bottom of a petri dish. Because ovums are heavier than follicular fluid, they are often found at the bottom of the petri dish, but they can also be found at the top. In such cases, if blood is mixed into the follicular fluid, it may be easier to view the ovum from above than from below. In such cases, providing both the camera 200 that photographs from vertically above and the camera 300 that photographs from vertically below can better prevent the ovum from being overlooked. As a result, even in an environment where it is difficult to obtain highly accurate detection results from each image alone, it is possible to obtain more accurate results by using the object detection results from the two images than by using the detection results from each image alone. Therefore, with the microscope system 1, it is possible to quickly find eggs with a high probability regardless of the state of the eggs in the sample, and it is possible to significantly reduce the number of eggs that are overlooked compared to conventional methods.

[0043] In the present embodiment, an example has been shown in which object detection is performed on both the image captured by camera 200 and the image captured by camera 300. However, the content of egg candling support using the object detection results may be switched as appropriate. That is, the information included in the auxiliary image may be switched between at least first information in which an existence area is identified by object detection on a first image, second information in which an existence area is identified by object detection on a second image, and third information in which an existence area is identified by object detection on both the first image and the second image, depending on the settings of the microscope system 1. These may be automatically switched depending on, for example, a predetermined task or an observation target. For example, when the object is stationary, such as in egg candling, the processing device 500 may generate an auxiliary image including the third information. On the other hand, when the object (sperm) is moving, such as in sperm sorting, the processing device 500 may generate an auxiliary image including the first information or the second information, thereby reducing the amount of calculation performed by the processing device 500.

[0044] Furthermore, such switching may be performed in accordance with a user's operation. The microscope system 1 may include an operation device that accepts a user's selection from at least a first support obtained by photographing the sample from vertically above, a second support obtained by photographing the sample from vertically below, and a third support obtained by photographing the sample from vertically above and vertically below. The processing device 500 may switch the information included in the auxiliary image among the above-mentioned first information, second information, and third information in accordance with the selection made by the user using the operation device.

[0045] The operation device may be, for example, any input device connected to the processing device 500, such as a keyboard or a mouse. The operation device may also be provided on the microscope 100, for example, on the binocular tube 110 so that it can be easily operated while looking through the eyepiece 111. An input to the operation device provided on the microscope 100 may be output to the processing device 500, which may then detect a selection by the user.

[0046] Furthermore, the support setting may be displayed within the field of view of the microscope 100 by including information indicating the current support setting in the auxiliary image. FIG. 8 is a diagram showing an example of changing the support setting. The processing device 500 may generate an auxiliary image including information C1 indicating whether egg candling support is being performed by the microscope system 1 and information C2 indicating the type of egg candling support, and output the image to the AR display 400. This allows the user to grasp the current support status without taking their eyes off the eyepiece 111, as shown in FIG. 8. Information C2 may also be used as a software button. Pressing information C2 using the above-mentioned operating device (e.g., a mouse) may switch the type of support setting, such as "upper," "lower," or "both," as shown in FIG. 8. For example, by operating a mouse connected to the processing device 500, a mouse pointer may be displayed within the field of view F, and the support setting may be switched by clicking while the mouse pointer is positioned on information C2. Here, the color of the bounding box B is determined by the color of light emitted from the AR display 400, and may be, for example, green.

[0047] (Second embodiment) In this embodiment, an AR microscope suitable for use in intracytoplasmic sperm injection, which is a type of ART, will also be described.

[0048] 9 is a diagram illustrating a microscope system 2 according to this embodiment. The microscope system 2 according to this embodiment differs from the microscope system 1 in that the light emission of the illumination device 120 and the illumination device 130 is controlled by a processing device 500. In other respects, it is the same as the microscope system 1.

[0049] In the microscope system 1, when the power of the illumination devices (illumination devices 120, 130) is turned on during egg candling, the illumination devices (illumination devices 120, 130) then emit light at all times, and observation with the microscope 100 and photography from two directions are performed simultaneously. In contrast, in the microscope system 2, the processing device 500 controls the illumination devices 120 and 130 so that the illumination devices 120 and 130 do not emit light at the same time. This is a major difference from the microscope system 1. The reason for performing such light emission control is to avoid a decrease in image contrast due to reflected light (stray light).

[0050] Fig. 10 is a diagram showing an example of a timing chart relating to illumination and exposure in the microscope system 2 shown in Fig. 9. Hereinafter, light emission control and exposure control performed in the microscope system 2 will be described with reference to Fig. 10.

[0051] In the microscope system 2, the processing device 500 controls the illumination device 120 and the illumination device 130 so that the illumination device 120 and the illumination device 130 alternately irradiate the sample with illumination light, as shown in Fig. 10. Fig. 10 shows that the light source 121 and the light source 131 alternately emit light over time. Furthermore, as shown in Fig. 10, the processing device 500 controls the exposure periods of the cameras 200 and 300 to match the light emission periods of the illumination device 120 and the illumination device 130. In other words, the camera 200 photographs the sample during the light emission period of the illumination device 120, and the camera 300 photographs the sample during the light emission period of the illumination device 130.

[0052] Most of the illumination light from the illumination device 130 is irradiated onto the sample. However, some of the light is reflected by optical elements on the way to the sample. For example, if light (stray light) reflected by the zoom lens 140 or the objective lens 102 is reflected by the beam splitter 201 and enters the camera 200, the contrast of the image of the sample captured by the camera 200 using transmitted illumination will decrease. This also applies to the case where an image is captured by the camera 300 while illumination light is being emitted from the illumination device 120.

[0053] Since a reduction in image contrast may adversely affect object detection, it is desirable to avoid a reduction in contrast due to stray light. In light of this, in the microscope system 2, the processing device 500 performs light emission control, so that the illumination device 130 stops emitting light while the camera 200 is capturing images. In addition, the illumination device 120 stops emitting light while the camera 300 is capturing images. Therefore, the microscope system 2 can avoid a reduction in contrast of captured images due to stray light, and ultimately improve the reliability of AR display using auxiliary images.

[0054] Note that, in FIG. 10, an example is shown in which the exposure period of camera 200 coincides with the light emission period of lighting device 120, and the exposure period of camera 300 coincides with the light emission period of lighting device 130, but the exposure period of the camera does not necessarily have to coincide as long as it is within the light emission period of the lighting device.

[0055] Furthermore, in the microscope system 2, the processing device 500 controls the illumination device 120 and the illumination device 130 so that the light-emitting period d1 of the illumination device 120 is longer than the light-emitting period d2 of the illumination device 130, as shown in Fig. 10. During the light-emitting period d1 of the illumination device 120, not only image capture by the camera 200 but also observation by the microscope 100 can be performed. By controlling the light-emitting period d1 to be longer than the light-emitting period d2, the period during which illumination for user observation is stopped can be shortened. Note that by shortening the light-emitting period d2 to a degree that the user does not notice that the light has been turned off, adverse effects on observation can be substantially avoided.

[0056] (Third embodiment) In this embodiment, an AR microscope suitable for use in intracytoplasmic sperm injection, which is a type of ART, will also be described.

[0057] 11 is a diagram for explaining the wavelength of illumination light in the microscope system according to this embodiment. The microscope system according to this embodiment differs from microscope system 1 in that the wavelengths of light used for imaging by camera 200 and camera 300 are different. In other respects, it is the same as microscope system 1.

[0058] In the microscope system according to this embodiment, similarly to the microscope system 1, the illumination devices 120 and 130 are constantly emitting light during candling, and observation by the microscope 100 and photographing from two directions are performed simultaneously. However, the illumination devices 120 and 130 irradiate the sample with illumination light of different wavelengths. Specifically, the illumination device 120 irradiates the sample from vertically below with visible light (see line L1 shown in FIG. 11 ), and the camera 200 photographs the sample using that visible light. Furthermore, the illumination device 130 irradiates the sample from vertically above with near-infrared light (see line L2 shown in FIG. 11 ), and the camera 300 photographs the sample using that near-infrared light. The camera 200 is a camera with high sensitivity in the visible range, and the camera 300 is a camera with high sensitivity in the near-infrared range.

[0059] By separating the wavelengths used for imaging by camera 200 and camera 300, even if light used for imaging by one camera enters the other camera as stray light, it is possible to block the stray light using a filter or the like provided on the camera. Also, a near-infrared light cut filter may be placed in front of camera 200, and a visible light cut filter may be placed in front of camera 300. By placing the filters, it is possible to obtain camera images with higher contrast. Therefore, unlike the microscope system 1, it is possible to prevent a decrease in image contrast even when illumination and imaging are performed from two directions simultaneously.

[0060] Furthermore, in the microscope system according to this embodiment, the illumination light is not stopped during egg candling, so there is no risk of the user feeling uncomfortable while observing through the eyepiece 111. Furthermore, because the stray light incident on the eyepiece 111 is near-infrared light to which the human eye has no sensitivity, there is no reduction in the contrast of the optical image observed visually.

[0061] The microscope system according to this embodiment can assist users in candling eggs, similar to the microscope system 1 and microscope system 2. Furthermore, the microscope system according to this embodiment can obtain good images even when observation and photography from two directions are performed simultaneously, without causing a decrease in image contrast in either observation or photography.

[0062] In this embodiment, an example of using visible light and near-infrared light has been shown, but it is also possible to use a combination of visible light and invisible light, and other invisible light, such as ultraviolet light, may be used instead of near-infrared light.

[0063] (Fourth embodiment) In this embodiment, an AR microscope suitable for use in intracytoplasmic sperm injection, which is a type of ART, will also be described.

[0064] 12 is a diagram illustrating a microscope system 4 according to this embodiment. The microscope system 4 according to this embodiment differs from the microscope system 1 in that the zoom coupler 141 is connected to a processing device 500. In other respects, it is the same as the microscope system 1. By connecting the zoom coupler 141 via the processing device 500, in the microscope system 4, the processing device 500 can grasp changes in zoom magnification made by the user operating the zoom coupler 141.

[0065] When the zoom magnification is changed, the optical magnification of the microscope 100 (magnification of the optical image) and the magnification of the first image acquired by the camera 200 change by the same amount, but the magnification of the second image acquired by the camera 300 does not change. As a result, the relationship between the magnifications of the three images changes.

[0066] Therefore, in the microscope system 4, the processing device 500 performs digital zoom processing on the second image, cropping and enlarging a range corresponding to the optical magnification of the microscope 100, thereby adjusting the magnification of the second image. This makes it possible to maintain a constant magnification ratio between the optical image, the first image, and the second image before changing the zoom magnification and a constant magnification ratio between the optical image, the first image, and the second image after changing the zoom magnification. If the magnification ratio is constant, the position of the ovum found on the second image can be easily converted to the correct position on the optical image, even if the zoom magnification is changed. Therefore, the bounding box can be displayed in the correct position even if the magnification changes.

[0067] The microscope system according to this embodiment can assist users in candling eggs, similar to the microscope systems 1 according to the first to third embodiments. Furthermore, the microscope system 4 according to this embodiment can avoid confusion that may arise from changing the optical magnification of the microscope 100, and can correctly assist in candling eggs regardless of the optical magnification.

[0068] In this embodiment, an example has been shown in which the optical magnification of the microscope 100 is changed by changing the zoom magnification, but the optical magnification may also be changed by changing the objective lens. In this case, the same effect can also be obtained by providing a configuration for detecting the objective lens in the revolver that holds the objective lens and outputting the detection result to the processing device 500.

[0069] Furthermore, in the present embodiment, an example has been shown in which the processing device 500 performs the digital zoom processing, but the digital zoom processing may also be performed by the camera 300. In this case, the processing device 500 may simply set a digital zoom magnification in the camera 300 that corresponds to the optical magnification of the microscope 100, and may acquire a second image after the digital zoom processing has been performed from the camera 300. When the digital zoom processing is performed by the camera 300, the same effect as when the processing device 500 performs the digital zoom processing can be obtained.

[0070] (Fifth embodiment) In this embodiment, an AR microscope suitable for use in intracytoplasmic sperm injection, which is a type of ART, will also be described.

[0071] FIG. 13 is a diagram showing a fourth example of a sample in a container, and FIG. 14 is a diagram showing an example of an auxiliary image displayed when observing the sample shown in FIG. 13 using the microscope system of this embodiment.

[0072] Figure 13 illustrates a case where, in the follicular fluid S2, the ovum S1-1 and the tissue fragment S3-1 overlap in the observation direction, and the ovum S1-2 and the tissue fragment S3-2 overlap in the observation direction. In particular, Figure 13 illustrates a case where the tissue fragment S3-1 is located vertically below the ovum S1-1, and the tissue fragment S3-2 is located vertically above the ovum S1-2. In this example, the ovum S1-1 is easy to distinguish in an image observed using upright observation, while the ovum S1-2 is easy to distinguish in an image observed using inverted observation.

[0073] Therefore, in this embodiment, an auxiliary image is generated by performing object detection on a digital image captured vertically from above, and an auxiliary image is generated by performing object detection on a digital image captured vertically from below, and the two generated auxiliary images are then superimposed on the optical image.

[0074] In this embodiment, the microscope system 1 shown in Figure 1 is used, and the processing device 500 performs object detection on both the digital image of the sample S taken by the camera 200 through upright observation and the digital image of the sample S taken by the camera 300 through inverted observation.

[0075] 14, auxiliary image D33 is generated by object detection on the image captured by camera 200, and shows a bounding box B1 representing the area where egg S1-1 exists. Furthermore, auxiliary image D43 is generated by object detection on the image captured by camera 300, and shows a bounding box B2 representing the area where egg S1-2 exists.

[0076] The AR display 400 superimposes both the auxiliary image D33 and the auxiliary image D43 output from the processing device 500 onto the optical image of the sample S. As a result, the optical image and the bounding boxes B1 and B2 are displayed superimposed within the field of view F of the microscope 100 shown in FIG. 14 when the user looks through the eyepiece 111. In this way, by superimposing and displaying the results of object detection for each of the auxiliary image D33 and the auxiliary image D43, it is possible to effectively detect the egg even when the egg and tissue fragments are mixed together.

[0077] Note that the object detection results for each of auxiliary image D33 and auxiliary image D43 (bounding boxes B1 and B2 in the example of FIG. 14) may be displayed on AR display 400 in different display modes. That is, in the example of FIG. 14, the bounding box B1 may be displayed in blue, for example, while the bounding box B2 may be displayed in red, for example, to display the object detection results in different colors. Furthermore, in the example of FIG. 14, the frame lines of the bounding boxes B1 and B2 may be displayed in different line widths or different line types (solid line, dashed line, dotted line, chain line, etc.).

[0078] (Sixth embodiment) In this embodiment, an AR microscope suitable for use in the inspection of specimens in the field of pathology will be described.

[0079] For example, Congo red staining is widely used in testing for the presence or absence of amyloid protein in tissue specimens. Congo red staining stains amyloid protein reddish-orange, but the stained portion is not necessarily amyloid protein. To confirm that the stained portion is amyloid protein, polarized light observation of the stained portion is performed. If green polarized light is confirmed from the stained portion through polarized light observation, the stained portion is confirmed to be amyloid protein.

[0080] Thus, when testing for the presence or absence of amyloid protein using Congo red staining, it is necessary to switch between bright-field observation to confirm the stained area and polarized observation to confirm the polarized light from the stained area.

[0081] Therefore, in this embodiment, object detection is performed on an image of a Congo red-stained sample using polarized light observation, and an auxiliary image that identifies the area where amyloid protein is present, obtained as a result of the detection, is superimposed on an optical image of the sample obtained by bright-field observation.

[0082] Fig. 15 is a diagram illustrating a microscope system 6 according to this embodiment. The microscope 100 in the microscope system 1 shown in Fig. 1 is a stereomicroscope, but the microscope 600 in the microscope system 6 in Fig. 6 is not a stereomicroscope but a normal upright microscope.

[0083] The microscope 600 includes a stage 101 , an objective lens 102 , a polarizer 202 , a condenser lens 103 , a lens barrel 610 , and an illumination device 120 .

[0084] The illumination device 120 irradiates the sample with illumination light from vertically below the sample. The illumination device 120 includes a light source 121 and a mirror 722. The illumination light emitted from the light source 121 passes through the mirror 722, an illumination lens (not shown), a polarizer 202, and a condenser lens 103, illuminating the sample from below. At this time, the light transmitted through the sample enters the objective lens 102 and is then imaged by the imaging lens 112 provided in the lens barrel 610. This allows a user of the microscope system 6 to look through the eyepiece 111 and magnify and observe the optical image of the sample formed on the image plane by the imaging lens 112 with the eyepiece 111. Note that instead of the monocular microscope shown in FIG. 15 in which observation is made with one eyepiece 111, a binocular microscope may be used in which observation is made with both left and right eyepieces.

[0085] In this way, in the microscope system 6, although the polarizer 202 is inserted in the optical path of the illumination light, it is possible to observe the sample through the eyepiece 111 by the transmitted light bright-field observation method, as in the microscope system 1. In the microscope system 6, the illumination device 120, the polarizer 202, the condenser lens 103, the objective lens 102, and the imaging lens 112 form a first optical system.

[0086] The AR display 400 is a superimposing device that superimposes the auxiliary image, which is auxiliary information output by the processing device 500, on an optical image of the sample obtained by the transmitted light bright-field observation method, which is formed on the optical axis of the first optical system of the microscope 600. The light emitted from the AR display 400 is converged into the observation light path by a beam splitter 401 arranged on the observation light path.

[0087] Furthermore, in the microscope system 6, a portion of the light that has passed through the sample and entered the objective lens 102 passes through the beam splitter 201 and is used for image capture by the camera 200. However, in the microscope 600, the analyzer 203 is arranged so that it can be inserted and removed freely from the optical path from the beam splitter 201 to the camera 200. When the analyzer 203 is inserted in this optical path, the image sensor included in the camera 200 acquires a digital image of the sample observed by polarization observation. On the other hand, when the analyzer 203 is removed from this optical path, the image sensor included in the camera 200 can acquire a digital image observed by transmission bright-field observation, which is similar to the optical image observed by a user of the microscope system 6 when looking through the eyepiece 111.

[0088] In the microscope system 6, the illumination device 120, the polarizer 202, the condenser lens 103, the objective lens 102, the beam splitter 201, and the analyzer 203 constitute a second optical system.

[0089] The sensor 204 detects a change in the state of insertion or removal of the analyzer 203 with respect to the optical path from the beam splitter 201 to the camera 200. That is, the sensor 204 detects a switch in the observation method (switch between polarized light observation and transmitted light bright-field observation) in observation with the second optical system in the microscope system 6. The detection result of this switch is notified to the processing device 500 by the sensor 204.

[0090] When the processing device 500 is notified by the sensor 204 that the observation method for observation using the second optical system in the microscope system 6 has switched to polarization observation, the processing device 500 generates an auxiliary image including information specifying an area where the presence of an object is estimated. The processing device 500 generates this auxiliary image based on a digital image obtained by the polarization observation method and acquired by the camera 200. More specifically, the processing device 500 performs object detection on the digital image obtained by the polarization observation method, and outputs the auxiliary image including information specifying an area where the presence of the object is estimated to the AR display 400 as auxiliary information including information on the result of the object detection.

[0091] The AR display 400 superimposes the auxiliary image output from the processing device 500 on the optical image formed on the image plane by the imaging lens 112. A display example of the superimposed display at this time will be described with reference to FIG.

[0092] In the example of FIG. 16 , the sample S is a patient tissue specimen containing amyloid protein and is stained with Congo red. The optical image O is an example of a bright-field transmission image of the sample S observed by a user of the microscope system 6 looking through the eyepiece 111, showing the structure of the tissue. Areas U1 and U2 in the optical image O are represented by small circles, and these small circles are Congo red-stained areas. However, the optical image O does not reveal which parts of the Congo red-stained small circles contain amyloid protein. Meanwhile, the digital image P is an example of a polarized light observation image captured by the camera 200 when the second optical system in the microscope system 6 is switched to observation using polarized light observation, showing how the shape of amyloid is specifically detected by polarized light observation. The digital image P reveals that amyloid protein is present in the small circle area in area U1 in the optical image O, and that amyloid protein is not present in the small circle area in area U2 in the optical image O.

[0093] The processing device 500 performs object detection on the digital image P, which is a polarized observation image output from the camera 200, and outputs an auxiliary image including information specifying an existence region A where the presence of amyloid protein is estimated, to the AR display 400. The AR display 400 displays an observation image Q in which the annular existence region A is superimposed on the optical image O by overlaying the auxiliary image output by the processing device 500 on an optical image O of the sample S formed on the optical axis of the first optical system by a transmitted light bright-field observation method.

[0094] A user of the microscope system 6 can easily confirm amyloid deposits in each tissue of the patient by observing the observation image Q in which the presence area A is superimposed on the optical image O through the eyepiece 111.

[0095] The processing device 500 may calculate the area of ​​the red-stained area and count the number of cells in the digital image of the sample output from the camera 200 after the observation method using the second optical system has been switched to the transmitted light bright-field observation method, and generate an auxiliary image containing the resulting information. At this time, the processing device 500 outputs the generated auxiliary image to the AR display 400, which then displays the auxiliary image superimposed on the optical image of the sample obtained by the transmitted light bright-field observation method. This allows the user to grasp this information without taking their eyes off the eyepiece 111.

[0096] Furthermore, the processing device 500 may determine which part of the human body the sample is (for example, a blood vessel) from a digital image of the sample obtained by the transmitted light bright-field observation method.

[0097] For example, if amyloid deposits are abundant in blood vessels in brain tissue, cerebral hemorrhage or dementia is predicted. Therefore, the processing device 500 performs object detection on a digital image of a sample, i.e., a blood vessel in brain tissue, output from the camera 200, whose observation method in the second optical system is switched to polarization observation, and determines whether amyloid deposits are abundant in the blood vessel. If amyloid deposits are determined to be abundant, the processing device 500 generates an auxiliary image containing a message such as "Caution: Cerebral Hemorrhage" and outputs it to the AR display 400. The AR display 400 displays this auxiliary image superimposed on an optical image of the sample obtained by transmission brightfield observation. This allows the user to grasp the information obtained from this message without taking their eyes off the eyepiece 111.

[0098] (Seventh embodiment) This embodiment also describes an AR microscope suitable for applications such as specimen inspection in the field of pathology.

[0099] 17 is a diagram illustrating a microscope system 7 according to this embodiment. The microscope system 7 according to this embodiment differs from the microscope system 6 shown in FIG. 15 in that the microscope 600 further includes a beam splitter 205 and a camera 210. Furthermore, the sensor 204 included in the microscope system 6 has been eliminated in the microscope system 7.

[0100] In the microscope system 6, the analyzer 203 is arranged so as to be freely insertable and detachable in the optical path from the beam splitter 201 to the camera 200, but in the microscope system 7, a beam splitter 205 is arranged instead of the analyzer 203.

[0101] In the microscope system 7, illumination light emitted from a light source 121 of an illumination device 120 passes through a beam splitter 122, a polarizer 202, and a condenser lens 103 to illuminate a sample from below. At this time, the light transmitted through the sample enters the objective lens 102 and is then imaged by an imaging lens 112 provided in a binocular tube 110. In this way, in the microscope system 7, like the microscope system 6, it is possible to observe a sample through the eyepiece lens 111 using the transmitted light bright-field observation method. In the microscope system 7, the illumination device 120, the polarizer 202, the condenser lens 103, the objective lens 102, and the imaging lens 112 constitute a first optical system.

[0102] Furthermore, in the microscope system 7, a portion of the light that has passed through the sample and entered the objective lens 102 passes through the beam splitters 201 and 205 and enters the camera 200, where it is used for capturing an image. Therefore, the image sensor (first image sensor) included in the camera 200 acquires a digital image of the sample observed by the transmitted light bright-field observation method, which is similar to the optical image observed by a user of the microscope system 7 by looking through the eyepiece 111.

[0103] On the other hand, in the microscope system 7, a portion of the light that passes through the sample and enters the objective lens 102 passes through the beam splitter 201, is reflected by the beam splitter 205, and then passes through the analyzer 203 and enters the camera 210. Therefore, the image sensor (second image sensor) included in the camera 210 acquires a digital image of the sample observed by the polarization observation method. In the microscope system 7, the illumination device 120, the polarizer 202, the condenser lens 103, the objective lens 102, the beam splitters 201 and 205, and the analyzer 203 constitute a second optical system.

[0104] The processing device 500 generates an auxiliary image including information specifying an area where the presence of an object is estimated, based on a digital image acquired by the camera 210 using the polarization observation method. More specifically, the processing device 500 performs object detection on the digital image acquired by the camera 210 using the polarization observation method, and generates an auxiliary image including information specifying an area where the presence of an object is estimated, as auxiliary information including information on the result of the object detection. The processing device 500 outputs the auxiliary image generated in this manner to the AR display 400.

[0105] The AR display 400 superimposes the auxiliary image output from the processing device 500 onto the optical image formed on the image plane by the imaging lens 112.

[0106] Since the microscope system 7 of this embodiment is configured as described above, it is possible to perform observations similar to those of the microscope system 6 without switching the observation method when observing with the second optical system, more specifically, without inserting or removing the analyzer 203.

[0107] In the microscope system 7 of this embodiment, part of the light that has passed through the sample and has passed through the beam splitter 205 is used for image capture by the camera 200. On the other hand, part of the light that has passed through the sample and has been reflected by the beam splitter 205 is used for image capture by the camera 210. For this reason, there is a mirror image relationship between the digital image of the optical image of the sample obtained by the transmitted light bright-field observation method, which is formed on the optical axis of the first optical system and acquired by image capture by the camera 200, and the digital image of the sample obtained by the polarized light observation method, which is acquired by image capture by the camera 210. In such a case, when detecting an object in a digital image, the processing device 500 performs mirror image inversion on either the digital image of the optical image of the sample obtained by the transmitted light bright-field observation method or the digital image of the sample obtained by the polarized light observation method, and then performs object detection.

[0108] Next, the processing device 500 used in each of the embodiments described above will be described in more detail.

[0109] Fig. 18 is a diagram illustrating an example of the hardware configuration of a computer 1000 for realizing the processing device 500. The hardware configuration illustrated in Fig. 18 includes, for example, a processor 1001, a memory 1002, a storage device 1003, a reading device 1004, a communication interface 1006, and an input / output interface 1007. The processor 1001, the memory 1002, the storage device 1003, the reading device 1004, the communication interface 1006, and the input / output interface 1007 are connected to one another via, for example, a bus 1008.

[0110] The processor 1001 is an arbitrary electric circuit, and may be, for example, a single processor, a multiprocessor, or a multi-core processor. The processor 1001 may perform the process of AR display of the auxiliary image described above by reading and executing a program stored in the storage device 1003.

[0111] The memory 1002 is, for example, a semiconductor memory, and may include a RAM area and a ROM area. The storage device 1003 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device.

[0112] The reading device 1004 accesses the removable storage medium 1005 in accordance with, for example, an instruction from the processor 1001. The removable storage medium 1005 is realized by, for example, a semiconductor device, a medium that inputs and outputs information by magnetic action, or a medium that inputs and outputs information by optical action. An example of a semiconductor device is a USB (Universal Serial Bus) memory. An example of a medium that inputs and outputs information by magnetic action is a magnetic disk. An example of a medium that inputs and outputs information by optical action is a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc), a Blu-ray Disc, etc. (Blu-ray is a registered trademark).

[0113] The communication interface 1006 communicates with other devices, for example, in accordance with instructions from the processor 1001. The input / output interface 1007 is, for example, an interface between an input device and an output device. The input device may be, for example, an operating device such as a keyboard, mouse, or touch panel that accepts instructions from a user. The output device is, for example, a display device such as a monitor, and an audio device such as a speaker.

[0114] The program executed by the processor 1001 is provided to the computer 1000 in the following form, for example. (1) It is pre-installed in the storage device 1003. (2) Provided by removable storage medium 1005. (3) Provided from a server such as a program server.

[0115] 18 is an example, and the embodiment is not limited thereto. For example, part of the above-described configuration may be deleted, or new configuration may be added. In another embodiment, for example, part or all of the functions of the above-described electrical circuits may be implemented as hardware using a field programmable gate array (FPGA), a system-on-a-chip (SoC), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or the like.

[0116] The above-described embodiments are illustrative examples provided to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modifications and alternatives to the above-described embodiments may be included. In other words, the components of each embodiment may be modified without departing from the spirit and scope of the invention. Furthermore, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. Furthermore, some components may be deleted from or added to the components shown in each embodiment. Furthermore, the order of the processing steps shown in each embodiment may be reversed as long as no contradictions are present. In other words, the microscope system of the present invention is susceptible to various modifications and variations without departing from the scope of the claims.

[0117] In some of the above-described embodiments, the object detection model has been described as a model that learns about eggs using deep learning. However, at the time of candling, an egg does not exist alone, but is usually covered by corona radiata cells and cumulus cells. Therefore, the training data used to train the object detection model is preferably image data (image data T1, image data T2) that shows an egg E1 together with corona radiata cells E3 and cumulus cells E4, as shown in FIG. 19. More specifically, the training data is preferably image data in which an egg E1 surrounded by corona radiata cells E3 in the image is labeled using an annotation tool. E2 is the zona pellucida. Using such training data is expected to further improve the accuracy of egg detection during egg candling.

[0118] In addition, in each of the above-described embodiments, the AR display 400 is configured to superimpose the auxiliary image output from the processing device 500 on the optical image of the sample. Here, for example, the processing device 500 may perform image synthesis by superimposing the auxiliary image obtained by object detection on a digital image of the optical image of the sample, and the synthesized image obtained by this image synthesis may be displayed on a monitor display device such as a liquid crystal display.

[0119] In this specification, the expression "based on A" does not mean "based only on A," but also means "based at least on A," and further means "based at least partially on A." That is, "based on A" may be based on B in addition to A, or may be based on a part of A.

[0120] The following supplementary notes are provided in relation to the embodiments described above. However, the present invention is not limited to the following supplementary notes.

[0121] [Appendix 1] a microscope for observing a sample containing an object from vertically above the sample; a first imaging device that captures an image of the sample from vertically above the sample; a second imaging device that captures an image of the sample from vertically below the sample; a processing device that performs object detection on at least one of the first image acquired by the first imaging device and the second image acquired by the second imaging device, and outputs an auxiliary image including information that identifies an existence region in which the existence of the object is estimated; and a superimposing device that superimposes the auxiliary image output by the processing device onto an optical image of the sample formed on the optical path of the microscope. A microscope system comprising: [Appendix 2] 2. The microscope system according to claim 1, The processing device performs the object detection on the first image and the second image, and outputs the auxiliary image including the information specifying the existence region. A microscope system comprising: [Appendix 3] 2. The microscope system according to claim 1, The information included in the auxiliary image is switched among at least first information in which the existence region is identified by the object detection for the first image, second information in which the existence region is identified by the object detection for the second image, and third information in which the existence region is identified by both the object detection for the first image and the object detection for the second image, depending on the setting of the microscope system. A microscope system comprising: [Appendix 4] The microscope system according to claim 3, further comprising: an operating device that accepts a user's selection from at least a first support obtained by photographing the sample from vertically above, a second support obtained by photographing the sample from vertically below, and a third support obtained by photographing the sample from vertically above and vertically below, The processing device switches the information included in the auxiliary image among the first information, the second information, and the third information in response to a selection made by the user using the operation device. A microscope system comprising: [Appendix 5] 2. The microscope system according to claim 1, The processing device performs digital zoom processing on the second image by cropping and enlarging a range according to the optical magnification of the microscope. A microscope system comprising: [Appendix 6] 2. The microscope system according to claim 1, The processing device sets a digital zoom magnification corresponding to the optical magnification of the microscope to the second imaging device. A microscope system comprising: [Appendix 7] 2. The microscope system according to claim 1, The microscope comprises: a first illumination device that irradiates illumination light onto the sample from vertically below the sample; a second illumination device that irradiates illumination light onto the sample from vertically above the sample, the first imaging device captures an image of the sample during a light emission period of the first illumination device; The second imaging device captures an image of the sample during the light emission period of the second illumination device. A microscope system comprising: [Appendix 8] 8. The microscope system according to claim 7, The processing device controls the first illumination device and the second illumination device so that the first illumination device and the second illumination device alternately irradiate the sample with the illumination light. A microscope system comprising: [Appendix 9] 9. The microscope system according to claim 8, The processing device controls the first lighting device and the second lighting device so that a light emission period of the first lighting device is longer than a light emission period of the second lighting device. A microscope system comprising: [Appendix 10] 2. The microscope system according to claim 1, The microscope comprises: a first illumination device that irradiates the sample with visible light from vertically below the sample; a second illumination device that irradiates the sample with invisible light from vertically above the sample, the first imaging device images the sample with the visible light; The second imaging device captures an image of the sample using the invisible light. A microscope system comprising: [Appendix 11] 11. The microscope system according to claim 10, The invisible light irradiated onto the sample by the second illumination device includes near-infrared light. A microscope system comprising: [Appendix 12] 2. The microscope system according to claim 1, the sample is follicular fluid, the object is an egg, The superimposing device superimposes the auxiliary image, which includes information for identifying the region where the egg is present, on the optical image of the follicular fluid during an egg candling operation for finding the egg from the follicular fluid. A microscope system comprising: [Appendix 13] 13. The microscope system according to claim 12, The microscope is a stereo microscope. A microscope system comprising: [Appendix 14] 2. The microscope system according to claim 1, The object detection performed by the processing device is performed using a trained model using a neural network that has learned the object through deep learning. A microscope system comprising: [Appendix 15] 15. The microscope system according to claim 14, The training data used to train the trained model is image data in which an ovum surrounded by corona radiata cells in the image is labeled. A microscope system comprising: [Explanation of symbols]

[0122] 1, 2, 4, 6, 7 Microscope System 100, 600 microscope 101 Stages 102 Objective Lens 103 Condenser Lens 104, 112 Imaging lenses 110 Binoculars 111 Eyepiece 120, 130 Lighting equipment 121, 131 light source 122, 132, 201, 205, 401 Beam splitter 140 zoom lens 141 Zoom Coupler 200, 210, 300 cameras 202 Polarizer 203 Analyzer 204 Sensors 400 AR Display 500 processing equipment 610 Telescope 722 Mirror 1000 computers 1001 processor 1002 memory 1003 Storage device 1004 Reading device 1005 Removable storage media 1006 Communication Interface 1007 Input / Output Interface 1008 Bus A Existence area B, B1, B2 bounding boxes C container C1, C2 information D1, D11, D12, D2, D21, D22, P Digital Images D3, D13, D23, D33, D43 auxiliary images E1, S1, S1-1, S1-2 eggs E2 zona pellucida E3 corona radiata cells E4 cumulus cells F field of view L1 and L2 lines Optical image Q Observation image S sample S2 follicular fluid S3, S3-1, S3-2, S4 tissue pieces T1 and T2 image data U1, U2 area d1, d2 Light emission period

Claims

1. a first optical system including a transmitted illumination light source and capable of observing an object through an eyepiece by a transmitted light bright-field observation method; a second optical system that shares at least a portion of its optical system with the first optical system and that allows observation of the object different from observation through the eyepiece; an imaging element that acquires at least the digital image obtained by the different observation via the second optical system; a processing device that performs object detection on the digital image acquired by the imaging element via the second optical system and outputs auxiliary information including information on the result of the object detection; a superimposing device that superimposes the auxiliary information on an optical image obtained by a transmitted light bright-field observation method that is formed on the optical axis of the first optical system; A microscope system comprising:

2. 2. The microscope system according to claim 1, the imaging element is further capable of acquiring a digital image of the optical image via the first optical system; the processing device also performs the object detection on the digital image acquired by the imaging element via the first optical system, and outputs the auxiliary information; the superimposing device superimposes the auxiliary information output by the processing device performing the object detection on each of the digital image acquired via the first optical system and the digital image acquired via the second optical system on the optical image; A microscope system comprising:

3. 3. The microscope system according to claim 1, the auxiliary information includes, as the result information, information representing an area where the target object is present, the area being estimated by the object detection; A microscope system comprising:

4. 3. The microscope system according to claim 2, a sensor that detects switching of the observation method in the observation using the second optical system and notifies the processing device of the detection result of the switching; A microscope system comprising:

5. 3. The microscope system according to claim 2, The imaging device includes at least a first imaging device that acquires a digital image of the optical image via the first optical system, and a second imaging device that acquires a digital image obtained by the different observation via the second optical system. A microscope system comprising:

6. 6. The microscope system according to claim 5, When the digital image of the optical image and the digital image obtained by the different observation are in a mirror image relationship, the processing device performs mirror image inversion on one of the digital image of the optical image and the digital image obtained by the different observation, and then performs the object detection. A microscope system comprising:

7. 3. The microscope system according to claim 2, the different observation includes observation by polarized light observation; the processing device performs the object detection on the digital image obtained by observation using a polarization observation method via the second optical system. A microscope system comprising:

8. 2. The microscope system according to claim 1, Observation through the eyepiece is upright observation, the different observation is an inverted observation, the processing device performs the object detection on the digital image obtained by inverted observation via the second optical system. A microscope system comprising:

Images