Microscope system, projection unit, and sorting support method

Abstract

To support a selection work of a target in a sample.SOLUTION: A microscope system 1 includes: a microscope 100 for forming an optical image of a sample including a selection target; an imaging device 143 for acquiring a digital image of the sample; a processor 200 for generating a target image on the basis of detection of an object in the digital image; and a projection device 153 for displaying the target image in an image surface on which the optical image is formed, so that the target image looks larger than the selection target in the optical image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The disclosure of this specification relates to a microscope system, a projection unit, and a sorting support method.

Background Art

[0002] At present, with the progress of late marriage and late childbirth, the number of patients receiving infertility treatment is increasing year by year, and the demand for assisted reproductive technology (ART) is also increasing.

[0003] ART is a general term for technologies that fertilize eggs and sperm taken from humans outside the body, such as in vitro fertilization (IVF) and microinjection, and is distinguished from general artificial insemination in which the collected sperm is injected into the uterus to be fertilized with eggs inside the body.

[0004] Technologies related to ART are described in, for example, Patent Document 1. Patent Document 1 describes a microscope suitable for intracytoplasmic sperm injection (ICSI), which is a type of ART. Note that ICSI is a method of directly injecting sperm into an egg by piercing an injection pipette containing sperm into the egg fixed with a holding pipette.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Incidentally, in order to increase the success rate of ICSI, it is important to select sperm and inject the sperm suitable for fertilization into the egg. However, whether the sperm obtained by the selection work is of good quality depends largely on the experience of the embryologist, who is the operator, and a gap in the fertilization rate is likely to occur among embryologists.

[0007] Note that, although the sperm selection work has been described as an example above, the above problems can occur not only in the sperm selection work but also in the selection work of any target.

[0008] From the above circumstances, an object according to one aspect of the present invention is to provide a technique for assisting the selection work of a target in a sample.

Means for Solving the Problems

[0009] A microscope system according to one aspect of the present invention includes Sperm a microscope that forms an optical image of a sample including the same, an imaging device that acquires a digital image of the sample, A processing device that performs object detection on the digital image, detects candidate sperm for selection in the sample based on the object detection, and includes the sperm to be selected, which is selected from the detected candidate sperm for selection a processing device that generates a target image, and an optical device that displays the target image The sperm to be selected in on an image plane where the optical image is formed, in a size larger than that in the The sperm to be selected optical image.

[0010] A projection unit according to one aspect of the present invention is a projection unit attached to a microscope, and includes Including sperm an imaging unit that acquires a digital image of a sample, A treatment unit that performs object detection on the digital image, detects candidate sperm for selection in the sample based on the object detection, and includes the sperm to be selected, which is selected from the detected candidate sperm for selection a processing unit that generates a target image, which is an image of a selection target, and a projection unit that displays the target image The sperm to be selected in on an image plane where the optical image of the sample formed by the microscope is formed, in a size larger than that in the The sperm to be selected optical image.

[0011] A selection support method according to one aspect of the present invention includes Sperm forming an optical image of a sample including the same, acquiring a digital image of the sample, Performing object detection on the digital image, detecting candidate sperm for selection in the sample based on the object detection, and including the sperm to be selected, which is selected from the detected candidate sperm for selection generating a target image, and The sperm to be selected in displaying the target image on an image plane where the optical image is formed, in a size larger than that in theThe sperm to be selected Display it in a larger size.

Advantages of the Invention

[0012] According to the above aspect, it is possible to assist in determining the quality of a sample in the sorting operation.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Embodiments for Carrying Out the Invention

[0014] [First Embodiment] FIG. 1 is a figure illustrating the configuration of the microscope system 1. FIG. 2 is a figure illustrating the configuration of the microscope 100. FIG. 3 is a figure illustrating the configuration of the operation unit of the input device 50. FIG. 4 is a figure illustrating the configuration of the processing device 200.

[0015] The microscope system 1 is a system for observing a sample by looking through an eyepiece 101. Specifically, the microscope system 1 is an inverted microscope system equipped with a transmitted illumination system 120, which is used for intracytoplasmic sperm injection, particularly sperm sorting. The microscope system 1 is utilized, for example, by embryologists. The sample to be observed is, during sperm sorting operations, a sperm suspension containing spermatozoa contained in a petri dish or the like.

[0016] The microscope system 1 includes at least a microscope 100, an imaging device 143, a projection device 153, and a processing device 200. The microscope 100 forms an optical image of a sample containing spermatozoa to be sorted. The imaging device 143 acquires a digital image of the sample. The processing device 200 generates an image of the object to be sorted (hereinafter referred to as the target image) based on object detection of the digital image. Here, the object to be sorted refers to an object whose quality is judged by the user, and an object for which selection or non-selection is determined as a result of the quality judgment. The projection device 153 is an example of an optical device that displays the target image on the image plane where the optical image is formed, in a size larger than that of the object to be sorted in the optical image. In this specification, "displaying an image" means forming the image so that it can be visually recognized, or in other words, forming the image and arranging it on a surface (position) where it can be visually recognized.

[0017] The microscope system 1 uses the projection device 153 to display the target image on the image plane where the optical image of the sample is formed by the microscope 100, in a size larger than that of the object to be sorted in the optical image. That is, when the object to be sorted (spermatozoa) in the optical image is displayed, for example, within a region of 1 mm × 1 mm on the image plane, the target image is displayed on the image plane in a size larger than 1 mm × 1 mm. As a result, a user observing the sample by looking through the eyepiece 101 can observe the spermatozoa of the object to be sorted contained in the optical image in detail with a target image of a larger size without taking their eyes off the eyepiece 101. Therefore, it becomes possible to accurately identify and collect good spermatozoa in a short time. Thus, according to the microscope system 1, the sperm sorting work of users such as embryologists can be supported.

[0018] Hereinafter, with reference to FIGS. 1 to 4, a specific example of the configuration of the microscope system 1 will be described in detail. As shown in FIG. 1, the microscope system 1 includes, in addition to the above-described microscope 100, imaging device 143, projection device 153, and processing device 200, microscope controller 10, display device 30, a plurality of input devices (input device 40, input device 50, input device 60, input device 70), and identification device 80. Further, the microscope system 1 is connected to a database server 20 in which various data are stored. In this example, the imaging device 143 and the projection device 153 are arranged inside the microscope main body 110 of the microscope 100.

[0019] The microscope 100 is an inverted microscope equipped with an eyepiece 101. As shown in FIG. 1, the microscope 100 includes a microscope main body 110, a plurality of objective lenses 102, a stage 111, a transmitted illumination system 120, and an eyepiece tube 170 attached to the microscope main body 110. Further, as will be described later, the microscope 100 is provided with modulation elements for visualizing unstained samples such as sperm and eggs in each of the illumination optical path and the observation optical path. Users such as embryologists can observe samples using the microscope 100 by four microscopy methods: bright field (BF) observation, polarization (PO) observation, differential interference contrast (DIC) observation, and modulation contrast (MC) observation. Note that modulation contrast observation is also referred to as relief contrast (RC) observation.

[0020] The plurality of objective lenses 102 are mounted on a revolver 112. As shown in FIG. 2, the plurality of objective lenses 102 include an objective lens 102a for BF observation, an objective lens 102b for PO observation and DIC observation, and an objective lens 102c for MC observation. Further, the objective lens 102c includes a modulator 104. The modulator 104 includes three regions with different transmittances (for example, a region with a transmittance of about 100%, a region with a transmittance of about 5%, and a region with a transmittance of about 0%).

[0021] FIG. 2 illustrates three objective lenses according to microscopy methods. The plurality of objective lenses 102 may include objective lenses with different magnifications for each microscopy method. Hereinafter, a case will be described as an example in which a 4x objective lens for BF observation, 10x, 20x, and 40x objective lenses for MC observation, a 20x objective lens for PO observation, and a 60x objective lens for DIC observation are included.

[0022] The revolver 112 is a switching device that switches the objective lens disposed on the optical path among the plurality of objective lenses 102. The revolver 112 switches the objective lens disposed on the optical path according to the microscopy method and the observation magnification. The objective lens disposed on the optical path by the revolver 112 guides the transmitted light that has passed through the sample to the eyepiece 101.

[0023] The stage 111 has a sample placed in a container thereon. The container is, for example, a petri dish, and the sample contains germ cells such as sperm and eggs. The stage 111 moves in the optical axis direction of the objective lens 102 disposed on the optical path and in a direction orthogonal to the optical axis of the objective lens 102. Note that the stage 111 may be a manual stage or a motorized stage.

[0024] The transmitted illumination system 120 illuminates the sample placed on the stage 111 from above the stage 111. The transmitted illumination system 120 includes a light source 121 and a universal condenser 122 as shown in FIGS. 1 and 2. The light source 121 may be, for example, an LED (Light Emitting Diode) light source or a lamp light source such as a halogen lamp light source.

[0025] As shown in Fig. 2, the universal condenser 122 includes a polarizer 123 (first polarizing plate), a plurality of optical elements housed in a turret 124, and a condenser lens 128. The polarizer 123 is used for MC observation, PO observation, and DIC observation. The turret 124 houses a plurality of optical elements that are switched and used according to the microscopy method. The DIC prism 125 is used for DIC observation. The aperture plate 126 is used for BF observation and PO observation. The optical element 127 is a combination of a slit plate 127a which is a light-shielding plate with a slit formed thereon, and a polarizing plate 127b (second polarizing plate) arranged to cover a part of the slit, and is used for MC observation.

[0026] The eyepiece tube 170 includes an eyepiece lens 101. The imaging lens 103 is arranged between the eyepiece lens 101 and the objective lens 102. The imaging lens 103 forms an optical image of the sample on the image plane IP between the eyepiece lens 101 and the imaging lens 103 based on the transmitted light. Also, on the image plane IP, a target image described later is formed based on the light from the projection device 153. Thereby, the optical image and the target image are displayed on the image plane IP. The user of the microscope system 1 observes the virtual images of the optical image and the target image formed on the image plane IP using the eyepiece lens 101.

[0027] As shown in Figs. 1 and 2, the microscope main body 110 includes a laser-assisted hatching unit 130, an imaging unit 140, and a projection unit 150. Also, as shown in Fig. 2, the microscope main body 110 includes an intermediate magnification-changing unit 160. Further, the microscope main body 110 includes a DIC prism 105 and an analyzer 106 in a manner that they can be inserted into and removed from the optical path.

[0028] As shown in FIG. 2, the laser-assisted hatching unit 130 is a laser unit disposed between the objective lens 102 and the imaging lens 103. The laser-assisted hatching unit 130 irradiates the sample with laser light by introducing the laser light from between the objective lens 102 and the imaging lens 103. More specifically, the laser-assisted hatching unit 130 irradiates, for example, the zona pellucida surrounding the embryo grown from the fertilized egg with laser light. The laser-assisted hatching unit 130 includes a splitter 131, a scanner 133, a lens 134, and a laser 135. The splitter 131 is, for example, a dichroic mirror. The scanner 133 is, for example, a galvanometer scanner, and adjusts the irradiation position of the laser light in a direction orthogonal to the optical axis of the objective lens 102. The lens 134 converts the laser light into a parallel light beam. Thereby, the laser light is focused on the sample by the objective lens 102.

[0029] As shown in FIG. 2, the imaging unit 140 includes a splitter 141 and an imaging device 143 that acquires a digital image of the sample based on transmitted light. The imaging unit 140 is disposed between the imaging lens 103 and the eyepiece lens 101. The splitter 141 is, for example, a half mirror. The imaging lens 103 forms an optical image of the sample on the light receiving surface of the image sensor included in the imaging device 143. The imaging device 143 is, for example, a digital camera that acquires a digital image, and the image sensor included in the imaging device 143 is, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, or the like. The image sensor detects light from the sample and converts the detected light into an electrical signal by photoelectric conversion. The imaging unit 140 outputs the digital image acquired by the imaging device 143 to the processing device 200.

[0030] The projection unit 150 is disposed between the imaging lens 103 and the eyepiece lens 101. As shown in FIG. 2, the projection unit 150 includes a splitter 151, a lens 152, and a projection device 153. The splitter 151 is, for example, a half mirror. The projection device 153 projects the target image generated by the processing device 200. More specifically, the projection device 153 projects the target image onto the image plane IP by condensing the light from the projection device 153 onto the image plane of the imaging lens 103, that is, the image plane IP where the optical image is formed.

[0031] The intermediate magnification unit 160 is disposed between the objective lens 102 and the imaging lens 103. As shown in FIG. 2, the intermediate magnification unit 160 includes a plurality of lenses (lens 161, lens 162, lens 163), and changes the magnification of the optical image formed on the image plane by switching the lenses disposed on the optical path among them. By using the intermediate magnification unit 160, the magnification of the optical image can be changed without switching the objective lens 102 located near the sample.

[0032] The DIC prism 105 and the analyzer 106 are disposed between the objective lens 102 and the imaging lens 103. The DIC prism 105 is used for DIC observation. The analyzer 106 is used for PO observation and DIC observation.

[0033] In microscope 100, when performing MC observation, a polarizer 123 and an optical element 127 are arranged on the illumination optical path as a modulation element (hereinafter referred to as the first modulation element) for modulating the illumination light irradiated on the sample, and a modulator 104 is arranged on the observation optical path as a modulation element (hereinafter referred to as the second modulation element) for modulating the transmitted light. When performing PO observation, a polarizer 123 is arranged on the illumination optical path as the first modulation element, and an analyzer 106 is arranged on the observation optical path as the second modulation element. When performing DIC observation, a polarizer 123 and a DIC prism 125 are arranged on the illumination optical path as the first modulation element, and an analyzer 106 and a DIC prism 105 are arranged on the observation optical path as the second modulation element. Thereby, it is possible to visualize an unstained sample, and for example, sperm sorting can be performed.

[0034] Microscope controller 10 is a device for controlling microscope 100. Microscope controller 10 is connected to processing device 200, input device 50, and microscope 100, and controls microscope 100 according to commands from processing device 200 or input device 50.

[0035] Display device 30 is, for example, a display device such as a liquid crystal display, a plasma display, an organic EL display, a CRT display, or an LED matrix panel.

[0036] Input device 40 includes handle 41 and handle 42. By operating handle 41 and handle 42, the operation of a micromanipulator (not shown) for moving pipette 43 and pipette 44 is controlled. Pipette 43 and pipette 44 are used for operating a sample in the work of microinjection including sperm sorting. Pipette 43 is, for example, a holding pipette, and pipette 44 is, for example, an injection pipette.

[0037] The input device 50 is a hand switch device for changing the settings related to the microscopy method and observation magnification of the microscope 100. As shown in FIG. 3, the input device 50 has, for example, six buttons (buttons 51 to 56), and the user can quickly switch the settings of the microscope 100 just by pressing these buttons.

[0038] When the user presses button 51, the settings of the microscope 100 switch to the setting of BF observation at a magnification of 4 times (hereinafter referred to as BF4× observation). When the user presses button 52, the settings of the microscope 100 switch to the setting of MC observation at a magnification of 10 times (hereinafter referred to as MC10× observation). When the user presses button 53, the settings of the microscope 100 switch to the setting of MC observation at a magnification of 20 times (hereinafter referred to as MC20× observation). When the user presses button 54, the settings of the microscope 100 switch to the setting of MC observation at a magnification of 40 times (hereinafter referred to as MC40× observation). When the user presses button 55, the settings of the microscope 100 switch to the setting of PO observation at a magnification of 20 times (hereinafter referred to as PO20× observation). When the user presses button 56, the settings of the microscope 100 switch to the setting of DIC observation at a magnification of 60 times (hereinafter referred to as DIC60× observation).

[0039] The input device 60 is a keyboard. The input device 70 is a mouse. The input device 60 and the input device 70 are each connected to the processing device 200. Note that the microscope system 1 may include other input devices (not shown) such as a touch panel, a voice input device, and a foot pedal.

[0040] The identification device 80 is a device that acquires identification information attached to the sample. Note that "attached to the sample" includes, for example, the case where the identification information is pasted on the container that houses the sample. The identification information is information for identifying the sample, and more specifically, for example, information for identifying the patient who provided the sample. The identification device 80 may be, for example, a barcode reader, an RFID (registered trademark) reader, a QR code (registered trademark) reader, or the like.

[0041] The processing device 200 generates a target image based on object detection for the digital image acquired by the imaging device 143. The generated target image is output to the projection device 153 of the microscope 100, either directly or via the microscope controller 10. As shown in FIG. 1, the processing device 200 is connected to the microscope 100, the microscope controller 10, the display device 30, the input device 60, the input device 70, and the identification device 80. The processing device 200 is also connected to the database server 20.

[0042] The processing device 200 includes an image analysis unit 210, an image generation unit 220, and a storage unit 230 as functional components related to the generation of the target image.

[0043] The image analysis unit 210 performs image analysis including object detection for the digital image and target determination for determining sperm (hereinafter referred to as selection candidates) to be the selection target from among the sperm detected by the object detection. The image analysis performed by the image analysis unit 210 may include candidate evaluation for evaluating the selection candidates detected by the object detection in addition to object detection and target determination. Note that the selection candidates refer to candidates for the selection target, and the selection target is determined from among the selection candidates.

[0044] The method of object detection is not particularly limited. For example, the image analysis unit 210 may perform object detection using a learned model stored in the storage unit 230 and detect an object classified as sperm as a selection candidate. The algorithm of the learned model is not particularly limited, but for example, a deep learning model such as SSD, YOLO, or FasterR-CNN may be used.

[0045] The method for determining the target is not particularly limited. For example, the image analysis unit 210 may determine the sorting target based on the information input by the user. Specifically, the image analysis unit 210 may determine the sperm specified by the user using the input device 70 or the like as the sorting target. Further, the image analysis unit 210 may determine the sorting target based on, for example, the position of the sorting candidate detected by object detection. Specifically, the image analysis unit 210 may determine the sperm closest to the center of the visual field as the sorting target, or may determine the sperm within the frame provided at a predetermined position within the visual field as the sorting target. Further, the image analysis unit 210 may determine the sorting target based on, for example, candidate evaluation. Specifically, the image analysis unit 210 may determine the sperm with the highest evaluation in the candidate evaluation as the sorting target. Further, the image analysis unit 210 may combine the above-described target determination methods. For example, the sperm closest to the center of the visual field may be determined as the sorting target from among the sperm with higher evaluations than the reference.

[0046] The method for candidate evaluation is not particularly limited. For example, the image analysis unit 210 may calculate the straight line velocity (VSL), curvilinear velocity (VCL), and average path velocity (VAP) of the sorting candidate tracked based on the digital image. The image analysis unit 210 may further evaluate the sorting candidate based on the linearity (LIN) and straightness (STR) calculated using these parameters. That is, the image analysis unit 210 may evaluate the motility of the sorting candidate.

[0047] The image analysis unit 210 may detect and further count the vacuole sites included in the sorting candidate based on the digital image.

[0048] The embryo culture technician restrains promising sperm with an instrument such as a pipette for detailed observation, and determines whether the sperm is suitable for microinsemination. For this purpose, the image analysis unit 210 may evaluate whether the screening candidate is restrained based on the digital image, and may determine the screening target from among the screening candidates based on the evaluation of whether the screening candidate is restrained. Note that the image analysis unit 210 may calculate the above-described evaluation (whether the screening candidate is restrained) based on, for example, image classification using a learned model stored in the storage unit 230. Further, the image analysis unit 210 may calculate an evaluation of whether the screening candidate is restrained using the positional relationship between an instrument such as a pipette detected by object detection and the screening candidate.

[0049] The image generation unit 220 generates a target image that is an image of the screening target determined by the image analysis unit 210 based on the digital image. The method for generating the target image is not particularly limited. For example, the image generation unit 220 may generate the target image by cutting out the area of the screening target determined by the image analysis unit 210 from the digital image. For example, the image generation unit 220 may generate the target image by further performing image processing on the cut-out image. Further, for example, the image generation unit 220 may generate the target image by cutting out the area of the screening target from an image obtained by performing image processing on the digital image. The image processing may be, for example, processing for adjusting brightness, processing for enhancing contrast, processing for removing noise, or the like, or any other arbitrary processing. Further, these processes may be performed in combination.

[0050] When the projection device 153 projects the target image onto the image plane, the image generation unit 220 may adjust the size of the target image to be generated by enlarging or reducing the image cut out from the digital image by digital zoom so that the target image projected onto the image plane is displayed in a size larger than that of the selection target in the optical image also projected onto the image plane. In addition, the size of the target image may be adjusted by changing the objective lens when acquiring the digital image, changing the projection magnification of the digital image onto the image plane, or other adjustment methods, and these methods may be combined. Also, the overall magnification of the target image arranged on the image plane may be determined according to the size of the selection target or according to the overall magnification of the optical image. Here, the overall magnification of the target image refers to the overall magnification with respect to the size of the selection target on the object plane. The overall magnification of the optical image refers to the overall magnification with respect to the size of the sample on the object plane and is the magnification of the image finally observed by the user. In both cases, it refers to the magnification of the image finally observed by the user. That is, the overall magnification of the target image is the magnification when the target image (real image) formed on the image plane is observed as a virtual image through the eyepiece lens and is the magnification of the virtual image obtained by enlarging the real image. Also, the overall magnification of the optical image is the magnification when the optical image (real image) formed on the image plane is observed as a virtual image through the eyepiece lens and is the magnification of the virtual image obtained by enlarging the real image. Specifically, the image generation unit 220 may change the overall magnification of the target image according to the size of the selection target detected by the image analysis unit 210, or the image generation unit 220 may change the overall magnification of the target image according to the overall magnification of the optical image determined by the settings of the microscope system 1, that is, the magnification of the virtual image.

[0051] The area of the region to be selected, which is cut out from the digital image, is, for example, the area including the entire sperm detected by object detection, but it is not always necessary to cut out the entire sperm. The range to be cut out from the selection target may be changeable according to the settings of the microscope system 1. Therefore, the image generation unit 220 may cut out the area of the entire sperm (from the head to the tail), the area from the head to the neck of the sperm, only the area of the head, only the area of the neck, only the area of the tail, or other specified areas of the sperm according to the settings of the microscope system 1. The settings of the microscope system 1 may be changed, for example, according to which part of the selection target the user focuses on for the selection to make a quality determination.

[0052] The target image generated by the image generation unit 220 is output to the projection device 153. Thereby, the projection device 153 projects the target image onto the image plane, and the target image is projected onto the image plane with a size larger than that of the selection target in the optical image. Thereby, desirably, the target image is displayed on the image plane with a higher overall magnification than the optical image.

[0053] The storage unit 230 stores the learned models used in the image analysis performed by the image analysis unit 210. Specifically, the storage unit 230 stores a learned model for object detection performed to detect sperm that are selection candidates and a learned model for candidate evaluation for evaluating sperm that are selection candidates. The image analysis unit 210 performs object detection and candidate evaluation using the learned models stored in the storage unit 230.

[0054] Note that the processing device 200 may be a general-purpose computer or a dedicated computer. The processing device 200 is not particularly limited to this configuration, but may have, for example, a physical configuration as shown in FIG. 4. Specifically, the processing device 200 may include a processor 201, a storage device 202, an input interface (I / F) 203, an output interface (I / F) 204, and a communication device 205, and they may be connected to each other by a bus 206.

[0055] Processor 201 may include hardware, which may include, for example, at least one of a circuit for processing digital signals and a circuit for processing analog signals. Processor 201 may include, for example, one or more circuit devices (e.g., ICs) or one or more circuit elements (e.g., resistors, capacitors) on a circuit board. Processor 201 may be a CPU (central processing unit). Also, various types of processors including a GPU (Graphics processing unit) and a DSP (Digital Signal Processor) may be used for Processor 201. Processor 201 may be a hardware circuit having an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). Processor 201 may include an amplification circuit, a filter circuit, etc. for processing analog signals. Processor 201 functions as the above-described image analysis unit 210 and image generation unit 220 by executing a program stored in the storage device 202.

[0056] The storage device 202 may include a memory and / or other storage devices. The memory may be, for example, a random access memory (RAM). The memory may be a semiconductor memory such as SRAM (Static Randam Access Memory) or DRAM (Dynamic Random Access Memory). The storage device 202 may be, for example, a register, a magnetic storage device such as a hard disk device, an optical storage device such as an optical disk device, an internal or external hard disk drive, a solid state storage device, a CD-ROM, a DVD, other optical or magnetic disk storage devices, or other storage devices. The storage device 202 stores a program, a learned model, and other data to be executed by the processor 201, and functions as the above-described storage unit 230. Note that the storage device 202 is an example of a non-transitory computer-readable storage medium.

[0057] The input I / F 203 is connected to an input device operated by a user (e.g., an embryo culturist) of the microscope system 1, receives an operation signal corresponding to an operation on the input device, and outputs it to the processor 201.

[0058] The output I / F 204 is connected to the display device 30. The output I / F 204 may further be connected to a voice output device such as a speaker that outputs voice, a light emitting device such as a lamp that outputs light, a vibration device such as a vibrator that outputs vibration, etc., which are not shown.

[0059] The communication device 205 is a device that exchanges data with the microscope 100 and other devices. The communication device 205 may be a communication device that exchanges data wiredly or a communication device that exchanges data wirelessly. The programs and learned models stored in the storage device 202 may be those acquired by the communication device 205 from other devices via the Internet.

[0060] FIG. 5 is a flowchart showing an example of the procedure of ICSI by an embryo culturist. FIG. 6 is a diagram illustrating the configuration of a drop formed as the sample 300 in the petri dish 310. FIG. 7 is a flowchart showing an example of the sperm selection procedure by an embryo culturist. FIG. 8 is a flowchart showing an example of the selection support process. FIG. 9 is a flowchart showing an example of the target image generation process. FIG. 10 is a flowchart showing an example of the image display process. FIG. 11 is a diagram showing an example of an optical image generated by the microscope 100. FIG. 12 is a diagram showing an example of a digital image acquired by the imaging device 143. FIG. 13 is a diagram showing an example of the object detection result for the digital image. FIG. 14 is a diagram for explaining an example of the method for generating a target image. FIG. 15 is a diagram showing an example of an image visible from the eyepiece 101. Hereinafter, with reference to FIGS. 5 to 15, the specific utilization of the sperm selection support method performed by the microscope system 1 in ICSI will be described.

[0061] First, the user prepares a sample (step S1). Here, for example, as shown in FIG. 6, the user creates a sample 300 containing a plurality of drops in a petri dish 310 and places it on the stage 111.

[0062] Drop 301 is a cleaning drop and is used for cleaning the pipette. Drop 302 is a sperm suspension drop, for example, obtained by dropping a sperm suspension into a PVP solution. Drop 303 is an egg manipulation drop, for example, obtained by placing an egg in an m-HTF solution. Note that the m-HTF solution is an HTF solution containing Hepps with 10% serum added. These drops are covered with mineral oil.

[0063] Next, the user sets up the microscope system 1 (step S2). Here, for example, the user presses the button 51 of the input device 50 to switch the setting of the microscope system 1 to BF4× observation. Then, the input device 40 is operated to adjust the positions of the pipettes 43 and 44 and focus the pipettes 43 and 44. Further, the stage 111 is moved to clean the pipettes 43 and 44 with drop 301 (cleaning drop).

[0064] When the setup is completed, the user checks the state of the eggs (oocytes) in drop 303 (egg manipulation drop) (step S3). Here, for example, the user presses the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20× observation. The morphology of the eggs is observed in MC20× observation to select the eggs. Further, for example, the user may press the button 55 of the input device 50 to switch the setting of the microscope system 1 to PO20× observation. By observing the spindle of the eggs in PO20× observation, the maturity of the eggs can be determined and the eggs can be further selected.

[0065] When the selection of eggs is completed, the user selects sperm according to the procedure shown in FIG. 7 (step S4). First, for example, the user presses the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20× observation. Then, the stage 111 is moved to move the observation position to the drop 302 (sperm suspension drop), and the sperm is focused with MC20× observation (step S11).

[0066] Next, the user selects sperm suitable for fertilization with MC20× observation (step S12). Conventionally, in this step, an embryologist judged the quality of sperm based on the morphology and motility of the sperm observed in the optical image, and selected sperm based on the judgment. However, in MC20×, a sufficient field of view is ensured for the size of the sperm. Therefore, while there is an advantage that it is easy to grasp the motility of the sperm, the magnification is too low to accurately grasp the morphology of the sperm. For this reason, it has been necessary to observe the sperm selected in step S12 at a higher magnification, and it has been a problem that the selection work takes time and there is a need to go back.

[0067] In view of such problems, in step S12, the microscope system 1 displays the target image on the image plane in a size larger than the selection target in the optical image. By being projected in a size larger than the selection target in the optical image, it is possible to assist the user in judging the morphology of the sperm, which has been difficult in this step conventionally. As a result, it is possible to suppress going back due to different judgments in the subsequent steps. For example, the switch to MC40x observation in step S14 can be omitted. Furthermore, since the optical image and the target image are projected on the same image plane, it is possible to accurately grasp the morphology of the sperm being focused on (the sperm to be selected) by the target image while mainly confirming the motility of the sperm with the optical image. Therefore, since it is possible to select sperm by confirming both motility and morphology, it is possible to suppress going back due to different judgments in the subsequent steps.

[0068] Specifically, in step S12, by performing the sorting support process shown in FIG. 8 by the microscope system 1, the target image is projected onto the image plane together with the optical image. In the sorting support process, first, the microscope system 1 projects the optical image of the sample onto the image plane (step S21). Here, the microscope 100 forms an optical image O1 shown in FIG. 11, for example, on the image plane. Note that the region 143R shown in FIG. 11 indicates the region photographed by the imaging device 143 in step S22.

[0069] At the same time as step S21, the microscope system 1 acquires a digital image (step S22). Here, the imaging device 143 acquires a digital image D1 of the sample shown in FIG. 12, for example, based on the light from the sample, and outputs the acquired digital image D1 to the processing device 200.

[0070] Thereafter, the microscope system 1 generates a target image based on the digital image (step S23). Here, the processing device 200 performs the target image generation process shown in FIG. 9. In the target image generation process, first, the processing device 200 performs object detection on the digital image D1 (step S31). Here, the image analysis unit 210 performs object detection by inputting the digital image D1 as an input image into a learned model, for example, and detects sperm as sorting candidates as shown in FIG. 13. Note that FIG. 13 shows a state in which a box B is attached to the position of the object (sorting candidate) classified as sperm by object detection.

[0071] When sorting candidates are detected by object detection, the processing device 200 evaluates the sorting candidates (step S32). Here, the image analysis unit 210 evaluates each of the sorting candidates. Specifically, the image analysis unit 210 may evaluate the motility of the sorting candidates. Also, it may be evaluated whether the sorting candidates are constrained. Hereinafter, a case will be described in which the image analysis unit 210 evaluates the constrained sorting candidates relatively highly and the unconstrained sorting candidates relatively lowly as an example.

[0072] Thereafter, the processing device 200 determines the selection target based on the evaluation performed in step S32 (step S33). Here, the image analysis unit 210 determines, based on the evaluation, the most highly evaluated selection candidate among the selection candidates as the selection target. Specifically, for example, the spermatozoa constrained by the pipette 44 shown in FIG. 13 are determined as the selection target.

[0073] When the selection target is determined, the processing device 200 cuts out the area of the selection target from the digital image (step S34). Here, the image generation unit 220 cuts out the area of the selection target from the digital image D1 and generates a target image T1, for example, as shown in FIG. 14. The size of the target image T1 may be adjusted so as to be displayed larger than the selection target in the optical image when projected onto the image plane in step S34.

[0074] When the target image is generated in step S23, the microscope system 1 displays the target image on the image plane in a size larger than the selection target in the optical image (step S24). Here, the processing device 200 performs the image display process shown in FIG. 10. In the image display process, the processing device 200 first determines the overall magnification of the target image T1 to be displayed on the image plane (step S41), and further determines the size of the target image T1 (step S42). In step S41, when the overall magnification of the optical image formed on the image plane is, for example, 200 times, the image analysis unit 210 determines the overall magnification of the target image T1 projected onto the image plane to be a magnification exceeding 200 times (for example, 1000 times). In step S42, when the selection target in the optical image formed on the image plane is projected onto an area of 1 mm × 1 mm, the image analysis unit 210 determines the size of the target image T1 to be a size of 1 mm × 1 mm or more.

[0075] Furthermore, the processing device 200 determines the position of the target image T1 to be displayed on the image plane (step S43). Here, the processing device 200 determines the position of the target image so as to avoid the area of the spermatozoa of the selection target included in the optical image, for example, so that the area of the spermatozoa of the selection target included in the optical image does not overlap with the target image. Note that the position of the target image may be arranged at a preset position such as above, below, right, or left with respect to the center of the field of view.

[0076] Finally, the processing device 200 displays the target image T1 of the size determined in step S42 at the comprehensive magnification determined in step S41 at the position determined in step S43 (step S44). Here, the processing device 200 assigns each pixel of the target image T1 to the pixels of the projection device 153 based on the comprehensive magnification determined in step S41, the size determined in step S42, and the position determined in step S43. As a result, the projection device 153 projects the target image T1 at a position that does not overlap with the selection target in the optical image of the image plane and at a size larger than the selection target in the optical image. Thereby, the user can simultaneously confirm the optical image O1 displayed on the image plane and the target image T1 as shown in, for example, FIG. 15. For this reason, it is possible to accurately select sperm suitable for fertilization by confirming both the morphology and motility of sperm. Further, in step S12, by repeatedly performing the selection support process shown in FIG. 8, it is possible to track the sperm to be selected that moves within the visual field, so that the sperm to be selected can be carefully observed with the target image.

[0077] When sperm are selected in step S12, the user immobilizes the sperm by damaging the tail of the sperm by observing with RC20× (step S13). Here, the user immobilizes the sperm by rubbing the tail of the sperm against the bottom surface of the petri dish 310 with a pipette.

[0078] Thereafter, the user may further observe the morphology of the immobilized sperm in more detail and further select the sperm (step S14). Here, for example, the user presses the button 54 of the input device 50 to switch the setting of the microscope system 1 to MC40× observation. Thereafter, the user may further select sperm by observing with MC40×. Also here, the microscope system 1 may assist the sperm selection work of the embryologist by performing the selection support process shown in FIG. 8 and projecting the target image onto the image plane at a size larger than the selection target in the optical image, in the same manner as in step S12.

[0079] When the sperm selection is completed, the user then takes the selected sperm into the pipette 44, which is an injection pipette, moves the observation position to the drop 303 (the drop for egg manipulation) (step S15), and ends the series of procedures for sperm selection shown in FIG. 7.

[0080] Note that in the microscope system 1, the target images projected in steps S12 and S14 can assist in determining the morphology such as the presence and size of vacuoles in the sperm head that could not be conventionally confirmed.

[0081] When the sperm selection is completed, the user checks the position of the spindle for sperm injection preparation (step S5). Here, the user observes the egg selected in step S3 present in the drop 303 and checks the position of the spindle of that egg. Specifically, for example, the user presses the button 55 of the input device 50 to switch the setting of the microscope system 1 to PO20× observation. Then, the user changes the orientation of the spindle by operating the pipette 43, which is a holding pipette, so that the spindle of the egg visualized in PO20× observation is positioned in the 12 o'clock or 6 o'clock direction. This is to avoid damaging the spindle by the pipette that will be pushed against the egg from the 3 o'clock or 9 o'clock direction in step S6 described later.

[0082] Finally, the user injects the sperm into the egg (step S6) and ends the ICSI. Here, for example, the user presses the button 53 of the input device 50 to switch the setting of the microscope system 1 to MC20× observation. Then, in MC20× observation, the user fixes the egg whose orientation was adjusted in step S5 with the pipette 43, which is a holding pipette, and pierces the pipette 44, which is an injection pipette. Then, good sperm is injected from the pipette 44 into the egg.

[0083] When the series of ICSI procedures shown in FIG. 5 is completed, the user returns the egg injected with sperm to the incubator and cultures it. Also, the user may operate the processing device 200 using the input device 60 and the input device 70 to store the information obtained by ICSI in the database server 20. For example, the patient information of sperm and egg (such as the clinical data of the mother, the test results of semen containing sperm), and the data of the culture medium of sperm and egg (such as type, concentration, PH, etc.) may be associated with the image of the egg injected with sperm, the image of the selected sperm, the working time of ICSI, etc., and stored in the database server 20.

[0084] As described above, in the microscope system 1, in ICSI, the target image, which is the image of the sperm to be selected, is projected onto the image plane in a size larger than the target to be selected in the optical image. The size of sperm is about 60 μm, and at least a 20-fold objective lens is used to distinguish good sperm. Generally, the field number of an inverted microscope is about 22, so the actual field of view is about Φ1 mm. In the region of this actual field of view Φ1 mm, the operation of selecting sperm with a size of about 60 μm moving freely is a very difficult operation. Generally, since sperm presumed to be good sperm have high motility and the ICSI operation needs to be performed in a short time, in the sperm selection operation, it is necessary to quickly observe the morphology of sperm moving relatively fast and judge good / bad. Even in such a working environment with such strict constraints, according to the microscope system 1, while mainly confirming the motility of sperm with the optical image, it is possible to simultaneously confirm the morphology of sperm with the target image projected onto the image plane at a total magnification higher than that of the optical image. Thereby, good sperm can be appropriately selected in a short time based on both the motility and morphology of sperm, so that an improvement in the fertilization success rate can be realized. Therefore, according to the microscope system 1, the sperm selection operation in the sample can be effectively supported.

[0085] [Second Embodiment] FIG. 16 is a diagram for explaining another example of a method for generating a target image. FIG. 17 is a diagram showing another example of an image visible from the eyepiece 101. Hereinafter, this embodiment will be described with reference to FIGS. 16 and 17. Note that the configuration of the microscope system according to this embodiment (hereinafter also simply referred to as the microscope system) is the same as that of the microscope system 1.

[0086] This embodiment is different from the first embodiment in that a plurality of target images are displayed on the image plane. Other points are the same as those of the first embodiment. Specifically, in step S33 of the target image generation process shown in FIG. 9, the processing device 200 determines a plurality of selection targets based on the evaluation performed in step S32. The number of selection targets may be determined in advance, for example, two. Also, the number of selection targets may be determined based on the evaluation in step S32, and the processing device 200 may determine, for example, all of the selection candidates evaluated as exceeding a certain criterion as the selection targets.

[0087] Thereby, in step S34, the processing device 200 cuts out regions of a plurality of selection targets from the digital image and generates a plurality of target images (target image T1, target image T2), for example, as shown in FIG. 16.

[0088] Also in the microscope system according to this embodiment, since the target image is projected on the image plane in a size larger than the selection target in the optical image, it is possible to assist the user in judging the morphology of sperm. Since both the motility and morphology of sperm can be confirmed simultaneously by the optical image and the target image, it is possible to appropriately and quickly select good sperm based on both the motility and morphology of sperm, which is the same as that of the microscope system 1 according to the first embodiment. Further, in the microscope system according to this embodiment, as shown in FIG. 17, a plurality of sperm included in the optical image O1 are determined as selection targets, and a plurality of target images (target image T1, target image T2) are displayed together with the optical image. Therefore, since the possibility of early discovery of good sperm exceeding the criterion for the user's pass / fail judgment is increased, sperm can be selected more efficiently than the microscope system 1.

[0089] [Embodiment 3] FIG. 18 is a flowchart showing another example of the sorting support process. FIG. 19 is a flowchart showing another example of the image display process. FIG. 20 is a diagram for explaining an example of a method for generating an auxiliary image. FIG. 21 is a diagram showing yet another example of the image visible from the eyepiece 101. Hereinafter, this embodiment will be described with reference to FIGS. 18 to 21. Note that the configuration of the microscope system according to this embodiment (hereinafter also simply referred to as the microscope system) is the same as that of the microscope system 1.

[0090] This embodiment is different from the first embodiment in that the sorting support process shown in FIG. 18 is performed instead of the sorting support process shown in FIG. 8. Other points are the same as those in the first embodiment.

[0091] In the sorting support process shown in FIG. 18, the microscope system projects the optical image of the sample onto the image plane (step S51), acquires a digital image (step S52), and generates a target image (step S53). Note that the processing from step S51 to step S53 is the same as the processing from step S21 to step S23 in FIG. 8.

[0092] Thereafter, the microscope system generates an auxiliary image related to the target image (step S54). Here, the processing device 200 generates the auxiliary image. The auxiliary image is an image that provides the user with information related to the target image, and typically, it is an image that provides the user with information related to the sorting target, such as an evaluation of the sorting target. However, the auxiliary image may be any image that provides information related to the target image, and may also be an image that provides the user with information such as the overall magnification of the target image that is not related to the sorting target.

[0093] In step S54, for example, as shown in FIG. 20, the processing device 200 may first generate an evaluation result ER by evaluating the sorting target based on the target image T1, and further generate an auxiliary image A1 based on the evaluation result ER. In FIG. 20, the auxiliary image A1 includes evaluation information E1 regarding the motility of the sorting target, a marker M1 indicating the position or area of vacuoles in the sorting target, and further includes information indicating the number of vacuoles.

[0094] Also, the auxiliary image A1 may include evaluation information regarding the DNA integrity of the sorting target. That is, the auxiliary image A1 may include evaluation information regarding the quality of the sorting target, and the evaluation result ER is not limited to the evaluation result regarding motility, and may be an evaluation result regarding any quality of the sorting target.

[0095] In FIG. 20, an example of generating the auxiliary image A1 based on the target image T1 is shown. However, the auxiliary image A1 may be generated based on information (for example, the result of candidate evaluation) obtained in the process of generating the target image T1 in step S53.

[0096] When the target image and the auxiliary image are generated, the microscope system displays the target image and the auxiliary image on the image plane (step S55). Here, by performing the image display process shown in FIG. 19 by the processing device 200, as shown in FIG. 21, the target image T1 and the auxiliary image A1 are displayed together with the optical image O1 on the image plane.

[0097] In the image display process, the processing device 200 first determines the overall magnification of the target image T1 arranged on the image plane (step S61), then determines the size of the target image T1 (step S62), and further determines the position of the target image T1 displayed on the image plane (step S63). The processing from step S61 to step S63 is the same as the processing from step S41 to step S43 in FIG. 10.

[0098] Furthermore, the processing device 200 determines the position of the auxiliary image A1 arranged on the image plane (step S64). In step S64, the processing device 200 may determine the position of the auxiliary image A1 in relation to the target image T1. For example, the processing device 200 may determine the positional relationship between the auxiliary image A1 and the target image T1 such that, for example, the marker M1 in the auxiliary image A1 overlaps the region of interest in the target image T1. That is, the processing device 200 generates a projection image including the auxiliary image A1 and the target image T1 based on the determined positional relationship.

[0099] Finally, the processing device 200 displays the target image T1 with the size determined in step S62 at the overall magnification determined in step S61 at the position determined in step S63, and also displays the auxiliary image A1 at the position determined in step S64 (step S65). That is, the processing device 200 projects the projection image generated in step S64 onto the image plane. Here, the processing device 200 assigns each pixel of the projection image to the pixels of the projection device 153. That is, each pixel of the target image T1 is assigned to the pixels of the projection device 153 based on the overall magnification determined in step S61, the size determined in step S62, and the position determined in step S63. Furthermore, each pixel of the auxiliary image A1 is assigned to the pixels of the projection device 153 based on the position determined in step S64. As a result, the projection device 153 projects the target image T1 at a size larger than the selection target in the optical image at a position that does not overlap the selection target in the optical image O1 on the image plane, and projects the auxiliary image A1 at an appropriate position on the image plane. Thereby, as shown in FIG. 21, the user can simultaneously view the optical image O1, the target image T1, and the auxiliary image A1.

[0100] Also, in the microscope system according to this embodiment, since the target image is projected onto the image plane with a size larger than that of the selection target in the optical image, it is possible to assist the user in determining the morphology of sperm. Also, since both the motility and morphology of sperm can be confirmed simultaneously with the optical image and the target image, it is possible to appropriately and quickly select good sperm based on both the motility and morphology of sperm, which is the same as the microscope system 1 according to the first embodiment. Further, as shown in FIG. 21, the microscope system according to this embodiment can further support the user's determination of the quality of sperm by projecting an auxiliary image that provides information related to the target image onto the image plane. In particular, by superimposing a marker indicating the position or region of interest of the selection target on the target image having a size larger than that of the selection target in the optical image, it is possible to inform the user of the part to be noted while avoiding the problem that it is too small to be seen clearly when superimposed on the optical image. Therefore, according to the microscope system according to this embodiment, it is possible to select sperm suitable for fertilization with high accuracy, and thus it is possible to suppress the difference in fertilization rate among embryologists.

[0101] FIGS. 22 to 25 are diagrams showing still another example of the image seen from the eyepiece 101. The information included in the auxiliary image is not limited to the evaluation information E1 and the marker M1 as described above. As shown in FIGS. 22 to 25, other information may be included.

[0102] The auxiliary image A2 shown in FIG. 22 includes a marker M2 in addition to the evaluation information E1 and the marker M1. The marker M2 is a marker indicating the position or region of the selection target in the optical image O1, and may be generated, for example, based on the information obtained by object detection. By including the marker M2 in the auxiliary image, the user can easily recognize which sperm in the optical image O1 the target image T1 is an image of. Therefore, it is possible to easily perform the operation of confirming the morphology of the selection target with the target image T1 while confirming the motility of the selection target with the optical image O1.

[0103] The auxiliary image A3 shown in FIG. 23 includes a scale S and magnification information M in addition to the evaluation information E1, the marker M1, and the marker M2. By including the scale S for measuring the size of the target image T1 and the magnification information M indicating the overall magnification of the target image T1, it is possible to grasp the absolute size of each part of the sperm to be selected. In addition, it becomes easier to grasp the ratio of the part of interest (for example, a vacuole) occupied by the sperm. For this reason, for example, it becomes easier to determine whether the sperm satisfies the requirements shown in the WHO guidelines.

[0104] The auxiliary image A4 shown in FIG. 24 includes patient information P in addition to the evaluation information E1, the marker M1, the marker M2, the scale S, and the magnification information M. By including the patient information P, the user can always grasp who's sperm is being selected. Note that the patient information P is obtained from the database server 20 using the identification information acquired by the identification device 80.

[0105] The auxiliary image A5 shown in FIG. 25 may include a reference image R1 to be compared with the target image T1 in addition to the marker M2. The reference image R1 is, for example, an image of good sperm that successfully fertilized in past intracytoplasmic sperm injection. By including the reference image R1 in the auxiliary image A5, the user can compare the sperm included in the reference image R1 with the sperm (to be selected) included in the target image T1, and can use the comparison result as a reference for determining whether the to-be-selected sperm is good sperm.

[0106] [Fourth Embodiment] FIG. 26 is a diagram showing an example of image output to the display device 30. Hereinafter, this embodiment will be described with reference to FIG. 26. Note that the configuration of the microscope system according to this embodiment (hereinafter, also simply referred to as the microscope system) is the same as that of the microscope system 1.

[0107] This embodiment is different from the first embodiment in that the target image and the auxiliary image projected onto the image plane are also displayed on the display device 30. Other points are the same as those in the first embodiment. As shown in FIG. 26, the images (target image T1 and auxiliary image A6) displayed on the display device 30 do not necessarily have to be exactly the same as the images (target image T1 and auxiliary image A4) projected onto the image plane.

[0108] According to the microscope system according to this embodiment, collaborative work by a plurality of embryo culturists is facilitated. Specifically, it becomes possible for another embryo culturist to give advice while looking at the image displayed on the display device 30 to an embryo culturist who is selecting sperm while looking through the eyepiece. Therefore, it can be used for applications such as an experienced embryo culturist guiding a less experienced embryo culturist.

[0109] Also, on the display device 30, the auxiliary image can be displayed larger. That is, the selection target can be enlarged and displayed at a higher magnification. Therefore, according to the microscope system according to this embodiment, by checking the display device 30, the form of the selection target can be observed in more detail, and sperm suitable for fertilization can be selected with higher accuracy.

[0110] Note that even when the target image T1 superimposed on the optical image O1 is difficult to see in the observation using the eyepiece 101, in the observation using the display device 30, the background of the target image T1 can be arbitrarily adjusted, so the visibility of the target image T1 can be surely ensured. Therefore, as a preliminary measure when the visibility of the target image T1 on the optical image O1 is poor, the image displayed on the display device 30 may be used.

[0111] The above-described embodiments are presented with specific examples to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modifications of the above-described embodiments and alternative forms that replace the above-described embodiments may be included. That is, each embodiment can be modified in its components without departing from the spirit and scope thereof. Further, by appropriately combining a plurality of components disclosed in one or more embodiments, a new embodiment can be implemented. Also, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in the embodiment. Furthermore, the processing procedures shown in each embodiment may be performed in a different order as long as there is no contradiction. That is, the microscope system, projection unit, and sorting support method of the present invention can be variously modified and changed without departing from the description of the claims.

[0112] In the above-described embodiment, the microscope system 1 was exemplified, but the configuration of the microscope system is not limited to this example. For example, the microscope system 2 shown in FIG. 27 may be used. The microscope system 2 is different from the microscope system 1 in that it includes a microscope 400 instead of the microscope 100. The microscope 400 includes a projection unit 500 between the microscope body 410 and the lens barrel 420.

[0113] The projection unit 500 is a projection unit for a microscope and includes a projection part (beam splitter 151, lens 152, and projection device 153) corresponding to the projection unit 150 shown in FIG. 1, an imaging part (beam splitter 141 and imaging device 143) corresponding to the imaging unit 140 shown in FIG. 1, and an image processing unit 510. The image processing unit 510 functions as the image analysis unit 210, image generation unit 220, and storage unit 230 shown in FIG. 2.

[0114] The projection unit 500 and the microscope system 2 can also achieve the same effects as the microscope system 1. In addition, by using the projection unit 500 to expand an existing microscope system, the above-described effects can be obtained, so that the existing microscope system can be effectively utilized.

[0115] In the above-described embodiment, the case of sorting sperm has been described as an example, but the sorting target is not limited to sperm. It is desirable that the sorting target be selected based on both motility and morphology.

[0116] In the above-described embodiment, an example in which the projection device 153 projects a target image onto the image plane has been shown, but it is sufficient that the target image can be displayed on the image plane, and a transmissive liquid crystal device placed on the image plane may be used instead of the projection device 153. Further, in the above-described embodiment, an example in which the target image is displayed on the image plane in a size larger than the sorting target in the optical image has been shown, but the target image may be displayed on the image plane on which the optical image is formed in a size equal to or larger than the sorting target in the optical image.

[0117] Also, in the above-described embodiment, an example in which one target image is generated for each of the sorting targets has been shown, but two or more target images may be generated for each of the sorting targets. For example, a target image obtained by cutting out the entire region of a sperm and a target image obtained by cutting out only the region of the head of the sperm may be generated, and these may be arranged on the image plane at different overall magnifications. Thereby, while observing a specific part of the sperm at a higher magnification, the whole sperm can also be observed at an appropriate magnification.

[0118] Note that the target image may be updated periodically based on, for example, the frame rate of the video captured by the imaging device 143, and the update frequency may also be changeable according to the settings. That is, the target image may be projected onto the image plane as a video or as a still image. By projecting the target image as a still image, the user can carefully observe the morphology of sperm. Also, by projecting the target image as a sperm image, it is possible to avoid a situation where the auxiliary image is projected at a position shifted from the predetermined position with respect to the target image due to a delay in creating the auxiliary image. On the other hand, by projecting the target image as a video, the user can observe the morphology of sperm in detail without being affected by the shooting timing. This means, in other words, that the processing device tracks the sorting target. Also, even when projecting as a video, by adjusting the update frequency, it is possible to avoid the inconvenience caused by the delay in creating the auxiliary image.

[0119] In this specification, the expression "based on A" does not mean "based only on A", but means "based at least on A", and further means "based at least partially on A". That is, "based on A" may be based on B in addition to A, or may be based on a part of A.

Explanation of Reference Numerals

[0120] 1, 2 Microscope System 10 Microscope Controller 20 Database Server 30 Display Device 40, 50, 60, 70 Input Device 41, 42 Handle 43, 44 Pipette 80 Identification Device 100 Microscope 101 Eyepiece 102, 102a~102c Objective Lens 103 Imaging Lens 104 Modulator 105, 125 DIC Prism 106 Analyzer 110 Microscope body 111 Stage 112 Revolver 120 Transmitted illumination system 121 Light source 122 Universal condenser 123 Polarizer 124 Turret 126 Aperture plate 127 Optical element 127a Slit plate 127b Polarizing plate 128 Condenser lens 130 Laser-assisted hatching unit 131, 141, 151 Splitter 133 Scanner 135 Laser 140 Imaging unit 143 Imaging device 143R Imaging area 150 Projection unit 153 Projection device 160 Intermediate magnification unit 170 Eyepiece tube 200 Processing device 201 Processor 202 Memory device 203 Input interface 204 Output interface 205 Communication control device 206 Bus 210 Image analysis section 220 Image generation section 230 Memory section 300 Sample 310 Petri dish 400 Microscope 410 Microscope body 420 Tube 500 Projection unit 510 Image processing section A1~A6 Auxiliary image B Box D1 Digital image ER evaluation result E1 evaluation information O1 optical image M magnification information M1, M2 markers P patient information T1, T2 target images R1 reference image S scale

Claims

1. A microscope that forms an optical image of a sample containing sperm, an imaging device that acquires a digital image of the sample, a processing device that performs object detection on the digital image, detects candidate sperm for selection in the sample based on the object detection, generates a target image including target sperm selected from the detected candidate sperm for selection, and an optical device that displays the target sperm in the target image on an image plane where the optical image is formed, at a size larger than that of the target sperm in the optical image. A microscope system characterized by comprising:

14. The microscope system according to claim 1, characterized in that 2. The microscope system according to claim 1, wherein the processing device tracks the candidate sperm for selection detected by the object detection. A microscope system characterized by comprising:

22. The microscope system according to claim 1 or 2, characterized in that 3. The microscope system according to claim 1 or 2, wherein the processing device determines the target sperm from among the candidate sperm for selection detected by the object detection. A microscope system characterized by comprising:

30. The microscope system according to claim 3, characterized in that 4. The microscope system according to claim 3, wherein the processing device determines the target sperm from among the candidate sperm for selection based on an evaluation of the motility of the candidate sperm for selection. A microscope system characterized by comprising:

38. The microscope system according to claim 3 or 4, characterized in that 5. The microscope system according to claim 3 or 4, wherein the processing device determines the target sperm from among the candidate sperm for selection based on an evaluation of whether the candidate sperm for selection is constrained. A microscope system characterized by comprising:

46. The microscope system according to claim 5, characterized in that 6. The microscope system according to claim 5, wherein In the microscope system according to claim 4 or claim 5, the processing device calculates the evaluation based on image classification using a learned model A microscope system characterized by the above.

7. In the microscope system according to any one of claims 3 to 6, the processing device determines the sperm to be selected from among the candidate sperm for selection based on the positions of the candidate sperm for selection A microscope system characterized by the above.

8. In the microscope system according to any one of claims 1 to 7, the processing device performs the object detection using a learned model A microscope system characterized by the above.

9. In the microscope system according to any one of claims 1 to 8, the processing device generates an auxiliary image related to the target image, and the optical device displays the auxiliary image on the image plane together with the target image A microscope system characterized by the above.

10. In the microscope system according to claim 9, the auxiliary image includes evaluation information regarding the motility of the sperm to be selected A microscope system characterized by the above.

11. In the microscope system according to claim 9 or claim 10, the auxiliary image includes a marker indicating a position or region of interest in the target image A microscope system characterized by the above.

12. In the microscope system according to claim 11, the auxiliary image includes a marker indicating a vacuole of the sperm in the target image A microscope system characterized by the above.

13. In the microscope system according to any one of Claims 9 to 12, the auxiliary image includes a marker indicating the position or region of the sperm to be selected in the optical image A microscope system characterized by this.

14. In the microscope system according to any one of Claims 9 to 13, the auxiliary image includes magnification information indicating the overall magnification of the target image, which is the overall magnification with respect to the size of the sperm to be selected on the object plane A microscope system characterized by this.

15. In the microscope system according to any one of Claims 9 to 14, the auxiliary image includes a scale attached to the target image A microscope system characterized by this.

16. In the microscope system according to any one of Claims 9 to 15, the auxiliary image includes a reference image for comparison with the target image A microscope system characterized by this.

17. In the microscope system according to any one of Claims 1 to 16, the target image is displayed at a position that does not overlap with the sperm to be selected in the optical image A microscope system characterized by this.

18. In the microscope system according to Claim 17, the target image is displayed at a position determined based on the digital image A microscope system characterized by this.

19. In the microscope system according to any one of Claims 1 to 18, The overall magnification of the target image displayed on the image plane, which is the overall magnification with respect to the size of the sperm to be sorted on the object plane, is determined according to the size of the sperm to be sorted on the object plane. A microscope system characterized by this.

20. In the microscope system according to any one of Claims 1 to 18, The overall magnification of the target image displayed on the image plane, which is the overall magnification with respect to the size of the sperm to be sorted on the object plane, is determined according to the overall magnification of the optical image, which is the overall magnification with respect to the size of the sample on the object plane. A microscope system characterized by this.

21. A projection unit attached to a microscope, An imaging unit that acquires a digital image of a sample containing sperm, A treatment unit, that performs object detection on the digital image, detects candidate sperm for sorting in the sample based on the object detection, and a processing unit that generates a target image including the sperm to be sorted selected from the detected candidate sperm for sorting, and a projection unit that displays the sperm to be sorted in the target image on the image plane where the optical image of the sample formed by the microscope is formed, in a size larger than the sperm to be sorted in the optical image. A projection unit characterized by this.

22. Forms an optical image of a sample containing sperm, acquires a digital image of the sample, performs object detection on the digital image, detects candidate sperm for sorting in the sample based on the object detection, generates a target image including the sperm to be sorted selected from the detected candidate sperm for sorting, and displays the sperm to be sorted in the target image on the image plane where the optical image is formed, in a size larger than the sperm to be sorted in the optical image. A sorting support method characterized by the following.

Images