Microscope system, program, and projection image generating method

Abstract

To provide a diagnosis assisting technology that assists pathological diagnosis based on optical images by pathologists.SOLUTION: A microscope system 1 comprises an ocular lens 104, an objective lens 102 that guides light from a sample to the ocular lens 104, an image-forming lens 103 that is located between the ocular lens 104 and the objective lens 102 and that forms an optical image of the sample on the basis of the light from the sample, an image analysis unit 22, a projection image generation unit 23, and a projector 133 that projects a projection image to the image surface of the optical image. The image analysis unit 22 performs at least one analysis process selected from a plurality of analysis processes on the digital image data of the sample, and outputs an analysis result that corresponds to the at least one analysis process. The projection image generation unit 23 generates projection image data on the basis of the analysis result and the at least one analysis process. The projection image expressed by the projection image data represents the analysis result in a display format that corresponds to the at least one analysis process.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The disclosure of this specification relates to a microscope system, a program, and a projection image generating method. [Background technology]

[0002] As one of the technologies that reduce the burden on pathologists in pathological diagnosis, WSI (Whole Slide Imaging) technology has been attracting attention. WSI technology is a technology that creates a WSI (Whole Slide Image), which is a digital image of the entire area of ​​a specimen on a glass slide. By displaying the WSI, which is a digital image, on a monitor and making a diagnosis, pathologists can enjoy various benefits. Specifically, the benefits include that the microscope body does not need to be operated during diagnosis, the display magnification can be easily changed, and multiple pathologists can be involved in the diagnosis at the same time. Such WSI technology is described, for example, in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2001-519944 Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, systems that apply WSI technology (hereafter referred to as WSI systems) require high performance. Specifically, because color and shade information is extremely important in pathological diagnosis, WSI systems are required to have high color reproducibility and a wide dynamic range. For this reason, each device that makes up a WSI system must be expensive and have high performance, and as a result, the number of users who can introduce a WSI system is limited.

[0005] Given this situation, there is a demand for new technology that reduces the burden on pathologists by assisting them in making pathological diagnoses based on optical images (analog images) obtained by optical microscopes.

[0006] An object according to one aspect of the present invention is to provide a diagnostic assistance technology that assists a pathologist in making a pathological diagnosis based on an optical image. [Means for solving the problem]

[0007] A microscope system according to one aspect of the present invention includes an eyepiece lens, an objective lens that guides light from a sample to the eyepiece lens, and an imaging lens that is disposed on an optical path between the eyepiece lens and the objective lens and forms an optical image of the sample based on the light from the sample. and a computer, wherein the computer acquires the digital image data of the sample from the imaging element, and identifies at least one of a plurality of analytical processes prepared in advance based on operation information based on a user's input operation or selection information including identification information added to the sample, the plurality of analytical processes being different processes each targeting a plurality of staining methods and / or biomarkers, performs at least one of the analytical processes identified from the plurality of analytical processes on the digital image data of the sample, outputs an analytical result corresponding to the identified at least one analytical process, generates projection image data based on the analytical result and the identified at least one analytical process, and displays the projection image data. and a projection device which projects the projection image onto an image plane on which the optical image is formed, the projection image being an image which represents the analysis result in a display format determined in accordance with the identified at least one analysis process, the display format including a color of the image and a figure which constitutes the image, and when the at least one analysis process is a process which targets a staining method, the computer determines a color of the projection image to be a color different from a color of the optical image which corresponds to a staining method which is a target of the at least one analysis process, and when the at least one analysis process is a process which targets a biomarker, the computer determines a position on the optical image where the figure which constitutes the projection image is projected to a stained location of the sample which corresponds to a biomarker which is a target of the at least one analysis process. .

[0011] The present invention A program according to one embodiment is a projection image generating program for a microscope having a projection device that projects a projection image onto an image plane on which an optical image is formed, the program instructing a computer to acquire digital image data of a sample from an image sensor, identify at least one of a plurality of analysis processes prepared in advance based on operation information based on a user's input operation or selection information including identification information added to the sample, the plurality of analysis processes being processes with different contents targeting a plurality of staining methods and / or biomarkers, perform at least one of the analysis processes identified from the plurality of analysis processes on the digital image data of the sample, output an analysis result corresponding to the at least one identified analysis process, and represent the projection image based on the analysis result and the at least one identified analysis process. generating projection image data, the projection image data showing the analysis results in a display format determined in accordance with the identified at least one analysis process, the display format including the color of the image and figures constituting the image; executing processing; and generating the projection image data includes: when the at least one analysis process is a process targeting a staining method, determining a color of the projection image to be a color different from a color of an optical image corresponding to a staining method targeted by the at least one analysis process; and when the at least one analysis process is a process targeting a biomarker, determining a stained location of the sample in the optical image onto which a figure constituting the projection image is projected, in accordance with a biomarker targeted by the at least one analysis process. .

[0012] An image projection method according to one aspect of the present invention includes the steps of: A projection image generating method for a microscope having a projection device which projects a projection image onto an image plane on which an optical image is formed, comprising: a computer acquiring digital image data of a sample from an image sensor; identifying at least one of a plurality of analytical processes prepared in advance based on operation information based on a user's input operation or selection information including identification information added to the sample; the plurality of analytical processes being processes with different contents targeted at a plurality of staining methods and / or biomarkers; performing at least one of the identified analytical processes on the digital image data of the sample; outputting an analytical result corresponding to the identified at least one analytical process; and generating a projection image representing the projection image based on the analytical result and the identified at least one analytical process. and generating projection image data, the projection image data showing the analysis results in a display format determined in accordance with the identified at least one analysis process, the display format including a color of an image and a figure constituting the image, and generating the projection image data includes, when the at least one analysis process is a process targeting a staining method, determining a color of the projection image to be a color different from a color of an optical image corresponding to a staining method targeted by the at least one analysis process, and, when the at least one analysis process is a process targeting a biomarker, determining a stained location of the sample in the optical image onto which a figure constituting the projection image is projected, in accordance with a biomarker targeted by the at least one analysis process. . Effect of the Invention

[0014] According to the above aspect, it is possible to assist a pathologist in making a pathological diagnosis based on an optical image. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram showing a configuration of a microscope system 1. FIG. [Diagram 2] FIG. 2 is a diagram showing the configuration of a computer 20. [Diagram 3] 11 is a flowchart of an image projection process performed by the microscope system 1. [Figure 4] FIG. 2 is a diagram illustrating an example of a selection screen 31. [Diagram 5] 1 is an example of an image seen through the eyepiece 104 of the microscope system 1. [Figure 6] 4 is another example of an image seen through the eyepiece 104 of the microscope system 1. [Figure 7] 13 is yet another example of an image seen through the eyepiece 104 of the microscope system 1. [Figure 8] 13 is yet another example of an image seen through the eyepiece 104 of the microscope system 1. [Figure 9] 13 is yet another example of an image seen through the eyepiece 104 of the microscope system 1. [Figure 10] 13 is yet another example of an image seen through the eyepiece 104 of the microscope system 1. [Figure 11] FIG. 1 is a diagram showing a configuration of a neural network. [Figure 12] FIG. 2 is a diagram showing a configuration of a microscope system 2. [Figure 13] 2 is a diagram showing the configuration of a computer 60 included in the microscope system 3. FIG. [Figure 14] 13 is a flowchart of an image projection process performed by the microscope system 3. [Figure 15] 1 is an example of an image seen through an eyepiece 104 of a microscope system 3. [Figure 16] 3 is another example of an image seen through the eyepiece 104 of the microscope system 3. FIG. [Figure 17] 13 is yet another example of an image seen through the eyepiece 104 of the microscope system 3. [Figure 18] 13 is yet another example of an image seen through the eyepiece 104 of the microscope system 3. [Figure 19] 1 is a diagram showing a configuration of a diagnostic support system including a microscope system 4 and an external viewing system 300. [Figure 20] FIG. 5 is a diagram showing the configuration of a microscope 500. [Figure 21] 5 is an example of a change in the image seen through the eyepiece 104 of a microscope system including the microscope 500. [Figure 22] FIG. 6 is a diagram showing the configuration of a microscope 600. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] [First embodiment] Fig. 1 is a diagram showing the configuration of a microscope system 1 according to this embodiment. Fig. 2 is a diagram showing the configuration of a computer 20. The microscope system 1 is a microscope system used by a pathologist for pathological diagnosis, and includes at least an objective lens 102, an imaging lens 103, an eyepiece lens 104, an image analysis unit 22, a projection image generation unit 23, and a projection device 133.

[0017] The microscope system 1 projects a projection image using the projection device 133 onto an image plane on which an optical image of the sample is formed by the objective lens 102 and the imaging lens 103. More specifically, the image analysis unit 22 performs an analysis process on the digital image data of the sample, the projection image generation unit 23 generates projection image data based on the analysis result and the analysis process, and the projection device 133 projects a projection image that displays the analysis result in a display format corresponding to the analysis process onto the image plane. As a result, the pathologist sees an image in which the projection image in a display format corresponding to the analysis process is superimposed on the optical image. Therefore, the microscope system 1 can provide various information that assists in pathological diagnosis to the pathologist who is observing the sample through the eyepiece 104 in an easily visible display format.

[0018] A specific example of the configuration of the microscope system 1 will be described in detail below with reference to Figures 1 and 2. As shown in Figure 1, the microscope system 1 includes a microscope 100, a microscope controller 10, a computer 20, a display device 30, an input device 40, and an identification device 50.

[0019] The microscope 100 is, for example, an upright microscope, and includes a microscope body 110, a lens barrel 120, and an intermediate lens barrel 130. The microscope 100 may be an inverted microscope.

[0020] The microscope body 110 includes a stage 101 on which a sample is placed, objective lenses (objective lens 102, objective lens 102a) that guide light from the sample to an eyepiece lens 104, an epi-illumination optical system, and a transmitted illumination optical system. The stage 101 may be a manual stage or an electric stage. It is desirable that a plurality of objective lenses with different magnifications are attached to the revolver. For example, the objective lens 102 is a 4x objective lens, and the objective lens 102a is a 20x objective lens. It is sufficient that the microscope body 110 includes at least one of an epi-illumination optical system and a transmitted illumination optical system.

[0021] The microscope body 110 further includes a turret 111 for switching between different microscopy methods. For example, a fluorescent cube used in the fluorescence observation method and a half mirror used in the bright field observation method are arranged on the turret 111. In addition, the microscope body 110 may include an optical element used in a specific microscopy method that can be freely inserted into and removed from the optical path. Specifically, the microscope body 110 may include, for example, a DIC prism, a polarizer, an analyzer, and the like used in the differential interference observation method.

[0022] The lens barrel 120 is a monocular or binocular tube equipped with an eyepiece 104. An imaging lens 103 is provided inside the lens barrel 120. The imaging lens 103 is disposed on the optical path between the objective lens 102 and the eyepiece 104. The imaging lens 103 forms an optical image of the sample on an image plane between the eyepiece 104 and the imaging lens 103 based on light from the sample. The imaging lens 103 also forms a projected image (described later) on the image plane based on light from a projection device 133. This causes the projected image to be superimposed on the optical image on the image plane.

[0023] The intermediate lens barrel 130 is provided between the microscope body 110 and the lens barrel 120. The intermediate lens barrel 130 includes an image pickup element 131, an optical deflection element 132, a projection device 133, and an optical deflection element 134.

[0024] The image sensor 131 is an example of a photodetector that detects light from a sample. The image sensor 131 is a two-dimensional image sensor, such as a CCD image sensor, a CMOS image sensor, etc. The image sensor 131 detects light from the sample and generates digital image data of the sample based on the detection result.

[0025] The optical deflection element 132 is an example of a first optical deflection element that deflects light from the sample toward the image sensor 131. The optical deflection element 132 is, for example, a beam splitter such as a half mirror. A variable beam splitter that varies the transmittance and reflectance may be used for the optical deflection element 132. The optical deflection element 132 is disposed on the optical path between the eyepiece lens 104 and the objective lens 102. This makes it possible to obtain a digital image of the sample viewed from the same direction as visual observation by the image sensor 131.

[0026] The projection device 133 is a projection device that projects a projection image, which will be described later, onto an image plane in accordance with an instruction from the computer 20. The projection device 133 is, for example, a projector using a liquid crystal device, a projector using a digital mirror device, a projector using LCOS, or the like.

[0027] The optical deflection element 134 is an example of a second optical deflection element that deflects the light emitted from the projection device 133 toward the image plane. The optical deflection element 134 is, for example, a beam splitter such as a half mirror. A variable beam splitter that varies the transmittance and reflectance may be used for the optical deflection element 134. A dichroic mirror or the like may be used for the optical deflection element 134. The optical deflection element 134 is disposed on the optical path between the image plane and the optical deflection element 132. This makes it possible to prevent the light from the projection device 133 from entering the image sensor 131.

[0028] The microscope controller 10 controls the microscope 100, particularly the microscope main body 110. The microscope controller 10 is connected to the computer 20 and the microscope 100, and controls the microscope 100 in response to commands from the computer 20.

[0029] The display device 30 is, for example, a liquid crystal display, an organic electroluminescence (OLED) display, a CRT (Cathode Ray Tube) display, etc. The input device 40 outputs an operation signal according to an input operation by a user to the computer 20. The input device 40 is, for example, a keyboard, but may also include a mouse, a joystick, a touch panel, etc.

[0030] The identification device 50 is a device that acquires identification information added to a sample. The identification information includes at least information for identifying the sample. The identification information may include information about an analysis method for the sample, a diagnostic protocol, and the like. The identification device 50 is, for example, a barcode reader, an RFID reader, a QR (registered trademark) code reader, or the like.

[0031] The computer 20 controls the entire microscope system 1. The computer 20 is connected to the microscope 100, the microscope controller 10, the display device 30, the input device 40, and the identification device 50. The computer 20 includes a camera control unit 21, an image analysis unit 22, a projection image generation unit 23, an information acquisition unit 24, a projection control unit 25, an image recording unit 26, an image synthesis unit 27, and a display control unit 28 as components mainly related to the control of the projection device 133, as shown in FIG.

[0032] The camera control unit 21 acquires digital image data of the sample by controlling the image sensor 131. The digital image data acquired by the camera control unit 21 is output to the image analysis unit 22, the image recording unit 26, and the image synthesis unit 27.

[0033] The image analysis unit 22 performs at least one analysis process selected from a plurality of analysis processes on the digital image data acquired by the camera control unit 21, and outputs an analysis result corresponding to the at least one analysis process to the projection image generation unit 23. The plurality of analysis processes to be selected may be processes targeting a plurality of different staining methods, such as HE staining and IHC staining. The plurality of analysis processes to be selected may also be processes targeting a plurality of different biomarkers, such as HER2, Ki-67, and ER / PgR. The plurality of analysis processes to be selected may also be processes targeting a plurality of different combinations of staining methods and biomarkers.

[0034] The image analysis unit 22 may select at least one analysis process based on an input operation by a user. More specifically, the image analysis unit 22 may select at least one analysis process based on operation information of the user acquired by the information acquisition unit 24. The image analysis unit 22 may also select at least one analysis process based on identification information acquired by the identification device 50. More specifically, the identification information acquired by the identification device 50 may be acquired from the information acquisition unit 24, and at least one analysis process may be selected based on an analysis method included in the identification information. Both the operation information and the identification information include information for selecting an analysis process. For this reason, hereinafter, these pieces of information are collectively referred to as selection information.

[0035] The content of the analysis process performed by the image analysis unit 22 is not particularly limited. For example, the image analysis unit 22 may classify one or more structures appearing in a digital image represented by digital image data into one or more classes, and generate an analysis result including position information that specifies the position of a structure classified into at least one of the one or more classes. More specifically, the image analysis unit 22 may classify cells appearing in a digital image according to staining intensity, and generate an analysis result including class information into which the cells are classified and position information that specifies the outline of the cell or the outline of the nucleus of the cell. In addition to the class information and position information, the image analysis unit 22 may generate an analysis result including statistical information of the structures classified into at least one class, such as the number of cells in each class and the ratio of cells in each class to the whole. It is preferable that the structures classified into at least one class are objects that serve as the basis for a pathologist's pathological diagnosis.

[0036] The projection image generating unit 23 generates projection image data based on the analysis result output from the image analyzing unit 22 and at least one analysis process specified by the selection information acquired from the information acquiring unit 24. The projection image represented by the projection image data is an image that represents the analysis result in a display format corresponding to at least one analysis process. The projection image generated by the projection image generating unit 23 is output to the projection control unit 25, the image recording unit 26, and the image synthesis unit 27.

[0037] The display format includes at least the color of the image. Therefore, the projection image generating unit 23 determines the color of the projection image in response to at least one analysis process. Furthermore, the display format may include the format of the figure (e.g., line) constituting the image in addition to the color of the image. Therefore, the projection image generating unit 23 may determine the format of the figure constituting the projection image in response to at least one analysis process. The format of the figure includes whether or not to fill the figure, the type of the figure, and the like. For example, if the figure is a line, the line type, the line thickness, and the like are included in the line format. Furthermore, the display format may include the position of the image in addition to the color of the image. Therefore, the projection image generating unit 23 may determine the positional relationship between the projection image and the optical image on the image plane in response to at least one analysis process, and may determine whether or not to project at least a part of the projection image outside the area of ​​the optical image.

[0038] More specifically, the projection image generating unit 23 generates the projection image data so that the color of the projection image is different from the color of the optical image. Since the color of the optical image differs depending on the staining method, the projection image generating unit 23 may change the color of the projection image depending on the staining method targeted by the selected analysis process. For example, since the optical image is blue-purple in the HE staining method, it is desirable to change the color of the projection image to a color different from blue-purple.

[0039] In addition, since the stained locations in the cells vary depending on the biomarker, the projection image generating unit 23 may change the format of the figures constituting the projection image depending on the biomarker targeted by the selected analysis process. For example, when analyzing the overexpression of HER2 protein, the cell membrane is stained, so the projection image may be composed of a hollow figure that traces the outline of the cell. In addition, when analyzing the expression of ER / PgR, the nucleus of the cell is stained, so the projection image may be composed of a figure that fills in the nucleus of the cell.

[0040] Furthermore, the HE staining method is often used for observing the morphology of cells. When observing the morphology of cells in detail, it is desirable that the projection image does not interfere with the observation of the optical image. For this reason, the projection image generating unit 23 may change the position of the projection image according to the staining method targeted by the selected analysis process. For example, in the case of the HE staining method, when the projection image includes supplementary character information, the position of the projection image may be changed so that the overlap between the character information and the optical image is reduced.

[0041] The information acquiring unit 24 acquires information from a device external to the computer 20. Specifically, the information acquiring unit 24 acquires user operation information based on an operation signal from the input device 40. The information acquiring unit 24 also acquires identification information from the identification device 50.

[0042] The projection control unit 25 controls the projection device 133 to control the projection of the projection image onto the image plane. The projection control unit 25 may control the projection device 133 according to the settings of the microscope system 1. Specifically, the projection control unit 25 may determine whether or not to project the projection image onto the image plane according to the settings of the microscope system 1, and may control the projection device 133 so that the projection device 133 projects the projection image onto the image plane when the microscope system 1 is in a predetermined setting. In other words, the microscope system 1 can change whether or not to project the projection image onto the image plane depending on the settings.

[0043] The image recording unit 26 records the digital image data and the projection image data. Specifically, the image recording unit 26 records the projection image data in an area different from the digital image data in association with the digital image data. This makes it possible to individually read out the digital image data and the projection image data, which are associated with each other, as necessary. Furthermore, the image recording unit 26 may acquire identification information added to the sample via the identification device 50 and the information acquisition unit 24, and record the acquired identification information in association with the digital image data. Furthermore, the image recording unit 26 may record the digital image data and the projection image data when it detects an input of a recording instruction by the user.

[0044] The image synthesis unit 27 generates image data of a synthetic image in which the digital image and the projection image are synthesized based on the digital image data and the projection image data, and outputs the generated image data to the display control unit .

[0045] The display control unit 28 displays the composite image on the display device 30 based on the composite image data output from the image synthesis unit 27. Note that the display control unit 28 may display the digital image solely on the display device 30 based on the digital image data.

[0046] The computer 20 may be a general-purpose device or a dedicated device. The computer 20 is not particularly limited to this configuration, but may have a physical configuration as shown in Fig. 2, for example. Specifically, the computer 20 may include a processor 20a, a memory 20b, an auxiliary storage device 20c, an input / output interface 20d, a medium drive device 20e, and a communication control device 20f, which may be connected to each other by a bus 20g.

[0047] The processor 20a is, for example, any processing circuit including a CPU (Central Processing Unit). The processor 20a may execute a program stored in the memory 20b, the auxiliary storage device 20c, or the storage medium 20h to perform the programmed processing, thereby realizing the components (camera control unit 21, image analysis unit 22, projection image generation unit 23, etc.) related to the control of the above-mentioned projection device 133. The processor 20a may also be configured using a dedicated processor such as an ASIC or FPGA.

[0048] The memory 20b is a working memory of the processor 20a. The memory 20b is, for example, any semiconductor memory such as a RAM (Random Access Memory). The auxiliary storage device 20c is a non-volatile memory such as an EPROM (Erasable Programmable ROM) or a hard disk drive. The input / output interface 20d exchanges information with external devices (the microscope 100, the microscope controller 10, the display device 30, the input device 40, and the identification device 50).

[0049] The medium drive device 20e can output data stored in the memory 20b and the auxiliary storage device 20c to the storage medium 20h, and can also read programs, data, and the like from the storage medium 20h. The storage medium 20h is any portable storage medium. Examples of the storage medium 20h include an SD card, a USB (Universal Serial Bus) flash memory, a CD (Compact Disc), and a DVD (Digital Versatile Disc).

[0050] The communication control device 20f inputs and outputs information to and from the network. For example, a NIC (Network Interface Card), a wireless LAN (Local Area Network) card, etc. can be adopted as the communication control device 20f. The bus 20g connects the processor 20a, the memory 20b, the auxiliary storage device 20c, etc. so that they can transmit and receive data to and from each other.

[0051] The microscope system 1 configured as above performs the image projection process shown in Fig. 3. Fig. 3 is a flowchart of the image projection process performed by the microscope system 1. Fig. 4 is a diagram showing an example of a selection screen 31. Hereinafter, the image projection method of the microscope system 1 will be described with reference to Figs. 3 and 4.

[0052] First, the microscope system 1 projects an optical image of a sample onto an image plane (step S1). Here, the imaging lens 103 focuses light from the sample captured by the objective lens 102 onto the image plane to form an optical image of the sample.

[0053] Furthermore, the microscope system 1 acquires digital image data of the sample (step S2). Here, the optical deflection element 132 deflects a part of the light from the sample captured by the objective lens 102 toward the image sensor 131. The image sensor 131 captures an image of the sample based on the light deflected by the optical deflection element 132, thereby generating digital image data.

[0054] Thereafter, the microscope system 1 specifies the selected analysis process from a plurality of analysis processes prepared in advance (step S3). Here, the user selects, for example, each menu (menu 32, menu 33, menu 34, menu 35) on the selection screen 31 shown in Fig. 4 and presses the button 36. The image analysis unit 22 specifies the selected analysis process based on this input operation by the user.

[0055] When the analysis process is specified, the microscope system 1 executes the specified analysis process (step S4). Here, the image analysis unit 22 performs the analysis process selected in step S3 on the digital image data acquired in step S2 to obtain an analysis result.

[0056] The microscope system 1 generates projection image data based on the analysis result acquired in step S4 and the analysis process specified in step S3 (step S5). Here, the projection image generating unit 23 generates projection image data representing a projection image that displays the analysis result acquired in step S4 in a display format according to the analysis process specified in step S3.

[0057] Finally, the microscope system 1 projects the projection image onto the image plane (step S6). The projection control unit 25 controls the projection device 133 based on the projection image data, and the projection device 133 projects the projection image onto the image plane. As a result, the projection image is superimposed on the optical image of the sample.

[0058] In the microscope system 1, the image analysis results by the computer are displayed on the optical image. This allows the pathologist to obtain various pieces of information that assist the diagnosis without taking his / her eyes off the eyepiece during the pathological diagnosis based on the optical image of the sample. In addition, the image analysis unit 22 performs an analysis process selected from a plurality of analysis processes, so that the microscope system 1 can handle various types of diagnoses. Furthermore, the projection image projected by the projection device 133 has a display format according to the analysis process. Therefore, the microscope system 1 can provide the pathologist with various pieces of information that assist the pathological diagnosis in an easily visible display format. Therefore, the microscope system 1 can assist the pathological diagnosis based on the optical image, and reduce the workload of the pathologist.

[0059] Furthermore, the microscope system 1 assists in pathological diagnosis by displaying additional information (projected image) on the optical image. Therefore, unlike a WSI system that performs pathological diagnosis based on a digital image, no expensive equipment is required. Therefore, the microscope system 1 can reduce the burden on pathologists while avoiding a significant increase in equipment costs. Also, in order to perform pathological diagnosis with a WSI system, it is necessary to create a WSI (Whole Slide Image) in advance, but with the microscope system 1, no advance preparation is required and diagnostic work can be started immediately.

[0060] 5 to 10 are diagrams illustrating examples of images seen through the eyepiece 104 of the microscope system 1. Hereinafter, the state of observation using the microscope system 1 that executes the image projection process shown in FIG. 3 will be specifically described with reference to FIGS. 5 to 10.

[0061] First, with reference to FIG. 5 to FIG. 7, a case where "Cell Diagnosis", "Breast Cancer", "IHC Staining", and "ER / PgR" are selected on the selection screen 31 shown in FIG. 4 will be described.

[0062] When observation using the microscope system 1 is started and the pathologist looks through the eyepiece 104, the pathologist can observe an image V1 shown in FIG. 5. The image V1 is an optical image M1 corresponding to the actual field of view formed on the image plane. The image V1 shows stained nuclei of cancer cells. A dark area R2 exists around the area R1 onto which the image V1 is projected. The area R2 is an area on the image plane observable through the eyepiece 104 through which the light from the objective lens 102 does not pass. At this time, an image corresponding to the image V1 may be displayed on the display device 30 based on the digital image data generated by the image sensor 131.

[0063] The computer 20 then analyzes the digital image data. The analysis identifies the nuclei of the cells and classifies them according to their staining intensity. For example, unstained nuclei are classified as class 0, which indicates negative; weakly stained nuclei are classified as class 1+, which indicates weak positivity; moderately stained nuclei are classified as class 2+, which indicates moderate positivity; and strongly stained nuclei are classified as class 3+, which indicates strong positivity.

[0064] The computer 20 generates projection image data based on the analysis results, and the projection device 133 projects the projection image represented by the projection image data onto an image plane. The projection image represented by the projection image data includes a position image consisting of a figure representing the position of the nucleus of the classified cells, and the figure has a different color for each class.

[0065] When the projection device 133 projects the projection image, the pathologist can observe the image V2 shown in FIG. 6. The image V2 is an image in which a projection image including the position image P1 is superimposed on the optical image M1. At this time, an image corresponding to the image V2 may be displayed on the display device 30. In the image V2 shown in FIG. 6, it is easier to distinguish the staining state of each cell than in the image V1 (optical image M1) shown in FIG. 5. This makes it easier to calculate a score using a predetermined scoring method such as J-score used in pathological diagnosis. Therefore, the microscope system 1 assists the pathologist in the task of determining whether a result is positive or negative, thereby reducing the burden on the pathologist.

[0066] Furthermore, the projection image represented by the projection image data may include, in addition to the position image P1, a statistical image T1 consisting of statistical information of the classified cells. In this case, the pathologist can observe the image V3 shown in FIG. 7. The image V3 is an image in which a projection image including the position image P1 and the statistical image T1 is superimposed on the optical image M1. At this time, an image corresponding to the image V3 may be displayed on the display device 30. In the image V3 shown in FIG. 7, the statistical image T1 makes it even easier to calculate the score. Therefore, the microscope system 1 better assists the pathologist in the task of determining whether a result is positive or negative, thereby further reducing the burden on the pathologist.

[0067] In addition, the thresholds for each scoring judgment of 0, 1+, 2+, and 3+, which indicate the degree of staining intensity, may differ depending on the guidelines of individual pathologists, hospital facilities, or diagnostic standards of each country. In light of this, when comparing the optical image M1, the position image P1, and the statistical image T1, and a doubt arises about the statistical image T1 based on the analysis result, the observer may change the threshold used in the analysis process using the input device. A new analysis process is performed based on the changed threshold, and the user can check the new statistical image T1' in which the result is reflected in real time, which better supports the task of setting an appropriate threshold. This further reduces the burden on the pathologist.

[0068] Next, a case where "Cell Diagnosis", "Breast Cancer", "IHC Staining", and "HER2" are selected on the selection screen 31 shown in FIG. 4 will be described with reference to FIG. 8 to FIG.

[0069] When starting observation using the microscope system 1 and looking through the eyepiece 104, the pathologist can observe image V4 shown in Fig. 8. Image V4 is an optical image M2 corresponding to the actual field of view formed on the image plane. Image V4 shows stained cell membranes of cancer cells. At this time, an image corresponding to image V4 may be displayed on the display device 30 based on the digital image data generated by the image sensor 131.

[0070] The computer 20 then analyzes the digital image data. The analysis identifies cell membranes and classifies them according to their staining intensity and staining pattern. For example, cells with completely unstained membranes are classified as Class 0, which indicates negative. Cells with partially or weakly stained membranes are classified as Class 1+, which indicates weak positivity. Cells with moderately stained membranes and entirely stained nuclei are classified as Class 2+, which indicates moderate positivity. Cells with strongly stained membranes and entirely stained nuclei are classified as Class 3+, which indicates strong positivity.

[0071] The computer 20 generates projection image data based on the analysis results, and the projection device 133 projects the projection image represented by the projection image data onto the image plane. The projection image represented by the projection image data includes a position image P2 consisting of a figure representing the position of the classified cells, and the figure has a different format (line type) for each class.

[0072] When the projection device 133 projects the projection image, the pathologist can observe the image V5 shown in FIG. 9. The image V5 is an image in which a projection image including the position image P2 is superimposed on the optical image M2. At this time, an image corresponding to the image V5 may be displayed on the display device 30. In the image V5 shown in FIG. 9, it is easier to distinguish the stained state of the cell membrane than in the image V4 (optical image M2) shown in FIG. 8. This makes it easier to calculate the score by a predetermined scoring method used in pathological diagnosis. Therefore, the microscope system 1 assists the pathologist in the task of determining whether the result is positive or negative, thereby reducing the burden on the pathologist.

[0073] Moreover, the projection image represented by the projection image data may include, in addition to the position image P2, a statistical image T2 consisting of statistical information of the classified cells. In this case, the pathologist can observe the image V6 shown in FIG. 10. The image V6 is an image in which a projection image including the position image P2 and the statistical image T2 is superimposed on the optical image M2. In this example, the statistical image T2 is projected outside the region R1 through which the light beam from the objective lens 102 passes. At this time, an image corresponding to the image V6 may be displayed on the display device 30. In the image V6 shown in FIG. 10, the statistical image T2 makes it even easier to calculate the score. Therefore, the microscope system 1 better assists the pathologist in the positive / negative judgment work, and the burden on the pathologist can be further reduced.

[0074] In the examples of Figures 5 to 10, projection images are shown that have different display formats due to differences in the format of the shapes that make up the image (whether or not they are filled in, the type of line), but the color of the projection images may also be changed depending on the analysis processing.

[0075] Furthermore, the image analysis section 22 of the microscope system 1 may perform a plurality of analysis processes using a plurality of predetermined algorithms, or may perform a plurality of analysis processes using a plurality of trained neural networks.

[0076] The parameters of each of the multiple trained neural networks may be generated by training each neural network in a device different from the microscope system 1, and the computer 20 may download the generated parameters and apply them to the image analysis unit 22. Furthermore, the computer 20 may add analysis processes selectable by the image analysis unit 22 as needed by downloading parameters of a new neural network.

[0077] FIG. 11 is a diagram showing the configuration of the neural network NN. The neural network NN has an input layer, multiple intermediate layers, and an output layer. Input data D1 is input to the input layer, and output data D2 output from the output layer is compared with correct answer data D3. Then, the parameters of the neural network NN are updated by learning using the backpropagation method. The set of input data D1 and correct answer data D3 is training data for supervised learning.

[0078] [Second embodiment] 12 is a diagram showing the configuration of a microscope system 2 according to this embodiment. The microscope system 2 differs from the microscope system 1 in that it includes a microscope 200 instead of the microscope 100. The microscope 200 includes a projection unit 140 between a microscope body 110 and a lens barrel 120.

[0079] The projection unit 140 is a projection unit for a microscope that includes an objective lens 102, an imaging lens 103, and an eyepiece lens 104, and includes an intermediate lens barrel 130. That is, the projection unit 140 includes an image sensor 131, which is an example of an image capturing device that acquires digital image data of a sample based on light from the sample, and a projection device 133 that projects a projection image onto an image plane on which an optical image is formed.

[0080] The projection unit 140 further includes an image analysis unit 142 and a projection image generation unit 143. The projection unit 140 may also include a camera control unit 141, an information acquisition unit 144, and a projection control unit 145.

[0081] The camera control unit 141, the image analysis unit 142, the projection image generation unit 143, and the projection control unit 145 are similar to the camera control unit 21, the image analysis unit 22, the projection image generation unit 23, and the projection control unit 25, respectively. Therefore, detailed description will be omitted.

[0082] The information acquiring unit 144 acquires user operation information based on an operation signal from the input device 40 acquired via the computer 20. The information acquiring unit 144 also acquires identification information from the identification device 50 via the computer 20.

[0083] In this embodiment, it is possible to obtain the same effect as the microscope system 1 by simply attaching the projection unit 140 to an existing microscope. Therefore, the projection unit 140 and the microscope system 2 can easily expand an existing microscope system to assist a pathologist in making a pathological diagnosis based on an optical image.

[0084] [Third embodiment] 13 is a diagram showing the configuration of a computer 60 included in a microscope system 3 according to this embodiment. Note that the microscope system 3 is similar to the microscope system 1 except that it includes the computer 60 instead of the computer 20.

[0085] The computer 60 controls the entire microscope system 3. The computer 60 is similar to the computer 20 in that it is connected to the microscope 100, the microscope controller 10, the display device 30, the input device 40, and the identification device 50.

[0086] The computer 60 includes components mainly related to the control of the projection device 133, such as a camera control unit 61, a projection image generation unit 63, an information acquisition unit 64, a projection control unit 65, an image recording unit 66, an image synthesis unit 67, and a display control unit 68.

[0087] In addition, the camera control unit 61, the information acquisition unit 64, the projection control unit 65, the image recording unit 66, the image synthesis unit 67, and the display control unit 68 correspond to the camera control unit 21, the information acquisition unit 24, the projection control unit 25, the image recording unit 26, the image synthesis unit 27, and the display control unit 28 contained in the computer 20, respectively.

[0088] The computer 60 is significantly different from the computer 20 in that it does not include a configuration corresponding to the image analysis unit 22. In addition, the computer 60 is also different from the computer 20 in that the projection image generation unit 63 performs processing different from that of the projection image generation unit 23.

[0089] The projection image generating unit 63 generates projection image data based on a diagnostic protocol selected from a plurality of diagnostic protocols. The diagnostic protocol is a series of agreements including procedures from the start to the end of a diagnosis, criteria for judgment, etc. The projection image represented by the projection image data is an image according to the selected diagnostic protocol. The projection image generated by the projection image generating unit 63 is output to the projection control unit 65, the image recording unit 66, and the image synthesis unit 67.

[0090] The method of selecting a diagnostic protocol in the projection image generating unit 63 is the same as the method of selecting an analysis process in the image analyzing unit 22 of the microscope system 1. That is, the projection image generating unit 63 may select a diagnostic protocol based on an input operation by a user, or may select a diagnostic protocol based on identification information acquired by the identification device 50.

[0091] The projection image preferably includes a guide to the diagnostic procedure of the selected diagnostic protocol, and the projection image generating unit 23 may determine the color of the guide to the diagnostic procedure according to the selected diagnostic protocol. This allows the pathologist to refer to the diagnostic procedure without taking his / her eyes off the eyepiece 104, and therefore allows the pathologist to efficiently proceed with the diagnosis without making any mistakes in the diagnostic procedure.

[0092] The projection image may also include a reference image showing the criteria for the selected diagnostic protocol, which allows the pathologist to simultaneously check the optical image and the reference image, thereby reducing the time required for diagnosis and improving diagnostic accuracy.

[0093] The projection image generation unit 63 may determine the positional relationship between the projection image and the optical image on the image plane depending on the selected diagnostic protocol, and may determine whether or not to project at least a portion of the projection image outside the area of ​​the optical image.

[0094] Furthermore, the projection image generating unit 63 may generate projection image data so that the projection image includes a warning display when the settings of the microscope system 3 do not satisfy the requirements of the selected diagnostic protocol. This allows the pathologist to make various diagnostic decisions in a correct environment, which is expected to improve diagnostic accuracy.

[0095] The projection control unit 65 may control the projection device 133 according to the settings of the microscope system 3. Specifically, the projection control unit 65 may determine whether or not to project the projection image onto the image plane according to the settings of the microscope system 3, and may control the projection device 133 so that the projection device 133 projects the projection image onto the image plane when the microscope system 3 is in a predetermined setting. In other words, the microscope system 3 can change whether or not to project the projection image onto the image plane depending on the settings.

[0096] In addition, some diagnostic protocols include a procedure for measuring the area and position of structures such as cancer cells, and the distance between structures. In this case, the projection image generating unit 63 may generate projection image data including the measurement results measured using the length measuring function of the microscope, and the projection control unit 65 may project the projection image including the measurement results onto the image plane.

[0097] The microscope system 3 configured as above performs the image projection process shown in Fig. 14. Fig. 14 is a flowchart of the image projection process performed by the microscope system 3. Hereinafter, the image projection method of the microscope system 3 will be described with reference to Fig. 14.

[0098] First, the microscope system 3 projects an optical image of a sample onto an image plane (step S11). This process is similar to step S1 shown in FIG.

[0099] Next, the microscope system 3 identifies a diagnostic protocol selected from a plurality of diagnostic protocols prepared in advance (step S12). Here, the user selects a diagnostic protocol on a selection screen, for example, and the projection image generating unit 63 identifies the selected diagnostic protocol based on the input operation by the user.

[0100] When the diagnostic protocol is specified, the microscope system 3 generates projection image data based on the specified diagnostic protocol (step S13). Here, the projection image generating unit 63 generates projection image data representing a projection image according to the diagnostic protocol specified in step S12.

[0101] Finally, the microscope system 3 projects the projection image onto the image plane (step S14). The projection control unit 65 controls the projection device 133 based on the projection image data, and the projection device 133 projects the projection image onto the image plane. As a result, the projection image is superimposed on the optical image of the sample.

[0102] In the microscope system 3, a projection image according to a diagnostic protocol is displayed on an optical image. This allows a pathologist to obtain various information that assists diagnosis, such as diagnostic procedures and criteria, without taking his / her eyes off the eyepiece during pathological diagnosis based on an optical image of a sample. In addition, the projection image generating unit 63 generates projection image data according to a diagnostic protocol selected from a plurality of diagnostic protocols, so that the microscope system 3 can handle various types of diagnostic protocols. Therefore, the microscope system 3 can assist in pathological diagnosis based on an optical image, and can reduce the workload of the pathologist.

[0103] In addition, like microscope system 1, microscope system 3 can reduce the burden on pathologists while avoiding a significant increase in equipment costs, and unlike WSI systems, it does not require any advance preparation and allows diagnostic work to begin immediately.

[0104] 15 to 18 are diagrams illustrating images seen through the eyepiece 104 of the microscope system 3. Hereinafter, the observation using the microscope system 3 that executes the image projection process shown in Fig. 14 will be specifically described with reference to Fig. 15 to 18. Note that here, a diagnostic protocol for determining overexpression of HER2 protein by IHC staining (hereinafter referred to as IHC-HER2 diagnostic protocol) will be described as an example.

[0105] When observation using the microscope system 3 is started and the pathologist looks through the eyepiece 104, the pathologist can observe an image V7 shown in FIG. 15. Image V7 is an image in which a projection image including a guide image G1 is superimposed on an optical image M2. The optical image M2 is an image acquired using, for example, a 4x objective lens 102, and shows a stained cell membrane of a cancer cell. The guide image G1 is an image that guides the diagnostic procedure of the IHC-HER2 diagnostic protocol. The IHC-HER2 diagnostic protocol specifies a procedure for observing a positive HER2 protein stained image, the intensity of the staining, and the proportion of positive cells using a 4x objective lens, and the guide image G1 guides this procedure.

[0106] In the microscope system 3, an image V8 may be projected on the image plane instead of the image V7. The image V8 is an image in which a projected image including a guide image G1 and a contrast image C1 is superimposed on the optical image M2. The contrast image C1 includes a plurality of reference images showing the judgment criteria in the IHC-HER2 diagnostic protocol. More specifically, the contrast image C1 includes four reference images showing examples of images to be judged as score 0, score 1+, score 2+, and score 3+. A pathologist can refer to the contrast image C1 when judging the proportion of positive cells.

[0107] Then, image V9 is projected onto the image plane. Image V9 is an image in which a projection image including guide image G2 is superimposed on optical image M2. Guide image G2 is an image that guides the diagnostic procedure of the IHC-HER2 diagnostic protocol. In the IHC-HER2 diagnostic protocol, it is prescribed that observation using a 4x objective lens should be followed by observation using a 10x objective lens, and guide image G2 guides this procedure.

[0108] If no switching from the 4x objective lens 102 to the 10x objective lens 102a is detected after the image V9 is projected, the image V10 is projected on the image plane. The image V10 is an image in which a projection image including a guide image G3 is superimposed on the optical image M2. The guide image G3 is a warning display that warns the pathologist that the diagnosis is not being made according to the diagnostic protocol. The warning display allows the pathologist to recognize an error in the procedure, and therefore the microscope system 3 makes it possible to prevent the diagnosis from being continued according to the incorrect procedure.

[0109] [Fourth embodiment] 19 is a diagram showing the configuration of a diagnostic support system including a microscope system 4 according to this embodiment and an external viewing system 300. The microscope system 4 differs from the microscope system 1 in that it includes a computer 70 instead of the computer 20.

[0110] The microscope system 4 is connected to one or more external viewing systems 300 via the Internet 400. The external viewing system 300 is a system including at least a computer 310 including a communication control unit 311, an input device 320, and a display device 330.

[0111] The Internet 400 is an example of a communication network. The microscope system 3 and the external viewing system 300 may be connected via, for example, a VPN (Virtual Private Network) or a dedicated line.

[0112] The computer 70 differs from the computer 20 in that the computer 70 includes a communication control unit 29. The communication control unit 29 exchanges data with an external browsing system 300.

[0113] The communication control unit 29 transmits, for example, image data to the external viewing system 300. The image data transmitted by the communication control unit 29 may be, for example, composite image data generated by the image synthesis unit 27. The digital image data and the projection image data may be transmitted separately. Also, only the digital image data may be transmitted. In the external viewing system 300, the computer 310 that receives the image data displays an image on the display device 330 based on the image data. The computer 310 may generate composite image data based on the digital image data and the projection image data, for example, and may display the composite image on the display device 330 based on the composite image data.

[0114] The communication control unit 29 receives, for example, operation information input by a user of the external viewing system 300. The image analysis unit 22 may select an analysis process based on the operation information received by the communication control unit 29. The microscope system 4 may project a projection image based on an input operation by the user of the external viewing system 300 onto an image plane using the projection device 133.

[0115] The microscope system 4 can communicate with an external viewing system 300 connected via a network. This allows users in different locations to communicate with each other while performing pathological diagnosis.

[0116] The above-described embodiments are illustrative examples for facilitating understanding of the invention, and the present invention is not limited to these embodiments. Various modifications and changes can be made to the microscope system, projection unit, and image projection method without departing from the scope of the claims.

[0117] 12 illustrates a projection unit 140 including an image analysis unit 142 that performs an analysis process selected from a plurality of analysis processes. However, the projection unit may include a projection image generation unit that generates projection image data based on a diagnostic protocol selected from a plurality of diagnostic protocols.

[0118] 19 illustrates a microscope system 4 in which a communication function with an external viewing system 300 is added to the microscope system 1. However, a new microscope system may be configured by adding a communication function with the external viewing system 300 to the microscope system 2.

[0119] 1 shows an example in which the projection image generating unit 23 generates the analysis result of the selected analysis process and the projection image data based on the analysis process. However, the projection image generating unit 23 may generate the first projection image data based on the analysis result of the selected analysis process and the analysis process, and generate the second projection image data based on the selected diagnostic protocol. Then, the projection device 133 may project the first projection image represented by the first projection image data and the second projection image represented by the second projection image data onto the optical image.

[0120] 5 to 10 show an example of pathological diagnosis of breast cancer, but the microscope system 1 can also be used for pathological diagnosis of other cancers such as cervical cancer. For example, when applied to the pathological diagnosis of cervical cancer, classification may be performed based on the Bethesda system, and classification results such as NILM, LSIL, HSIL, SCC, and ASC may be displayed as projection images. In genomic diagnosis, the number of tumor cells, the total number of cells, and their ratios may be displayed as projection images.

[0121] Although an example has been given in which the display format of the projection image is changed in response to the analysis process, the microscope system 1 may change the display format in response to a change in the settings of the illumination optical system or the observation optical system.

[0122] Although the microscope system 1 has been described as an example in which the image analysis unit 22 is provided in the computer 20 of the microscope system 1, the image analysis unit 22 may be realized by the computer 20 in the microscope system 1 and a remote module outside the microscope system 1. The remote module is, for example, a server placed on a cloud. The computer 20 may respond to new analysis processing by updating the analysis program by downloading the latest program from the remote module. In addition, the computer 20 may respond to a new diagnostic protocol by downloading the latest program from the remote module.

[0123] Further, the microscope included in the microscope system 1 may be, for example, a microscope 500 shown in FIG. 20. In the above-mentioned embodiment, the configuration in which the intermediate lens barrel 130 is provided with the image sensor 131 is exemplified, but the image sensor 151 that acquires digital image data used for image analysis may be provided in the digital camera 150 attached to the lens barrel 120a, which is a trinocular lens barrel, as shown in FIG. 12. In this case, however, light emitted from the projection device 133 included in the intermediate lens barrel 130a is incident on the image sensor 151. For this reason, the digital camera 150 may be controlled so that the light emission period of the projection device 133 and the exposure period of the image sensor 151 do not overlap. This makes it possible to prevent the projection image from being captured in the digital image.

[0124] Furthermore, the microscope system including the microscope 500 may be used to diagnose moving objects, such as sperm selection performed in artificial insemination. In sperm selection, the quality of sperm is judged based on information that can be determined based on still images, such as the shape of the sperm (hereinafter referred to as shape information), and information that can be determined based on video or multiple still images, such as the straightness and speed of the sperm's movement (hereinafter referred to as movement information). For this reason, in an analysis process to assist the sperm selection process, sperm movement information may be output to be reflected in a projected image, and furthermore, sperm candidates to be selected by the user may be identified using the sperm shape information and sperm movement information.

[0125] More specifically, when a user first starts observing the sperm to be selected, the image analysis unit 22 analyzes multiple digital image data acquired at different times to calculate the trajectory of the sperm's movement. After that, the projection image generation unit 23 generates projection image data based on the analysis results, and the projection device 133 projects a projection image including a trajectory display MT onto the image plane. The trajectory display MT is a display showing the trajectory of the movement of each sperm up to its current position. The trajectory display MT may also show the trajectory of the movement from a time going back a predetermined time (e.g., 3 seconds) to the current time.

[0126] As a result, the user first observes the image V11 including the optical image M3 shown in Fig. 21, and when the analysis process by the image analysis unit 22 is completed, the user can observe the image V12 including the optical image M4 and the auxiliary image A1 shown in Fig. 22. Since the auxiliary image A1 includes the trajectory display MT of each sperm, the user who observes the image V12 can grasp at a glance the characteristics of the sperm's movement as well as the shape characteristics of the sperm. This makes it easier for the user to judge whether the sperm are good or bad, and reduces the burden of the selection work.

[0127] Furthermore, the image analysis unit 22 identifies candidates for sperm to be selected by the user by analyzing the digital image data of the superimposed image in which the auxiliary image A1 is superimposed on the optical image M4. After that, the projection image generation unit 23 generates projection image data based on the analysis result including the position information of the identified sperm, and the projection device 133 projects the projection image including the attention region display ROI on the image plane. The attention region display ROI is a display for encouraging the user to pay attention to the sperm identified by the image analysis unit 22, and is, for example, a rectangular or circular figure surrounding the sperm. In addition, the color of the figure may be different to indicate the likelihood that the sperm is of good quality.

[0128] This allows the user to observe image V13, which includes optical image M4 and auxiliary image A2 shown in FIG. 23. Auxiliary image A2 includes region of interest display ROI in addition to trajectory display MT. Therefore, similar to image V12, the user can grasp at a glance not only the shape characteristics of the sperm but also the movement characteristics of the sperm. Furthermore, the user can select high-quality sperm early by preferentially observing sperm identified by region of interest display ROI. This further reduces the burden of the selection work.

[0129] Furthermore, the microscope included in the microscope system 1 may be, for example, a microscope 600 shown in Fig. 22. The microscope 600 includes an intermediate lens barrel 130b including a projection device 135 using a transmissive liquid crystal device, instead of the intermediate lens barrel 130. In the above-described embodiment, a configuration has been exemplified in which the projection image is projected onto the image plane by deflecting the light emitted from the projection device 133 with the optical deflection element 134 arranged on the optical path between the objective lens 102 and the eyepiece lens 104, but as shown in Fig. 22, the projection device 135 may be arranged on the optical path between the objective lens 102 and the eyepiece lens 104.

[0130] In the above-described embodiment, an example is shown in which an image sensor is included as a photodetector, but the photodetector is not limited to an image sensor. For example, the above-described technology may be provided for a scanning microscope, in which case the photodetector may be a photomultiplier tube (PMT) or the like.

[0131] Furthermore, the above-mentioned microscope system may adjust the brightness of at least one of the optical image and the projected image in accordance with the selected analysis process, or may adjust the brightness of at least one of the optical image and the projected image in accordance with the selected diagnostic protocol. The brightness adjustment may be performed by controlling the amount of light from the light source, or by controlling the amount of transmitted light by a variable ND filter or the like.

[0132] In the above-described embodiment, a keyboard, a mouse, a joystick, a touch panel, etc. are exemplified as the input device 40, but the input device 40 may be a device that accepts voice input, such as a microphone. In this case, the computer 20 may have a function for recognizing a voice instruction input from the input device 40. For example, the information acquisition unit 24 included in the computer 20 may convert voice data into operation information using a voice recognition technology and output the operation information to the projection image generation unit 23. [Explanation of symbols]

[0133] 1, 2, 3, 4 Microscope System 10 Microscope Controller 20, 60, 70, 310 Computer 20a processor 20b Memory 20c Auxiliary storage 20d Input / Output Interface 20e Media drive unit 20f Communication control device 20g Bus 20h storage medium 21, 61, 141 Camera control unit 22, 142 Image analysis section 23, 63, 143 Projection image generator 24, 64, 144 Information acquisition section 25, 65, 145 Shadow control section 26, 66 Image recording unit 27, 67 Image synthesis unit 28, 68 Display control unit 29, 311 Communication control section 30, 330 display device 31 Selection Screen 32, 33, 34, 35 Menu 36 Buttons 40, 320 Input device 50 Identification device 100, 200, 500, 600 Microscope Stage 101 102, 102a Objective lens 103 Imaging Lens 104 Eyepiece 110 Microscope body 111 Turret 120, 120a telescope 130, 130a, 130b Intermediate telescope 131, 151 Image sensor 132, 134 Light deflection element 133, 135 Projection device 140 Projection Unit 150 Digital Camera 300 External Viewing System 400 Internet A1, A2 auxiliary images C1 Comparison image D1 Input data D2 Output data D3 Correct answer data G1, G2, G3 guide images M1, M2, M3, M4 optical images MT trajectory display NN Neural Network P1, P2 position images R1, R2 area ROI region of interest display T1, T2 statistical images V1~V13 Images

Claims

1. An eyepiece; an objective lens that guides light from a sample to the eyepiece; an imaging lens disposed on an optical path between the eyepiece lens and the objective lens, the imaging lens forming an optical image of the sample based on light from the sample; an imager for generating digital image data of the sample; A computer, comprising: acquiring digital image data of the sample from the imaging device; identifying at least one of a plurality of analytical processes prepared in advance based on operation information based on an input operation by a user or selection information including identification information added to the sample, the plurality of analytical processes being different processes targeting a plurality of staining methods and / or biomarkers, performing at least one analytical process identified from the plurality of analytical processes on the digital image data of the sample; outputting an analysis result corresponding to the at least one identified analysis process; The computer generates projection image data based on the analysis result and the at least one of the specified analysis processes, and a projection image represented by the projection image data is an image that represents the analysis result in a display format determined in accordance with the at least one of the specified analysis processes; a projection device that projects the projection image onto an image plane on which the optical image is formed; the display format includes a color of the image and a figure constituting the image; The computer includes: When the at least one analysis process is a process targeting a staining method, a color of the projection image is determined to be a color different from a color of an optical image corresponding to the staining method targeted by the at least one analysis process, in accordance with the staining method targeted by the at least one analysis process; When the at least one analysis process is a process targeting a biomarker, a position on the optical image onto which a figure constituting the projection image is projected is determined to be a stained portion of the sample corresponding to the biomarker targeted by the at least one analysis process. A microscope system comprising:

2. In the microscope system according to claim 1, The computer includes: generating the projection image so as to fill in the nucleus when the at least one analysis process is a process targeting a biomarker and a stained portion corresponding to the biomarker targeted by the at least one analysis process is a nucleus of a cell; When the at least one analysis process is a process targeting a biomarker and the stained portion corresponding to the biomarker targeted by the at least one analysis process is a cell membrane, the projection image is generated so as to trace the outline of the cell. A microscope system comprising:

3. 3. The microscope system according to claim 1, The microscope system adjusts brightness of at least one of the optical image and the projected image in response to the at least one analysis process. A microscope system comprising:

4. 4. The microscope system according to claim 1, The computer includes: classifying one or more structures depicted in a digital image represented by the digital image data into one or more classes; generating position information for identifying a position of a structure classified into at least one of the one or more classes; outputting the analysis result including the location information; The projected image includes a figure representing the position of the structure classified into the at least one class. A microscope system comprising:

5. 5. The microscope system according to claim 4, The computer includes: generating statistical information for the structures classified into the at least one class; outputting the analysis result including the location information and the statistical information; The projection image includes a figure representing the position of the structure classified into at least one class and the statistical information. A microscope system comprising:

6. 6. The microscope system according to claim 4, The structure classified into at least one class is an object that serves as a basis for a pathologist's pathological diagnosis. A microscope system comprising:

7. 7. The microscope system according to claim 1, The computer includes: determining whether or not to project the projection image onto the image plane according to a setting of the microscope system; Controlling the projection device so that the projection device projects the projection image onto the image plane when the microscope system is set to a predetermined setting. A microscope system comprising:

8. The microscope system according to any one of claims 1 to 7, further comprising: an input device that outputs an operation signal in response to an input operation by a user; The computer changes a part of the threshold value used in the at least one analysis process in accordance with an operation signal output from the input device. A microscope system comprising:

9. 9. The microscope system according to claim 8, The computer further generates second projection image data based on a diagnostic protocol selected from a plurality of diagnostic protocols, and a second projection image represented by the second projection image data is an image according to the selected diagnostic protocol. A microscope system comprising:

10. 10. The microscope system according to claim 9, The projection image includes a diagnostic procedure guide for the selected diagnostic protocol. A microscope system comprising:

11. A projection image generating program for a microscope having a projection device that projects a projection image onto an image plane on which an optical image is formed, comprising: On the computer, Acquire digital image data of the sample from the image sensor; identifying at least one of a plurality of analytical processes prepared in advance based on operation information based on an input operation by a user or selection information including identification information added to the sample, the plurality of analytical processes being processes with different contents targeted at a plurality of staining methods and / or biomarkers; performing at least one analytical process identified from a plurality of analytical processes on the digital image data of the sample; outputting an analysis result corresponding to the at least one identified analysis process; generating projection image data representing the projection image based on the analysis result and the at least one of the specified analysis processes, the projection image data showing the analysis result in a display format determined in response to the at least one of the specified analysis processes, the display format including the color of the image and the figures constituting the image; Execute the process, The generating of the projection image data includes: When the at least one analysis process is a process targeting a staining method, a color of the projection image is determined to be a color different from a color of an optical image corresponding to the staining method targeted by the at least one analysis process, in accordance with the staining method targeted by the at least one analysis process; and when the at least one analysis process is a process targeting a biomarker, determining a stained location of the sample in the optical image onto which a figure constituting the projection image is projected, according to a biomarker targeted by the at least one analysis process. program.

12. 1. A method for generating a projection image for a microscope having a projection device for projecting a projection image onto an image plane on which an optical image is formed, comprising the steps of: The computer Acquire digital image data of the sample from the image sensor; identifying at least one of a plurality of analytical processes prepared in advance based on operation information based on an input operation by a user or selection information including identification information added to the sample, the plurality of analytical processes being processes with different contents targeted at a plurality of staining methods and / or biomarkers; performing at least one analytical process identified from a plurality of analytical processes on the digital image data of the sample; outputting an analysis result corresponding to the at least one identified analysis process; generating projection image data representing the projection image based on the analysis result and the at least one of the specified analysis processes, the projection image data showing the analysis result in a display format determined in response to the at least one of the specified analysis processes, the display format including the color of the image and the figures constituting the image; The generating of the projection image data includes: When the at least one analysis process is a process targeting a staining method, a color of the projection image is determined to be a color different from a color of an optical image corresponding to the staining method targeted by the at least one analysis process, in accordance with the staining method targeted by the at least one analysis process; and when the at least one analysis process is a process targeting a biomarker, determining a stained location of the sample in the optical image onto which a figure constituting the projection image is projected, according to a biomarker targeted by the at least one analysis process. Projection image generation method.

Images