Microscope system, control method, and computer readable medium

Abstract

The present invention provides a microscope system, a control method, and a computer-readable medium, which enable a user to start observing a specimen using the microscope system without having to perform complicated operations. The microscope system (1) comprises a computer (10) and a microscope (100) that acquires images at least at a first magnification and a second magnification that is higher than the first magnification. The computer (10) determines the type of container in which the specimen is placed. When the computer (10) starts observing the specimen at the second magnification, the computer (10) determines an observation start position corresponding to the type of container by performing object detection corresponding to the type of container on a macroscopic image including the container acquired by the microscope (100) at the first magnification, so as to control the relative position of the microscope (100) with respect to the specimen in such a manner that the observation start position is included in the field of view of the microscope (100) at the second magnification.

Description

Technical Field

[0001] The disclosure of this specification relates to a microscope system, a control method, and a computer-readable medium. Background Art

[0002] By using a microscope system, the user can observe an image of a magnified object. On the other hand, since the field of view of the microscope system becomes narrower as the object is magnified, it is difficult to bring the object to be observed within the field of view. In addition, the higher the resolution required, the shallower the depth of focus, making it difficult to focus the microscope system on the object to be observed. Therefore, in order to make the microscope system perform at its high performance, various adjustments and settings need to be made before starting formal observation, and the time required for these operations becomes a great burden on the user.

[0003] For example, a technology related to such a technical problem is described in Patent Document 1. Patent Document 1 describes a technology that acquires a macroscopic image of a subject, sets an image acquisition range that includes a specimen, and the like as a microscopic image pickup condition by referring to the acquired macroscopic image. In other words, a technology that automatically sets a microscopic image pickup condition by detecting a specimen from a macroscopic image is described.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2007-233098 Summary of the invention

[0007] Problem that the invention aims to solve

[0008] However, it is not always possible to detect specimens in macroscopic observation. The observation magnification and resolution of macroscopic observation are lower than those of microscopic observation, so sometimes it is impossible to detect small specimens such as cells. In the technology described in Patent Document 1, when the specimen is not detected, the user has to make troublesome adjustments and settings by himself.

[0009] In view of the above circumstances, an object of one aspect of the present invention is to provide a technology that enables a user to start observing a specimen using a microscope system without performing complicated operations.

[0010] Solutions for solving problems

[0011] A microscope system according to one embodiment of the present invention includes: an image acquisition unit that acquires images at least at a first magnification and a second magnification higher than the first magnification; and a control unit. The control unit determines the type of a container in which a specimen is placed, and when starting observation of the specimen placed in the container at the second magnification, the control unit determines an observation start position corresponding to the determined type of the container by performing object detection corresponding to the type of the container on a first image including the container acquired by the image acquisition unit at the first magnification, and controls the relative position of the image acquisition unit with respect to the specimen in such a manner that the observation start position is included in the field of view of the image acquisition unit at the second magnification.

[0012] A control method involved in one embodiment of the present invention is a control method for a microscope system including an image acquisition unit that acquires images at least at a first magnification and a second magnification higher than the first magnification, wherein a computer determines the type of a container in which a specimen is placed, and when the computer starts observing the specimen placed in the container at the second magnification, the computer determines an observation start position corresponding to the determined type of the container by performing object detection corresponding to the type of the container on a first image including the container acquired by the image acquisition unit at the first magnification, and controls the relative position of the image acquisition unit with respect to the specimen in such a manner that the observation start position is included in the field of view of the image acquisition unit at the second magnification.

[0013] A computer-readable medium according to one embodiment of the present invention stores a program that causes a computer of a microscope system having an image acquisition unit that acquires images at least at a first magnification and a second magnification higher than the first magnification to execute the following processing: determining the type of a container in which a specimen is placed, and when starting observation of the specimen placed in the container at the second magnification, determining an observation start position corresponding to the determined type of the container by performing object detection corresponding to the type of the container on a first image including the container acquired by the image acquisition unit at the first magnification, and controlling the relative position of the image acquisition unit with respect to the specimen in such a manner that the observation start position is included in the field of view of the image acquisition unit at the second magnification.

[0014] Effects of the Invention

[0015] According to the above aspect, the user can start observing a specimen using the microscope system without performing complicated operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 2 is a diagram illustrating the configuration of the microscope system 1 .

[0017] Figure 2 The diagram exemplifies the configuration of the computer 10 .

[0018] Figure 3 This is a diagram for explaining a method of acquiring identification information for identifying a specimen container.

[0019] Figure 4 1 is a diagram showing an example of a flowchart of the observation preparation process performed by the microscope system 1 .

[0020] Figure 5 2 is a diagram illustrating a table for selecting parameters of the electric stage 101 .

[0021] Figure 6 This is an example of a screen displayed on the display device 14 .

[0022] Figure 7 1 is a diagram showing an example of a flowchart of macro image analysis processing performed by the microscope system 1 .

[0023] Figure 8 1 is a diagram showing an example of a flowchart of label processing performed by the microscope system 1 .

[0024] Fig. 9 2 is a diagram illustrating examples of histograms before and after brightness correction.

[0025] Fig.10 It is a diagram for explaining the object detection result for the slide specimen.

[0026] Fig.11 This is a diagram for explaining the object detection result for the petri dish specimen.

[0027] Fig.12 It is a diagram for explaining the object detection result for the multi-well plate sample.

[0028] Fig.13 It is a diagram showing another example of a table for selecting parameters of the electric stage 101 .

[0029] Fig.14 It is a diagram showing still another example of a table for selecting parameters of the electric stage 101 .

[0030] Fig.15 This is a diagram illustrating a table for selecting an observation method for microscopic observation.

[0031] Fig.16 This is another example of the screen displayed on the display device 14.

[0032] Fig.17 This is another example of the screen displayed on the display device 14.

[0033] Fig.18 This is a diagram for explaining an example of a scanning method for scanning a mapping area.

[0034] Fig.19 It is a diagram for explaining another example of the scanning method of scanning the mapping area.

[0035] Fig. 20 This is a diagram for explaining still another example of the scanning method of scanning the mapping area.

[0036] Fig.21 This is another example of the screen displayed on the display device 14.

[0037] Fig. 22 This is another example of the screen displayed on the display device 14.

[0038] Fig.23 This is another example of the screen displayed on the display device 14.

[0039] Fig.24 2 is a diagram illustrating the configuration of the microscope system 2 .

[0040] Description of Reference Numerals

[0041] 1, 2: microscope system; 10: computer; 11: processor; 12: memory; 14: display device; 20: microscope controller; 100, 500: microscope; 110: container support holder; 112, 112a to 112c: Hall element; 120: slide support; 122, 122a to 122c, 132, 132a, 132b, 150: magnet; 130: culture dish support; 140: multi-well plate; 141, 141a to 141f: wells; 150: universal container support; 160, 161-163, 161a-161d: glass slide; 164-166, 164a-164d, 173b: glass cover; 167-169: label; 170, 171a-171c: culture dish; 172b, 172c: well; 180: flask; 200: macro image acquisition unit; 300: micro image acquisition unit; C: cell; M1, M2, M5: macro image; M3: micro image; M4, M6: mapping image; MT: multiple tissues; S: specimen; ST: stained tissue; UT: unstained tissue. DETAILED DESCRIPTION

[0042] Figure 1 2 is a diagram illustrating the configuration of the microscope system 1 . Figure 21 is a diagram showing an example of the structure of a computer 10 included in the microscope system 1. The microscope system 1 can reduce the user's workload and enable specimen observation using high-quality microscope images by automating at least a portion of the various adjustments and settings that the user needs to perform before starting observation in conventional microscope systems (hereinafter referred to as observation preparation work). Figure 1 and Figure 2 The structure of the microscope system 1 will be described.

[0043] like Figure 1 As shown, the microscope system 1 includes a microscope 100, a computer 10 for controlling the entire microscope system 1, and a microscope controller 20 for controlling the microscope 100. The microscope 100 and the microscope controller 20 are communicably connected, and further, the computer 10 and the microscope controller 20 are communicably connected to each other.

[0044] The microscope 100 includes a macro image acquisition unit 200 that acquires an image at a relatively low observation magnification (first magnification) and a micro image acquisition unit 300 that acquires an image at an observation magnification (second magnification) higher than the observation magnification of the macro image acquisition unit 200. That is, the microscope 100 is an example of an image acquisition unit that acquires an image at least at the first magnification and the second magnification. The microscope 100 also includes an electric stage 101 provided with a container support holding unit 110. In addition, the image acquired by the macro image acquisition unit 200 is recorded as a macro image or a first image, and the image acquired by the micro image acquisition unit 300 is recorded as a micro image or a second image as needed.

[0045] The electric stage 101 has a motor 102 and an origin sensor 103. The electric stage 101 moves in a direction (XY direction) orthogonal to the optical axes of the macro image acquisition unit 200 and the micro image acquisition unit 300 by the rotation of the motor 102. The electric stage 101 is controlled by the microscope controller 20. In more detail, the microscope controller 20 controls the motor 102 based on the origin in the XY direction detected by the origin sensor 103, thereby the electric stage 101 transports the object placed on the container support holding part 110 onto the optical axis of the macro image acquisition unit 200 or the optical axis of the micro image acquisition unit 300. Here, the subject means an object to be photographed, and is distinguished from the object mainly observed by the user, that is, the specimen. In addition, the subject can include a specimen. Furthermore, by controlling the motor 102 by the microscope controller 20, the electric stage 101 can align an arbitrary position on the object with the center of the field of view of the macro image acquisition unit 200 or the center of the field of view of the micro image acquisition unit 300. In addition, the microscope controller 20 outputs the position information of the electric stage 101 to the computer 10 as appropriate or upon request.

[0046] The macro image acquisition unit 200 includes a macro light source 201, a macro optical system 202, and an imaging device 203. In addition, the macro image acquisition unit 200 may further include an illumination optical system for irradiating an object with illumination light emitted from the macro light source 201. The macro light source 201 is, for example, a light source such as a xenon lamp or a light emitting diode (LED). The observation magnification (first magnification) of the macro optical system 202 may be a magnification that allows at least a specimen container described later to be contained within the field of view of the macro optical system 202, and more preferably, a magnification that allows a container support held by the container support holding unit 110 to be contained within one or two macro images. The imaging device 203 is a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary MOS: Complementary Metal Oxide Semiconductor) image sensor, or the like.

[0047] In the macro image acquisition unit 200, the light transmitted through the subject is focused on the camera device 203 through the macro optical system 202, and the camera device 203 captures the subject to acquire a macro image of the subject. The camera device 203 also outputs the image data of the subject to the computer 10 via the microscope controller 20. In addition, the amount of illumination light emitted from the macro light source 201 or the amount of illumination light irradiated to the subject is controlled by the microscope controller 20. In addition, the imaging operation performed by the camera device 203 is also controlled by the microscope controller 20. For example, the microscope controller 20 can control the camera device 203 based on the input from the computer 10 to switch the automatic gain control on / off, set the gain, switch the automatic exposure control on / off, set the exposure time, set the edge enhancement degree, and set the gamma correction.

[0048] The microscopic image acquisition unit 300 includes a microscopic light source 301, a condenser 302, a plurality of objective lenses (objective lens 304, objective lens 305) mounted on a revolving nosepiece 306, and an imaging device 307. The microscopic image acquisition unit 300 can acquire images using a plurality of observation methods. In this example, the image can be acquired using a bright field observation method and a phase difference observation method. The condenser 302 corresponds to a plurality of observation methods. For example, when acquiring an image using the phase difference observation method, Figure 1As shown, an annular slit is arranged in the condenser 302. In addition, the annular slit is arranged to be freely inserted and removed. For example, when acquiring an image by bright field observation, the condenser 302 is used in a state where the annular slit is removed outside the optical path. In addition, the microscopic image acquisition unit 300 can have a plurality of condensers corresponding to the observation method. In addition, the objective lens 304 is a phase difference objective lens used in the phase difference observation method, and includes a phase film at a position corresponding to the annular slit. The objective lens 305 is an objective lens used in the bright field observation method. The lens changer 306 is a switching mechanism for switching the objective lens arranged on the optical axis. In addition, the lens changer 306 also functions as a focusing device that moves the objective lens in the direction of the optical axis by moving along the optical axis. The camera device 307 is a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary MOS: Complementary Metal Oxide Semiconductor) image sensor, etc.

[0049] In the microscopic image acquisition unit 300, the illumination light emitted from the microscopic light source 301 is irradiated to the subject via a condenser. Thereafter, the light transmitted through the subject is condensed onto the camera device 307 via an objective lens and an imaging lens (not shown), and the camera device 307 captures the subject to acquire a microscopic image of the subject. The camera device 307 also outputs the image data of the subject to the computer 10 via the microscope controller 20. In addition, the amount of illumination light emitted from the microscopic light source 301 or the amount of illumination light irradiated to the subject is controlled by the microscope controller 20. In addition, the imaging operation performed by the camera device 307 is also controlled by the microscope controller 20. For example, the microscope controller 20 can control the camera device 307 according to the input from the computer 10 to perform on / off switching of automatic gain control, setting of gain, on / off switching of automatic exposure control, setting of exposure time, setting of edge enhancement degree, setting of gamma correction, and the like.

[0050] The lens changer 306 has a motor and an origin sensor which are not shown in the figure. The lens changer 306 moves in the optical axis direction (Z direction) of the microscopic image acquisition unit 300 by rotating the motor. The lens changer 306 is controlled by the microscope controller 20. In more detail, the microscope controller 20 controls the motor based on the origin in the Z direction detected by the origin sensor, thereby enabling the lens changer 306 to move the objective lens in the optical axis direction of the microscopic image acquisition unit 300 to adjust the focus. In this case, the lens changer 306 functions as a focusing device. In addition, the microscope controller 20 outputs the position information of the lens changer 306 to the computer 10 appropriately or upon request. In this example, an example in which the lens changer 306 functions as a focusing device is shown, but the electric stage 101 may be used to replace the lens changer 306 to move in the optical axis direction.

[0051] The microscope controller 20 is a controller for controlling the microscope 100, and controls all electric devices (except the imaging device 203 and the imaging device 307) included in the microscope 100 under the control of the computer 10. Specifically, the microscope controller 20 controls the electric stage 101 to change the observation position, controls the nosepiece 306 to change the focus, switches the objective lens and the condenser to change the observation magnification or the observation method, controls the light source (the macro light source 201 and the micro light source 301) to adjust the light, and controls the imaging operation performed by the imaging device 203 and the imaging device 307. In addition, the microscope controller 20 also notifies the computer 10 of the state of the microscope 100.

[0052] The computer 10 is an example of a control unit that controls the entire microscope system 1, and controls the microscope 100 and the microscope controller 20. Figure 2 As shown, the computer 10 includes a processor 11, a memory 12, an input device 13, a display device 14, and a communication I / F 15, which are connected to each other via a bus 16. Figure 2 This is an example of the structure of the computer 10, and the computer 10 is not limited to this structure. The computer 10 can be implemented by a general-purpose computer such as a workstation or a personal computer, or can be implemented by a dedicated computer.

[0053] The processor 11 includes one or more processors. The one or more processors may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), etc. In addition, it may include an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA), etc. The processor 11 may, for example, execute a prescribed software program to perform, for example, Figure 4 The processing described later is shown.

[0054] The memory 12 includes a non-transitory computer-readable medium storing a software program executed by the processor 11. The processor 11 may include, for example, one or more arbitrary semiconductor memories, and may also include one or more other storage devices. Semiconductor memories include, for example, volatile memories such as RAM (Random Access Memory), ROM (Read Only Memory), programmable ROM, flash memory and other non-volatile memories. RAM may include, for example, DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), etc. Other storage devices may include, for example, magnetic storage devices including magnetic disks, optical storage devices including optical disks, etc.

[0055] The input device 13 is a device for detecting the operation of the user of the microscope system 1 to input operation information. The input device 13 is, for example, a keyboard, a mouse, a handle, a joystick, various switches, etc. The display device 14 is an example of a display unit that displays a screen based on a display signal from the processor 11. The display device 14 is, for example, a liquid crystal display device, an organic EL (OLE: Organic Electro-Luminescence: organic electroluminescent element) display device, a CRT (Cathode RayTube: cathode ray tube) display device, etc. The display device 14 can have a touch panel sensor, in which case the display device 14 also functions as the input device 13. The communication I / F 15 sends and receives signals between the microscope 100 and the microscope controller 20.

[0056] In addition, Figure 1 , the microscope 100 included in the microscope system 1 is shown as an example of an inverted microscope, but the microscope 100 is not limited to an inverted microscope, and may be an upright microscope. Figure 1 , an example of the microscope 100 using the transmitted illumination method is shown, but the microscope 100 may use the epi-illumination method. In addition, an example of the microscope 100 using a dedicated objective lens to switch between the bright field observation method and the phase difference observation method is shown, but it is also possible to insert and remove the phase modulation element (phase film) relative to the pupil position, thereby switching between the bright field observation method and the phase difference observation method without switching the objective lens to another objective lens. In addition, an example of the microscope 100 corresponding to the bright field observation method and the phase difference observation method is shown, but the microscope 100 may also correspond to other observation methods, such as the fluorescence observation method, the differential interference observation method, the polarization observation method, the dark field observation method, etc.

[0057] In addition, an example is shown in which the macro image acquisition unit 200 and the micro image acquisition unit 300 each have an imaging device, but the macro image acquisition unit 200 and the micro image acquisition unit 300 can share the imaging device by adjusting the image positions of each other to be located in the same plane. In addition, an example is shown in which the micro image acquisition unit 300 includes two objective lenses, but it can include more than three objective lenses, and can also include objective lenses corresponding to multiple different observation magnifications. In addition, the condenser can also include more than two.

[0058] In addition, although the example in which the computer 10, the microscope controller 20, and the microscope 100 are configured as separate devices in the microscope system 1 is shown, a part or all of these may be configured as an integral unit. For example, the microscope 100 and the microscope controller 20 may be configured as an integral unit, or the microscope 100, the computer 10, and the microscope controller 20 may be configured as an integral unit.

[0059] In addition, although the example in which the computer 10 includes the input device 13 and the display device 14 is shown, the input device 13 and the display device 14 may be separate devices from the computer 10. The computer 10 may be connected to the input device 13 and the display device 14 which are independent from the computer 10 so as to be communicable.

[0060] Figure 3 1 is a diagram for explaining a method of acquiring identification information for identifying a specimen container. The microscope system 1 has a function of acquiring identification information for identifying a specimen container and determining the type of the specimen container based on the acquired identification information. Figure 3 To illustrate the method of obtaining identification information.

[0061] like Figure 3 As shown, an opening 111 is formed in a container support holding portion 110 provided on the electric stage 101. By providing a container support for holding a specimen container in a manner that closes the opening 111, a user can observe specimens contained in various specimen containers using the microscope system 1. In addition, a plurality of Hall elements 112 (Hall element 112a, Hall element 112b, Hall element 112c) as non-contact type magnetic sensors are provided at predetermined positions of the container support holding portion 110.

[0062] When observing the specimen S fixed to the slide glass 160, the slide glass support 120, which is a kind of container support, can be set in the container support holder 110. By inserting the slide glass into the openings 121 (opening 121a, opening 121b, opening 121c) formed in the slide glass support 120, the slide glass support 120 can hold up to three slide glass at the same time. In addition, three magnets 122 are fixed to the slide glass support 120. More specifically, when the slide glass support 120 is set in the container support holder 110, the magnet 122a is fixed at a position opposite to the Hall element 112a, the magnet 122b is fixed at a position opposite to the Hall element 112b, and the magnet 122c is fixed at a position opposite to the Hall element 112c. Therefore, when the slide glass support 120 is set in the container support holder 110, all three Hall elements 112 detect the magnets.

[0063] In addition, when observing the specimen S contained in the culture dish 170, the culture dish support 130, which is a kind of container support, may be set in the container support holder 110. By inserting the culture dish into the openings 131 (opening 131a, opening 131b, opening 131c) formed in the culture dish support 130, the culture dish support 130 can hold up to three culture dishes at the same time. In addition, two magnets 132 are fixed to the culture dish support 130. More specifically, when the culture dish support 130 is set in the container support holder 110, the magnet 132a is fixed at a position opposite to the Hall element 112a, and the magnet 132b is fixed at a position opposite to the Hall element 112b. Therefore, when the culture dish support 130 is set in the container support holder 110, the two Hall elements 112 detect the magnets.

[0064] In addition, when observing the specimen S accommodated in the holes 141 of the porous plate 140, the porous plate 140 itself may be set on the container support holding part 110. In addition, no magnet is fixed to the porous plate 140. Therefore, when the porous plate 140 is set on the container support holding part 110, none of the three Hall elements detects the magnet.

[0065] In addition, when observing the specimen S contained in the flask 180, the universal container support 150, which is a kind of container support, may be set on the container support holding part 110. The universal container support 150 has an opening 151, so the specimen S can be observed by placing the flask 180 on the universal container support 150 so that the specimen S is located on the opening 151. In addition, one magnet 152 is fixed to the universal container support 150. More specifically, the magnet 152 is fixed at a position opposite to the Hall element 112a when the universal container support 150 is set on the container support holding part 110. Therefore, when the universal container support 150 is set on the container support holding part 110, one Hall element 112 (Hall element 112a) detects the magnet. In addition, a specimen container other than the flask 180 may be arranged on the universal container support 150.

[0066] As described above, in the microscope system 1, the number of magnets detected by the Hall element 112 varies depending on the type of the container support (slide support 120, culture dish support 130, universal container support 150) provided in the container support holding portion 110 or the type of the specimen container (multi-well plate 140). Therefore, the computer 10 can determine the type of the specimen container placed on the electric stage 101 based on the acquired information by acquiring information related to the number of magnets detected by the Hall element 112 through the microscope controller 20. That is, in the microscope system 1, the computer 10 acquires information related to the number of detected magnets as identification information for identifying the type of the specimen container in which the specimen is placed, and determines the type of the specimen container in which the specimen is placed based on the acquired identification information.

[0067] Furthermore, the microscope system 1 may include a detection unit that directly or indirectly outputs identification information to the computer 10, and the detection unit may detect an identification structure provided on a container support that holds the specimen container or on the specimen container itself, corresponding to the type of the specimen container. Figure 3 In the example of FIG. 1 , a magnet provided in the container support or the specimen container itself functions as an identification structure, and a Hall element functions as a detection unit for detecting the identification structure. However, for example, different numbers of openings may be formed at positions where magnets are fixed according to the type of container support, and a light sensor for detecting light incident through these openings may be provided in the container support holding portion 110. In this case, the opening provided in the container support or the specimen container itself functions as an identification structure, and the light sensor functions as a detection unit for detecting the identification structure.

[0068] In addition, Figure 3, the detection unit is provided in the container support holding portion 110, but the detection unit can be arranged at any position as long as it can detect the identification structure. The detection unit may not be provided in the container support holding portion 110, but may be provided in the electric stage 101, for example, or may be provided in other positions in the microscope 100. In addition, the number of containers held by the container support is not limited to Figure 3 For example, a slide glass support capable of inserting four or more slide glasses may be held by the container support holding portion 110, and a culture dish support capable of inserting four or more culture dishes may be held by the container support holding portion 110. In addition, multi-well plates having various numbers of wells (e.g., 6, 12, 24, 48, 96, etc.) may be held by the container support holding portion 110.

[0069] The microscope system 1 configured as described above automates at least a part of the observation preparation work that the user needs to perform before the observation starts in the previous microscope system, so that the user can start the observation of the specimen without performing complicated work. More specifically, the microscope system 1 automates the specimen search, which brings a particularly large workload in the observation preparation work. As a result, the burden on the user can be greatly reduced, and even a lightweight user who does not have professional knowledge about the microscope can easily and quickly start the observation of the specimen. In addition, the microscope system 1 automatically determines the range (mapping area) in which the mapping image should be constructed and proposes it to the user. In the case of formally observing the specimen at a high observation magnification, the field of view is narrow, so it is very useful to grasp the mapping image of the relationship between the overall image of the specimen and the current observation position. By proposing the range in which the mapping image should be constructed by the microscope system 1, the user can save the effort of exploring and setting the range in which the mapping image should be constructed. As a result, the burden on the user can be greatly reduced, and even a lightweight user who does not have professional knowledge about the microscope can easily and quickly start the formal observation of the specimen.

[0070] Figure 4 1 is a diagram showing an example of a flowchart of the observation preparation process performed by the microscope system 1. Figure 4 The following describes an observation preparation process that automates at least a portion of the observation preparation work. The observation preparation process is started, for example, when a user places a specimen container on the container support holding portion 110 and instructs the microscope system 1 to start observation using the input device 13. Figure 4 The observation preparation process shown is an example of a method of controlling the microscope system 1 by the computer 10 , and is performed by, for example, the processor 11 executing a program stored in the memory 12 .

[0071] When the observation preparation process starts, the microscope system 1 first moves the specimen container arranged on the container support holding part 110 to the macro imaging position (step S1). Here, the microscope controller 20 is controlled by the computer 10, and the microscope controller 20 moves the electric stage 101 so that the container support holding part 110 is located on the optical axis of the macro image acquisition part 200. At this time, the microscope controller 20 controls the movement of the electric stage 101 according to the initial setting regardless of the type of the specimen container.

[0072] When the specimen container moves to the macrophotography position, the microscope system 1 determines the type of the specimen container (step S2). Here, the Hall element 112 provided in the container support holding portion 110 detects the number of magnets corresponding to the container support to output identification information for identifying the type of the specimen container, and the computer 10 determines the type of the specimen container based on the input identification information. Specifically, when the identification information indicates that the number of magnets is three, the computer 10 determines the type of the specimen container as a slide. In addition, when the identification information indicates that the number of magnets is two, the computer 10 determines the type of the specimen container as a culture dish. In addition, when the identification information indicates that the number of magnets is one, the computer 10 determines the type of the specimen container as other specimen containers such as a flask. In addition, when the identification information indicates that the number of magnets is 0, the computer 10 determines the type of the specimen container as a multi-well plate.

[0073] When the type of the specimen container is determined, the microscope system 1 changes the setting of the microscope system 1 to a setting corresponding to the determined type of the specimen container (step S3). Figure 5 The parameter group corresponding to the type of the specimen container determined in step S2 is read from the table T1 shown, and is set in the microscope system 1 .

[0074] Figure 5 1 is a diagram showing an example of a table for selecting parameters of the electric stage 101. Figure 5Table T1 shown stores parameter groups (parameter groups PS1 to PS4) related to the desired drive of the electric stage 101 for each type of specimen container. The parameter groups related to the drive of the electric stage 101 include parameters such as acceleration, initial velocity, and maximum velocity of the electric stage 101. When comparing the parameter groups stored in Table T1, the parameter groups (parameter groups PS2 to PS4) for specimen containers other than glass slides show values ​​that make the electric stage 101 move more gently or slowly than the parameter group PS1 for glass slides. This is because, in specimen containers other than glass slides, it is highly likely that the specimen is contained in a state of being immersed in liquid, and when the electric stage 101 is moved quickly or suddenly, the specimen and the liquid in which the specimen is immersed will shake greatly along with the movement of the electric stage 101, and there is a possibility that the quality of the image will be adversely affected. In addition, an example of storing parameter groups in the table is shown, but the storage method of the parameter groups is not particularly limited to this example. The parameter groups can be stored in any form that can identify the parameter groups corresponding to the type of specimen container.

[0075] By setting the parameter group corresponding to the specimen container in step S3, when a slide glass in which the specimen is fixed and not easily shaken is placed on the electric stage 101, the electric stage 101 moves at a high speed, and the time required for image acquisition can be shortened. In addition, when a specimen container other than a slide glass is placed on the electric stage 101, the electric stage 101 moves relatively gently in a manner that does not affect the image quality. By controlling the drive of the electric stage 101 according to the specimen container in this way, it is possible to acquire an image in the shortest possible time while avoiding a decrease in image quality.

[0076] When the setting is changed in step S3, the microscope system 1 performs macro photography (step S4). Here, the microscope 100 and the microscope controller 20 are controlled by the computer 10, and the macro image acquisition unit 200 photographs the subject at a first magnification. In more detail, the macro image acquisition unit 200 first photographs the upper half of the container support held by the container support holding unit 110. Next, the microscope controller 20 moves the electric stage 101 so that the lower half of the container support enters the field of view of the macro image acquisition unit 200. At this time, the electric stage 101 operates according to the parameters set by step S3. Thereafter, the macro image acquisition unit 200 photographs the lower half of the container support held by the container support holding unit 110. Finally, the macro image acquisition unit 200 outputs two macro images of the entire container support to the computer 10.

[0077] In addition, in macro photography, the entire container support can be photographed in one image. It is expected that most of the container support can be photographed in the macro image obtained by step S4, but it is not necessary to photograph the entire container support. At least the specimen container can be photographed in the macro image. That is, the macro image is an image including the specimen container. In addition, the computer 10 can perform various correction processing on the macro image obtained by macro photography as needed. For example, the computer 10 can perform correction processing such as brightness correction, distortion correction, and magnification chromatic aberration correction on the macro image.

[0078] When the macro photography is finished, the microscope system 1 displays the macro image (step S5). Here, the computer 10 displays the macro image acquired in step S4 on the display device 14. Figure 6 Screen 400 is shown.

[0079] Figure 6 This is an example of the screen displayed on the display device 14 in step S5. Figure 6 The screen 400 shown includes a microscopic image area 410, a mapping image area 420, a macroscopic image area 430, and an operation area 440. The microscopic image area 410 is an area for displaying a microscopic image acquired by the microscopic image acquisition unit 300. The mapping image area 420 is an area for displaying a mapping image generated by stitching microscopic images. The macroscopic image area 430 is an area for displaying a macroscopic image acquired by the macroscopic image acquisition unit 200. The operation area 440 is an area for configuring a GUI (Graphical User Interface) controller for the user to input instructions to the microscope system 1. In step S5, in the macroscopic image area 430, a macroscopic image M1 generated by stitching two macroscopic images acquired in step S4 is displayed in a state of being enlarged or reduced according to the size of the macroscopic image area 430.

[0080] exist Figure 6 The macro image M1 shown shows that the container support is a glass slide support 120, and three glass slides (glass slide 161, glass slide 162, glass slide 163) are arranged on the glass slide support 120. In addition, it is also shown that a glass cover (glass cover 164, glass cover 165, glass cover 166) covering a specimen is placed on each glass slide, and a label (label 167, label 168, label 169) is further attached. In addition, the label is printed or recorded with information for managing the glass slide specimen. The label is usually made of a member such as paper that has a lower transmittance than the glass slide, and therefore appears as a particularly dark area in the macro image M1.

[0081] When the macro image is displayed, the microscope system 1 performs macro image analysis processing (step S6). Here, the computer 10 performs image analysis processing on the macro image acquired in step S4.

[0082] Figure 7 1 is a diagram showing an example of a flowchart of macro image analysis processing performed by the microscope system 1 . Figure 8 1 is a diagram showing an example of a flowchart of label processing performed by the microscope system 1 . Fig. 9 2 is a diagram illustrating examples of histograms before and after brightness correction. Fig.10 It is a diagram for explaining the object detection result for the slide specimen. Fig.11 This is a diagram for explaining the object detection result for the petri dish specimen. Fig.12 is a diagram for explaining the object detection results for a multi-well plate sample. Figures 7 to 12 The macro image analysis process performed in step S6 will be described.

[0083] When to start Figure 7 In the macro image analysis process shown in FIG. 1 , the microscope system 1 first determines whether the type of the specimen container is a glass slide (step S20). Here, the computer 10 determines whether it is a glass slide based on the type of the specimen container determined in step S2. If the type of the specimen container is determined to be a glass slide, the microscope system 1 performs Figure 8 On the other hand, when it is determined that the type of the specimen container is not a slide glass, the microscope system 1 omits step S21.

[0084] When to start Figure 8During the label processing shown, the microscope system 1 first detects the label portion on the slide (step S30). Here, the computer 10 detects the area (label portion) of the label attached to the slide based on the macro image M1. The label hardly transmits light, so the label portion appears as a particularly dark area in the macro image M1. Therefore, the computer 10 can detect the area with a pixel value lower than a specified threshold included in the macro image M1 as the label portion. In addition, assuming that the label portion has a certain size, the computer 10 can detect the continuous area with a pixel value lower than a specified threshold and an area above a fixed value included in the macro image M1 as the label portion. In addition, the detection method of the label portion is not particularly limited to these methods. The label portion can be detected by other methods. For example, the computer 10 can detect a preset area as the label portion, or detect the label portion by pattern matching processing such as template matching. In addition, the computer 10 can detect the label portion using an object detection method using deep learning such as YOLO (You Only Look Once) and SSD (Single Shot Detector).

[0085] When the label portion is detected, the microscope system 1 performs image processing on the label portion (step S31). Here, the computer 10 can adjust the brightness of the label portion of the macro image M1, for example, the gain of the label portion can be made larger than the gain of other portions (for example, 10 times, etc.) to enhance the slight brightness difference in the label portion. Thus, the macro image M1 is converted into a macro image M2 in which the information printed or recorded on the label floats relative to the background. In addition, in order to make it easier to see the information on the label, the label portion of the macro image M2 can be smoothed to remove noise, and the contrast can also be corrected.

[0086] For example, in the case where brightness adjustment is performed in the image processing performed in step S31, refer to Fig. 9 Let's explain in further detail. Fig. 9 The shown histogram H1 is an example of a histogram of pixel values ​​of the macro image M1 before the image processing of step S31 is performed. Fig. 9 The histogram H2 shown is an example of a histogram of pixel values ​​of the macro image M2 after the image processing of step S31 is performed. Fig. 9As shown, the histogram H1 of the macro image M1 of the slide specimen with a label attached includes a peak P1 formed by pixel values ​​other than the label part and a peak P2 formed by pixel values ​​of the label part. The pixel values ​​determined to be the label part that are less than the threshold value TH are adjusted by image processing, so that in the histogram H2 after image processing, the peak P3 corresponding to the peak P2 has a higher pixel value than the peak P2. In addition, the pixel value is adjusted by adjusting the gain, so that the distribution including the peak P3 has a larger dispersion than the distribution including the peak P2. Therefore, in the macro image M2 after image processing, the brightness of the label part becomes brighter, and the difference in brightness between the pixels of the label part becomes larger, so that the information printed or recorded on the label can be visually confirmed. In addition, the information of the label part is obtained by adjusting the brightness of a part of the image by image processing. Figure 8 The label processing shown has the following advantages compared to methods such as acquiring images through transmitted illumination and incident illumination respectively: since the number of photographs can be reduced, the required information can be obtained in a short time, and since a complex device structure is not required, the cost can be reduced.

[0087] The microscope system 1 then performs object detection on the macro image (step S22). Here, the computer 10 detects the structure corresponding to the type of the specimen container determined in step S2 based on the macro image M1 or the macro image M2. The computer 10 can further detect the specimen placed in the specimen container based on the macro image M1 or the macro image M2, and can also detect the staining state of the specimen.

[0088] In step S22, if the type of the specimen container is a glass slide, the computer 10 detects the glass cover slip as a structure corresponding to the type of the specimen container. Further, the specimen is detected from the area within the glass cover slip. Regarding the specimen, it is expected that, for example, specimens stained by HE (Hematoxylin Eosin) staining (hereinafter referred to as stained specimens) will be distinguished from other non-stained specimens for detection. In addition, non-stained specimens are, for example, transparent specimens observed by phase contrast observation, but can also be specimens observed by fluorescence observation combined with fluorescent pigments such as DAPI. That is, it is expected that the computer 10 determines the position of the object based on the macro image and classifies the object into at least three categories including stained specimens, non-stained specimens, and glass cover slips.

[0089] For the object detection of the glass slide, for example, an object detection algorithm using deep learning such as YOLO (You Only Look Once) and SSD (Single Shot Detector) can be used. Thus, for example, if a glass slide specimen with a stained tissue ST sandwiched between the glass slide 161a and the cover glass 164a is captured in a macro image, Fig.10 As shown in (a), the range R1 of the cover glass 164a overlapping the slide glass 161a and the range R2 of the stained tissue ST are detected. In addition, for example, if a slide specimen in which the non-stained tissue UT is sandwiched between the slide glass 161b and the cover glass 164b is captured in the macro image, as shown in FIG. Fig.10 As shown in (b), the range R3 of the cover glass 164b and the range R4 of the non-stained tissue UT are detected. In addition, for example, if a slide specimen in which a cell C is sandwiched between the slide glass 161c and the cover glass 164c is captured in the macro image, since the macro image does not have a resolution that can identify the cell C, Fig.10 As shown in (c), only the range R5 of the cover glass 164c is detected. In addition, for example, if a slide specimen in which a plurality of tissues MT are sandwiched between the slide glass 161d and the cover glass 164d is captured in the macro image, then Fig.10 As shown in (d), the range R7 of the cover glass 164d and the range R8 of each of the multiple tissues MT are detected. The above-mentioned object detection can be performed on the entire macro image or only on the area within the slide in the macro image. In addition, the position of the slide is limited by the slide support 120 and is therefore known.

[0090] In step S22, if the type of the specimen container is a culture dish, the computer 10 detects the hole portion provided in the culture dish as a structure corresponding to the type of the specimen container. Further, the specimen is detected from the area of ​​the hole portion. It is expected that the specimen is distinguished as a stained specimen and an unstained specimen for detection, for example. That is, it is expected that the computer 10 determines the position of the object according to the macroscopic image and classifies the object into at least three categories including a stained specimen, an unstained specimen and a hole portion. In addition, the culture dish is also called a shallow bottom dish, a petri dish, and is usually composed of glass, transparent plastic, etc. The culture dish can be roughly divided into three types according to the structure of the bottom surface: a common culture dish, a glass bottom culture dish, and a membrane bottom culture dish. Among them, there is a hole portion in the glass bottom culture dish and the membrane bottom culture dish. The glass bottom culture dish forms a hole on the bottom surface of the culture dish, and a glass cover is attached from the back side of the culture dish in a manner of closing the hole. On the other hand, the membrane bottom culture dish is a culture dish in which a hole portion is formed on the bottom surface of the culture dish by integral molding.

[0091] Object detection for a culture dish is similar to object detection for a glass slide, and for example, an object detection algorithm using deep learning such as YOLO (You Only Look Once) or SSD (Single Shot Detector) can be used. Thus, for example, if a culture dish specimen containing cells C in a normal culture dish 171a is captured in a macro image, Fig.11 As shown in (a), the hole portion cannot be detected, and the cell C cannot be detected. In addition, for example, if a culture dish specimen in which a cell C is contained in the hole portion 172b of a glass-bottomed culture dish 171b with a glass cover 173b attached to the bottom is captured in a macroscopic image, as shown in FIG. Fig.11 As shown in (b), the cell C is not detected, but the range R11 of the hole portion 172b is detected. In addition, for example, if a culture dish specimen containing a cell C in a membrane bottom culture dish 171c integrally formed with a hole portion 172c is captured in a macro image, as shown in FIG. Fig.11 As shown in (c), the cell C is not detected, but the range R12 of the hole 172c is detected. The above-mentioned object detection can be performed on the entire macro image or only on the area within the culture dish in the macro image. In addition, the position of the culture dish is limited by the culture dish support 130 and is therefore known.

[0092] If it is known in advance that the specimen placed in the culture dish is a specimen such as a cell that is difficult to detect from a macroscopic image, the computer 10 can only determine the position of the hole. In this case, since it is sufficient to detect the shape of the hole, that is, the circular shape, other well-known techniques such as circle detection based on Hough transform can be used instead of the object detection algorithm using deep learning. In addition, the specimen can be detected by the object detection algorithm using deep learning, and the hole can be detected by Hough transform.

[0093] In step S22, if the type of the specimen container is a multi-well plate, the computer 10 detects the wells as a structure corresponding to the type of the specimen container. Further, the specimen is detected from the area of ​​the wells. It is expected that the specimens are distinguished as stained specimens and non-stained specimens for detection, for example. That is, it is expected that the computer 10 determines the position of the object based on the macro image and classifies the object into at least three categories including stained specimens, non-stained specimens, and wells.

[0094] Object detection for a multi-well plate is similar to object detection for a glass slide or a culture dish, and uses, for example, an object detection algorithm using deep learning such as YOLO (You Only Look Once) or SSD (Single Shot Detector). Thus, for example, if a multi-well plate specimen containing cells C in a well of a 6-well multi-well plate 140 is captured in a macro image, Fig.12 As shown, no cell C is detected, and the respective ranges R20 of the six holes (141a to 141f) are detected. In addition, the positions of the holes of the multi-well plate are different depending on the number of holes. In addition, even for multi-well plates with the same number of holes, the positions are sometimes different due to different manufacturers. Therefore, it is desired to perform object detection on the multi-well plate on the entire macroscopic image.

[0095] In addition, Fig.10 , an example of using a rectangle to determine the range of a detected object is shown, but the range of a detected object can be determined using a circle or other shapes, for example. Fig.11 and Fig.12 As shown, the range of the detected object can be determined in units of 1 pixel based on the shape of the object without using a predetermined shape.

[0096] In addition, object detection may be performed on at least one specimen container. For example, when there are multiple specimen containers in the macro image, object detection may be performed only on the specimen container at the position with the highest priority according to a predetermined priority. When there is no specimen container at the position with the highest priority, object detection may be performed on the specimen container at the position with the second highest priority. In addition, object detection may be performed on all specimen containers.

[0097] When the object detection is completed, the microscope system 1 determines the observation start position and the mapping area (step S23, step S24), and ends. Figure 7 The computer 10 performs macroscopic image analysis processing as shown. Here, the computer 10 determines the observation start position corresponding to the type of the specimen container and the mapping area corresponding to the type of the specimen container based on the macroscopic image. More specifically, the computer 10 performs object detection corresponding to the type of the specimen container on the macroscopic image, and determines the observation start position and the mapping area corresponding to the type of the specimen container based on the result of the object detection.

[0098] In addition, the observation start position is the following position: after the user instructs the microscope system 1 to start observation, a series of processes automatically performed by the microscope system 1 are completed and the user can manually operate the microscope system 1, the position on the subject that the user can first observe using the microscope system 1. That is, it is the position of the field of view of the microscope system 1 when the user himself can operate it or the center position of the field of view. In addition, the mapping area is an area that becomes the object of the mapping image, usually an area where the specimen exists or an area where the specimen is assumed to exist. The mapping image is an image displayed in order to grasp the relationship between the overall image of the specimen and the current observation position when the user observes. The mapping image is an image constructed by stitching a plurality of images acquired at an observation magnification higher than the observation magnification of the macro image acquisition unit 200. In addition, the plurality of images acquired to construct the mapping image are, for example, images acquired at the observation magnification (second magnification) at the start of observation, but are not limited to this. The plurality of images acquired to construct the mapping image may be images acquired at a higher magnification than the macro image (first magnification) and may be images acquired at a magnification lower than the observation magnification (second magnification) at the start of observation.

[0099] In step S23, the computer 10 determines the observation start position based on the result of the object detection performed in step S22. Specifically, the computer 10 may determine the observation start position based on at least the detection result of the structure corresponding to the type of the specimen container detected in step S22. More preferably, the computer 10 determines the observation start position based on the detection result of the structure corresponding to the type of the specimen container detected in step S22 and the detection result of the specimen in the specimen container. That is, when the specimen can be detected, the computer 10 uses the information of the position and size of the specimen to determine the position where the specimen exists as the observation start position. When the specimen cannot be detected, the computer 10 uses the information of the position and size of the structure of the specimen container to estimate the position where the specimen exists, and determines the estimated position as the observation start position.

[0100] For example, if the type of the specimen container is a slide glass, the computer 10 determines the observation start position from within the cover glass detected in step S22. More specifically, Fig.10 (a) Fig.10 (b) and Fig.10 As shown in (d) of FIG. 1 , if both the cover glass and the specimen are detected in step S22, the computer 10 determines the center of the detected specimen as the observation start position. Fig.10As shown in (d) of the example, if multiple specimens are detected, the center of any specimen can be determined as the observation start position. In addition, since the observation start position can be determined as a position on the specimen, the computer 10 can also determine these positions (center of gravity position, position of the lesion) as the observation start position based on further identifying the center of gravity position of the specimen and the lesion part in the specimen. In addition, as Fig.10 As illustrated in (c) of FIG. 1 , if only the cover glass is detected in step S22 , the computer 10 determines the center of the cover glass where the specimen is likely to exist as the observation start position.

[0101] For example, if the type of the specimen container is a culture dish, the computer 10 determines the observation start position from within the culture dish, and if a hole is also detected in step S22, the observation start position is determined from within the hole. More specifically, if Fig.11 (b) and Fig.11 As shown in (c) of FIG. 1 , if a hole is detected in step S22, the computer 10 determines the center of the detected hole as the observation start position. Fig.11 As illustrated in (a) of FIG. 1 , if the hole portion is not detected in step S22 , the computer 10 determines the center of the culture dish as the observation start position.

[0102] For example, if the type of the specimen container is a multi-well plate, the computer 10 determines the observation start position from the holes of the multi-well plate. More specifically, the computer 10 selects a hole from the plurality of holes detected in step S22, and determines the center of the hole as the observation start position. The selection of the hole may be made according to a predetermined rule, such as selecting the hole at the center of the multi-well plate, or may be made based on the detection result of the object detection, such as selecting the hole where the specimen is detected.

[0103] For example, if the type of the specimen container is another specimen container, the computer 10 determines the center of the opening 151 of the universal container support 150 as the observation start position. In addition, since the position of the opening 151 of the universal container support 150 is known, the object detection of step S22 can be omitted when the type of the specimen container is another specimen container. However, since the opening 151 can also be estimated based on the distribution of pixel values ​​of the macro image, the observation position can be determined using the detection result of the object detection even when the type of the specimen container is another specimen container hole.

[0104] In step S24, the computer 10 determines the mapping area based on the result of the object detection performed in step S22. Specifically, the computer 10 may determine the mapping area based on at least the detection result of the structure corresponding to the type of the specimen container detected in step S22. More preferably, the computer 10 determines the mapping area based on the detection result of the structure corresponding to the type of the specimen container detected in step S22 and the detection result of the specimen in the specimen container. That is, when the specimen can be detected, the computer 10 uses the information of the position and size of the specimen to determine the area where the specimen exists as the mapping area. When the specimen cannot be detected, the computer 10 uses the information of the position and size of the structure of the specimen container to estimate the area where the specimen exists and determines the estimated area as the mapping area.

[0105] For example, if the type of the specimen container is a glass slide, the computer 10 determines the mapping area from within the glass cover detected in step S22. Fig.10 (a) Fig.10 (b) and Fig.10 As shown in (d) of FIG. 1 , if both the cover glass and the specimen are detected in step S22, the computer 10 determines the range of the detected specimen (range R2, range R4, range R8) as the mapping area. In addition, the computer 10 may determine the range of the lesion as the mapping area based on further identifying the lesion in the specimen. Fig.10 As illustrated in (c) of FIG. 1 , if only the cover glass is detected in step S22 , the computer 10 determines the entire or a portion of the range of the cover glass as the mapping area.

[0106] For example, if the type of the specimen container is a culture dish, the computer 10 determines the mapping area from within the culture dish, and if a hole is also detected in step S22, the computer 10 determines the mapping area from within the hole. Fig.11 (b) and Fig.11 As shown in (c) of FIG. 1 , if a hole is detected in step S22, the computer 10 determines the entire range of the detected hole (range R11, range R12) or a predetermined range from the center of the hole as the mapping area. Fig.11 As illustrated in (a) of FIG. 1 , if the hole is not detected in step S22 , the computer 10 determines a predetermined range from the center of the culture dish as the mapping area.

[0107] For example, if the type of the specimen container is a multi-well plate, the computer 10 determines the mapping area from within the wells of the multi-well plate. More specifically, the computer 10 determines the entire range of the well selected in step S23 or a predetermined range from the center of the well as the mapping area.

[0108] For example, if the type of the specimen container is another specimen container, the computer 10 determines the mapping area from within the range of the opening 151 of the universal container support 150. More specifically, the computer 10 may determine the entire range of the opening 151 or a portion thereof as the mapping area.

[0109] When the macro-image analysis process is completed, the microscope system 1 changes the settings of the microscope system 1 to settings suitable for observation by the user (step S7 ). Fig.13 It is a diagram showing another example of a table for selecting parameters of the electric stage 101 . Fig.14 It is a diagram showing still another example of a table for selecting parameters of the electric stage 101 . Fig.15 is a diagram showing an example of a table for selecting an observation method for microscopic observation. Figures 13 to 15 A specific example of setting change will be described.

[0110] In step S7, the computer 10 may update the parameters of the electric stage 101 based on the detection result of the object detection. If information related to the size of the specimen container is obtained through object detection, such as Fig.13 As shown, instead of referring to Table T1 in step S3, Tables T2 to T5 can be referred to to read out a parameter group that takes into account the size of the specimen container in addition to the type of the specimen container and set it to the microscope system 1. In addition, if information related to the type of the specimen is obtained by object detection, such as Fig.14 As shown, the table T6 may be referred to instead of the table T1 referred to in step S3 , and a parameter set taking the type of the specimen into consideration may be read out and set to the microscope system 1 .

[0111] In addition, the parameter group PS5 stored in Table T2 for slides and the parameter group PS15 stored in Table T5 for other specimen containers in Tables T2 to T5 are respectively the same as the parameter group PS1 for slides and the parameter group PS4 for other specimen containers stored in Table T1. In contrast, different parameter groups (parameter groups PS6 to PS9) are stored in Tables T3 and T4 according to the size of the specimen container. Specifically, in Table T3 for culture dishes, a parameter group is stored for making the electric stage 101 move more gently or slowly as the diameter of the culture dish increases. This is because the liquid surface of a culture dish with a large diameter tends to shake more gently and greatly than that of a culture dish with a small diameter, which has a greater adverse effect on the image quality. In addition, in Table T4 for multi-well plates, a parameter group (parameter group PS10 to PS14) is stored for making the electric stage 101 move more gently or slowly as the number of holes decreases (that is, the holes become larger). This is because the liquid surface tends to fluctuate more easily and greatly when the number of holes is small than when the number of holes is large, which has a greater adverse effect on image quality.

[0112] In addition, parameter groups are stored in Table T6 according to the type of specimen. Specifically, parameter group PS16 for floating specimens, parameter group PS17 for bonded specimens, and parameter group PS18 for specimens not in liquid are stored, and the values ​​shown in these parameter groups make the electric stage 101 move faster or more violently in the above order. This is because specimens not in liquid are not easily shaken by the movement of the electric stage 101, so even if the electric stage 101 moves at a high speed, the impact on image quality is small. In addition, compared with floating specimens, bonded specimens are not easily shaken by the shaking of the liquid, so the impact on image quality caused by the movement of the electric stage 101 can be suppressed to a small extent.

[0113] In step S7, the computer 10 may also change the setting of the observation method of the microscope 100 based on the detection result of the object detection, and the microscope controller 20 may arrange the objective lens and the condenser lens corresponding to the observation method on the optical path according to the instruction of the computer 10. If the staining state of the specimen is detected by the object detection, the computer 10 may refer to the Fig.15 Table T7 shown here changes the setting of the observation method of the microscopic image acquisition unit 300 to an observation method according to the detection result of the staining state.

[0114] In Table T7, parameter groups representing the desired observation methods are stored according to the staining state of the specimen. Specifically, parameter group PS21 for stained specimens, parameter group PS22 for unstained specimens, and parameter group PS23 for when the specimen is not detected are stored. In addition, parameter group PS21 represents bright field observation. This is because, if it is a stained specimen, it can be observed by bright field observation. In addition, parameter group PS22 represents phase difference observation. This is because, if it is an unstained specimen, it is believed that better observation can be performed by phase difference observation compared to bright field observation. In addition, parameter group PS23 represents phase difference observation. This is because it is difficult to perform good observation by bright field observation when the specimen is not detected, and it is desired to observe by phase difference observation.

[0115] When the setting change of step S7 is completed, the microscope system 1 updates the macro image display (step S8). Here, the computer 10 displays a macro image, a display indicating the observation start position (cross line CL), and a display indicating the range of the mapping area (bounding box BB) on the display device 14. Fig.16 Screen 400a is shown.

[0116] Fig.16 This is an example of a screen displayed on the display device 14 in step S8. Fig.16 The screen 400a shown is different from the screen 400 in that it includes a microscopic image area 410, a mapping image area 420, a macroscopic image area 430, and an operation area 440. Figure 6 The screen 400a is different from the screen 400 in that a macro image M2 is displayed in the macro image area 430 instead of the macro image M1, and a display (cross line CL) indicating the observation start position and a display (bounding box BB) indicating the range of the mapping area are superimposed on the macro image M2.

[0117] In addition, the macro image M2 is an image after labeling the macro image M1. Fig.16 As shown, the information recorded in the labels 167, 168, and 169 can be visually confirmed, which is different from the macro image M1. In addition, in the screen 400a, the crosshairs CL are illustrated as a display indicating the observation start position, but the display method of the observation start position is not particularly limited. The observation start position can be displayed by other methods. In addition, in the screen 400a, the bounding box BB is illustrated as a display indicating the range of the mapping area, but the display method of the mapping area is not particularly limited. The mapping area can be displayed by other methods.

[0118] After that, the microscope system 1 moves the specimen container disposed on the container support holding portion 110 to the microphotography position (step S9). Here, the microscope controller 20 is controlled by the computer 10, whereby the microscope controller 20 moves the electric stage 101 in such a manner that the container support holding portion 110 is located on the optical axis of the microscopic image acquisition portion 300. In more detail, the computer 10 controls the relative position of the microscopic image acquisition portion 300 with respect to the specimen in such a manner that the observation start position determined by step S23 is included in the field of view range of the microscopic image acquisition portion 300 at the second magnification. It is desirable that the relative position is controlled in such a manner that the observation start position determined by step S23 is located at the center of the field of view range. In addition, at this time, the microscope controller 20 controls the movement of the electric stage 101 according to the setting of step S7 regardless of the type of the specimen container.

[0119] When moving to the micro-photography position, the microscope system 1 performs focusing (step S10). The focusing method is not particularly limited, and the focus can be adjusted by moving the lens changer 306 to a position predetermined by the specimen container, or by using a known automatic focusing technique. In addition, when the specimen container is a multi-well plate, the height to the bottom surface of the hole is prone to deviation due to manufacturer and individual differences. Therefore, it is desirable to use an automatic focusing technique rather than moving the lens changer 306 to a predetermined position.

[0120] When the focusing at the microphotography position is completed, the microscope system 1 starts real-time display of the microscopic image (step S11). Here, the computer 10 controls the camera 307 via the microscope controller 20, so that the microscopic image acquisition unit 300 acquires the microscopic image at a specified frame rate. In addition, the display device 14 displays the acquired microscopic image M3. Fig.17 Screen 400b is shown.

[0121] Fig.17 This is an example of the screen displayed on the display device 14 in step S11. Fig.17 The screen 400b shown is different from the screen 400 in that it includes a microscopic image area 410, a mapping image area 420, a macroscopic image area 430, and an operation area 440. Fig.16 The screen 400 b is the same as the screen 400 a shown in FIG. 4. The screen 400 b is different from the screen 400 a in that the microscopic image M3 is displayed in the microscopic image area 410 .

[0122] When the real-time display starts, the user can operate the microscope system 1, and the microscope system 1 temporarily enters a state of waiting for the user's input. That is, when the user instructs the microscope system 1 to start observation, the microscope system 1 performs a series of processes from step S1 to step S11 to display a screen including a macro image and a micro image on the display device 14. In this way, the user can start observing the specimen from the observation position corresponding to the specimen container without performing troublesome operations.

[0123] The user, while viewing the screen 400b, determines whether the mapping area shown in the boundary box BB is appropriate, and adjusts the mapping area as needed. In addition, the user can also adjust other conditions for mapping image photography. The microscope system 1 determines whether the user has indicated a setting change of the mapping image photography condition (step S12), and if a setting change is indicated, changes the setting according to the instruction (step S13). In addition, here, the changed mapping image photography conditions include, in addition to the mapping area, various settings of the microscopic image acquisition unit 300, such as observation magnification, observation method, objective lens used in photography, illumination light amount, exposure time, exposure mode, gain, resolution, etc.

[0124] The mapping area can be adjusted by the user specifying an arbitrary range as the mapping area, or by the user selecting a mapping area candidate from a plurality of different mapping area candidates determined by the microscope system 1 as the mapping area. Fig.10As shown in (d), if multiple tissues MT are detected by object detection, each of the multiple regions corresponding to the multiple tissues MT is set as a mapping region candidate, and one region can be selected from these multiple mapping region candidates as a mapping region. In addition, two or more regions corresponding to two or more tissues can be selected from the multiple mapping region candidates as a mapping region. In this case, it is desirable to select a region newly created by connecting two or more separated regions as a mapping region.

[0125] When the user instructs to shoot the mapping image, the microscope system 1 starts shooting the mapping image (step S14). Here, the computer 10 controls the microscope controller 20 to move the field of view of the microscopic image acquisition unit 300 to each position of the mapping area in sequence, so that the microscopic image acquisition unit 300 acquires a microscopic image at each position, thereby shooting the mapping area without gaps. The computer 10 also constructs a mapping image by splicing a plurality of microscopic images acquired by the microscopic image acquisition unit 300.

[0126] Furthermore, in step S14, the computer 10 determines the position at which the microscopic image acquisition unit 300 acquires the microscopic image in such a manner that the acquired multiple microscopic images overlap each other by a fixed amount or more. By determining the photographing position in this way, the computer 10 can well correct the positional deviation that may occur when stitching the microscopic images. In determining the stitching position in which the positional deviation is well corrected, a known template matching method or the like can be used, and in addition, so-called feathering that synthesizes images with weights corresponding to the distance from the center of the image can be used to stitch the images.

[0127] In addition, in step S14, the computer 10 can Figures 18 to 20 The method for determining the order of movement between the imaging positions (i.e., the method for scanning the mapping area) is selected from among the plurality of methods shown. The computer 10 may, for example, Fig.18 The tornado mode shown in FIG. 1 moves the field of view FV of the microscopic image acquisition unit 300. In the tornado mode, the field of view FV can be moved from the inside to the outside while rotating clockwise or counterclockwise from the center of the mapping area. Fig.19 The bending mode shown in FIG. 1 is to move the field of view FV. In the bending mode, the field of view FV is moved from one end to the other end of the mapping area in the sub-scanning direction, and then the field of view FV is displaced in the main scanning direction. The movement is repeated in such a way that the direction of movement in the sub-scanning direction is switched to the opposite direction each time the movement is repeated. In addition, the computer 10 can be, for example, Fig. 20The conventional raster scanning mode as shown in the figure moves the field of view FV. In addition, no matter which mode is used, the user can end the mapping image photography at any time. In the case where the mapping image photography is terminated halfway, the computer 10 can splice the completed images to construct the mapping image. In addition, after the mapping image photography is completed, the computer 10 controls the relative position of the microscopic image acquisition unit 300 with respect to the specimen in such a way that the observation start position determined by step S23 is included in the field of view range of the microscopic image acquisition unit 300. In this way, the current observation position can be returned to the observation position before the mapping image photography was started.

[0128] When the mapping image photography is completed, the microscope system 1 displays the mapping image (step S15), and the process ends. Figure 4 Here, the computer 10 displays the mapping image M4 constructed in step S14 on the display device 14. Fig.21 Screen 400c is shown.

[0129] Fig.21 This is an example of the screen displayed on the display device 14 in step S15. Fig.21 The screen 400c shown is different from the screen 400 in that it includes a microscopic image area 410, a mapping image area 420, a macroscopic image area 430, and an operation area 440. Fig.17 The screen 400c is the same as the screen 400b shown. The screen 400c is different from the screen 400b in that the mapping image M4 is displayed in the mapping image area 420. In addition, it is desirable that the mapping image M4 includes a display CP indicating the current observation position. Thus, the user can easily understand which part of the entire specimen is being observed.

[0130] Fig. 22 This is another example of the screen displayed on the display device 14 in step S8. Fig.16 4 shows an example in which a display (cross line CL) indicating the observation start position and a display (bounding box BB) indicating the range of the mapping area are overlapped and displayed on the macro image M2 updated in step S8, but the adjustment operation of the mapping area by the user can be performed on the mapping image area 420 instead of the macro image area 430. For example, in step S8, in addition to updating the macro image M2, an image of the specimen container including the mapping area (hereinafter referred to as macro image M5) is extracted from the macro image M2, and the following is performed: Fig. 22The process of displaying the extracted macro image M5 in the mapping image area 420 is shown. In this case, a display (bounding box BB) indicating the range of the mapping area can be displayed overlappingly on the displayed macro image M5. The user can determine whether the mapping area is appropriate by confirming the macro image M5 overlapped with the boundary box BB, and can also manually adjust the mapping area as needed. In this way, by displaying a part of the macro image M2 and the boundary box together in the mapping image area 420 before shooting the mapping image, the mapping area can be further enlarged and displayed, so that the area within the screen 400d can be effectively used to confirm whether the mapping area is appropriate and adjust the mapping area.

[0131] In addition, it is sufficient to display an image of one specimen container in the mapping image area 420. When the macro image area 430 includes images of a plurality of specimen containers, an image of any specimen container selected by the user may be displayed in the mapping image area 420, and the user may specify a mapping area from the displayed images of the specimen containers.

[0132] Fig.23 This is another example of the screen displayed on the display device 14 in step S15. Fig.21 4 shows an example in which the mapping image M4 corresponding to the mapping area is displayed in an enlarged or reduced manner according to the size of the mapping image area 420, but the mapping image displayed in the mapping image area 420 may include an image outside the mapping area. Fig. 22 As shown, in the case where the macro image M5 is displayed in the mapping image area 420 before the mapping image is captured, the mapping area in the macro image M5 can be updated by the captured mapping image after the mapping image is captured, thereby displaying the mapping image M6 in the mapping image area 420. In this case, the mapping image M6 is composed of the mapping image of the mapping area and the macro image outside the mapping area.

[0133] As above, in Figure 4 In the observation preparation process shown, a position suitable for specimen observation is determined by using information such as the type of specimen container, and the field of view is moved to the determined position. Therefore, the manual specimen search performed by the user in the conventional microscope system can be saved, and the user can start observing the specimen using the microscope system 1 without performing complicated operations. Figure 4In the observation preparation process shown, the range where the specimen exists is determined by using information such as the type of specimen container, and the determined range is proposed to the user as the range where the mapping image should be acquired, that is, the mapping area. Therefore, in most cases, the user can make the microscope system 1 construct the mapping image only by confirming that the mapping area is properly set, which can save the effort of searching and setting the mapping area. In addition, the mapping image is constructed and displayed on the screen through simple operations, so the user can easily and quickly start the formal observation of the specimen.

[0134] The above-mentioned embodiments are specific examples for facilitating understanding of the invention, and the embodiments of the present invention are not limited to these. The microscope system, the control method, and the computer-readable medium can be variously modified and changed without departing from the scope of the claims.

[0135] exist Figure 1 , a microscope system 1 having a macro image acquisition unit 200 and a micro image acquisition unit 300 is illustrated, but the microscope system only needs to have an image acquisition unit that acquires images at least at a first magnification and a second magnification higher than the first magnification. Therefore, it is not necessary to have a macro image acquisition unit and a micro image acquisition unit independently, and for example, Fig.24 2, a microscope 500 is provided for obtaining macroscopic images and microscopic images by switching objective lenses for shooting. In addition, the microscope system 2 includes a light source 501, a condenser lens 502, a plurality of objective lenses (objective lenses 504 to 506), a revolving lens 507, a camera device 508, etc., and can obtain images by a plurality of observation methods. The same observation preparation process as the microscope system 1 can also be performed by the microscope system 2, so that the user can start observing the specimen without performing complicated operations. In addition, the microscope included in the microscope system can be a scanning microscope.

[0136] In addition, Figure 4 2 shows an example in which identification information for identifying the type of the specimen container is acquired in step S2 after the user instructs to start observation, and the type of the specimen container is determined based on the identification information. However, the process of acquiring the identification information to determine the type of the specimen container may be performed before the user instructs to start observation.

[0137] In addition, Figure 4, an example is shown in which the identification information is obtained by the detection unit provided in the microscope system 1 detecting the identification structure provided in the container support or the specimen container in step S2, but the identification information can also be obtained by the user operating the input device. The computer 10 can, for example, obtain information input by the user inputting the type of the specimen container using the input device as the identification information. In addition, an example is shown in which the detection unit is a sensor such as a Hall element in step S2, but the detection unit can also be an image acquisition unit. For example, the identification information can be a macro image acquired by the image acquisition unit, and the computer 10 can identify the type of the specimen container based on the macro image.

[0138] In addition, Figure 4 , the type of the specimen container is determined after the specimen is moved to the macro imaging position. However, the process of determining the type of the specimen container may be performed at least before the macro image analysis process. Therefore, for example, the type of the specimen container may be determined before the specimen is moved to the macro imaging position.

[0139] In addition, Figure 7 , the labeling process is performed only when the specimen container is a slide glass, but the labeling process can also be performed on specimen containers other than slide glass. For example, the label attached to the area other than the well of the multi-well plate can be used as the object for labeling process.

Claims

1. A microscope system, characterized in that: have: an image acquisition unit that acquires an image at least at a first magnification and a second magnification higher than the first magnification; and Control Department, The control unit determines the type of container in which the specimen is placed, and when starting to observe the specimen placed in the container at the second magnification, the control unit performs object detection of a structure corresponding to the type of the container on the first image including the container acquired by the image acquisition unit at the first magnification, determines an observation start position based on the type of the container and a result of the object detection, and controls the relative position of the image acquisition unit with respect to the specimen in such a way that the observation start position is included in the field of view of the image acquisition unit at the second magnification.

2. The microscope system according to claim 1, characterized in that The control unit detects the specimen placed in the container based on the first image, and determines the observation start position based on the type of the container, the detection result of the structure, and the detection result of the specimen.

3. The microscope system according to claim 1 or 2, characterized in that: Regarding the structure corresponding to the type of the container, When the container is a glass slide, the structure is a glass cover slide. When the type of the container is a culture dish, the structure is a hole provided in the culture dish, When the type of the container is a well plate, the structure is a well.

4. The microscope system according to claim 1 or 2, characterized in that: When the container is a glass slide, The control unit determines the observation start position from within a cover glass overlapped on the slide glass, and controls the relative position of the image acquisition unit with respect to the specimen in such a manner that the observation start position determined from within the cover glass is included in the field of view range at the second magnification.

5. The microscope system according to claim 1 or 2, characterized in that: When the type of the container is a culture dish, The control unit determines the observation start position from within the petri dish and controls the relative position so that the observation start position determined from within the petri dish is included in the field of view range at the second magnification.

6. The microscope system according to claim 1 or 2, characterized in that: When the type of the container is a culture dish, The control unit determines the observation start position from within a hole provided in the culture dish and controls the relative position so that the observation start position determined from within the hole is included in the field of view range at the second magnification.

7. The microscope system according to claim 1 or 2, characterized in that: When the type of the container is a multiwell plate, The control unit determines the observation start position from within a well provided in the multi-well plate and controls the relative position so that the observation start position determined from within the well is included in the field of view range at the second magnification.

8. The microscope system according to claim 1 or 2, characterized in that: The control unit detects a staining state of the specimen placed in the container based on the first image, and changes a setting of an observation method of the image acquisition unit to an observation method according to a detection result of the staining state.

9. The microscope system according to claim 1 or 2, characterized in that: further comprising a detection unit configured to output identification information for identifying the type of the container by detecting an identification structure provided on a container support holding the container or the container, The control unit specifies the type of the container based on the identification information output from the detection unit.

10. The microscope system according to claim 9, characterized in that The identification structure is a magnet, The detection unit is a magnetic sensor.

11. The microscope system according to claim 1 or 2, characterized in that: The control unit determines the type of the container based on the first image.

12. A control method for a microscope system including an image acquisition unit that acquires images at least at a first magnification and a second magnification higher than the first magnification, the control method comprising: The computer determines the type of container in which the specimen is placed. When the computer starts observing the specimen placed in the container at the second magnification, the computer performs object detection of a structure corresponding to the type of the container on the first image including the container acquired by the image acquisition unit at the first magnification, determines an observation start position according to the type of the container and the result of the object detection, and controls the relative position of the image acquisition unit with respect to the specimen in such a manner that the observation start position is included in the field of view of the image acquisition unit at the second magnification.

13. A computer-readable medium storing a program, wherein the program causes a computer of a microscope system including an image acquisition unit that acquires an image at least at a first magnification and a second magnification higher than the first magnification to execute the following processing: Determine the type of container in which the specimen is placed, When observation of the specimen placed in the container is started at the second magnification, object detection of a structure corresponding to the type of the container is performed on the first image including the container acquired by the image acquisition unit at the first magnification, and an observation start position is determined according to the type of the container and the result of the object detection, so that the relative position of the image acquisition unit with respect to the specimen is controlled in such a way that the observation start position is included in the field of view of the image acquisition unit at the second magnification.

Images