Observation system and observation support method

Abstract

To provide auxiliary information, together with an optical image, to users without adding significant changes to the configuration of an existing observation device.SOLUTION: An observation system 1000 includes an observation device 100 and an auxiliary observation device 200. The observation device 100 includes an eyepiece lens 102 and an objective lens 101, and forms the real image of a sample on an optical path between the eyepiece lens 102 and the objective lens 101. The auxiliary observation device 200 is worn on a user and outputs auxiliary information. The auxiliary observation device 200 stacks auxiliary information on top of a virtual image of the sample observed by a user via the eyepiece lens 102, on the basis of the relative position of the auxiliary observation device 200 to the observation device 100.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The disclosure of this specification relates to an observation system and an observation support method. [Background technology]

[0002] In pathological diagnosis, a technology is known that displays an image in which auxiliary information based on the results of image analysis is superimposed on a digital image of a sample on a display device to assist the pathologist in his or her diagnosis. This technology for displaying auxiliary information is applicable not only to pathological diagnosis but also to various other fields, and it is expected that its use will become more widespread as image analysis technology advances.

[0003] On the other hand, there is still a need to view optical images of a sample through an eyepiece, because digital images are generally inferior to optical images in terms of color reproduction and dynamic range.

[0004] A technology related to such a problem is described, for example, in Patent Document 1. By using the technology described in Patent Document 1, auxiliary information is superimposed on an optical image, so that it is possible to obtain necessary support from the auxiliary information while observing a sample using the optical image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 231204 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in order to utilize the technology described in Patent Document 1, it is necessary to incorporate a projection device into an existing observation device, which requires significant changes to the configuration of the existing observation device.

[0007] In light of the above-described circumstances, an object of one aspect of the present invention is to provide a technique for providing auxiliary information to a user together with an optical image without making significant changes to the configuration of an existing observation device. [Means for solving the problem]

[0008] An observation system according to one aspect of the present invention comprises an observation device that includes an eyepiece and an objective lens and forms a real image of a sample on an optical path between the eyepiece and the objective lens, and an observation auxiliary device that is worn by a user and outputs auxiliary information to the user, superimposing the auxiliary information on a virtual image of the sample observed by the user through the eyepiece based on the relative position of the observation auxiliary device with respect to the observation device.

[0009] In an observation assistance method according to one aspect of the present invention, an observation device including an eyepiece and an objective lens forms a real image of a sample on an optical path between the eyepiece and the objective lens, and an observation assistance device worn by a user outputs auxiliary information to the user and superimposes the auxiliary information on a virtual image of the sample observed by the user through the eyepiece based on the relative position of the observation assistance device with respect to the observation device. Effect of the Invention

[0010] According to the above aspect, it is possible to provide auxiliary information to a user together with an optical image without making significant changes to the configuration of an existing observation device. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a system 1000 according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a configuration of a superimposed image provided to a user. [Diagram 3] 10 is a flowchart showing an example of processing performed in the system 1000. [Figure 4] 13 is a flowchart showing an example of auxiliary information acquisition processing performed in the system 1000. [Diagram 5] 10 is a flowchart showing an example of a relative position detection process performed in the system 1000. [Figure 6] 13 is a flowchart showing an example of auxiliary information output processing performed in the system 1000. [Figure 7] 7 is a diagram showing an example of adjustment of the output position of auxiliary information 3 by the auxiliary information output process shown in FIG. 6. FIG. [Figure 8] 13 is a flowchart showing another example of the auxiliary information output process performed in the system 1000. [Figure 9] 9 is a diagram showing an example of adjustment of the output position of auxiliary information 3 by the auxiliary information output process shown in FIG. 8. FIG. [Figure 10] FIG. 1 is a diagram illustrating a functional configuration of a system 1000 according to a first embodiment. [Figure 11] 2 is a diagram illustrating an example of the configuration of an output unit 220. FIG. [Figure 12] 13 is a diagram illustrating another example of the configuration of the output unit 220. FIG. [Figure 13] 13 is a diagram illustrating yet another example of the configuration of the output unit 220. FIG. [Figure 14] FIG. 11 is a diagram illustrating a functional configuration of a system according to a second embodiment. [Figure 15] FIG. 13 is a diagram illustrating a functional configuration of a system according to a third embodiment. [Figure 16] 2 is a diagram illustrating an example of the configuration of an output unit 260 and an imaging unit 270. FIG. [Figure 17] 13 is a diagram showing an example of an image captured by the imaging section 270. FIG. [Figure 18] FIG. 13 is a diagram illustrating a functional configuration of a system according to a fourth embodiment. [Figure 19] 13 is a diagram showing an example of output of auxiliary information in the observation auxiliary device 200c. FIG. [Figure 20] FIG. 13 is a diagram illustrating a functional configuration of a system according to a fifth embodiment. [Figure 21] 13A and 13B are diagrams showing an example of output of auxiliary information in the observation auxiliary device 200d. [Figure 22]FIG. 13 is a diagram showing another example of a superimposed image. [Figure 23] FIG. 13 is a diagram showing yet another example of a superimposed image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [First embodiment] Fig. 1 is a diagram for explaining the configuration of a system 1000 according to this embodiment. Fig. 2 is a diagram for explaining the configuration of a superimposed image provided to a user. The configuration of the system 1000 will be explained below with reference to Figs. 1 and 2.

[0013] 1, the system 1000 includes an observation device 100 and an observation assistant device 200 capable of communicating with the observation device 100. In the following, a case will be described in which communication between the observation device 100 and the observation assistant device 200 is realized by wireless communication, but communication between the observation device 100 and the observation assistant device 200 may also be realized by wired communication.

[0014] The observation device 100 is, for example, a microscope device. The observation device 100 is equipped with an objective lens 101 and an eyepiece lens 102. The observation device 100 forms an optical image (real image) of a sample on an optical path between the objective lens 101 and the eyepiece lens 102. A user of the observation device 100 observes an optical image 1 (virtual image) that is an enlarged version of the optical image (real image) formed by the observation device 100, through the eyepiece lens 102, as shown in FIG. 2.

[0015] The observation device 100 may also include a digital camera 103. The observation device 100 may use a triplet tube to branch light from the objective lens 101 into an optical path that guides the light to an eyepiece lens 102 and an optical path that guides the light to the digital camera 103. The digital image 2 acquired by the digital camera 103 is output to, for example, a control device 105 that controls the microscope body.

[0016] In the observation device 100, the control device 105 may generate auxiliary information 3 based on the result of image analysis performed on the digital image 2. The auxiliary information 3 may be information that displays the outline of a specific object (e.g., a cell, a cell nucleus, etc.) contained in the sample as shown in FIG. 2, and may be generated based on the result of object detection performed on the digital image 2, for example.

[0017] The observation device 100 may further include a transmitter 104 for position detection. The transmitter 104 may be used to detect the relative position of the observation assistance device 200 with respect to the observation device 100 by combining with a receiver 201 described later. Hereinafter, the relative position of the observation assistance device 200 with respect to the observation device 100 is simply referred to as the relative position. In order to know how far the user's eyes are from the eye point, it is preferable that the transmitter 104 is provided in the eyepiece 102 or in the vicinity of the eyepiece 102 as shown in FIG. 1. It is preferable that the transmitter 104 emits a signal that the user cannot sense, for example, at least one of electromagnetic waves outside the visible range or ultrasonic waves. The transmitter 104 may be a magnetic field generating device that generates a magnetic field.

[0018] The observation auxiliary device 200 is, for example, smart glasses. The observation auxiliary device 200 is a device worn by a user of the observation device 100, and outputs auxiliary information to the user when the user uses the observation device 100. It is sufficient for the observation auxiliary device 200 to be able to output the auxiliary information in a manner that allows the user to visually recognize the auxiliary information. More specifically, it is sufficient for the observation auxiliary device 200 to be able to output the auxiliary information in a manner that allows the user to visually recognize the auxiliary information while observing an optical image of a sample through the eyepiece 102.

[0019] For this reason, the observation assistance device 200 is not limited to smart glasses, and may include, for example, a see-through head mounted display, or may include a contact lens type device (smart contact lens) that is directly attached to the eyeball, or may include at least one of these. The method of outputting the auxiliary information to the user is not particularly limited. The observation assistance device 200 may include, for example, at least one of a retinal projection type, a retinal scanning type, a hologram type, or a micro LED type device, and may output auxiliary information to the user by each of the methods.

[0020] 2, based on the relative position, the observation auxiliary device 200 superimposes auxiliary information 3 on the optical image 1 (virtual image) observed by the user through the eyepiece 102. As a result, the user observes a superimposed image 4 in which the auxiliary information 3 is superimposed on the optical image 1. By arranging the auxiliary information 3 at an appropriate position with respect to the optical image 1 using the relative position, the user can obtain appropriate information from the auxiliary information 3 included in the superimposed image 4.

[0021] The observation assistance device 200 may include a receiver 201 for position detection. The receiver 201 may be used in combination with the above-mentioned transmitter 104 to detect the relative position. The method of position detection by the transmitter 104 and the receiver 201 is not particularly limited. For example, the observation device 100 may be provided with three or more transmitters 104, and the positioning method may be three-point positioning using signals from those transmitters 104. Furthermore, the observation assistance device 200 may be provided with three or more receivers 201, and the positioning method may be three-point positioning using signals received by those receivers 201. The observation device 100 and the observation assistance device 200 may detect their relative positions by three-point positioning.

[0022] Fig. 3 is a flowchart showing an example of processing performed by the system 1000. Fig. 4 is a flowchart showing an example of auxiliary information acquisition processing performed by the system 1000. Fig. 5 is a flowchart showing an example of relative position detection processing performed by the system 1000. Fig. 6 is a flowchart showing an example of auxiliary information output processing performed by the system 1000. Fig. 7 is a diagram showing an example of adjustment of the output position of auxiliary information 3 by the auxiliary information output processing shown in Fig. 6. An observation support method performed by the system 1000 will be described with reference to Figs. 3 to 7.

[0023] 3, the observation device 100 forms an optical image of a sample placed on a stage (step S10). Here, the observation device 100 forms an optical image of the sample by collecting light from the sample using an objective lens 101 and an imaging lens (not shown). The optical image is formed, for example, on an optical path leading to an eyepiece lens 102 that branches in a triplet tube, and on an imaging surface of a digital camera 103. The optical image formed on the optical path leading to the eyepiece lens 102 is observed by a user through the eyepiece lens 102.

[0024] Moreover, the system 1000 acquires auxiliary information (step S20). The system 1000 may acquire auxiliary information by performing, for example, the auxiliary information acquisition process shown in FIG. 4. When the process shown in FIG. 4 is started, the observation device 100 acquires a digital image (step S21). In step S21, the digital camera 103 converts an optical image formed on the imaging surface into a digital image and outputs it to the control device 105, so that the observation device 100 acquires the digital image. Next, the observation device 100 analyzes the digital image (step S22). In step S22, the control device 105 performs, for example, object detection on the digital image to detect the position of a predetermined object in the digital image. Finally, the observation device 100 generates auxiliary information (step S23). In step S23, the control device 105 generates auxiliary information that depicts the outline of a predetermined object based on, for example, position information of the predetermined object detected by object detection. The generated auxiliary information is transmitted from the observation device 100 to the observation auxiliary device 200.

[0025] 4 shows an example of the auxiliary information acquisition process in which the observation device 100 generates auxiliary information and the observation auxiliary device 200 acquires the auxiliary information from the observation device 100, but the auxiliary information may be generated by the observation auxiliary device 200. The observation auxiliary device 200 may acquire information different from the auxiliary information (e.g., a digital image) from the observation device 100, and generate the auxiliary information based on the acquired information.

[0026] Furthermore, the system 1000 detects the relative position (step S30). The system 1000 may detect the relative position by, for example, performing a relative position detection process shown in FIG. 5. In the process shown in FIG. 5, the observation device 100 transmits a signal (step S31), and the observation assistance device 200 receives the signal (step S32). More specifically, the transmitter 104 transmits a signal for position detection (hereinafter, referred to as a position signal), and the receiver 201 receives the position signal. Thereafter, the observation assistance device 200 calculates the relative position based on the position signal detected by the receiver 201 (step S33). The relative position may be calculated by any known method.

[0027] 5 shows an example of the relative position detection process in which the observation assistant device 200 receives a signal transmitted by the observation assistant device 100 and calculates the relative position, but the relative position may be calculated by the observation device 100. For example, a transmitter may be provided in the observation assistant device 200 and a receiver may be provided in the observation device 100, and the observation device 100 may calculate the relative position based on the signal received from the observation assistant device 200. In this case, information on the calculated relative position is transmitted from the observation assistant device 100 to the observation assistant device 200.

[0028] When the processes of steps S10 to S30 are completed, the system 1000 outputs the auxiliary information and superimposes the auxiliary information on the optical image (step S40). The order of the processes of steps S10 to S30 is not limited to this example. Each process of steps S10 to S30 may be performed before the process of step S40. The processes of steps S10 to S30 may be performed in an order different from the order of the processes shown in FIG. 3, and one or more processes may be performed in parallel in time.

[0029] In step S40, the system 1000 may superimpose auxiliary information on the optical image by performing, for example, the auxiliary information output process shown in Fig. 6. In the process shown in Fig. 6, the observation auxiliary device 200 calculates the distance and direction from the optical axis of the eyepiece 102 to the observation auxiliary device 200 based on the relative position calculated in step S33 (step S41), and determines the output position of the auxiliary information based on the calculated distance and direction (step S42). This is because even if auxiliary information is output from a fixed position of the observation auxiliary device 200, the position of the auxiliary information relative to the optical image observed by the user through the eyepiece 102 changes depending on the distance and direction from the optical axis of the eyepiece 102 to the observation auxiliary device 200.

[0030] Finally, the observation auxiliary device 200 outputs the auxiliary information acquired in step S23 according to the output position determined in step S42 (step S43). As a result, the output position of the auxiliary information is appropriately changed based on the distance and direction calculated in step S41, so that the auxiliary information can be output at a position according to the position of the observation auxiliary device 200 relative to the optical axis of the eyepiece 102.

[0031] The system 1000 can display the auxiliary information 3 at an appropriate position on the optical image 1 by performing the process shown in FIG. 3 described above. This point will be described with reference to FIG. 7. First, consider a case where the output position of the auxiliary information 3 from the observation auxiliary device 200 does not change. In this case, when a user wearing the observation auxiliary device 200 moves relative to the eyepiece 102 while observing the superimposed image 4 in which the auxiliary information 3 is superimposed at an appropriate position on the optical image 1, the observation auxiliary device 200 also moves relative to the eyepiece 102 together with the user, and as a result, the auxiliary information 3 moves relative to the optical image 1. For this reason, the user will observe the superimposed image 5 in which the auxiliary information 3 is superimposed at a position shifted from the appropriate position on the optical image 1, as shown in FIG. 7. However, in the system 1000, the output position of the auxiliary information 3 from the observation auxiliary device 200 changes based on the distance and direction from the eyepiece 102, thereby eliminating the deviation seen in the superimposed image 5 described above. As a result, even if the user moves relative to the eyepiece 102, the user can observe the superimposed image 6 in which the auxiliary information 3 is superimposed at an appropriate position on the optical image 1, as shown in FIG.

[0032] Furthermore, in the system 1000, the observation device 100 can be configured without making significant changes to an existing observation device. Specifically, the observation device 100 can be configured, for example, by simply adding a transmitter 104 to an existing observation device. Other necessary changes can be realized by software processing performed by the control device 105.

[0033] Therefore, according to the system 1000, it is possible to provide the user with auxiliary information together with the optical image without making significant changes to the configuration of the existing observation device. In particular, since the auxiliary information can be displayed superimposed on an appropriate position on the optical image, the user can correctly recognize the auxiliary information. In addition, it is possible to avoid the device becoming larger than the existing device.

[0034] Furthermore, the observation assistance device 200 does not require significant changes to the configuration of existing smart glasses. The observation assistance device 200 can be configured, for example, by simply adding a receiver 201 to existing smart glasses. Other necessary changes can be realized by software processing performed by a processor included in the observation assistance device 200.

[0035] Therefore, according to the system 1000, existing smart glasses used for other purposes can be used with necessary modifications as the observation assistance device 200 included in the system 1000. Therefore, a user can use one pair of smart glasses in multiple systems including the system 1000, and can avoid the inconvenience of having to change the smart glasses depending on the system being used.

[0036] Fig. 8 is a flowchart showing another example of the auxiliary information output process performed in the system 1000. Fig. 9 is a diagram showing an example of adjustment of the output position of the auxiliary information 3 by the auxiliary information output process shown in Fig. 8. The system 1000 may perform the auxiliary information output process shown in Fig. 8 instead of the auxiliary information output process shown in Fig. 6.

[0037] The observation auxiliary device 200 calculates the distance from the observation device 100 to the observation auxiliary device 200 (hereinafter, referred to as the first distance) based on the relative position calculated in step S33 (step S51), and judges whether the calculated first distance is equal to or greater than a predetermined distance (step S52). This predetermined distance may be a distance indicating that the user (observation auxiliary device 200) is far enough away from the observation device 100 that it can be judged that the user is not looking into the eyepiece 102. More specifically, the predetermined distance is determined in advance based on, for example, the eye relief, which is the distance from the eyepiece 102 to the eye point. That is, in step S52, the observation auxiliary device 200 judges whether the distance from the eyepiece 102 to the user's eye (for example, the pupil) is sufficiently longer than the eye relief.

[0038] When the observation auxiliary device 200 determines in step S52 that the first distance is equal to or greater than the predetermined distance (step S52 YES), it further determines whether the first distance has increased across the predetermined distance when compared with the first distance calculated previously (step S53). When the observation auxiliary device 200 determines that the first distance has increased across the predetermined distance (step S53 YES), it stops outputting the auxiliary information 3 (step S54), and then ends the auxiliary information output process shown in Fig. 8. On the other hand, when the observation auxiliary device 200 determines that the first distance has not increased across the predetermined distance (step S53 NO), it omits step S54 and ends the auxiliary information output process.

[0039] On the other hand, if the observation auxiliary device 200 determines in step S52 that the first distance is not equal to or greater than the predetermined distance (step S52 NO), it calculates the distance (hereinafter referred to as the second distance) and direction from the optical axis of the eyepiece 102 to the observation auxiliary device 200 based on the relative position calculated in step S33 (step S55), and determines the output position of the auxiliary information based on the calculated second distance and direction (step S56). Note that the processes in steps S55 and S56 are similar to those in steps S41 and S42 in FIG. 6.

[0040] Thereafter, the observation auxiliary device 200 judges whether the first distance has decreased across a predetermined distance when compared with the first distance calculated previously (step S57). When the observation auxiliary device 200 judges that the first distance has decreased across a predetermined distance (step S57 YES), it starts outputting the auxiliary information 3 (step S58), outputs the auxiliary information according to the output position determined in step S56 (step S59), and then ends the auxiliary information output process shown in Fig. 8. On the other hand, when the observation auxiliary device 200 judges that the first distance has not decreased across a predetermined distance (step S57 3 NO), it omits step S58, outputs the auxiliary information according to the output position determined in step S56 (step S59), and then ends the auxiliary information output process.

[0041] In the auxiliary information output process shown in FIG. 8, the observation auxiliary device 200 stops outputting the auxiliary information 3 based on the distance from the observation device 100 to the observation auxiliary device 200, and starts outputting the auxiliary information 3 based on the distance from the observation device 100 to the observation auxiliary device 200. More specifically, the observation auxiliary device 200 stops outputting the auxiliary information 3 when the observation auxiliary device 200 changes from a state in which it is located relatively close to the observation device 100 (for example, a state in which the user's eyes are located at the eye point) to a state in which it is located relatively far away from the observation device 100 (for example, a state in which the user's eyes are located away from the eye point), and the stop of output is maintained until the observation auxiliary device 200 changes to a state in which it is located relatively close to the observation device 100. Also, the observation auxiliary device 200 starts outputting the auxiliary information 3 when the observation auxiliary device 200 changes from a state in which it is located relatively far from the observation device 100 to a state in which it is located relatively close, and the auxiliary information 3 is continuously output thereafter until the observation auxiliary device 200 changes to a state in which it is located relatively far from the observation device 100.

[0042] As a result, when the user's eyes are located at the eye point, the auxiliary information 3 is output from the observation auxiliary device 200, so that the user can observe the superimposed image 4 in which the auxiliary information 3 is superimposed on the optical image 1, as shown in FIG. 9. Furthermore, when the user is not looking into the eyepiece 102, the user's eyes are away from the eye point, so the auxiliary information 3 is not output from the observation auxiliary device 200, and the auxiliary information 3 does not unnecessarily obstruct the user's field of vision. Furthermore, even if the user looks into the eyepiece 102 with his or her eyes away from the eye point, the optical image 7 that can be observed through the eyepiece 102 is an incomplete image with vignetting around the field of vision, as shown in FIG. 9. For this reason, there is no substantial adverse effect in that the auxiliary information 3 is not output when the user's eyes are away from the eye point.

[0043] Even when the auxiliary information output process shown in FIG. 8 is performed instead of the auxiliary information output process shown in FIG. 6, the system 1000 can obtain the same effect as when the auxiliary information output process shown in FIG. 6 is performed. In addition, by performing the auxiliary information output process shown in FIG. 8, the auxiliary information 3 is not output from the observation auxiliary device 200 when the user is away from the observation device 100, so that it is possible to avoid a situation in which the auxiliary information 3 is displayed and the user's view is obstructed when the user is not observing a sample with the observation device 100. Furthermore, by performing the auxiliary information output process shown in FIG. 8, the output of the auxiliary information 3 is started simply by approaching the observation device 100 and looking through the eyepiece 102. Therefore, the auxiliary information 3 is automatically output only when the user observes a sample, so that the auxiliary information 3 can be provided at the necessary timing without imposing a burden on the user.

[0044] 10 is a diagram illustrating a functional configuration of a system 1000 according to this embodiment. The functions of the system 1000 will be described below with reference to FIG.

[0045] The observation device 100 included in the system 1000 includes a control unit 110, an image capturing unit 120, a transmission unit 130, and a transmission unit 140, as shown in FIG. 10. The control unit 110 is, for example, configured by the control device 105 shown in FIG. 1. The control unit 110 includes a processor and a memory, and controls the operation of the observation device 100. The image capturing unit 120 is, for example, configured by the digital camera 103 shown in FIG. 1. The image capturing unit 120 acquires a digital image of the sample and outputs it to the control unit 110. The transmission unit 130 is, for example, a wireless communication device, and transmits auxiliary information to the observation auxiliary device 200. Note that the auxiliary information is generated by the control unit 110 by analyzing the digital image captured by the image capturing unit 120, for example, but does not necessarily have to be generated based on the digital image. The transmission unit 140 is, for example, configured by the transmitter 104. The transmission unit 140 transmits a position signal for detecting the relative position.

[0046] The observation assistance device 200 included in the system 1000 includes a control unit 210, an output unit 220, a receiving unit 230, and a receiving unit 240, as shown in Fig. 10. The control unit 210 includes a processor and a memory, and controls the operation of the observation assistance device 200. The output unit 220 outputs auxiliary information. The receiving unit 230 is, for example, a wireless communication device, and receives auxiliary information from the observation device 100. The receiving unit 240 is, for example, configured by a receiver 201. The receiving unit 240 receives a position signal for detecting a relative position.

[0047] 11 to 13 are diagrams for explaining a configuration example of the output unit 220. In the example shown in FIG. 11, the output unit 220 includes a display element 281, a lens 284, and a concave mirror 285. The display element 281 is, for example, a liquid crystal display (LCD). Light from the display element 281 enters the eye 300 from the pupil via the lens 284 and the concave mirror 285, and an image of the display element 281 is projected onto the retina. The device shown in FIG. 11 is an example of a retinal projection type device. The observation assistance device 200 may include a retinal projection type device.

[0048] 12, the output section 220 includes a MEMS (Micro Electro Mechanical Systems) mirror 286 and a concave mirror 285. By controlling the MEMS mirror 286, the retina is scanned with light incident on the eye 300 from the pupil. The device shown in Fig. 12 is an example of a retinal scanning type device. The observation assistance device 200 may include a retinal scanning type device.

[0049] In the example shown in FIG. 13, the output unit 220 includes a display element 287, a lens 288, and a diffractive light guide element 289. The display element 287 is, for example, a liquid crystal display (LCD). Light from the display element 287 is incident on the diffractive light guide element 289 via the lens 288. The diffractive light guide element 289 includes, for example, a transparent plate and a diffractive optical element. The light incident on the diffractive light guide element 289 travels through repeated diffraction reflections, and reaches the diffractive optical element located in front of the eye 300, whereby an image (auxiliary information 8) is reproduced. The device shown in FIG. 13 is an example of a holographic device. The observation auxiliary device 200 may include a holographic device.

[0050] [Second embodiment] Fig. 14 is a diagram illustrating the functional configuration of the system according to this embodiment. As shown in Fig. 14, the system according to this embodiment (hereinafter simply referred to as the present system) includes an observation device 100a and an observation assistant device 200a capable of communicating with the observation device 100a.

[0051] The observation device 100a differs from the observation device 100 in that it includes a receiving unit 150 instead of the transmitting unit 140. The observation assistant device 200a differs from the observation assistant device 200 in that it includes a transmitting unit 250 instead of the receiving unit 240.

[0052] That is, this system differs from the system 1000 in that the observation device 100a functions as the receiver of a position signal for detecting the relative position, and the observation assistant device 200a functions as the transmitter. Furthermore, in this system, since the relative position is calculated by the observation device 100a, information on the relative position is also transmitted from the observation device 100a to the observation assistant device 200a together with the auxiliary information. This system can also obtain the same effects as the system 1000.

[0053] [Third embodiment] Fig. 15 is a diagram illustrating the functional configuration of the system according to this embodiment. As shown in Fig. 15, the system according to this embodiment (hereinafter simply referred to as the present system) includes an observation device 100b and an observation assistant device 200b capable of communicating with the observation device 100b.

[0054] The observation device 100b differs from the observation device 100 in that it does not include a transmission unit 140. The observation assistance device 200b differs from the observation assistance device 200 in that it includes a shooting unit 270 instead of the receiving unit 240. The shooting unit 270 is a shooting device that shoots an image of the retina, and shoots an image of the retina of the user wearing the observation assistance device 200b, and the control unit 210 detects the relative position based on the image shot by the shooting unit 270. That is, this system differs from the system 1000 in that the observation assistance device 200b alone detects the relative position.

[0055] FIG. 16 is a diagram for explaining an example of the configuration of the output unit 260 and the photographing unit 270. FIG. 17 is a diagram showing an example of an image photographed by the photographing unit 270. In the example shown in FIG. 16, the output unit 260 has the same configuration as the output unit 220 shown in FIG. 11, and is a retinal projection device that projects an image onto the retina of a user. That is, the output unit 260 includes a display element 281, a lens 284, and a concave mirror 285. Light from the display element 281 enters the eye 300 from the pupil via the lens 284 and the concave mirror 285, and an image of the display element 281 is projected onto the retina.

[0056] The photographing section 270 includes an image sensor 282, a half mirror 283, a lens 284, and a concave mirror 285. Light from the retina passes through the pupil and enters the half mirror 283 via the concave mirror 285 and the lens 284. Furthermore, the light reflected by the half mirror 283 is irradiated onto the image sensor 282, so that an image of the retina is projected onto the image sensor 282.

[0057] When the photographing unit 270 photographs an image of the retina, the output unit 260 projects the image for position detection onto the retina. Specifically, for example, when the user approaches the eyepiece 102, the output unit 260 displays a ring of a size that matches the number of fields of view of the eyepiece 102 on the display element 281, and projects the image for relative position detection 10 onto the retina. It is preferable that the display element 281 emits light outside the visible range, so that the output unit 260 projects the image for relative position detection 10 onto the retina with light outside the visible range. For example, infrared light is preferably used as the light outside the visible range. The photographing unit 270 photographs an image of the retina during a period when the output unit 260 projects the image for relative position detection 10 onto the retina. As a result, an image including the image 9 projected onto the retina via the eyepiece 102 and the image for relative position detection 10 is photographed by the photographing unit 270.

[0058] In the observation assistance device 200b, the control unit 210 detects the relative position based on an image of the retina captured by the capturing unit 270 while the output unit 260 is projecting the relative position detection image 10 onto the retina. More specifically, the relative position may be detected based on the image 9 and the relative position detection image 10, and for example, the relative position may be detected by comparing the center positions of the image 9 and the relative position detection image 10.

[0059] This system can also provide the same effects as the system 1000. Furthermore, this system can minimize the modification of the conventional observation device 100.

[0060] [Fourth embodiment] Fig. 18 is a diagram for explaining the functional configuration of the system according to this embodiment. Fig. 19 is a diagram showing an example of output of auxiliary information in the observation auxiliary device 200c. As shown in Fig. 18, the system according to this embodiment (hereinafter simply referred to as the present system) includes an observation device 100 and an observation auxiliary device 200c capable of communicating with the observation device 100. Note that in the observation device 100, the transmission unit 130 transmits a digital image together with auxiliary information to the observation auxiliary device 200c.

[0061] The observation auxiliary device 200c differs from the observation auxiliary device 200 in that it includes a direction detection unit 291. The direction detection unit 291 includes a sensor that detects the direction of the observation auxiliary device 200c. When the relative position and the direction detected by the direction detection unit 291 satisfy a predetermined condition, the observation auxiliary device 200c outputs a digital image captured by the observation device 100 as auxiliary information. This can assist the user in setting a sample on the stage.

[0062] The predetermined condition may be, for example, that the distance between the observation device 100 and the observation assistant device 200 is equal to or greater than a predetermined distance (third distance) and less than a predetermined distance (fourth distance (>third distance)), and that the detected orientation is a predetermined orientation (for example, diagonally downward). The predetermined condition may be any condition that is satisfied when the user sets the sample (container) on the stage.

[0063] 19, the user can check the digital image 2 while setting the sample on the stage, making it possible to place the sample at an appropriate position on the stage without looking through the eyepiece 102. The third distance may be the same as the first distance shown in FIG.

[0064] [Fifth embodiment] Fig. 20 is a diagram for explaining the functional configuration of the system according to this embodiment. Fig. 21 is a diagram showing an example of output of auxiliary information in an observation assistant device 200d. The system according to this embodiment (hereinafter simply referred to as the present system) includes an observation device 100 and an observation assistant device 200d capable of communicating with the observation device 100, as shown in Fig. 20.

[0065] The observation auxiliary device 200d differs from the observation auxiliary device 200 in that it includes an illuminance detection unit 292. The illuminance detection unit 292 includes an illuminance sensor that detects the brightness around the observation auxiliary device 200. As shown in Fig. 21, the observation auxiliary device 200d outputs an image (wire frame 11) indicating the position of the observation device 100 as an auxiliary image when the relative position satisfies a predetermined condition. More preferably, the observation auxiliary device 200d outputs an image indicating the position of the observation device 100 as an auxiliary image when the relative position and the illuminance detected by the illuminance detection unit 292 satisfy a predetermined condition.

[0066] The predetermined condition may be, for example, that the distance between the observation device 100 and the observation auxiliary device 200 is equal to or greater than a predetermined distance (third distance) and less than a predetermined distance (fourth distance (>third distance)). Alternatively, the distance between the observation device 100 and the observation auxiliary device 200 may be equal to or greater than a predetermined distance (third distance) and less than a predetermined distance (fourth distance (>third distance)), and the detected illuminance is less than a predetermined value. The predetermined condition may be any condition that is satisfied when the user approaches the observation device 100 to a certain degree, and more preferably, any condition that is satisfied when the user approaches the observation device 100 to a certain degree in a dark environment.

[0067] 21, the wire frame 11 of the observation device 100 is displayed, so that the user can grasp the position of the observation device 100 through the observation assistant device 200 even when observing a sample in a darkroom. Therefore, the observation work in the darkroom can be performed smoothly. The third distance may be the same as the first distance shown in FIG.

[0068] The above-mentioned embodiments are specific examples shown to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modifications of the above-mentioned embodiments and alternative forms that replace the above-mentioned embodiments may be included. That is, each embodiment can have its components modified within the scope of its purpose and scope. In addition, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. In addition, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in the embodiment. Furthermore, the processing procedures shown in each embodiment may be performed in a different order as long as there is no contradiction. That is, the observation system and observation support method of the present invention can be variously modified and changed within the scope of the claims.

[0069] In the above-described embodiment, for example, in FIG. 2, an example is shown in which the auxiliary information 3 is generated based on the digital image 2 acquired by the observation device, but the auxiliary information does not necessarily have to be generated based on a digital image. The auxiliary information may be information about the settings of the observation device 100, such as the magnification of the objective lens 101, and the user may observe a superimposed image 12 in which the auxiliary information "20x" indicating the magnification of the objective lens 101 is superimposed on the optical image, as shown in FIG. 22. In addition, the auxiliary information is not limited to information acquired from the observation device as shown in FIG. 22. The auxiliary information may be information about time, such as the elapsed time from a predetermined time or the remaining time until a predetermined time, and the user may observe a superimposed image 13 in which the auxiliary information "00:01:35" indicating the time is superimposed on the optical image, as shown in FIG. 23.

[0070] In the above-described embodiment, Fig. 10 illustrates a configuration in which the observation device 100 includes the transmitting unit 140 and the observation auxiliary device 200 includes the receiving unit 240, and Fig. 14 illustrates a configuration in which the observation device 100a includes the receiving unit 150 and the observation auxiliary device 200 includes the transmitting unit 250. That is, in the system, it is sufficient that one of the observation device and the observation auxiliary device includes a transmitting unit that transmits a signal for detecting the relative position, and the other includes a receiving unit that receives the signal for detecting the relative position.

[0071] In the above-described embodiment, smart glasses, a see-through head mounted display, and smart contact lenses have been given as specific examples of the observation auxiliary device, but when smart glasses or a see-through head mounted display are used as the observation auxiliary device, if the eye relief is shorter than when looking through the eyepiece 102 with the naked eye, the observation auxiliary device may interfere and make it impossible to place the eye at the eyepoint. For this reason, it is desirable for the observation device to include a so-called high eyepoint eyepiece 102, and for example, it is desirable for the observation device to have an eye relief of 40 mm or more.

[0072] In this specification, the expression "based on A" does not mean "based only on A" but means "based on at least A." That is, "based on A" may be based on B in addition to A. [Explanation of symbols]

[0073] 1 Optical Image 2. Digital Images 3, 8 Supplementary Information 4~6, 12, 13 Overlay images Images 7 and 9 10 Relative position detection image 11. Wireframe 100, 100a to 100c Observation device 101 Objective Lens 102 Eyepiece 103 Digital Camera 104 Transmitter 105 Control device 110, 210 Control unit 120, 270 Photography Department 130 Transmitter 140, 250 Transmission Section 150, 230, 240 Receiver 200, 200a-200d Observation aids 201 Receiver 210 Control section 220, 260 Output section 281, 287 Display elements 282 Image sensor 283 Half Mirror 284, 288 Lenses 285 Concave mirror 286 MEMS Mirror 289 Diffractive light guide element 291 Orientation detection unit 292 Illuminance detection unit 300 eyes 1000 Systems

Claims

1. an observation device including an eyepiece and an objective lens, and forming a real image of a sample on an optical path between the eyepiece and the objective lens; an observation auxiliary device that is worn by a user and outputs auxiliary information to the user, and that superimposes the auxiliary information on a virtual image of the sample observed by the user through the eyepiece based on a relative position of the observation auxiliary device with respect to the observation device. An observation system comprising:

2. 2. The observation system according to claim 1, The observation auxiliary device changes an output position of the auxiliary information based on a distance and a direction from the optical axis of the eyepiece to the observation auxiliary device. An observation system comprising:

3. The observation system according to claim 1 or 2, The observation auxiliary device stops output of the auxiliary information based on a distance from the observation device to the observation auxiliary device. An observation system comprising:

4. The observation system according to any one of claims 1 to 3, one of the observation device and the observation auxiliary device includes a transmitter that transmits a signal for detecting the relative position, The other of the observation device and the observation auxiliary device includes a receiving unit that receives a signal for detecting the relative position. An observation system comprising:

5. 5. The observation system according to claim 4, The signal includes at least one of electromagnetic waves outside the visible range and ultrasonic waves. An observation system comprising:

6. The observation system according to claim 4 or 5, The observation device and the observation assistant device detect the relative positions by three-point positioning. An observation system comprising:

7. The observation system according to any one of claims 1 to 3, The observation assistance device is a retinal projection device for projecting an image onto the user's retina; an imaging device for capturing an image of the retina; The relative position is detected based on an image of the retina captured by the image capturing device during a period in which the retinal projection device projects an image for detecting the relative position onto the retina. An observation system comprising:

8. 8. The observation system according to claim 7, The observation assistance device detects the relative position based on the relative position detection image projected onto the retina and the image of the sample projected onto the retina, both of which are included in the image of the retina. An observation system comprising:

9. The observation system according to claim 7 or 8, The retinal projection device projects the image for relative position detection onto the retina using light outside the visible range. An observation system comprising:

10. 10. The observation system according to claim 9, The light outside the visible range is infrared light. An observation system comprising:

11. The observation system according to any one of claims 1 to 10, The observation assistance device includes at least one of smart glasses, a see-through head-mounted display, or a smart contact lens. An observation system comprising:

12. The observation system according to any one of claims 1 to 11, The observation assistance device includes at least one of a retinal projection type, a retinal scanning type, a hologram type, or a micro LED type device. An observation system comprising:

13. The observation system according to any one of claims 1 to 12, The observation assistance device is Further, a sensor for detecting the orientation of the observation auxiliary device is provided, When the relative position and the orientation detected by the sensor satisfy a predetermined condition, an image of the sample captured by the observation device is output as the auxiliary information. An observation system comprising:

14. The observation system according to any one of claims 1 to 13, When the relative position satisfies a predetermined condition, the observation auxiliary device outputs an image indicating the position of the observation device as the auxiliary information. An observation system comprising:

15. The observation system according to any one of claims 1 to 14, The observation device has an eye relief of 40 mm or more. An observation system comprising:

16. an observation device including an eyepiece and an objective lens, forming a real image of a sample on an optical path between the eyepiece and the objective lens; An observation assistance device worn by a user outputs auxiliary information to the user, and superimposes the auxiliary information on a virtual image of the sample observed by the user through the eyepiece based on a relative position of the observation assistance device with respect to the observation device. An observation support method comprising:

Images