System and method for overlaying a virtual image onto a real-time image

Abstract

Disclosed are systems and methods for overlaying a virtual image on a real-time image. The system for overlaying a virtual image on a real-time image includes a real-time image module and a virtual image module. The real-time image module includes a magnification assembly that generates a real-time image of an object at a first location and a first depth at a predetermined magnification. The virtual image module generates the virtual image by projecting a right light signal to a viewer's right eye and a corresponding left light signal to the viewer's left eye, respectively. The right light signal and the corresponding left light signal are perceived by the viewer to display a virtual image at a second location and a second depth. The second depth is related to the angle between the right light signal and the corresponding left light signal projected to the viewer's eye. The second depth is approximately the same as the first depth.

Description

Technical Field

[0001] The present disclosure generally relates to methods and systems for overlaying virtual images on real-time images, and more particularly to methods and systems for overlaying virtual images with depth, generated by projecting a plurality of right light signals and corresponding left light signals onto a real-time image into the viewer's eyes.

Background Art

[0002] In recent years, many visualization assistance systems and methods have been developed to assist medical personnel during health diagnosis or during surgical procedures including ophthalmic surgery. During medical procedures, visualization assistance systems can provide additional visual information of patients such as surgical parameters like medical records, photographs, magnetic resonance imaging (MRI), X-ray photographs, computed tomography (CT), or optical coherence tomography (OCT). In some cases, the additional visual information is a processed image of the patient, such as a CT image with some markings. Visualization assistance systems are often used together with other medical devices that can provide a real-time image of the patient. Medical personnel may receive additional information separated from the real-time image provided by the visualization assistance system. For example, the additional information is displayed separately on a monitor rather than from a surgical microscope that can observe the real-time image of the patient. The monitor can usually only provide a two-dimensional image. However, during a medical procedure, medical personnel desire to observe additional visual information (e.g., a processed image of the patient) overlaid on the real-time image of the patient. In addition, conventional visualization assistance systems can only provide additional visual information as 2D images. As a result, the ability to create three-dimensional images of patients' real-time images superimposed with additional visual information has become a major concern in the medical industry. For example, in ophthalmic examinations and surgeries, medical professionals perform surgery by viewing a real-time optical image of the patient's eye through the eyepiece of an ophthalmic microscope. However, surgeons cannot simultaneously observe the processed retinal image of the patient's eye through the microscope during the procedure; they must turn their heads to observe a separate monitor and then return to the microscope. Therefore, there remains a need to incorporate additional visual information of the patient, provided by visualization assistance systems, along with the real-time optical image seen by medical professionals. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. PCT / US20 / 59317 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The object of this disclosure is to provide a system and method for overlaying a virtual image onto a real-time image. The system for overlaying a virtual image onto a real-time image comprises a real-time image module and a virtual image module. The real-time image module comprises an enlargement assembly that generates a real-time image of an object at a predetermined magnification and a first location and depth. [Means for solving the problem]

[0005] The virtual image module generates a virtual image by projecting a right light signal onto the viewer's right eye and a left light signal corresponding to the viewer's left eye. The right light signal and the corresponding left light signal are perceived by the viewer to display the virtual image at a second location and a second depth. The second depth is related to the angle between the right light signal and the corresponding left light signal projected onto the viewer's eye. In one embodiment, the second depth is approximately the same as the first depth. The virtual image is superimposed on the real-time image to provide the viewer with more information. As a result, in one embodiment, the virtual image is a processed image of the object.

[0006] The magnification of the real-time image is adjustable. After the real-time image has been enlarged, the virtual image may be enlarged manually or automatically to maintain the initial overlay between the virtual image and the real-time image. An automatic mode for overlay may be selected.

[0007] In order to overlay a virtual image onto a real-time image, the system must first be calibrated with respect to the viewer. Since every viewer's eye has different physical characteristics, including interpupillary distance, the system must be specifically calibrated with respect to the viewer, using the right and left light signals projected into the viewer's eye, to ensure that the viewer perceives a virtual image displayed at a second location and a second depth.

[0008] The process of overlaying a virtual image onto a real-time image includes the steps of (a) selecting a first point on the real-time image as a first marker, (b) displaying the real-time image at a predetermined magnification and a first location and depth, and (c) projecting a virtual image by projecting a right light signal to the viewer's right eye and a left light signal corresponding to the viewer's left eye, so that the viewer perceives the virtual image at a second location and a second depth, such that the corresponding first marker on the virtual image overlaps with the first marker on the real-time image. In one embodiment, the depth of the first marker on the real-time image is approximately the same as the depth of the corresponding first marker on the virtual image. For more accurate overlaying, second and third markers may be used by similar methods.

[0009] Additional features and advantages of this disclosure are described below, some of which may be evident from the description or learned through practice. The purposes and other advantages of this disclosure are realized and achieved by the structures and methods detailed in the accompanying drawings as well as in the written descriptions and claims. It should be understood that both the above general description and the following detailed description are representative and for illustrative purposes only, and are intended to further provide a description of the claimed invention. [Brief explanation of the drawing]

[0010] [Figure 1A] This is a schematic diagram illustrating an embodiment of the system according to the present invention. [Figure 1B] This is a schematic diagram illustrating another embodiment of the system according to the present invention. [Figure 1C] This is a schematic diagram illustrating a collimator in the virtual image module of the system according to the present invention. [Figure 2] This is a structural diagram illustrating an embodiment of a system comprising various modules according to the present invention. [Figure 3A] This is a schematic diagram illustrating possible embodiments of the system according to the present invention. [Figure 3B]This is a schematic diagram illustrating possible embodiments of the system according to the present invention. [Figure 4] This is a schematic diagram illustrating an embodiment of the relationship between an object, a real-time image, and a virtual image according to the present invention. [Figure 5] This is a photograph illustrating the virtual image superposition of the retina on a real-time image according to the present invention. [Figure 6] This flowchart illustrates an embodiment of the process for overlaying a virtual image onto a real-time image according to the present invention. [Figure 7] This flowchart illustrates another embodiment of the process for overlaying a virtual image onto a real-time image according to the present invention. [Figure 8] This is a schematic diagram illustrating an embodiment of a virtual image module according to the present invention. [Figure 9] This is a schematic diagram illustrating the relationship between virtual binocular pixels and corresponding pairs of right and left pixels according to the present invention. [Figure 10] This is a schematic diagram illustrating the optical path from the optical signal generator to the beam splitter and to the viewer's retina according to the present invention. [Figure 11] This is a schematic diagram illustrating a virtual binocular pixel formed by a right optical signal and a left optical signal according to the present invention. [Figure 12] This table illustrates an embodiment of a look-up table according to the present invention. [Modes for carrying out the invention]

[0011] The technical terms used in the following descriptions are intended to be interpreted in the broadest reasonable manner, even if they are used in connection with a detailed description of certain specific embodiments of the technology. Certain terms may even be emphasized below, however any technical terms intended to be interpreted in any restricted manner are specifically defined as such in this “Modes for Carrying Out the Invention” section.

[0012] The present disclosure relates to a system and method for superimposing a virtual image on a real-time image. A virtual image with depth may be superimposed on the real-time image to provide a viewer with more information related to the real-time image, such as guidance, instructions, navigation, etc. for surgery. A real-time image is an image that reflects changes in an object in real time. The real-time image may be a two-dimensional (2D) image or a three-dimensional (3D) image. In one embodiment, the real-time image is generated by light reflected or emitted from an object, for example, an image observed by a microscope or a telescope. In another embodiment, the real-time image is generated by a display device that receives an image of an object captured in real time, perhaps by a camera, such as an image on a display device obtained from an endoscope. In addition to this, the real-time image may be a real image or a virtual image. A virtual image with depth is generated by projecting optical signals onto both eyes of the viewer. The depth of the virtual image is related to the angle between the right optical signal projected onto the viewer's eye and the corresponding left optical signal. The virtual image may be a 2D image or a 3D image. When the virtual image is superimposed on the real-time image, a portion of the virtual image overlaps with the real-time image.

[0013] A system for superimposing a virtual image on a real-time image includes a real-time image module and a virtual image module. The real-time image module includes a magnifying assembly that generates a real-time image of an object at a first location and a first depth at a predetermined magnification. Magnification is a process of increasing the apparent size rather than the physical size of an object. This expansion is quantified by a calculated value called "magnification", which is the ratio between the apparent size of the object (in the real-time image) and the observed size of the object without magnification. The magnification is adjustable and may be any positive value, such as 0.5, 1, and 10. When the magnification is less than 1, it is called size reduction and may sometimes be called minification or de-magnification.

[0014] The virtual image module generates a virtual image by projecting a right light signal onto the viewer's right eye and a corresponding left light signal onto the viewer's left eye. The right light signal and the corresponding left light signal are perceived by the viewer to display the virtual image at a second location and a second depth. The second depth is related to the angle between the right light signal projected onto the viewer's eye and the corresponding left light signal. In one embodiment, the second depth is approximately the same as the first depth.

[0015] The virtual image is superimposed on the real-time image to provide the viewer with more information. For example, in one embodiment, the virtual image is a processed image of an object. For example, the object may be a brain, and the real-time image is a brain image generated in real time by a surgical microscope. The virtual image may be a CT image or an MRI image of the brain taken before the surgery, with a mark placed at the location of the brain tumor to be removed during the surgery. The marked virtual image is superimposed on the real-time image of the brain during the surgery to assist the surgeon in identifying the location of the brain tumor to be removed. In this situation, in order to be accurate with respect to the location of the surgery, the second depth of the virtual image (the marked CT image or MRI image) is approximately the same as the first depth of the real-time image, that is, the actual brain image obtained from the surgical microscope. The virtual image may further include some text information, marks, and pointers for guidance and explanation to assist in diagnosis and treatment. In addition, by image overlay, the viewer may be able to compare the previous image of the object presented by the virtual image with the current state of the object presented by the real-time image, and as a result, may be able to estimate the disease progression and treatment outcome.

[0016] The magnification of the real-time image is adjustable. In one embodiment, such adjustment can be achieved manually by rotating a knob, changing the objective lens, controlling a virtual switch, or by giving verbal instructions. After the real-time image has been magnified, the virtual image may be magnified manually or automatically to maintain the initial overlay between the virtual image and the real-time image. An automatic mode for overlay may be selected.

[0017] In order to overlay a virtual image onto a real-time image, the system must first be calibrated with respect to the viewer. Since every viewer's eye has different physical characteristics, including interpupillary distance (IPD), the system must be calibrated specifically with respect to the viewer to ensure that the viewer perceives a virtual image displayed at a second location and a second depth using the right and left light signals projected into the viewer's eye. For example, the distance between the right and left eyepieces of the microscope must be adjusted to match the viewer's interpupillary distance, and the angle between the right and corresponding left light signals must be adjusted so that the viewer perceives the virtual image precisely at the second depth.

[0018] The process of overlaying a virtual image onto a real-time image includes (a) selecting a first point on the real-time image as a first marker; (b) displaying the real-time image at a predetermined magnification and a first depth; and (c) projecting a virtual image by projecting a right light signal to the viewer's right eye and a left light signal corresponding to the viewer's left eye, so that the viewer perceives the virtual image at a second location and a second depth, such that the corresponding first marker on the virtual image overlaps with the first marker on the real-time image. As discussed above, the second depth is related to the angle between the right light signal and the corresponding left light signal projected onto the viewer's eye. In one embodiment, the depth of the first marker on the real-time image is approximately the same as the depth of the corresponding first marker on the virtual image. For more accurate overlaying, second and third markers may be used in a similar manner.

[0019] As shown in Figures 1A and 1B, the system 100 for placing a virtual image 165 on a real-time image 115 includes a real-time image module 110 and a virtual image module 160. The real-time image module 110 may include a magnification assembly 120 that generates a magnified real-time image of an object 105, such as a brain, for both eyes of the viewer. The magnification assembly 120 may include a number of optical units and assemblies, such as various types of lenses, including an objective lens 113. In another embodiment, the magnification assembly 120 may use electronic circuitry to process and magnify the real-time image of the object 105. The magnification of the real-time image module may be determined before observation and may be adjustable during observation. The magnification may be 1 / 2, 1, 3, 10, 100, etc. Magnification adjustment may be performed via a user interface that is in communication with the real-time image module. The real-time imaging module may have one set of optical units and assemblies that generate real-time images for both eyes of the viewer, or two separate sets of optical units and assemblies that generate real-time images for the right and left eyes of the viewer, respectively. The real-time imaging module 110 may further include a prism assembly 130 that changes the direction of light, beam splitters 140, 145 that split the light, an observation tube 150 that guides the light, and eyepieces 152, 154 that further magnify the image. Furthermore, the real-time image may be generated from light reflected or emitted from an object 105, such as a real-time image generated by a microscope, including a surgical microscope. In another embodiment, the real-time image may be generated by an image acquisition and display device, such as an endoscope and its associated display device. Depending on the size and resolution of the image, the real-time image may actually or conceptually contain 921,600 pixels in a 1280 × 720 array. Each pixel may have a slightly different location and depth from its neighboring pixels. A representative pixel, such as a first marker, may be selected for the real-time image. Markers such as the first and second markers are typically unique points in the real-time image with identifiable features that should be easily recognized by the viewer, such as a center point or the intersection of two specific blood vessels. The markers may be pixels or comprise a number of pixels that are close to each other. In one embodiment, the location and depth of a representative pixel may be used as the location and depth of the real-time image, i.e., the first location and first depth.

[0020] The virtual image module 160, configured to connect to the real-time image module 110, includes a right light signal generator 170 and a left light signal generator 175. The right light signal generator 170 generates a number of right light signals relating to the virtual image and is likely located near the right-hand portion of the real-time image module. Similarly, the left light signal generator 175 generates a number of left light signals relating to the virtual image and is likely located near the left-hand portion of the real-time image module. The right light signals are then redirected by the right beam splitter 140 towards one eye of the viewer. Similarly, the left light signals are then redirected by the left beam splitter 145 towards the other eye of the viewer. The redirected right light signals and the corresponding redirected left light signals are perceived by the viewer to display the virtual image at a second depth. Depending on the size and resolution of the image, the virtual image may actually contain 921,600 virtual binocular pixels within a 1280 × 720 array. Each virtual binocular pixel may have a location and depth slightly different from its neighboring pixels. A representative virtual binocular pixel, such as a first marker, may be selected with respect to the virtual image. In one embodiment, a second location and second depth may be used for the location and depth of the representative virtual binocular pixel with respect to the virtual image. After the viewer's eyes receive the reoriented right light signal and the reoriented corresponding left light signal of the representative virtual binocular pixel, the viewer perceives the representative virtual binocular pixel at a second depth related to the angle between such reoriented right light signal and reoriented corresponding left light signal.

[0021] The light beam of the real-time image may also be directed towards the viewer's eyes through the right beam splitter 140 and the left beam splitter 145. As a result, to some extent, the right beam splitter 140 and the left beam splitter 145 are shared by both the real-time image module and the virtual image module. In one embodiment, a beam splitter initially installed in the real-time image module to share the real-time image with other viewers can be rotated by an appropriate angle to direct the light signal generated from the virtual image module towards the viewer's eyes.

[0022] As shown in Figures 1B and 1C, the virtual image module 160 may further include a right focus adjustment unit 182 between the right optical signal generator 170 (or right collimator 180, if available) and the right beam splitter 140, and a left focus adjustment unit 187 between the left optical signal generator 175 (or right optical collimator 185, if available) and the left beam splitter 145, in order to improve the clarity of the virtual image for the viewer. The right / left focus adjustment units may include optical units such as various types of lenses, including convex lenses. In one embodiment using a convex lens as a focusing unit, assuming the distance between the optical signal generator and the beam splitter remains constant, the focal point of the light beam changes by adjusting the distance between the optical signal generator and the convex lens. The closer the focal point of the light beam is to the retina, the clearer the virtual image becomes for the viewer. Since the axial length of a viewer's eye can vary, the preferred focal point of the light beam, and consequently the distance between the optical signal generator and the focusing unit, will vary accordingly. In other words, for a viewer with a longer axial length, the focusing unit needs to be further away from the optical signal generator so that the focal point of the light beam is closer to the viewer's retina. When a collimator is available, the focusing unit is positioned between the collimator and the beam splitter. After passing through the collimator, the light beam from the optical signal generator becomes substantially parallel, and then focuses after passing through the focusing unit. In addition, since the focusing unit does not change the angle of incidence of the light beam, the depth of the virtual image is unaffected.

[0023] As partially shown in Figure 1C, the virtual image module 160 may further include a right collimator 180 and a left collimator 185 to narrow the optical beams of multiple optical signals, for example, to better align the direction of movement in a specific direction, or to narrow the spatial cross-section of the optical beams. The right collimator 180 may be positioned between the right optical signal generator 170 and the right beam splitter 140, and the left collimator 185 may be positioned between the left optical signal generator 175 and the left beam splitter 145. The collimators may be curved mirrors or lenses.

[0024] In addition, the virtual image module 160 may include a control module 190 that controls the virtual image signals related to the right optical signal generator 175 and the left optical signal generator 175. The control module 190 is communicably connected to the virtual image module 160 and adjusts the right optical signal and the corresponding left optical signal so that the virtual image may be automatically modified in order to superimpose the virtual image on top of the real-time image based on fluctuations in the real-time image. Fluctuations in the real-time image include variations in field of view, magnification, or location. For example, when the magnification of the real-time image is adjusted from 3x to 10x, the control module 190 processes the image signal to enlarge the virtual image to the same size and uses at least a first marker to ensure that the virtual image continues to be superimposed on the real-time image. The control module 190 includes one or more processors, but may use an external computing server 250 for complex signal processing.

[0025] The virtual image may be stored in the memory module 195. In one embodiment, the virtual image is a processed image of the object, such as an X-ray, ultrasound, CT, and MRI image of the object, with some markers or highlights in the area of ​​interest. The virtual image may further include some text information and pointers for guidance or explanation. For example, the virtual image may be a previously taken and processed retinal image of a patient with markers on bleeding vessels to be blocked by the laser. The system 100 may overlay such a virtual image onto a real-time image of the same retina obtained from a slit-lamp microscope. The control module 190 may, whenever necessary, retrieve the virtual image stored in the memory module 195 and then generate virtual images relating to the right optical signal generator 170 and the left optical signal generator 175.

[0026] As shown in Figure 2, in addition to the real-time image module 110 and the virtual image module 160, the system 100 may further include a recording module 210 for recording either or both real-time and virtual images, an object measurement module 220 for measuring the location and depth of an object, a surgical module 230 for physically performing surgical operations on the object 105, and a user interface 240 for a viewer to communicate with the various modules of the system 100 and control the various functions of the system 100. All modules of system 100 may have electronic communication with each other via wired or wireless methods. Wireless methods may include Wi-Fi®, Bluetooth®, near-field communication (NFC), the Internet, telecommunications, radio frequency (RF), etc. The real-time image module 110, the virtual image module 160, and the recording module 210 may have optical communication with each other via light beams and optical signals. A viewer may observe real-time and virtual images through system 100 and then control system 100 through physical interaction with the user interface 240. System 100 may have optical communication with an object 105, such as receiving a light beam reflected or emitted from the object and projecting a light beam onto the object. System 100 may have physical interaction with an object, such as performing laser surgery on the object.

[0027] As described above, the system 100 may further include a recording module 210 for recording either or both real-time images and virtual images. In one embodiment, the recording module 210 may be positioned between the right beam splitter 140 and the left beam splitter 145 to record real-time images, i.e., the light beams obtained from the object, reflected by the right beam splitter and the left beam splitter, respectively, during a surgical procedure. The recording module 210 may include a digital camera or a charge-coupled device (CCD) for capturing images. In another embodiment, the recording module 210 may be positioned close to the eyepiece to record light beams that pass through the eyepiece but before reaching the viewer's eye, including both beams that form the real-time image and the virtual image. The recording module 210 may be connected to a control unit to directly record the virtual image signal, and related information, and parameters for further display.

[0028] As described above, the system 100 may further include an object measurement module 220 that measures the location and depth of an object. The object measurement module 220, configured to be connected to the system, may continuously or periodically measure the location and depth of an object relative to the object measurement module (or the viewer) and transmit relevant information to the virtual image module to adjust the virtual image. Upon receiving such information, the control module 190 may process the virtual image signal based on the updated location and depth of the object relative to the object measurement module and the viewer. As a result, the virtual image may remain superimposed on the real-time image. The distance or relative location between the object 105 and the object measurement module 220 (or the viewer's eye) may change over time. In some situations, the object 105, such as a part of the human body like an eyeball, may move during surgery. In other situations, the system 100 may be worn by a viewer, such as a surgeon, who may move their head during surgery. As a result, the relative location and distance between the object 105 and the viewer's eye need to be measured and calculated to maintain a virtual image overlay on a real-time image. The object measurement module 220 may include a gyroscope, an indoor / outdoor global positioning system (GPS), and distance measuring components (e.g., radiators and sensors) that accurately track such relative location and depth variations of the object 105.

[0029] As described above, the system 100 may further include a surgical module 230 for physically performing surgical operations on the object 105. The surgical module 230 may include a laser for removing tissue or blocking bleeding vessels, and / or a scalpel for cutting tissue. The surgical module 230 may work in conjunction with the real-time imaging module 110 to position the laser and / or scalpel toward a point of interest identified by a viewer, such as a surgeon, as shown in the real-time image.

[0030] As described above, system 100 may further include a user interface 240 that allows a viewer to control various functions of system 100, such as real-time image magnification, a second location and second depth of a virtual image, a focus adjustment unit, a recording module 210, an object measurement module 220, and so on. The user interface 240 may be operated by voice, hand gestures, finger / foot movements, and in the form of pedals, keyboards, mice, knobs, switches, styluses®, buttons, sticks, and touch screens. The user interface 240 may communicate with other modules of the system 100 (including the real-time image module 110, virtual module 160, recording module 210, object measurement module 220, and surgical module 230) via wired or wireless means. Wireless methods may include Wi-Fi, Bluetooth, near-field communication (NFC), the Internet, telecommunications, radio frequency (RF), etc. The observer may use the user interface 240, for example by controlling a stick, to move a cursor to a point of interest on the real-time image, and then use the user interface 240 to activate a laser beam toward a corresponding point of interest on the object 105 to remove tissue or block a bleeding vessel, for example by pressing a pedal.

[0031] In one embodiment, the system 100 may be an AR microscope for surgical and / or diagnostic purposes, such as an AR ophthalmoscope and an AR slit-lamp microscope. Figure 3A shows an example of a fixed AR surgical microscope 310 including a user interface pedal 320. Figure 3B shows an example of a portable AR surgical microscope 350, i.e., a head-mountable device including a real-time imaging module 370 and a virtual imaging module 360. The real-time imaging module 370 is attached to the virtual imaging module 360 ​​but is detachable from the virtual imaging module 360.

[0032] As shown in Figure 4, the object 105, the real-time image 115 generated by the real-time image module 110, and the virtual image 165 generated by the virtual image module 160 may have different locations and depths. In this embodiment, the virtual image 165 is a processed partial image of the object 105. The virtual image module 160 may generate only the virtual image 165 relating to a field of view or region of interest of the object. Images of the object may be captured and processed, for example, by an artificial intelligence (AI) module to generate virtual images within a very short time interval, such as one second.

[0033] As already described, depending on the resolution, the object 105, the real-time image 115, and the virtual image 165 may conceptually or actually have a large number of pixels, such as 921,600 pixels in a 1280 × 720 array. In this embodiment, the location and depth of the object 105, the real-time image 115, and the virtual image 165 are represented by the location and depth of their corresponding first landmarks, respectively. The depth is measured based on the distance between the eyepiece 152 and the object 105, or the real-time image 115, or the virtual image 165. Accordingly, as shown in Figure 4, the object 105 is located at object location L(o) and object depth D(o), the real-time image 115, i.e., the magnified image of the object 105, is located at a first location L(r) and a first depth D(r), and the virtual image 165 is located at a second location L(v) and a second depth D(v). Depending on the optical characteristics of the real-time image module, the depth of the real-time image 115 may be closer to or further away from the viewer's eye. In this embodiment, the depth D(r) of the real-time image is greater than the depth D(o) of the object. However, in other embodiments, the depth D(r) of the real-time image may be less than or almost the same as the depth D(o) of the object. In this case, the virtual image 165 is generated by the virtual image module 160 with a depth D(v) closer to the eyepiece than the real-time image 115.

[0034] Using the information of L(r) and D(r), the virtual image module 160 of system 100 may superimpose a virtual image onto a real-time image by superimposing a first marker LM1(r) on the real-time image with the corresponding first marker LM1(v) on the virtual image, as shown in Figure 4. For a more advanced superimposition, the virtual image module 160 of system 100 may further superimpose a second marker LM2(r) on the real-time image with the corresponding second marker LM2(v) on the virtual image. In another embodiment where the superposition extends beyond simply overlapping the markers with respect to their locations, the depth of the corresponding first marker on the virtual image may be approximately the same as the depth of the first marker on the real-time image. Similarly, the depth of the corresponding second marker on the virtual image may be approximately the same as the depth of the second marker on the real-time image. In order to accurately and completely superimpose the 3D virtual image onto the 3D real-time image, a third marker on the real-time image is selected in addition to the first and second markers. In this case, the virtual image module ensures that the location and depth of the corresponding third marker on the virtual image are approximately the same as the location and depth of the third marker on the real-time image.

[0035] Figure 5 illustrates three images: a real-time image of the patient's retina, a processed virtual image of the retina, and an image with both superimposed. In one embodiment, an angiographic image of the patient's retina is likely captured and processed by a slit-lamp microscope. In this case, the virtual image module 160 may use such processed images to project the superimposed virtual image onto the real-time image of the patient's retina during surgery, which may help in the identification and visualization of the edges of choroidal neovascularization membranes. AR / MR microscopes can greatly facilitate the diagnosis and treatment of various ophthalmic disorders and diseases.

[0036] As shown in Figure 6, the process of overlaying a virtual image onto a real-time image involves four steps. In step 610, a first point on the real-time image is selected as the first landmark by the viewer, specialist, computer, or system 100. For example, the viewer selects the first landmark on the real-time image by controlling a mouse or stick to move a cursor or pointer visible through the eyepiece. As described above, the landmarks, including the first, second, and third landmarks, are unique points with identifiable features that should be easily recognized by the viewer in the real-time image, such as a central point or the intersection of two specific blood vessels. The landmarks may be defined manually by a specialist or automatically by a computer program. There are three basic types of landmarks: anatomical landmarks, mathematical landmarks, and pseudo-landmarks. Anatomical landmarks are biologically significant points within a living organism. Any anatomical feature, namely folds, protrusions, tubes, blood vessels, etc., is always present in tissue that helps indicate a specific structure or location. Anatomical landmarks may be used by surgical pathologists for specimen orientation. Mathematical markers are points within a shape that are positioned according to some mathematical or geometric property, such as large curve points or limiting points in practice. A computer program may determine which mathematical markers to use for automatic pattern recognition. Pseudo-markers are structured points located between anatomical or mathematical markers. A typical example is a pair of points equally spaced between two anatomical markers to obtain more sample points from a shape. Pseudo-markers are useful during shape matching when the matching process requires a large number of points. Markers may be pixels, or they may consist of a number of pixels that are close to each other.

[0037] In step 620, a real-time image of the object is displayed at a predetermined magnification at a first location and a first depth. As described above, there are at least two types of real-time images. The first type of real-time image is generated by light reflected or emitted from an image of the object, for example, observed by a microscope or telescope. In this situation, the first location and first depth may be determined by the optical features of the real-time image module. The viewer may observe the real-time image through an eyepiece. The second type of real-time image is generated by a display device that receives an image of the object, possibly taken in real time by a camera, on a display device obtained from an endoscope, for example, a gastroscopy, colonoscopy, or proctoscope. The endoscope may have two image acquisition devices, one positioned separately to obtain a 3D image and the other to generate a 3D image. The real-time image may be a two-dimensional (2D) image or a three-dimensional (3D) image. Steps 610 and 620 are interchangeable.

[0038] In step 630, the virtual image module is calibrated for a particular viewer. As already described, certain physical characteristics of each viewer, such as interpupillary distance, may affect the location and depth of the virtual image that the viewer perceives with the same right light signal and corresponding left light signal. In one embodiment, the control module may adjust the virtual image signal based on the viewer's IPD to ensure that the right light signal generator 170 and the left light signal generator 175 project the light signals to the appropriate location and angle so that the viewer perceives the virtual image accurately at a second location and second depth.

[0039] In step 640, the virtual image module projects a virtual image by projecting a right light signal to the viewer's right eye and a left light signal corresponding to the viewer's left eye, so that the viewer perceives the virtual image at a second location and a second depth, with the corresponding first marker on the virtual image overlapping the first marker on the real-time image. In other words, the virtual image module projects a virtual image and superimposes it onto the real-time image. The location of the corresponding first marker on the virtual image (the second location) is approximately the same as the location of the first marker on the real-time image (the first location). Generally, the virtual image is divided into a number of virtual binocular images depending on the resolution, for example, 921,600 virtual binocular pixels in a 1280 x 720 array. For each right light signal and its corresponding left light signal projected onto the viewer's retina, the viewer perceives a virtual binocular pixel at a specific location and depth. The depth is related to the angle between the right light signal and its corresponding left light signal projected into the viewer's eye. When the first marker on the real-time image is at the first location and first depth, the virtual binocular pixel of the corresponding first marker on the virtual image is projected so that it is perceived by the viewer as being at the second location and second depth. For the initial overlay, the location of the corresponding first marker on the virtual image (second location) is set to be approximately the same as the location of the first marker on the real-time image (first location), although the depths of the two locations may differ. This overlay can be achieved manually by the viewer or automatically by system 100 using shape recognition technology, including artificial intelligence (AI) algorithms. To further improve the superposition, the second depth is set to be approximately the same as the first depth. In addition, if the real-time image is enlarged from the actual size of the object, the virtual image needs to be enlarged to the same extent for superposition. Furthermore, to further improve the superposition, the field of view of the virtual image needs to match the field of view of the real-time image. The relationship between the optical signal generated by the optical signal generator and the depth perceived by the viewer is described in detail below.

[0040] In step 650, if the location, magnification, or field of view of the real-time image changes, the virtual image module corrects the virtual image to maintain the overlay between the virtual image and the real-time image. Changes in the location, magnification, and field of view of the real-time image may be caused by the viewer's actions or the movement of the object or the viewer. System 100 constantly monitors the first location and first depth of the real-time image, as well as the second location and second depth of the virtual image. If any change occurs in the real-time image, the virtual image module corrects the virtual image signal to maintain the overlay between the virtual image and the real-time image.

[0041] As shown in Figure 7, the alternative process of overlaying a virtual image onto a real-time image includes six steps. Some steps are the same as or similar to those described in the preceding embodiment shown in Figure 6. Some steps are optional and can be further modified. In step 710, the first, second, and third points on the real-time image are selected as the first marker, second marker, and third marker, respectively, by the viewer, specialist, computer, or system 100. Three markers are used in this case for the most accurate superposition. Some surgical procedures, such as neurosurgery, require a very high level of precision, and as a result, three markers may be required to ensure that the virtual image is perfectly superimposed on the real-time image. However, the process may include two markers if necessary. Step 720 may be the same as step 620, and step 730 may be the same as step 630. Step 740 follows the same principle as described for Step 640. However, the locations and depths of the corresponding first, second, and third landmarks on the virtual image are approximately the same as the locations and depths of the first, second, and third landmarks on the real-time image, respectively. In Step 750, the first location and first depth are repeatedly monitored and measured. The first location and first depth may be calculated based on the location and depth of the object relative to the object measurement module (or the viewer), as measured by the object measurement module. As a result, the virtual image can remain superimposed on the real-time image. In Step 760, the viewer, for example a surgeon, performs surgery on the object with a laser or scalpel at the point of interest identified by the viewer.

[0042] The virtual image module 160, as well as the method for generating the virtual image 165 at a second location and a second depth, and the method for moving the virtual image as desired, are discussed in detail below. International Publication PCT / US20 / 59317, “SYSTEM AND METHOD FOR DISPLAYING AN OBJECT WITH DEPTHS,” filed on 6 November 2020, is incorporated herein by reference in its entirety. As shown in Figure 8, the virtual image module 160 includes a right light signal generator 170 that generates multiple right light signals, such as 12 for RLS_1, 14 for RLS_2, and 16 for RLS_3; a right beam splitter 140 that receives multiple right light signals and directs them toward the viewer's right retina 54; a left light signal generator 175 that generates multiple left light signals, such as 32 for LLS_1, 34 for LLS_2, and 36 for LLS_3; and a left beam splitter 145 that receives multiple left light signals and directs them toward the viewer's left retina 64. The viewer has a right eye 50 that includes the right pupil 52 and right retina 54, and a left eye 60 that includes the left pupil 62 and left retina 64. The diameter of a human pupil can generally range from 2 mm to 8 mm, depending partly on ambient light. Normal pupil size in adults varies from 2 mm to 4 mm in diameter in bright light and from 4 mm to 8 mm in darkness. Numerous right-facing light signals are reoriented by the right beam splitter 140, pass through the right pupil 52, and are finally received by the right retina 54. Right-facing light signal RLS_1 is the light signal furthest to the right that the viewer's right eye can see on a specific horizontal plane. Right-facing light signal RLS_2 is the light signal furthest to the left that the viewer's right eye can see on the same horizontal plane. Upon receiving the reoriented right-facing light signals, the viewer perceives numerous right pixels relating to the object 105 within a region A defined by the ranges of the reoriented right-facing light signals RLS_1 and RLS_2. Region A is called the field of view (FOV) for the right eye 50. Similarly, numerous left-facing light signals are reoriented by the left beam splitter 145, pass through the center of the left pupil 62, and are finally received by the left retina 64. Left-facing light signal LLS_1 is the light signal furthest to the right that the viewer's left eye can see on a specific horizontal plane. The left light signal LLS_2 is the light signal that is furthest to the left that the viewer's left eye can see on the same horizontal plane. When the oriented left light signal is received, the viewer perceives a number of left pixels relating to the object 105 within a region B, which is defined by the ranges of the oriented left light signals LLS_1 and LLS_2. Region B is called the field of view (FOV) for the left eye 60. When a large number of right pixels and a large number of left pixels are displayed within region C, which overlaps with regions A and B, at least one right light signal displaying one right pixel and a corresponding left light signal displaying one left pixel merge to display a virtual binocular pixel with a specific depth within region C. The depth is related to the angle between the oriented right light signal and the oriented left light signal projected onto the viewer's retina. Such an angle is also called the convergence angle.

[0043] As shown in Figures 8 and 9, the viewer perceives a virtual image of a brain object 105 with multiple depths within region C in front of them. The image of the brain object 105 includes a first virtual binocular pixel 72 displayed at a first depth D1 and a second virtual binocular pixel 74 displayed at a second depth D2. The first angle between the first oriented right light signal 16' and the corresponding first oriented left light signal 26' is θ1. The first depth D1 is related to the first angle θ1. In detail, the first depth of the first virtual binocular pixel of object 105 can be determined by the first angle θ1 between the spread of the optical paths of the first oriented right light signal and the corresponding first oriented left light signal. As a result, the first depth D1 of the first virtual binocular pixel 72 can be calculated approximately by the following equation. TIFF0007837556000001.tif9150 The distance between the right pupil 52 and the left pupil 62 is the interpupillary distance (IPD). Similarly, the angle between the second reoriented right light signal 18' and the corresponding second reoriented left light signal 38' is θ2. The second depth D2 is related to the second angle θ2. In detail, the second depth D2 of the second virtual binocular pixel of object 105 can be approximately determined by the second angle θ2 between the spread of the optical paths of the second reoriented right light signal and the corresponding second reoriented left light signal using the same formula. Since the second virtual binocular pixel 74 is perceived by the viewer as being farther away from the viewer (i.e., at a greater depth) than the first virtual binocular pixel 72, the second angle θ2 is smaller than the first angle θ1.

[0044] Furthermore, however, the reoriented right light signal 16' with respect to RLG_2 and the corresponding reoriented left light signal 36' with respect to LLS_2 together display the first virtual binocular pixel 72 at a first depth D1. The reoriented right light signal 16' with respect to RLG_2 may have the same or a different field of view as the corresponding reoriented left light signal 36' with respect to LLS_2. In other words, the first angle θ1 determines the depth of the first virtual binocular pixel, but the reoriented right light signal 16' with respect to RLG_2 may or may not be the parallax of the corresponding reoriented left light signal 36' with respect to LLS_2. As a result, the intensity and / or brightness of the red, blue, and green (RBG) colors of the right and left light signals may be approximately the same or slightly different for shadows, field of view, etc., which better present some 3D effects.

[0045] As described above, numerous right optical signals are generated by a right optical signal generator, redirected by a right beam splitter, and then scanned directly over the right retina to form a right retinal image on the right retina. Similarly, numerous left optical signals are generated by a left optical signal generator, redirected by a left beam splitter, and then scanned over the left retina to form a left retinal image on the left retina. In the embodiment shown in Figure 9, the right retinal image 80 contains 36 right pixels in a 6x6 array, and the left retinal image 90 also contains 36 left pixels in a 6x6 array. In another embodiment, the right retinal image 80 contains 921,600 right pixels in a 1280x720 array, and the left retinal image 90 also contains 921,600 left pixels in a 1280x720 array. The virtual image module 160 is configured to generate a number of right optical signals and a number of corresponding left optical signals that form a right retinal image on the right retina and a left retinal image on the left retina, respectively. As a result, the viewer perceives a virtual image at a specific depth within region C for image fusion.

[0046] Refer to Figure 9. The first right light signal 16 from the right light signal generator 170 is received and reflected by the right beam splitter 140. The first reoriented right light signal 16' reaches the viewer's right retina through the right pupil 52 and displays the right pixel R43. The corresponding left light signal 36 from the left light signal generator 175 is received and reflected by the left beam splitter 145. The first reoriented light signal 36' reaches the viewer's left retina through the left pupil 62 and displays the left retinal pixel L33. As a result of image fusion, the viewer perceives a virtual image with multiple depths, where the depth is determined by the angles between the multiple reoriented right light signals and the corresponding multiple reoriented left light signals. The angle between the reoriented right light signal and the corresponding left light signal is determined by the relative horizontal distance between the right and left pixels. Consequently, the depth of the virtual binocular pixels is inversely correlated with the relative horizontal distance between the right pixels and the corresponding left pixels that form the virtual binocular pixels. In other words, the shorter the relative horizontal distance along the X-axis between the right and left pixels that form such virtual binocular pixels, the more deeply the virtual binocular pixels are perceived by the viewer. For example, as shown in Figure 9, the second virtual binocular pixel 74 is perceived by the viewer as having a greater depth (i.e., being further away from the viewer) than the first virtual binocular pixel 72. As a result, the horizontal distance between the second right pixel and the second left pixel is smaller than the horizontal distance between the first right pixel and the first left pixel on the retinal image. Specifically, the horizontal distance between the second right pixel R41 and the second left pixel L51, which form the second virtual binocular pixel, is the length of 4 pixels. However, the distance between the first right pixel R43 and the first left pixel L33, which form the first virtual binocular pixel, is the length of 6 pixels.

[0047] In one embodiment shown in Figure 10, the optical paths of multiple right and multiple left optical signals from an optical signal generator to the retina are illustrated. The multiple right optical signals generated from the right optical signal generator 170 are projected onto the right beam splitter 140 to form a right splitter image (RSI) 82. These multiple right optical signals are reoriented by the right beam splitter 140, focused into a small right pupil image (RPI) 84, pass through the right pupil 52, and then finally reach the right retina 54 to form a right retina image (RRI) 86. Each of the RSI, RPI, and RRI comprises i × j pixels. Each right optical signal RLS(i,j) travels from RSI(i,j) to RPI(i,j) and then to RRI(x,y) through the same corresponding pixels. For example, RLS(5,3) moves from RSI(5,3) to RPI(5,3) and then to RRI(2,4). Similarly, multiple left optical signals generated from the left optical signal generator 175 are projected onto the left beam splitter 145 to form a left splitter image (LSI) 92. These multiple left optical signals are redirected by the left beam splitter 145, focused into a small left pupil image (LPI) 94, pass through the left pupil 62, and then finally reach the left retina 64 to form a right retina image (LRI) 96. Each of the LSI, LPI, and LRI comprises i × j pixels. Each left optical signal LLS(i,j) travels from LCI(i,j) to LPI(i,j) and then to LRI(x,y) through the same corresponding pixel. For example, LLS(3,1) travels from LCI(3,1) to LPI(3,1) and then to LRI(4,6). The (0,0) pixel is the pixel at the top left corner of each image. Pixels in the retinal image are inverted left-right and up-down with respect to their corresponding pixels in the splitter image. Based on the proper arrangement of the relative positions and angles of the optical signal generators and beam splitters, each optical signal has its own optical path from the optical signal generator to the retina. The combination of one right optical signal displaying one right pixel on the right retina and one corresponding left optical signal displaying one left pixel on the left retina forms a virtual binocular pixel at a specific depth perceived by the viewer. As a result, virtual binocular pixels in space can be represented by pairs of right retinal pixels and left retinal pixels, or pairs of right splitter pixels and left splitter pixels.

[0048] The virtual image perceived by the viewer within region C includes a number of virtual binocular pixels. To accurately describe the location of each virtual binocular pixel in space, each location in space is provided with three-dimensional (3D) coordinates, e.g., XYZ coordinates. In another embodiment, other 3D coordinate systems can be used. As a result, each virtual binocular pixel has 3D coordinates, namely horizontal, vertical, and depth directions. The horizontal direction (or X-axis direction) is along the direction of the interpupillary line. The vertical direction (or Y-axis direction) is perpendicular to the horizontal direction along the midline of the face. The depth direction (or Z-axis direction) is perpendicular to the frontal plane and perpendicular to both the horizontal and vertical directions. The horizontal and vertical coordinates are collectively referred to as locations in this invention.

[0049] Figure 11 illustrates the relationship between pixels in the right splitter image, pixels in the left splitter image, and virtual binocular pixels. As described above, pixels in the right splitter image have a one-to-one correspondence with pixels in the right retinal image (right pixels). Pixels in the left splitter image have a one-to-one correspondence with pixels in the left retinal image (left pixels). However, pixels in the retinal image are inverted left-to-right and up-to-down with respect to their corresponding pixels in the combiner image. However, if eyepieces 152 and 154 are available in system 100, the relationship between pixels in the splitter image and their corresponding pixels in the retinal image may be further modified by the optical characteristics of the eyepieces. For a right retinal image with 36 (6×6) right pixels and a left retinal image with 36 (6×6) right pixels, assuming all light signals are within the field of view (FOV) of both eyes of the viewer, there are 216 (6×6×6) virtual binocular pixels (shown as dots) within region C. The spread of the optical path of one reoriented right light signal intersects with the spread of the optical path of each reoriented left light signal in the same row of the image. Similarly, the spread of the optical path of one reoriented left light signal intersects with the spread of the optical path of each reoriented right light signal in the same row of the image. As a result, there are 36 (6×6) virtual binocular pixels on one layer and six layers in space. Typically, two adjacent lines representing the spreads of optical paths that intersect to form virtual binocular pixels are shown as parallel lines in Figure 11, but there is a small angle between these lines. Right pixels and corresponding left pixels at approximately the same height in each retina (i.e., in the same row of the right and left retinal images) tend to merge earlier. As a result, the right pixel pairs with the left pixel in the same row of the retinal image to form a virtual binocular pixel.

[0050] As shown in Figure 12, a look-up table is created to facilitate the identification of the right and left pixels for each virtual binocular pixel. For example, 36 (6x6) right pixels and 36 (6x6) left pixels form 216 virtual binocular pixels numbered from 1 to 216. The first virtual binocular pixel VBP(1) represents the pair of right pixel RRI(1,1) and left pixel LRI(1,1). The second virtual binocular pixel VBP(2) represents the pair of right pixel RRI(2,1) and left pixel LRI(1,1). The seventh virtual binocular pixel VBP(7) represents the pair of right pixel RRI(1,1) and left pixel LRI(2,1). The 37th virtual binocular pixel VBP(37) represents the pair of right pixel RRI(1,2) and left pixel LRI(1,2). The 216th virtual binocular pixel VBP(216) represents the pair of right pixel RRI(6,6) and left pixel LRI(6,6). As a result, it is determined which pairs of right and left pixels can be used to generate the corresponding right and left light signals in order to display the specific binocular pixels of the virtual image in space for the viewer. In addition, each row of virtual binocular pixels on the look-up table contains a pointer that leads to a memory address that stores the perceived depth (z) and perceived position (x, y) of the VBP. Additional information, such as the size scale, the number of overlapping objects, and the depth within the sequence depth, can also be stored with respect to the VBP. The size scale may be relative size information of the specific VBP compared to a standard VBP. For example, the size scale may be set to 1 when the virtual image is displayed on a standard VBP 1 m in front of the viewer. As a result, the size scale may be set to 1.2 for a specific VBP 90 cm in front of the viewer. Similarly, the size scale may be set to 0.8 for a specific VBP 1.5 m in front of the viewer. The size scale can be used to determine the size of the virtual image to be displayed when the virtual image is moved from a first depth to a second depth. In this invention, the size scale may be a magnification. The number of overlapping objects is the number of objects that overlap each other so that one object is completely or partially hidden behind another. The depth in sequence provides information about the sequence of depths of the various overlapping images. For example, three images overlap each other. The sequence depth of the first image in the foreground may be set to 1, and the sequence depth of the second image hidden behind the first image may be set to 2. The number of overlapping images and the sequence depths may be used to determine which parts of the images should be used for display when the various overlapping images are in motion.

[0051] A look-up table may be created by the following process: In the first step, individual virtual maps based on the viewer's IPD are obtained, created by the virtual image module during startup or calibration, which specify the boundaries of the region C that a viewer can perceive as a virtual image with depth due to the fusion of the right and left retinal images. In the second step, a convergence angle is calculated to identify pairs of right and left pixels on each of the right and left retinal images, respectively, for each depth in the Z-axis direction (each point in the Z-coordinate), regardless of the location of the X and Y coordinates. In the third step, the pairs of right and left pixels are moved along the X-axis direction to specify the X and Z coordinates of each pair of right and left pixels at a depth unique to them, regardless of the location of the Y coordinate. In the fourth step, the pairs of right and left pixels are moved along the Y-axis direction to determine the Y coordinate of each pair of right and left pixels. As a result, the 3D coordinate system, such as XYZ, of each pair of right and left pixels on each of the right and left retinal images can be determined, and a look-up table can be created. In addition, the third and fourth steps are interchangeable.

[0052] The light signal generators 170 and 175 may use a laser, or a light-emitting diode ("LED") including a mini and micro light-emitting diode ("LED"), an organic light-emitting diode ("OLED"), or a superluminescent diode ("SLD"), or an LCoS (Liquid Crystal on Silicon), or a liquid crystal display ("LCD"), or any combination thereof, as their light source. In one embodiment, the optical signal generators 170 and 175 may be a laser beam scanning projector (LBS projector) equipped with a light source including a red light laser, a green light laser, and a blue light laser, a light color corrector such as a dichroism coupler and a polarization coupler, and a 2D adjustable reflector such as a two-dimensional (2D) electromechanical system ("MEMS") mirror. The 2D adjustable reflector can be replaced with two 1D reflectors such as two one-dimensional (1D) MEMS mirrors. The LBS projector sequentially generates and scans optical signals one by one to form a 2D image at a predetermined resolution, for example, 1280 × 720 pixels per frame. As a result, one optical signal is generated for each pixel and projected at once toward the beam splitters 140, 145. For a viewer to see such a 2D image with one eye, the LBS projector must sequentially generate optical signals for each pixel, for example, 1280 × 720 optical signals, within the duration of vision, for example, within a range of 1 / 18 of a second. As a result, the duration of each optical signal is approximately 60.28 nanoseconds.

[0053] In another embodiment, the optical signal generators 170 and 175 may be digital light processing projectors ("DLP projectors") capable of generating a 2D color image in a single step. Texas Instruments' DLP technology is one of several technologies that can be used to manufacture DLP projectors. For example, a 2D monochrome image frame, which may have 1280 x 720 pixels, is projected simultaneously toward splitters 140 and 145.

[0054] The beam splitters 140 and 145 receive and redirect multiple optical signals generated by the optical signal generators 170 and 175. In one embodiment, the beam splitters 140 and 145 reflect multiple optical signals so that the redirected optical signals are on the same side as the incident optical signal with respect to the beam splitters 140 and 145. In another embodiment, the beam splitters 140 and 145 refract multiple optical signals so that the redirected optical signals are on a different side from the incident optical signal with respect to the beam splitters 140 and 145. In this case, the beam splitters 140 and 145 function as refractors. The reflectivity can vary over a wide range, such as 20% to 80%, depending in part on the power of the optical signal generators. Those skilled in the art know how to determine an appropriate reflectivity based on the characteristics of the optical signal generators and splitters. Furthermore, in one embodiment, the beam splitters 140 and 145 are optically transparent to ambient light from the opposite side of the incident optical signal so that a viewer can simultaneously observe a real-time image. Transparency can vary widely depending on the application. For AR / MR applications, transparency is preferably greater than 50%, such as about 75% in one embodiment. In addition to changing the direction of the light signal, the focusing units 182 and 187 may focus multiple light signals so that multiple light signals pass through the pupil and reach both eyes of the viewer.

[0055] The beam splitters 140 and 145 may be made from glass or plastic material, such as lenses, coated with certain materials, such as metal, to make them partially transparent and partially reflective. One advantage of using reflective splitters instead of conventional waveguides to guide the light signal to the viewer's eye is that it eliminates problems of undesirable diffraction effects, such as numerous shadows and color shifts.

[0056] The above description of embodiments is provided to enable those skilled in the art to fabricate and use the subject matter. Various modifications of these embodiments will be readily apparent to those skilled in the art, and the novel principles and subject matter disclosed herein may be applied to other embodiments without using innovative capabilities. The claimed subject matter as shown in the claims is not intended to be limited to the embodiments shown herein, but should adhere to the broadest scope consistent with the principles and novel features disclosed herein. Additional embodiments are intended to fall within the spirit and true scope of the disclosed subject matter. Accordingly, the present invention is intended to include modifications and variations that fall within the scope of the appended claims and their equivalents.

Claims

1. A virtual image module for generating a virtual image with depth, A right-side light signal generator that generates a right-side light signal directed towards one retina of the viewer, A left optical signal generator that generates a left optical signal corresponding to the right optical signal, which is directed towards the other retina of the viewer, The system comprises an object measurement module configured to measure the location and depth of an object, The right light signal and the left light signal form binocular pixels of a virtual image having a second depth, and the second depth perceived by the viewer is modified by changing the convergence angle between the optical path extension of the right light signal and the optical path extension of the left light signal projected onto the viewer's eye, based on the interpupillary distance. The virtual image is superimposed on a real-time image of the object having a first depth and magnification, which is generated by a real-time image module. The second depth of the virtual image is changed in accordance with the variation in the first depth of the real-time image, and the magnification of the virtual image is changed in accordance with the variation in the magnification of the real-time image. Virtual image module.

2. The virtual image module according to claim 1, wherein the real-time image module is a surgical microscope, an endoscope, a slit-lamp microscope, or an ophthalmic microscope.

3. The virtual image module according to claim 1, wherein the virtual image module is configured to be connectable to and decoupled from the real-time image module.

4. The virtual image module according to claim 1, wherein the first depth is perceived by the viewer to be approximately the same as the second depth perceived by the viewer.

5. The virtual image module according to claim 1, wherein the right optical signal generator is located near the right portion of the real-time image module, the left optical signal generator is located near the left portion of the real-time image module, the right optical signal and the left optical signal are directed towards the viewer's retina by the right beam splitter and the left beam splitter of the real-time image module, respectively, and the orientation of the right beam splitter and the left beam splitter is configured to be changeable.

6. The virtual image module according to claim 5, wherein the right beam splitter and the left beam splitter are shared by both the real-time image module and the virtual image module.

7. The virtual image module according to claim 1, wherein the real-time image is generated by light reflected or emitted from an object.

8. The virtual image module according to claim 1, wherein the virtual image is a photograph of a body organ or tissue, a magnetic resonance image, an X-ray, a computed tomography, and an optical coherence tomography provided by the virtual image module.

9. The virtual image module according to claim 1, wherein a first point on the real-time image is selected as a first marker for superimposing the virtual image onto the real-time image by superimposing a corresponding first marker on the virtual image onto the real-time image.

10. The virtual image module according to claim 1, further comprising a control module that processes the right optical signal and the corresponding left optical signal so that the virtual image is modified to be superimposed on the real-time image based on the field of view angle, location and magnification of the real-time image.

11. The virtual image module according to claim 1, further comprising a user interface configured for the viewer to control the location of the virtual image and the second depth.

12. The virtual image module according to claim 1, further comprising a recording module that records either or both of the real-time image and the virtual image.

13. A method for overlaying a virtual image onto a real-time image, The object measurement module measures the location and depth of the object, The steps include selecting a first point on the real-time image as a first landmark, The steps include displaying the real-time image at a predetermined magnification at a first location and a first depth, A step of projecting a virtual image by projecting a right light signal to the viewer's right eye and a left light signal corresponding to the viewer's left eye, wherein the right light signal and the corresponding left light signal are perceived by the viewer to display the virtual image at a second location and a second depth, The aforementioned virtual image is a processed image of the object. The second depth of the virtual image as perceived by the viewer is changed in accordance with the variation in the first depth of the real-time image, and the magnification of the virtual image is changed in accordance with the variation in the predetermined magnification of the real-time image. The second depth perceived by the viewer is modified by changing the convergence angle between the optical path extension of the right light signal and the corresponding optical path extension of the left light signal, based on the interpupillary distance. The virtual image is characterized by superimposing on the real-time image by superimposing a corresponding first marker on the virtual image onto a first marker on the real-time image. method.

14. The method according to claim 13, characterized in that the second depth is the same as the first depth.

15. The method according to claim 13, further comprising the step of selecting a second point on the real-time image as a second marker, wherein the virtual image is superimposed on the real-time image by superimposing a corresponding second marker on the virtual image onto the second marker on the real-time image.

16. The method according to claim 13, further comprising the step of adjusting the magnification such that both the real-time image and the virtual image are enlarged and the corresponding first marker on the virtual image remains superimposed on the first marker on the real-time image at the same depth.

17. The method according to claim 13, further comprising the step of repeatedly monitoring the first location and first depth of the virtual image so that it remains superimposed on the real-time image.

Images