System and method for superimposing virtual image on instant image

Abstract

The invention discloses a system and a method for superimposing a virtual image on a real-time image. The system for superposing the virtual image on the instant image comprises an instant image module and a virtual image module. The real-time image module is composed of an amplification assembly and generates a real-time image of an object at a first position and a first depth at a preset amplification factor. The virtual image module generates a virtual image by projecting a right optical signal to the right eye of a viewer and projecting a corresponding left optical signal to the left eye of the viewer. The right optical signal and the corresponding left optical signal are perceived by the viewer and the virtual image is displayed at a second position and a second depth. The second depth is related to an angle between the right light signal projected to the eyes of the viewer and the corresponding left light signal. The second depth may be substantially the same as the first depth.

Description

[0001] This application is a divisional application of the invention patent application with the application date of August 16, 2021, application number 202180004252.X, and invention name “System and method for superimposing virtual images on real-time images”. Technical Field

[0002] The present invention relates to a system and method for superimposing a virtual image on a real-time image, and in particular, the system and method superimpose a virtual image with depth on a real-time image by projecting a plurality of right light signals and a corresponding plurality of left light signals into the eyes of a viewer. Background Art

[0003] In recent years, many visualization assistance systems and methods for assisting medical personnel during medical examinations or operations (including ophthalmic operations) have been developed. During the medical process, the visualization assistance system can provide additional image information of the patient, such as medical records, surgical parameters, such as photos, magnetic resonance imaging (MRI), X-rays, computed tomography (CT) or optical coherence tomography (OCT), etc. In some cases, the additional image information is a processed image of the patient, such as a CT image with some markings. The visualization assistance system is often used together with other medical devices that can provide real-time images of the patient. The medical staff can receive additional information provided by the visualization assistance system and separated from the real-time image. For example, the additional information can be displayed by a display alone, rather than from a surgical microscope that observes the real-time image of the patient. The display can usually only provide a plane image. However, during the medical process, the medical staff expects to be able to superimpose the additional image information (i.e., the pre-processed image of the patient) on the real-time image of the patient and observe it together. In addition, traditional visualization assistance systems can only provide additional image information in the form of a plane image. Therefore, it has become a major goal in the medical community to produce a stereoscopic image that overlaps with the real-time image of the patient and provides additional image information. For example, during an eye examination or surgery, the medical staff sees a real-time optical image of the patient's eye through the eyepiece of an ophthalmic microscope. However, the surgeon cannot see a processed retinal image of the patient's eye through the microscope during the procedure, and the surgeon must turn his head to look at another display and then back to the microscope. Therefore, there is still a need to combine the additional image information of the patient provided by the visualization assistance system with the real-time image viewed by the medical staff. Summary of the invention

[0004] The object of the present invention is to provide a system and method for superimposing a virtual image on a real-time image. The system for superimposing a virtual image on a real-time image comprises a real-time image module and a virtual image module. The real-time image module is composed of a magnifying component, which generates a real-time image of an object at a first position and a first depth with a preset magnification.

[0005] The virtual image module generates a virtual image by projecting a right light signal to the viewer's right eye and a corresponding left light signal to the viewer's left eye. The right light signal and the corresponding left light signal are perceived by the viewer and the virtual image is displayed at a second position and a second depth. The second depth is related to the angle between the right light signal and the corresponding left light signal projected to the viewer's eyes. In one embodiment, the second depth is approximately the same as the first depth. The virtual image is superimposed on the real-time image to provide the viewer with more information. Therefore, in one embodiment, the virtual image is a processed image of the object.

[0006] The magnification of the real-time image can be adjusted. After the real-time image is magnified, the virtual image can be manually or automatically magnified to maintain the superposition of the virtual image and the real-time image. The automatic mode can be selected for superposition.

[0007] In order to superimpose the virtual image on the real-time image, the system first needs to be calibrated for the viewer. Because each observer's eyes have different physical characteristics, including pupil distance, the system needs to be calibrated specifically for the viewer to ensure that the right light signal and the left light signal are projected into the viewer's eyes so that the viewer can perceive the virtual image displayed at the second position and the second depth.

[0008] The process of superimposing a virtual image on a real-time image includes: (a) selecting a first point in the real-time image as a first positioning mark; (b) displaying the real-time image at a first position and a first depth at a preset magnification; (c) projecting a right light signal to the viewer's right eye and a left light signal to the viewer's left eye to project a virtual image, so that the viewer perceives the virtual image at a second position and a second depth, and the corresponding first positioning mark on the virtual image overlaps the first positioning mark of the real-time image. In one embodiment, the depth of the first positioning mark on the real-time image is approximately the same as the depth of the corresponding first positioning mark on the virtual image. For more accurate superposition, the second positioning mark or the third positioning mark can also be applied in a similar manner.

[0009] Other features and advantages of the present invention will be described later, and some of them can be learned from the description or examples of the present invention. The objectives and other advantages of the present invention will be achieved by the structures and methods particularly pointed out in the specification, claims, and drawings. It is understood that the above general description and the following detailed description are exemplary and illustrative, and are intended to provide further explanation of the invention claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A is a schematic diagram illustrating an embodiment of a system in the present invention.

[0011] Figure 1B FIG. 4 is a schematic diagram illustrating another embodiment of a system in the present invention.

[0012] Figure 1C FIG. 4 is a schematic diagram illustrating a collimator in a virtual imaging module of a system in the present invention.

[0013] Figure 2 is a block diagram illustrating an embodiment of a system with various modules in the present invention.

[0014] Figure 3A and Figure 3B It is a schematic diagram illustrating a possible embodiment of a system in the present invention.

[0015] Figure 4 It is a schematic diagram illustrating the relationship among the object, the real-time image and the virtual image in the present invention.

[0016] Figure 5 The following are photos illustrating superimposing a virtual image of the retina on a real-time image in the present invention.

[0017] Figure 6 is a flow chart illustrating an embodiment of the process of superimposing a virtual image on a real-time image in the present invention.

[0018] Figure 7 FIG. 4 is a flow chart illustrating another embodiment of the process of superimposing a virtual image on a real-time image in the present invention.

[0019] Figure 8 FIG. 4 is a schematic diagram illustrating an embodiment of a virtual image module in the present invention.

[0020] Fig. 9 FIG. 4 is a schematic diagram illustrating the relationship between the virtual binocular pixels and the corresponding right pixel and left pixel pair in the present invention.

[0021] Fig.10 FIG. 4 is a schematic diagram illustrating the light path from the optical signal generator to the beam splitter and then to the viewer's retina in the present invention.

[0022] Fig.11 FIG. 4 is a schematic diagram illustrating the virtual binocular pixel formed by the right light signal and the left light signal in the present invention.

[0023] Fig.12 is a table illustrating an embodiment of the query table in the present invention. DETAILED DESCRIPTION

[0024] The words used in this article are used to describe the details of the specific embodiments of the present invention, and all words should be interpreted reasonably with the widest scope. Certain words will be particularly emphasized below; any restrictive terms will be defined by the specific embodiments.

[0025] The present invention relates to a system and method for superimposing a virtual image on a real-time image. A virtual image with depth can be superimposed on a real-time image to provide a viewer with more information related to the real-time image, such as surgical guidance, instructions, navigation, etc. The real-time image is an image that reflects the changes of an object in a real-time manner. The real-time image can be a plane image or a stereoscopic image. In one embodiment, the real-time image is generated by light reflected or emitted by the object, for example, the image observed from a microscope or a telescope. In another embodiment, the real-time image is generated by a display that receives an image of the object, which may be captured in real time by a camera, such as an image on an endoscope display. In addition, the real-time image can be a real image or a virtual image. The virtual image with depth is generated by projecting a light signal to both eyes of the viewer. The depth of the virtual image is related to the angle between the right light signal projected to the viewer's eyes and the corresponding left light signal. The virtual image can be a plane image or a stereoscopic image. When the virtual image is superimposed on the real-time image, a portion of the virtual image overlaps the real-time image.

[0026] A system for superimposing a virtual image on a real-time image includes a real-time image module and a virtual image module. The real-time image module includes an amplification component that generates a real-time image at a first position and a first depth with a preset magnification. Amplification is the process of making the surface size of an object larger, rather than the physical size. This amplification can be quantified by a calculated number, which is also called the "magnification ratio", which is the ratio between the surface (real-time image) size of the object and the size of the object observed without magnification. The magnification ratio is adjustable and can be any positive number, such as 0.5, 1, and 10. When the magnification ratio is less than one, it represents a reduction in size, sometimes referred to as downsizing or de-magnification.

[0027] The virtual image module projects a right light signal to the viewer's right eye and a corresponding left light signal to the viewer's left eye to generate a virtual image. The right light signal and the corresponding left light signal are perceived by the viewer and the virtual image is displayed at a second position and a second depth. The second depth is related to the angle between the right light signal and the corresponding left light signal projected to the viewer's eye. In one embodiment, the second depth is substantially the same as the first depth.

[0028] The virtual image is superimposed on the real-time image to provide the viewer with more information. Therefore, in one embodiment, the virtual image is an image processed by the object. For example, the object can be a brain and the real-time image is an image of the brain generated by a surgical microscope in real time. The virtual image can be a CT or MRI image of the brain taken before the operation, and the position of the brain tumor to be removed during the operation is marked on it. The marked virtual image will be superimposed on the real-time image of the brain during the operation to help the surgeon identify the position of the brain tumor to be removed. In this case, in order to make the surgical position accurate, the second depth of the virtual image (the marked CT or MRI image) will be roughly the same as the first depth of the real-time image (the actual image of the brain from the surgical microscope). The virtual image can further include some text information, marks, or indicators for guidance or explanation to assist in diagnosis and treatment. In addition, the superposition of images allows the viewer to compare the previous image of the object represented by the virtual image and the current state of the object represented by the real-time image, thereby predicting the development of the disease and the treatment results.

[0029] The magnification of the real-time image is adjustable. In one embodiment, it can be adjusted by manually turning a knob, changing an objective lens, controlling a virtual switch, or issuing a verbal command. After the real-time image is magnified, the virtual image can be manually or automatically magnified to maintain the superposition of the virtual image and the real-time image. An automatic mode can be selected for superposition.

[0030] In order to superimpose the virtual image on the real-time image, the system first needs to be calibrated for the viewer. Because each observer's eyes have different physical characteristics, including pupil distance, the system needs to be calibrated specifically for the viewer to ensure that the right light signal and the left light signal are projected into the viewer's eyes so that the viewer can perceive the virtual image displayed at the second position and the second depth. For example, the distance between the left eyepiece and the right eyepiece of the microscope needs to be adjusted to match the viewer's pupil distance; the angle between the right light signal and the corresponding left light signal needs to be adjusted so that the viewer can perceive the virtual image exactly at the second depth.

[0031] The process of superimposing a virtual image on a real-time image includes: (a) selecting a first point in the real-time image as a first positioning mark; (b) displaying the real-time image at a preset magnification at a first position and a first depth; (c) projecting a right light signal to the viewer's right eye and a left light signal to the viewer's left eye to project a virtual image, so that the viewer perceives the virtual image at a second position and a second depth, and the corresponding first positioning mark on the virtual image overlaps the first positioning mark of the real-time image. As described above, the second depth is related to the angle between the right light signal and the corresponding left light signal projected on the viewer's eye. In one embodiment, the depth of the first positioning mark on the real-time image is approximately the same as the depth of the corresponding first positioning mark on the virtual image. For more precise superposition, the second positioning mark or the third positioning mark can also be applied in a similar manner.

[0032] like Figure 1A and Figure 1BAs shown, the system 100 for superimposing a virtual image 165 on a real-time image 115 includes a real-time image module 110 and a virtual image module 160. The real-time image module 110 may include a magnifying component 120 to generate a magnified real-time image of an object 105 (such as a brain) for both eyes of a viewer. The magnifying component 120 may include a plurality of optical units and components, such as different types of lenses, including an objective lens 113. In another embodiment, the magnifying component 120 may use an electronic circuit to process and magnify the real-time image of the object 105. The magnification of the real-time image can be determined before observation or adjusted during observation. The magnification can be 1 / 2, 3, 10, 100, etc. The magnification can be adjusted through a user interface linked to the real-time image module. The real-time image module can have a set of optical units and components to generate the real-time image for both eyes of the viewer; or there can be two separate sets of optical units and components to generate the real-time image for the right eye and the left eye of the viewer respectively. The real-time image module 110 may further include a prism assembly 130 to redirect the direction of light, a beam splitter 140, 145 to split light, an observation tube 150 to guide light, and eyepieces 152, 154 to further magnify the image. Similarly, the real-time image is generated by light reflected or emitted by the object 105, such as the real-time image generated from a microscope (including a surgical microscope). In another embodiment, the real-time image is generated by an image capture device and a display device, such as an endoscope and its associated display. Depending on the size and resolution of the image, the real-time image may actually or conceptually contain 921,600 pixels (1280x720 matrix). Each pixel may be slightly different from adjacent pixels in position and depth. Representative pixels may be selected for the real-time image, such as a first positioning mark. Positioning marks, such as the first positioning mark and the second positioning mark, are usually a special point in the real-time image with features that are easily recognizable by the viewer, such as a midpoint or the intersection of two specific blood vessels. Positioning marks may be pixels or composed of several adjacent pixels. In one embodiment, the position and depth of the representative pixel are used as the position and depth of the real-time image, ie, the first position and the first depth.

[0033] The virtual image module 160 is connected to the real-time image module 110 and includes a right light signal generator 170 and a left light signal generator 175. The right light signal generator 170 generates a plurality of right light signals for the virtual image and may be close to the right portion of the real-time image module. Similarly, the left light signal generator 175 generates a plurality of left light signals for the virtual image and may be close to the left portion of the real-time image module. The right light signal is then redirected by the right beam splitter 140 to one eye of the viewer. Similarly, the left light signal is then redirected by the left beam splitter 145 to the other eye of the viewer. The redirected right light signal and the corresponding redirected left light signal are perceived by the viewer and the virtual image is displayed at a second depth. Depending on the size and resolution of the image, the virtual image may include 921,600 virtual binocular pixels (1280x720 matrix). Each virtual binocular pixel may be slightly different in position and depth from adjacent pixels. Representative virtual binocular pixels, such as first positioning markers, may be selected for the virtual image. In one embodiment, the position and depth of the representative virtual binocular pixel are used as the position and depth of the virtual image, i.e., the second position and the second depth. After the viewer's eyes receive the redirected right light signal and the corresponding redirected left light signal of the representative virtual binocular pixel, the viewer perceives the representative virtual binocular pixel at the second depth, and the second depth is related to the angle between the redirected right light signal and the corresponding redirected left light signal.

[0034] The beam of the real-time image can also pass through the right beam splitter 140 and the left beam splitter 145 to the eyes of the viewer. Therefore, to a certain extent, the real-time image module and the virtual image module share the right beam splitter 140 and the left beam splitter 145. In one embodiment, the beam splitter originally installed in the real-time image module to share the real-time image with other viewers can be rotated at an appropriate angle so as to redirect the light signal generated by the virtual image module to the eyes of the viewer.

[0035] like Figure 1B and Figure 1CAs shown, the virtual image module 160 may further include a right focus adjustment unit 182, located between the right optical signal generator 170 (or the right collimator 180, if any) and the right beam splitter 140; and a left focus adjustment unit 187, located between the left optical signal generator 175 (or the left collimator 185, if any) and the left beam splitter 145, to improve the clarity of the virtual image to the viewer. The right / left focus adjustment unit may include an optical unit, such as different types of lenses, including a convex lens. In one embodiment, the convex lens is used as a focus adjustment unit, and the distance between it and the optical signal generator can be adjusted to change the focal position of the light beam. Assuming that the distance between the optical signal generator and the beam splitter remains unchanged, the closer the focal position of the light beam is to the retina, the clearer the virtual image seen by the viewer. Since the axial length of the viewer's eyes may be different, the optimal focal position of the light beam and the distance between the optical signal generator and the focus adjustment unit will also change accordingly. In other words, if the viewer's eye axis is longer, the focus adjustment unit needs to be farther from the optical signal generator so that the focus position of the light beam is closer to the viewer's retina. If a collimator is provided, the focus adjustment unit is located between the collimator and the beam splitter. After passing through the collimator, the light beams from the optical signal generator become roughly parallel and then converge after passing through the focus adjustment unit. In addition, because the focus adjustment unit does not change the incident angle of the light beam, the depth of the virtual image is not affected.

[0036] like Figure 1C As shown in the figure, the virtual image module 160 may further include a right collimator 180 and a left collimator to reduce the beams of the plurality of light signals, for example, to make the movement direction more consistent in a specific direction or to reduce the spatial cross-section of the light beam. The right collimator 180 may be disposed between the right light signal generator 170 and the right beam splitter 140; the left collimator 180 may be disposed between the left light signal generator 175. The collimator may be a curved mirror or lens.

[0037] In addition, the virtual image module 160 may include a control module 190 for controlling the virtual image signals of the right light signal generator 170 and the left light signal generator 175. The control module 190 is connected to the virtual image module 160 in communication to adjust the right light signal and the corresponding left light signal, so that the virtual image can be automatically modified according to the change of the real-time image to superimpose the virtual image on the real-time image. The change of the real-time image includes a change in viewing angle, magnification or position. For example, when the magnification of the real-time image is adjusted (for example, from three times to ten times), the control module 190 will process the image signal, enlarge the virtual image to the same size, and use the first positioning mark to allow the virtual image to continue to be superimposed on the real-time image. Although the control module 190 includes one or more processors, for complex signal processing, the control module 190 can use an external server 250 for calculation.

[0038] The virtual image module can be stored in the memory module 195. In one embodiment, the virtual image is a processed image of the object, such as an X-ray image, an ultrasound image, a CT image or an MRI image of the object, with some marks or highlights in the area of ​​interest. The virtual image can further include some text information and marks for guidance or explanation. For example, the virtual image can be a pre-taken and processed retinal image of a patient with marks of bleeding blood vessels to be sealed with a laser. The system 100 can superimpose the virtual image on a real-time image of the same retina from a slit lamp microscope. The control module 190 can retrieve the virtual image stored in the memory module 195 and, when necessary, allow the right optical signal generator 170 and the left optical signal generator 175 to generate virtual image signals.

[0039] like Figure 2As shown, in addition to the real-time image module 110 and the virtual image module 160, the system 100 may further include a recording module 210 to record the real-time image and the virtual image; an object measurement module 220 for measuring the position and depth of the object; a surgery module 230 for performing surgery on the object; and a user interface 240, so that the viewer can communicate with various modules in the system 100 and control various functions of the system 100. All modules of the system 100 can communicate with each other by wire or wirelessly. Wireless methods may include WiFi, Bluetooth, near field communication (NFC), network, telecommunications, radio waves, etc. The real-time image module 110, the virtual image module 160 and the recording module 210 can communicate optically through light beams and light signals. The viewer can observe the real-time image and the virtual image through the system 100, and control the system 100 by physically interacting with the user interface 240. The system 100 can communicate optically with the object 105, such as receiving a light beam reflected or emitted from the object and projecting the light beam on the object. The system may physically interact with the object 105, such as performing laser surgery on the object.

[0040] As described above, the system 100 may further include a recording module 210 to record the real-time image and / or the virtual image. In one embodiment, the recording module 210 may be placed on the right beam splitter 140 and the left beam splitter 145 to record the real-time image, that is, the light beam from the object reflected by the right beam splitter and the left beam splitter during the operation. The recording module 210 may include a digital camera or a light coupling device (CCD) to capture the image. In another embodiment, the recording module 210 is located next to the eyepiece to record the light beam that passes through the eyepiece but has not yet reached the observer's eyes, including the light beam that forms the real-time image and the virtual image. The recording module 210 may be connected to the control unit to directly record the virtual image signal and related information and parameters for later display.

[0041] As described above, the system 100 may further include an object measurement module 220 to measure the position or depth of an object. The object measurement module 220 connected to the system may continuously or periodically measure the position and depth of the object relative to the object measurement module (or the viewer), and transmit relevant information to the virtual image module to adjust the virtual image. Based on the received information, the control module 190 processes the virtual image signal according to the updated position of the object relative to the object measurement module and the viewer. Therefore, the virtual image can be maintained superimposed on the real-time image. The distance or relative position between the object 105 and the object measurement module 220 (or the observer's eyes) may change over time. In one case, the object 105, such as a part of the human body—an eyeball, may move during surgery. In another case, the system 100 may be worn by a viewer (such as a surgeon), and the viewer may move his head during surgery. Therefore, in order to maintain the virtual image superimposed on the real-time image, the relative position and distance between the object 105 and the observer's eyes need to be measured and calculated. The object measurement module 220 may include a gyroscope, an indoor / outdoor global positioning system (GPS), and distance measurement components (ie, a transmitter and a sensor) to accurately track the relative position and depth change of the object 105 .

[0042] As described above, the system 100 may further include a surgery module 230 to physically perform surgery on the object 105. The surgery module 230 may include a laser for removing tissue or sealing a bleeding blood vessel; and / or a scalpel for cutting tissue. The surgery module 230 may cooperate with the real-time imaging module 110 to position the laser and / or scalpel at a point of interest to the viewer (i.e., the surgeon) in the real-time image.

[0043] As described above, the system 100 may further include a user interface that allows the viewer to control various functions of the system 100, such as the magnification of the real object, the second position and the second depth of the virtual image, the focus adjustment unit, the recording module 210, the object measurement module 220, etc. The user interface 140 may be operated by voice, gesture, finger / foot movement, etc. in the form of a pedal, keyboard, mouse, knob, switch, stylus, button, stick, touch screen, etc. The user interface communicates with other modules of the system 100 (including the real-time imaging module 110, the virtual imaging module 160, the recording module 210, the object recording module 220, and the surgical module 230) by wired or wireless means. Wireless methods may include WiFi, Bluetooth, near field communication (NFC), network, telecommunications, radio waves, etc. The viewer can use the user interface 240, such as a joystick, to move a cursor to a point of interest on the real-time image, and then use the user interface 240, such as a pedal, to activate a laser to the point of interest corresponding to the object 105 to remove the tissue or seal a bleeding blood vessel.

[0044] In one embodiment, the system 100 may be an AR microscope for surgery or diagnosis, such as an AR ophthalmoscope and an AR slit lamp. Figure 3A An example of a stationary AR surgical microscope 310 including a user interface pedal 320 is described. Figure 3B The example of a portable AR surgical microscope 350 is described, which is a head mounted device, and includes a real-time imaging module 370 and a virtual imaging module 360. The real-time imaging module is connected to the virtual imaging module 360 ​​but can be separated.

[0045] like Figure 4 As shown, the object 105, the real-time image 115 generated by the real-time image module 110, and the virtual image 165 generated by the virtual image module 160 may have different positions and depths. In this embodiment, the virtual image 165 is a processed partial image of the object 105. The virtual image module 160 may only generate an interesting portion of the object as the virtual image. The image of the object may be intercepted and processed, such as by an artificial intelligence (AI) module, so as to generate the virtual image in a very short time, such as within one second.

[0046] As described above, depending on the resolution, the object 105, the real-time image 115, and the virtual image 165 may include a large number of pixels, such as 921,600 pixels (1280x720 matrix), conceptually or actually. In this embodiment, the position and depth of the object 105, the real-time image 115, and the virtual image 165 are represented by the position and depth of the corresponding first positioning mark, respectively. The depth is measured based on the distance between the eyepiece 152 and the object 105, the real-time image 115, or the virtual image 165. Therefore, if Figure 4 As shown, the object 105 is located at the object position L(o) and the object depth D(o); the real-time image 115, the magnified image of the object 105, is located at the first position L(r) and the first depth D(r); and the virtual image 165 is located at the second position L(v) and the second depth D(v). Depending on the optical characteristics of the real-time image module, the depth of the real-time image D(r) may be closer to or farther away from the viewer's eyes. In this embodiment, the depth D(r) of the real-time image is deeper than the depth D(o) of the object. However, in other embodiments, the depth D(r) of the real-time image may be smaller than or approximately the same as the depth D(o) of the object, and then the virtual image module 160 will generate the virtual image 165 at a depth D(v) closer to the eyepiece than the real-time image 115.

[0047] like Figure 4 As shown, with the information of L(r) and D(r), the virtual image module 160 of the system 100 can superimpose the virtual image on the real-time image by overlapping the first positioning mark LM1(v) corresponding to the virtual image with the first positioning mark LM1(r) on the real-time image. For more accurate superposition, the virtual image module 160 of the system 100 can further overlap the second positioning mark LM2(v) corresponding to the virtual image with the second positioning mark LM2(r) on the real-time image. In another embodiment, the superposition is not limited to overlapping the positions of the positioning marks, and the depth of the corresponding first positioning mark in the virtual image may be approximately the same as the depth of the first positioning mark in the real-time image. Similarly, the depth of the corresponding second positioning mark in the virtual image may be approximately the same as the depth of the second positioning mark in the real-time image. In order to accurately and completely superimpose the stereoscopic virtual image on the stereoscopic real-time image, in addition to the first positioning mark and the second positioning mark, the third positioning mark on the real-time image must also be selected, and then the virtual image module makes the position and depth of the corresponding third positioning mark in the virtual image roughly the same as the third positioning mark in the real-time image.

[0048] Figure 5Three images are shown, namely a live image of the patient's retina, a processed virtual image of the retina, and a superimposed image of the two images. In one embodiment, the angiographic image of the patient's retina may be captured and processed by a slit lamp biomicroscope, and then the virtual image module 160 may use the processed image as a virtual image superimposed on the live image of the patient's retina during surgery to assist in identifying and visualizing the edge of the choroidal neovascular membrane. The AR / MR microscope can greatly facilitate the diagnosis and treatment of various ophthalmic diseases.

[0049] like Figure 6 As shown, the process of superimposing a virtual image on a real-time image includes four steps. In step 610, a viewer, an expert, a computer, or the system 100 selects a first point on the real-time image as a first positioning mark. For example, the viewer can use a mouse or a joystick to move a cursor or pointer that can be seen from the eyepiece to select the first positioning mark on the real-time image. As described above, positioning marks, including the first positioning mark, the second positioning mark, and the third positioning mark, are usually special points with identification features that can be easily identified by the viewer in the real-time image, such as the midpoint, the intersection of two specific blood vessels. Positioning marks can be manually defined by an expert or automatically defined by a computer program. There are three basic types of defined marks: anatomical positioning marks, mathematical positioning marks, and pseudo-positioning marks. Anatomical positioning marks are points with biological significance in the body. Any anatomical feature, folds, protrusions, ducts, blood vessels, etc., will always exist in the tissue and can be used to indicate a specific structure or position. Surgical pathologists can use anatomical positioning marks to confirm the direction of the specimen. Mathematical markers are points within a shape that are located according to some mathematical or geographical characteristics, such as points of high curvature or endpoints. Mathematical markers can be used by computer programs for automatic shape recognition. Pseudo-markers are points between anatomical markers and mathematical markers. A typical example is a set of equidistant points between two anatomical markers to get more sample points from a shape. Pseudo-markers are useful in the shape matching process when a large number of points are needed for the matching process. Markers can be pixels or composed of several adjacent pixels.

[0050] In step 620, the real-time image of the object is displayed at a first position and a first depth with a preset magnification. As described above, there are at least two types of real-time images. The first real-time image is generated by the object reflecting or emitting light, such as an image seen from a microscope or a telescope. In this case, the first position and the first depth may be determined by the optical properties of the real-time imaging module. The viewer can see the real-time image through an eyepiece. The second real-time image is generated by a display that receives an image of the object captured in real time by a camera, such as an image on a display of an endoscope (including a gastroscope, a colonoscope, or a proctoscope). The endoscope may have two image capture devices located in different places for capturing and generating stereoscopic images. The real-time image can be a planar image or a stereoscopic image. Steps 610 and 620 can be swapped.

[0051] In step 630, the virtual image module may be calibrated for a specific viewer. As described above, some physical characteristics of each viewer, such as pupil distance, may affect the position and depth of the virtual image displayed by the viewer's perception of the same right light signal and the corresponding left light signal. In one embodiment, the control module may adjust the virtual image signal according to the viewer's pupil distance so that the right light signal generator 170 and the left light signal generator 175 can project the light signal to an appropriate position and angle to ensure that the viewer perceives the virtual image at the second position and the second depth.

[0052] In step 640, the virtual image module projects a virtual image by projecting a right light signal to the viewer's right eye and a corresponding left light signal to the viewer's left eye, respectively, so that the viewer sees the virtual image at a position and a second depth, so that the first positioning mark corresponding to the virtual image will overlap with the first positioning mark on the real-time image. In other words, the virtual image module projects a virtual image and superimposes it on the real-time image. At least the position (second position) of the corresponding first positioning mark on the virtual image will be approximately the same as the position (first position) of the first positioning mark on the real-time image. Generally speaking, according to its resolution, the virtual image will be divided into a number of virtual binocular pixels, such as 921600 pixels (1280x720 matrix). For each right light signal and corresponding left light signal projected to the viewer's retina, the viewer perceives a virtual binocular pixel at a specific position and depth. The depth is related to the angle between the right light signal and the corresponding left light signal. When the first positioning mark on the real-time image is located at the first position and the first depth, the virtual binocular pixel corresponding to the first positioning mark on the virtual image is projected and perceived by the viewer at the second position and the second depth. For initial superposition, the position (second position) corresponding to the first positioning mark on the virtual image is set to be approximately the same as the position (first position) of the first positioning mark on the real-time image, and their depths may be different. This superposition can be done manually by the viewer or by the system 100 using graphic recognition technology, including artificial intelligence (AI) algorithms. In order to further improve the quality of superposition, the second depth is set to be approximately the same as the first depth. In addition, if the real-time image is enlarged from the actual size of the object, the virtual image also needs to be enlarged to the same extent for superposition. In addition, in order to further improve the quality of superposition, the viewing angle of the virtual image needs to be consistent with the viewing angle of the real-time image. The relationship between the light signal generated by the light signal generator and the depth perceived by the viewer will be described in detail below.

[0053] In step 650, if the position, magnification or viewing angle of the real-time image changes, the virtual image module modifies the virtual image to maintain the superposition between the virtual image and the real-time image. The change in the position, magnification and viewing angle of the real-time image may be caused by the operation of the viewer or the movement of the object or the viewer. The system 100 needs to continuously monitor the first position and the first depth of the real-time image, and the second position and the second depth of the virtual image. Once the real-time image has any changes, the virtual image module will modify the virtual image to maintain the superposition between the virtual image and the real-time image.

[0054] like Figure 7 Another process for superimposing a virtual image on a live image includes six steps. Figure 6 The same or similar to the previous embodiment shown. Some steps are optional and further modified. In step 710, the first point, the second point, and the third point are selected as the first positioning mark, the second positioning mark, and the third positioning mark by the viewer, the expert, the computer or the system 100, respectively. For the most accurate superposition, three positioning marks are used here. Some surgeries, such as cranial nerve surgery, require very high precision, so three positioning marks are required to ensure that the virtual image is completely superimposed on the real-time image. However, depending on the needs, the process may only include two positioning marks. Step 720 is the same as step 620, and step 730 is the same as step 630. Step 740 is the same as the principle described in step 640, however, the position and depth of the corresponding first positioning mark, the second positioning mark, and the third positioning mark on the virtual image are respectively substantially the same as the position and depth of the first positioning mark, the second positioning mark, and the third positioning mark on the real-time image. At step 750, the first position and the first depth are repeatedly monitored and measured, and the first position and the first depth are calculated based on the position and depth of the object relative to the object measurement module (or the viewer) as measured by the object measurement module. Thus, the virtual image can remain superimposed on the real-time image. At step 760, the viewer, i.e., the surgeon, performs surgery using a laser or a scalpel at the point of interest determined by the viewer.

[0055] The virtual image module 160 and the method for generating the virtual image 165 at the second position and the second depth, as well as the method for moving the virtual image as required, will be discussed in detail below. PCT International Application No. PCT / US20 / 59317, filed on November 6, 2020, entitled “SYSTEM AND METHOD FOR DISPLAYING AN OBJECT WITH DEPTHS”, is incorporated herein by reference in its entirety. Figure 8As shown, the virtual image module 160 includes a right light signal generator 170 to generate a plurality of right light signals such as RLS_1, RLS_2, and RLS_3, which are 12, 14, and 16 respectively; a right beam splitter 140 to receive and redirect the plurality of right light signals to the right retina 54 of the viewer; a left light signal generator 175 to generate a plurality of left light signals such as LLS_1, LLS_2, and LLS_3, which are 32, 34, and 36 respectively; and a left beam splitter 145 to receive and redirect the plurality of right light signals to the left retina 64 of the viewer. The viewer has a right eye 50, including a right pupil 52 and a right retina 54, and a left eye 60, including a left pupil 62 and a left retina 64. The diameter of a human pupil is generally between 2 cm and 8 cm, which depends in part on the amount of light in the environment. The normal pupil size of an adult is about 2 to 4 cm in a bright environment and about 4 to 8 cm in a dark environment. The several right light signals are redirected by the right beam splitter 140, pass through the right pupil 52, and are finally received by the right retina 52. The right light signal RLS_1 refers to the rightmost light signal that the viewer's right eye can see on a specific horizontal plane. The right light signal RLS_2 refers to the leftmost light signal that the viewer's right eye can see on a specific horizontal plane. After receiving the redirected right light signal, the viewer perceives several right light pixels of the object 105 in the area A included in the extended parts of the redirected right light signals RLS_1 and RLS_2. The area A is called the field of view (FOV) of the right eye 50. Similarly, the several left light signals are redirected by the left beam splitter 145, pass through the left pupil 62, and are finally received by the left retina 64. The left light signal LLS_1 refers to the rightmost light signal that the viewer's left eye can see on a specific horizontal plane. The left light signal LLS_2 refers to the leftmost light signal that the viewer's left eye can see on a specific horizontal plane. After receiving the redirected left light signal, the viewer perceives several left light pixels of the object 105 in the area B included in the extended part of the redirected left light signals LLS_1 and LLS_2. The area B is called the field of view of the left eye 60. When several right pixels and several left pixels on both sides are displayed in the overlapping area C of area A and area B, at least the right light signal of the right pixel and the left light signal of the corresponding left pixel are fused and the virtual binocular pixels are displayed at a specific depth in area C. The depth is related to the angle between the redirected right light signal and the redirected left light signal projected to the viewer's retina, which is also called the convergence angle.

[0056] like Figure 8 and Fig. 9As shown, the viewer perceives a virtual image of a brain object 105 with several depths in an area C in front of the viewer. The image of the brain object 105 includes a first virtual binocular pixel 72 displayed at a first depth D1, and a second virtual binocular pixel 74 displayed at a second depth D2. The first angle between the first redirected right light signal 16' and the corresponding first redirected left light signal 26' is θ1. The first depth D1 is related to the first angle θ1. In particular, the first depth of the first virtual binocular pixel of the object 105 can be determined by the first angle θ1 between the light extension paths of the first redirected right light signal and the corresponding first redirected left light signal. As a result, the first depth D1 of the first virtual binocular pixel 72 can be approximately calculated by the following formula

[0057]

[0058] The distance between the right pupil 52 and the left pupil 62 is the interpupillary distance (IPD). Similarly, the second angle between the second redirected right light signal 18' and the corresponding second redirected left light signal 38' is θ2. The second depth D1 is related to the second angle θ2. In particular, the second depth D2 of the second virtual binocular pixel of the object 105 can be determined by the same formula, by the second angle θ2 between the second redirected right light signal and the corresponding first redirected left light signal light extension path. Because the second virtual binocular pixel 74 is perceived farther from the viewer than the first virtual binocular pixel 72 (greater depth), the second angle θ2 is smaller than the first angle θ1.

[0059] In addition, although the redirected right light signal 16' of RLG_2 and the redirected left light signal 36' of LLS_2 jointly display the first virtual binocular pixel 72 at the first depth D1, the redirected right light signal 16' of RLG_2 may have the same or different viewing angles as the corresponding redirected left light signal 36' of LLS_2. In other words, although the first angle θ1 determines the depth of the first virtual binocular pixel 72, the redirected right light signal 16' of RLG_2 and the corresponding redirected left light signal 36' of LLS_2 may have parallax. Therefore, the intensity of red, green and blue (RGB) color light and / or the brightness of the right light signal and the left light signal may be substantially the same, or may be slightly different due to shadows, viewing angles, etc., to achieve a better 3D effect.

[0060] As described above, the right optical signals are generated by the right optical signal generator, redirected by the right beam splitter, and then scanned by the right retina to form a right retinal image on the right retina. Similarly, the left optical signals are generated by the left optical signal generator, redirected by the left beam splitter, and then scanned by the left retina to form a left retinal image on the left retina. Fig. 9 In one embodiment shown, the right retinal image 80 includes 36 right pixels (6x6 matrix) and a left retinal image 90 also includes 36 left pixels (6x6 matrix). In another embodiment, the right retina 80 includes 921,600 right pixels (1280x720 matrix) and the left retinal image also includes 921,600 left pixels (1280x720 matrix). The virtual image module 160 is used to generate a plurality of right light signals and a corresponding plurality of left light signals, which respectively form the right retinal image on the right retina and the left retinal image on the left retina. As a result, the viewer perceives a virtual object with a specific depth in area C due to image fusion.

[0061] refer to Fig. 9 , the first right light signal 16 from the right light signal generator 170 is received and reflected by the right beam splitter 140. The first redirected right light signal 16' passes through the right pupil 52 to reach the right retina of the viewer and displays the right pixel R43. The corresponding left light signal 36 from the left light signal generator 175 is received and reflected by the left beam splitter 145. The first redirected left light signal 36' passes through the left pupil 62 to reach the left retina of the viewer and displays the left pixel L33. As a result of the image fusion, the viewer perceives a virtual object with several depths, the depth of which is determined by the angles between the several redirected right light signals and the corresponding several redirected left light signals. The angle between the redirected right light signal and the corresponding left light signal is determined by the relative horizontal distance between the right pixel and the left pixel. Therefore, the depth of the virtual binocular pixel is inversely proportional to the horizontal distance between the right pixel and the corresponding left pixel forming the virtual binocular pixel. In other words, the deeper the virtual binocular pixel perceived by the viewer, the smaller the horizontal distance on the x-axis between the right pixel and the left pixel forming the virtual binocular pixel. Fig. 9As shown, the second virtual binocular pixel 74 perceived by the viewer is deeper than the first virtual binocular vision pixel 72 (i.e., farther than the viewer). Therefore, on the retinal image, the horizontal distance between the second right pixel and the second left pixel is smaller than the horizontal distance between the first right pixel and the first left pixel. Specifically, the horizontal distance between the second right pixel R41 and the second left pixel L51 forming the second virtual binocular pixel is four pixels long. However, the horizontal distance between the first right pixel R43 and the first left pixel L33 forming the first virtual binocular pixel is six pixels long.

[0062] like Fig.10The embodiment shown illustrates the light paths of several right light signals and several left light signals from the light signal generator. The several right light signals are generated by the right light signal generator 170 and projected to the right beam splitter 140 to form a right beam splitter image (RSI) 82. The several right light signals are redirected and converged by the right beam splitter 140 into a tiny right pupil image (RPI) 84, pass through the right pupil 52, and finally reach the right retina 54 to form a right retinal image (RRI) 86. RSI, RPI, and RRI are all composed of ixj pixels. Each right light signal RLS(i,j) will pass through the corresponding pixel, from RSI(i,j) to RPI(i,j), and then to RRI(x,y). For example, RLS(5,3) will go from RSI(5,3) to RPI(5,3), and then to RRI(2,4). Similarly, the left light signals are generated by the left light signal generator 175 and projected to the left beam splitter 145 to form a left beam splitter image (LSI) 92. The left light signals are redirected and converged by the left beam splitter 145 into a small left pupil image (LPI) 94, pass through the left pupil 62, and finally reach the left retina 64 to form a left retinal image (LRI) 96. LSI, LPI, and LRI are all composed of ixj pixels. Each left light signal LLS(i,j) will pass through the corresponding pixel, from LSI(i,j) to LPI(i,j), and then to LRI(x,y). For example, ALS(3,1) will go from LSI(3,1) to LPI(3,1), and then to LRI(4,6). The (0,0) pixel is the upper leftmost pixel of each image. The pixels in the retinal image are opposite to the corresponding pixels in the beam splitter image and are upside down. When the relative positions of the optical signal generator and the beam splitter are arranged, each optical signal has its own light path from the optical signal generator to the retina. The right optical signal that displays the right pixel on the right retina and the corresponding left optical signal that displays the left pixel on the left retina together form a virtual binocular pixel with a specific depth and are perceived by the viewer. Therefore, the virtual binocular pixel in space can be represented by a pair of right retinal pixels and left retinal pixels or a pair of right beam splitter pixels and left beam splitter pixels.

[0063] The virtual object perceived by the viewer in region C may include several virtual binocular pixels. In order to accurately describe the position of the virtual binocular pixels in space, each position in space will have a stereo coordinate, such as an XYZ coordinate. Other stereo coordinate systems can be used in other embodiments. Therefore, each virtual binocular pixel has a stereo coordinate: a horizontal direction, a vertical direction, and a depth direction. The horizontal direction (or X-axis direction) is the direction along the interpupillary line; the vertical direction (or Y-axis direction) is the direction along the midline of the face and is perpendicular to the horizontal direction; the depth direction (or Z-axis direction) refers to the normal of the front plane and is perpendicular to both the horizontal and vertical directions. The horizontal coordinate and the vertical coordinate are collectively referred to as positions in the present invention.

[0064] Fig.11 The relationship between the pixels in the right spectroscopic image, the pixels in the left spectroscopic image, and the virtual binocular pixels is described. As described above, each pixel in the right spectroscopic image corresponds to a pixel in the right retinal image (right pixel). Each pixel in the left spectroscopic image corresponds to a pixel in the left retinal image (left pixel). However, the pixels in the retinal image are opposite to the corresponding pixels in the spectroscopic image and are upside down. However, if the system 100 has eyepieces 152, 154, the pixels in the spectroscopic image and the corresponding pixels in the retinal image can be further modified due to the optical properties of the eyepieces. For a right retinal image composed of 36 (6x6) right pixels and a left retinal image composed of 36 (6x6) left pixels, assuming that all light signals are within the field of view (FOV) of the viewer's two eyes, there are 216 (6x6x6) virtual binocular pixels (displayed as a point) in area C. The ray extension path of the redirected right light signal will intersect with the ray extension paths of all redirected left light signals in the same row of the image. Similarly, the ray extension path of the redirected left light signal will intersect with the ray extension paths of all redirected right light signals in the same row of the image. Therefore, there will be 36 (6x6) virtual binocular visions in one layer and there are six layers in space. Although in Fig.11 In the image, it looks like parallel lines, but the intersection of two adjacent light rays and the formation of virtual binocular pixels means that there is a small angle between them. The right pixel and the corresponding left pixel at approximately the same height in the retina (i.e., in the same row of the right retinal image and the left retinal image) usually fuse earlier. Therefore, the right pixel is paired with the left pixel in the same row of the retinal image and forms a virtual binocular pixel.

[0065] like Fig.12As shown, a lookup table is created to facilitate identification of the right pixel and left pixel pair of each virtual binocular pixel. For example, 216 virtual binocular pixels are composed of 36 right pixels and 36 left pixels, numbered from 1 to 216. The first (1st) virtual binocular pixel VBP(1) represents the right pixel RRI(1,1) and the left pixel LRI(1,1). The second (2nd) virtual binocular pixel VBP(2) represents the right pixel RRI(2,1) and the left pixel LRI(1,1). The seventh (7th) virtual binocular pixel VBP(7) represents the right pixel RRI(1,1) and the left pixel LRI(2,1). The thirty-seventh (37th) virtual binocular pixel VBP(37) represents the right pixel RRI(1,2) and the left pixel LRI(1,2). The two hundred and sixteenth (216th) virtual binocular pixel VBP(216) represents the right pixel RRI(6,6) and the left pixel LRI(6,6). Therefore, in order to display a specific virtual binocular pixel of a virtual object in space to the viewer, it is necessary to determine which pair of right pixels and left pixels can be used to generate corresponding right light signals and left light signals. In addition, each row of virtual binocular pixels in the query table includes an indicator that leads to a memory address storing the perceived depth (z) and perceived position (x, y) of the VBP. Additional information can also be stored in the VBP, such as size ratio, number of overlapping objects, and sequence depth. The size ratio is the relative size information of a specific VBP compared to a standard VBP. For example, when the virtual object is displayed at the standard VBP one meter in front of the viewer, the size ratio can be set to 1. Therefore, for the specific VBP 90 centimeters in front of the viewer, the size ratio can be set to 1.2. Similarly, for the specific VBP 1.5 meters in front of the viewer, the size ratio can be set to 0.8. When the virtual object moves from a first depth to a second depth, the size ratio can be used to determine the size of the virtual object display. The size ratio can be a magnification in the present invention. The number of overlapping objects is the number of objects that are partially or completely covered by other objects due to overlap. Sequence depth includes the depth order of each overlapping image. For example, there are three images overlapping each other. The first image at the front has a sequence depth of 1, and the second image covered by the first image has a sequence depth of 2. The number of overlapping images and the sequence depth are used to determine which image and which part of the image to display when the various overlapping objects are moved.

[0066] The query table is created by the following steps. The first step: obtaining a virtual map of a person according to the observer's IPD, the virtual map is created by the virtual image module at startup or calibration, and specifies the boundary of area C in which the observer can perceive a virtual object with depth due to the fusion of the right retinal image and the left retinal image. The second step: calculating the convergence angle for each depth in the Z-axis direction (each point on the Z-axis) to determine a pair of right pixels and left pixels on the right retinal image and the left retinal image, regardless of their X coordinates and Y coordinates. The third step: moving the pair of right pixels and left pixels along the X-axis direction to determine the X coordinates and Z coordinates of each pair of right pixels and left pixels at a specific depth, regardless of their Y coordinates. The fourth step: moving the pair of right pixels and left pixels along the Y-axis direction to determine the Y coordinates of each pair of right pixels and left pixels. Therefore, the three-dimensional coordinate system, such as XYZ, of each pair of right pixels and left pixels on the right retinal image and the left retinal image can be determined to establish the query table. In addition, the third step and the fourth step can be interchangeable.

[0067] The optical signal generators 170 and 175 can be lasers, light emitting diodes (LEDs), including mini LEDs and microLEDs, organic light emitting diodes (OLEDs), or super luminescent diodes (SLDs), liquid crystal on silicon (LCoS), liquid crystal displays (LCDs), or any combination thereof as their light sources. In one embodiment, the optical signal generators 170 and 175 are laser scanning projectors (LBS projectors), which are composed of a light source (including a red laser, a green laser, and a blue laser), a color modifier (such as a two-color light combiner and a polarized light combiner), and a two-dimensional adjustable reflector (such as a two-dimensional micro-electromechanical system mirror). The two-dimensional adjustable reflector can be replaced by two one-dimensional reflectors, such as two one-dimensional micro-electromechanical system (MEMS) mirrors. The LBS projector sequentially generates and scans optical signals to form a two-dimensional image at a preset resolution, such as 1280x720 pixels per frame. Therefore, the projector generates a pixel optical signal and projects it onto the beam splitter 140 and 145 at one time. In order to allow the viewer to see the planar image at a glance, the LBS projector needs to sequentially generate a light signal for each pixel (eg, 1280x720 light signals) within the visual persistence time (eg, 1 / 18 second). Therefore, the duration of each light signal is approximately 60.28 nanoseconds.

[0068] In another embodiment, the optical signal generators 170 and 175 may be digital light processing projectors (DLP projectors), which can generate a flat color image at a time. Texas Instruments' DLP technology is one of several technologies that can be used to manufacture DLP projectors. The complete flat color image (which may consist of 1280x720 pixels) is projected onto the beam splitters 140 and 145 at the same time.

[0069] The beam splitter 140, 145 receives and redirects the plurality of light signals generated by the light signal generator 170, 175. In one embodiment, the beam splitter 20, 40 reflects the plurality of light signals so that the redirected light signal is on the same side as the incident light signal. In another embodiment, the beam splitter 140, 145 refracts the plurality of light signals so that the redirected light signal is on a different side of the beam splitter than the incident light signal. When the beam splitter 140, 145 acts as a refractor, its reflectivity varies widely, such as 20% to 80%, depending in part on the power of the light signal generator. Those skilled in the art know how to determine the appropriate reflectivity based on the characteristics of the light signal generator and the beam splitter. In addition, the beam splitter 140, 145 is optically transparent to the ambient light on the other side of the incident light signal so that the viewer can see the real-time image at the same time. Depending on the application, the range of transparency varies widely. If applied in AR / MR, its transparency is preferably greater than 50%, for example, about 75% in one embodiment. In addition to redirecting the light signals, the focus left unit 182, 187 can converge the light signals so that they can pass through the pupil and reach the retina of the viewer's eyes.

[0070] The beam splitters 140, 145 can be made of glasses or plastic materials like lenses, coated with a special material (such as metal) to make it partially transparent and partially reflective. One advantage of using a reflective beam splitter to direct the light signal to the viewer's eyes instead of a waveguide as in the prior art is that it eliminates undesirable diffraction effects such as ghosting, color displacement, etc.

[0071] The description of the embodiments provided above is intended to enable those skilled in the art to make and use the invention. Various modifications to the embodiments will be apparent to those skilled in the art, and the basic principles established herein may be applied to other embodiments without creative effort. Therefore, the subject matter claimed herein is not limited to the embodiments shown herein, but is to be consistent with the widest scope consistent with the principles and novel features disclosed herein. Other embodiments are envisioned to be within the spirit and scope of the invention disclosed herein. Therefore, the invention is intended to cover modifications and variations within the scope of the appended claims and their equivalents.

Claims

1. A virtual image module for generating a virtual image with depth, characterized in that: Include: a right optical signal generator that generates a right optical signal that is redirected toward a retina of a viewer; and a left optical signal generator, generating a left optical signal, the left optical signal corresponding to the right optical signal, the left optical signal being redirected toward another retina of the viewer; wherein the right light signal and the left light signal form virtual binocular pixels of the virtual image, the virtual binocular pixels having a second depth, the second depth perceived by the viewer being modulated by changing an angle between a light extension path of the right light signal and a light extension path of the left light signal entering the viewer's eyes according to the pupil distance of the viewer, The virtual image module is a head mounted device, the head mounted device has a real-time image module, the real-time image module has a light combining lens, The virtual image is superimposed on a real-time image, the real-time image has a first depth and a magnification, and the real-time image is generated by light reflected or emitted by an object, The second depth is changed according to the magnification of the real-time image or the first depth.

2. The virtual image module according to claim 1, characterized in that: The real-time imaging module is a surgical microscope.

3. The virtual image module according to claim 1, characterized in that: The real-time image module is configured to be detachably connected to the virtual image module.

4. The virtual image module according to claim 1, characterized in that: The viewer perceives the first depth to be the same as the second depth.

5. The virtual image module according to claim 1, characterized in that: The right light signal generator is close to the right part of the real-time imaging module, and the left light signal generator is close to the left part of the real-time imaging module. The right light signal and the left light signal are redirected to the retina and the other retina of the viewer respectively by the right beam splitter and the left beam splitter of the real-time imaging module.

6. The virtual image module according to claim 5, characterized in that: The real-time image module and the virtual image module share the right beam splitter and the left beam splitter.

7. The virtual image module according to claim 1, characterized in that: The real-time image is generated by receiving a light beam reflected or emitted from an object.

8. The virtual image module according to claim 1, characterized in that: When the magnification is adjusted, the size of the virtual image is adjusted accordingly.

9. The virtual image module according to claim 1, characterized in that: The virtual image is a medical image of a body organ or tissue provided to the virtual image module.

10. The virtual image module according to claim 9, characterized in that: The virtual image indicates the location where surgery will be performed.

11. The virtual image module according to claim 1, characterized in that: The first point in the real-time image is selected as a first positioning mark, and a corresponding first positioning mark of the virtual image is overlapped on the first positioning mark of the real-time image to achieve superimposing the virtual image on the real-time image.

12. The virtual image module according to claim 11, characterized in that: A second point in the real-time image is selected as a second positioning mark, and a corresponding second positioning mark of the virtual image is overlapped on the second positioning mark of the real-time image to achieve superimposing the virtual image on the real-time image.

13. The virtual image module according to claim 1, characterized in that: The virtual image module also includes a control module for processing the right light signal and the corresponding left light signal, and changing and modifying the virtual image according to the viewing angle and superimposing the virtual image on the real-time image.

14. The virtual image module according to claim 1, characterized in that: Also included is a user interface for allowing the viewer to control the second position and the second depth of the virtual image.

15. The virtual image module according to claim 1, characterized in that: The device also includes an object measuring module for measuring the position and depth of the object.

16. The virtual image module according to claim 1, characterized in that: The invention also comprises a recording module, which is used for recording both or one of the real-time image and the virtual image.

17. The virtual image module according to claim 1, characterized in that: The virtual image module further includes a support structure capable of being worn on the viewer's head; wherein the right optical signal generator and the left optical signal generator are carried by the supporting structure; and The right beam splitter and the left beam splitter are carried by the supporting structure and are located in the viewer's field of vision.

18. A method for superimposing a virtual image on a real-time image, characterized in that: include: Selecting a first point in the real-time image as a first positioning mark; Displaying the real-time image at a first position and a first depth with a preset magnification; Projecting a virtual image by respectively projecting a right light signal to a right eye of a viewer and projecting a corresponding left light signal to a left eye of the viewer, the right light signal and the corresponding left light signal being perceived by the viewer and displaying the virtual image at a second position and a second depth; The virtual image is a processed image of the object; The second depth perceived by the viewer changes according to the preset magnification or the first depth of the real-time image, wherein the second depth perceived by the viewer is modulated by changing the angle between the light-extending path of the right light signal and the light-extending path of the corresponding left light signal projected onto the viewer's eyes according to a pupil distance of the viewer; and The virtual image is superimposed on the real-time image by overlapping the corresponding first positioning mark of the virtual image on the first positioning mark of the real-time image.

19. The method according to claim 18, characterized in that The second depth is substantially the same as the first depth.

20. The method according to claim 18, characterized in that Also includes: Selecting a second point in the real-time image as a second positioning mark; The virtual image is superimposed on the real-time image by overlapping the corresponding second positioning mark of the virtual image on the second positioning mark of the real-time image.

21. The method according to claim 18, characterized in that Also includes: The magnification is adjusted so that both the real-time image and the virtual image are magnified, and the corresponding first positioning mark of the virtual image is still superimposed on the first positioning mark of the real-time image at the same depth.

22. The method according to claim 18, characterized in that Also includes: The first position and the depth of the virtual image are repeatedly monitored and measured to maintain the virtual image superimposed on the real-time image.

Images