A coaxial augmented reality microscope system and method of use

Abstract

The application belongs to the technical field of transparent display, and discloses a coaxial augmented reality microscope system and an application method. The system is tightly integrated with a microscopic objective lens, a transparent display panel and an ocular lens along an optical axis from an object side to an image side. The microscopic objective lens is used for receiving light from a sample and forming an intermediate real image of the sample. The transparent display panel is located between the microscopic objective lens and the ocular lens and coincides with the intermediate real image. The ocular lens synchronously magnifies the two for observation by the human eye or a photoelectric detector. The application breaks through the limitation of traditional off-axis optical path to realize AR function, seamlessly integrates high-transmittance micro display technology in the optical path of a traditional optical microscope, eliminates stray light interference and system volume redundancy, and realizes pixel-level accurate superposition of microscopic object magnification imaging and virtual digital information. The system has extremely high light energy utilization, compact mechanical structure and wide compatibility, and is suitable for biological medical diagnosis, precision industrial detection and frontier scientific education.

Description

Technical Field

[0001] This invention belongs to the field of transparent display technology, and more specifically, relates to a coaxial augmented reality microscope system and its application method. Background Technology

[0002] Since its invention in the 17th century, the optical microscope has been a core tool for humankind to explore the microscopic world (such as cell biology, materials science, and micro / nano fabrication). From the initial single-lens magnification to the compound achromatic microscope, and then to the modern confocal and super-resolution microscopes, its core function has always been to provide a "physical optical magnified image" of the sample. However, in today's world where the digital wave is sweeping the globe, traditional pure optical microscopes face the dilemma of "information silos."

[0003] In typical microsurgical procedures (such as pathological slide diagnosis or wafer defect detection), observers often face a dual task: meticulously observing the microscopic details of the sample through the eyepiece while frequently shifting their gaze to a nearby computer monitor or paper notebook to obtain reference images, verify patient information, record measurement data, or view AI-assisted diagnostic results. This frequent eye-switching not only disrupts the observer's thought process and reduces work efficiency but also easily leads to visual fatigue and cervical strain. Furthermore, in high-precision scenarios such as microsurgery, taking one's gaze away from the eyepiece can even pose operational risks.

[0004] To address the aforementioned issue of human-machine separation, researchers have attempted to introduce augmented reality (AR) technology into microscope systems, directly overlaying virtual information within the microscope's field of view (FOV). Current mainstream solutions are based on off-axis optical path architectures, requiring the display screen to be placed on the side in conjunction with a combiner to achieve image overlay. While this approach achieves basic AR functionality, it suffers from significant drawbacks, such as large size, complex structure, and low system efficiency, resulting in less than ideal AR display effects. Achieving clear microscopic imaging and display imaging simultaneously, while meeting the demands for high efficiency, intelligence, and high performance, remains a critical technical challenge that urgently needs to be addressed. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a coaxial augmented reality microscope system and application method, which aims to solve the problems of large size and complex structure in the existing AR technology introduced into the microscope system.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a coaxial augmented reality microscope system structure is provided, comprising, in sequence along the optical axis from the object side to the image side: an illumination source, a stage, a microscope objective, an image output unit, a transparent display panel, and an eyepiece; The microscope objective is positioned above the stage to receive the emitted light beam formed by the specimen placed on the stage after transmission or reflection of light from the illumination source, and to form a magnified intermediate real image of the specimen behind its image-side focal plane. The transparent display panel is an active light-emitting device with a pixel array, positioned along the optical axis between the microscope objective and the eyepiece, and located on the imaging plane of the intermediate real image formed by the microscope objective, so that the light rays of the specimen's real image transmitted through the transparent display panel and the virtual image light rays emitted by the transparent display panel are coaxially superimposed. The transparent display panel is configured to allow the light rays of the intermediate real image of the specimen to physically transmit through, and simultaneously according to... An external driving signal emits virtual image light rays; the image output unit is directly connected to the transparent display panel and is used to output virtual images, which are random two-dimensional images obtained by digital generation or camera capture, and outputs corresponding driving signals to the transparent display panel to control the pixel array of the transparent display panel to emit virtual image light rays carrying the virtual image; the eyepiece is disposed behind the transparent display panel and is configured to simultaneously receive the real image light rays of the specimen in the center through the transparent display panel and the virtual image light rays emitted by the pixel array of the transparent display panel, and synchronously couple and magnify the real image and the virtual image into a superimposed virtual image, which is then projected to the exit pupil position for observation.

[0007] Furthermore, the specific position of the transparent display panel along the optical axis satisfies the following optical condition: the light-emitting surface of the transparent display panel is located within the object-side focal plane of the eyepiece, that is, the optical distance from the transparent display panel to the first principal plane of the eyepiece is less than the focal length of the eyepiece (f). e This ensures that the virtual image is magnified through the eyepiece to form an upright virtual image.

[0008] Furthermore, the transparent display panel and the intermediate real image plane of the microscope objective are aligned in the optical axis direction or have a slight defocus, the defocus being less than the optical distance corresponding to the depth-of-field adjustment range of the human eye, so as to ensure that the observer can simultaneously see the microscopic image of the specimen and the digital content on the transparent display panel.

[0009] Furthermore, the light-emitting unit of the transparent display panel is selected from one or a combination of transparent organic light-emitting diodes, transparent micro light-emitting diodes, or transparent quantum dot light-emitting diodes. The overall visible light transmittance of the transparent display panel is greater than 40%, and its pixel gaps are filled with a transparent dielectric material with high light transmittance to reduce the diffraction effect of microscopic imaging light passing through the panel.

[0010] Furthermore, the coaxial augmented reality microscope system also includes a tube lens or an equivalent imaging lens group, which together with the microscope objective forms an infinity-corrected optical system. The tube lens converts the light from the specimen into a parallel beam, and the microscope objective then converges the light from the specimen to form an intermediate real image, which overlaps with the transparent display panel.

[0011] According to another aspect of the present invention, an augmented reality microscopic observation method is provided, comprising the following steps: S1: Turn on the illumination source and adjust the relative position of the specimen and the microscope objective so that the central real image of the specimen falls on the plane of the transparent display panel; S2: Observe through the eyepiece. At this time, the human eye simultaneously receives the transmitted light from the specimen and the transmitted light from the background of the transparent display panel. S3: The image output unit acquires the auxiliary information that needs to be displayed and outputs it as a virtual image; S4: Drive the transparent display panel to emit light and display the virtual image, and the light from the virtual image and the light from the specimen imaging are directly and physically superimposed in space; S5: The image is superimposed on the real scene seen by the human eye to obtain an enhanced image. The observer can simultaneously observe the high-resolution optical details of the specimen and the clear digital information superimposed on or around the specimen surface through the eyepiece.

[0012] According to another aspect of the present invention, an augmented reality method is provided, comprising: a wearer using the above-described coaxial augmented reality microscope system to form a complete and clear image; The image is superimposed on the real scene as seen by the human eye to obtain an enhanced image.

[0013] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly have the following beneficial effects: 1. The coaxial augmented reality microscope system provided by this invention utilizes the physical light transmittance of the transparent display panel, placing it directly on the primary imaging surface of the microscope objective along the optical axis, thereby achieving seamless fusion of dual imaging functions within the coaxial optical path. This invention eliminates the need for additional beam splitters or projection optical paths, utilizing only the transparent display panel along the main optical axis to complete the fusion of virtual and real images, eliminating volume redundancy, significantly reducing system size and weight, facilitating portable and modular design, and overcoming the limitations of traditional off-axis optical paths in achieving AR functionality.

[0014] 2. The transparent display panel of the present invention is configured to be self-emissive and has a high overall visible light transmittance. It seamlessly integrates high transmittance micro-display technology into the optical path of a traditional optical microscope, eliminates stray light interference, and realizes pixel-level precise superposition of magnified imaging of micro-objects and virtual digital information (such as AI diagnostic results, real-time measurement data, and dynamic annotations).

[0015] 3. Users can directly observe the natural spatial overlap between the specimen's actual morphology and the virtual augmented information through the eyepiece, without needing to switch fields of view or change observation posture. This aligns with human visual perception and reduces operational burden. The system does not introduce additional laser projection or external lighting interference, posing no photobiological safety risks to the specimen or operator. With its extremely high light energy utilization rate, compact mechanical structure, and broad compatibility, this system is suitable for fields such as biomedical diagnostics, precision industrial testing, and cutting-edge scientific education.

[0016] 4. In summary, this invention, through the rational design of the microscope objective, the transparent display array, and the position of the eyepiece layer, has achieved a highly integrated, high-contrast microscopic AR display device, which also has the advantages of being seamless, intelligent, compact, portable, and safe. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a coaxial augmented reality microscope system according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram illustrating the working method and principle of the coaxial augmented reality microscope system in an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the transparent display panel structure in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of the integration of a coaxial augmented reality microscope system and an intelligent interactive system in an embodiment of the present invention.

[0021] In all the accompanying drawings, the same reference numerals are used to denote the same elements or structures, wherein: 101-Specimen, 102-Microscope objective, 103-Transparent display panel, 104-Eyepiece, 105-Field of view, 106-Illumination source, 110-Refractive index matching plate, 111-CMOS camera. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] This invention provides a coaxial augmented reality microscope system structure, which includes, in sequence along the optical axis from the object side to the image side: an illumination source, a stage, a microscope objective, an image output unit, a transparent display panel, and an eyepiece; The microscope objective is positioned above the stage to receive the emitted light beam formed by the specimen placed on the stage after transmission or reflection of light from the illumination source, and to form a magnified intermediate real image of the specimen behind its image-side focal plane. The transparent display panel is an active light-emitting device with a pixel array, positioned along the optical axis between the microscope objective and the eyepiece, and located on the imaging plane of the intermediate real image formed by the microscope objective, so that the light rays of the specimen's real image transmitted through the transparent display panel and the virtual image light rays emitted by the transparent display panel are coaxially superimposed. The transparent display panel is configured to allow the light rays of the intermediate real image of the specimen to physically transmit through, and simultaneously according to... An external driving signal emits virtual image light rays; the image output unit is directly connected to the transparent display panel and is used to output virtual images, which are random two-dimensional images obtained by digital generation or camera capture, and outputs corresponding driving signals to the transparent display panel to control the pixel array of the transparent display panel to emit virtual image light rays carrying the virtual image; the eyepiece is disposed behind the transparent display panel and is configured to simultaneously receive the real image light rays of the specimen in the center through the transparent display panel and the virtual image light rays emitted by the pixel array of the transparent display panel, and synchronously couple and magnify the real image and the virtual image into a superimposed virtual image, which is then projected to the exit pupil position for observation.

[0024] Specifically, the specific position of the transparent display panel along the optical axis satisfies the following optical condition: the light-emitting surface of the transparent display panel is located within the object-side focal plane of the eyepiece, that is, the optical distance from the transparent display panel to the first principal plane of the eyepiece is less than the focal length of the eyepiece (f). e This ensures that the virtual image is magnified through the eyepiece to form an upright virtual image.

[0025] Specifically, the transparent display panel and the intermediate real image plane of the microscope objective are aligned in the optical axis direction or have a small amount of defocus. The amount of defocus is less than the optical distance corresponding to the depth-of-field adjustment range of the human eye, so as to ensure that the observer can simultaneously see the microscopic image of the specimen and the digital content on the transparent display panel.

[0026] Specifically, the light-emitting units of the transparent display panel are selected from one or a combination of transparent organic light-emitting diodes, transparent micro light-emitting diodes, or transparent quantum dot light-emitting diodes. The overall visible light transmittance of the transparent display panel is greater than 40%, and its pixel gaps are filled with a transparent dielectric material with high light transmittance to reduce the diffraction effect of microscopic imaging light passing through the panel.

[0027] Specifically, the coaxial augmented reality microscope system also includes a tube lens or an equivalent imaging lens group, which together with the microscope objective forms an infinity-corrected optical system. The tube lens converts the light from the specimen into a parallel beam, and the microscope objective then converges the light from the specimen to form an intermediate real image, which overlaps with the transparent display panel.

[0028] The present invention also provides an augmented reality microscopic observation method comprising the following steps: S1: Turn on the illumination source and adjust the relative position of the specimen and the microscope objective so that the central real image of the specimen falls on the plane of the transparent display panel; S2: Observe through the eyepiece. At this time, the human eye simultaneously receives the transmitted light from the specimen and the transmitted light from the background of the transparent display panel. S3: The image output unit acquires the auxiliary information that needs to be displayed and outputs it as a virtual image; S4: Drive the transparent display panel to emit light and display the virtual image, and the light from the virtual image and the light from the specimen imaging are directly and physically superimposed in space; S5: The image is superimposed on the real scene seen by the human eye to obtain an enhanced image. The observer can simultaneously observe the high-resolution optical details of the specimen and the clear digital information superimposed on or around the specimen surface through the eyepiece.

[0029] The present invention also provides an augmented reality method, comprising: a wearer using the above-described coaxial augmented reality microscope system to form a complete and clear image; The image is superimposed on the real scene as seen by the human eye to obtain an enhanced image.

[0030] Example 1 like Figure 1 and Figure 2 As shown, this embodiment constructs a standard transmission coaxial augmented reality microscope system. The main optical axis of the system is defined as the Z-axis. Along the optical axis from the object side to the image side, it includes: an illumination source 106, a stage, a microscope objective 102, a transparent display panel 103, and an eyepiece 104.

[0031] The lighting system 106 uses a white LED light source. This lighting method can provide uniform and bright background light, which is especially important for observation against the background of a transparent display panel, because it can "suppress" the weak scattering that may exist in the panel itself through the high brightness background.

[0032] The microscope objective 102 is a 40X achromatic objective with a numerical aperture (NA) of 0.65. Its working distance is approximately 0.6 mm. The objective collects and converges the light emitted from the specimen 101. In this embodiment, a finite-distance conjugate objective is used, with a mechanical tube length of 160 mm, which means it forms a magnified inverted real image (intermediate image) approximately 150 mm from the objective positioning surface.

[0033] The transparent display panel 103 uses a 1.5-inch diagonal OLED transparent screen. The screen has a resolution of 128×128 and a pixel pitch of approximately 0.2 mm. A key parameter is its transmittance design: the pixel area consists of a light-emitting material (such as an organic light-emitting layer) and a transparent cathode, while non-pixel areas are completely hollowed out or filled with transparent resin (such as...). Figure 3 As shown in the figure, the overall average visible light transmittance is approximately 45%. The panel is connected to an external drive board via an ultra-thin flexible printed circuit board (FPC).

[0034] Eyepiece 104 is a 10X Huygens eyepiece or a wide field eyepiece.

[0035] In this embodiment, the microscope objective 102 and eyepiece 104 are fixed so that their optical axes coincide. The distance between them is adjusted so that a clear edge of the field stop can be seen in the eyepiece.

[0036] Furthermore, a transparent display panel 103 is introduced and mounted on a three-dimensional precision displacement stage, inserted between the objective lens and the eyepiece.

[0037] Next, determine the axial position (Z-axis), turn on the transparent display panel, and display a test pattern (such as "bio"). Move the panel back and forth until the test pattern observed through the eyepiece is clearest. At this point, the luminescent layer of the panel is located on the object-side focal plane of the eyepiece.

[0038] Further, determine the conjugate relationship and place specimen 101 (such as a biological slide) on the stage. Adjust the height of the stage (focus) until the microscopic image of the specimen is also clearly presented in the eyepiece field of view. At this point, the central real image plane of the specimen coincides (or is very close to) the luminescent surface of the transparent display panel in space.

[0039] Initially, in the microscopic imaging optical path, light from the light source penetrates the specimen, carrying its microscopic structural information. After being converged by the microscope objective, it physically passes through the transparent display panel (at this point, the panel only serves as a highly transparent medium), forming a clear intermediate real image in front of the eyepiece. Simultaneously, in the information display optical path, the pixel array on the transparent display panel is controlled to actively emit light, and the virtual image light radiated backward is directly collected by the eyepiece. Finally, the microscopic intermediate real image and the virtual image emitted by the panel are synchronously magnified into a virtual image through the same eyepiece, forming an augmented reality view on the human eye's retina with precise superposition of microscopic details and digital information. The observer receives two sets of wavefronts simultaneously on the retina. Since both the intermediate real image and the virtual image are located at the focal plane of the eyepiece, according to the lens imaging formula, the human eye can see both clearly simultaneously in a relaxed state (adjusted to infinity), achieving true augmented reality superposition.

[0040] Example 2 like Figure 4 As shown, an intelligent sensing and interaction module is additionally integrated into the original architecture. The transparent display panel 103 is connected to the embedded computing unit 111 (such as an NVIDIA Jetson series or a high-performance FPGA) via an HDMI or MIPI interface. Simultaneously, a CMOS camera is connected to the eyepiece for real-time acquisition of microscopic images. Eye-tracking and voice control modules further enhance the device's intelligence.

[0041] A CMOS camera captures images of the specimen in real time and transmits them to the computing unit. The computing unit runs deep learning algorithms (such as U-Net or YOLO models) for inference and analysis. For example, in blood smear examination, the algorithm automatically identifies red blood cells and white blood cells and calculates their number and proportion; in pathological sections, the algorithm identifies regions of nuclear atypia suspected of being cancerous. An AR rendering engine (based on OpenGL or Vulkan) generates graphics layers based on the AI ​​analysis results.

[0042] Further, spatial registration and display are performed. Since there is a fixed geometric transformation relationship between camera images and the human eye's field of view, the software performs inverse distortion correction and coordinate transformation on the generated graphic layer, and then outputs it to the transparent display panel.

[0043] In one practical application of this invention, when doctors observe a slide through an eyepiece, they can directly see an AI-generated diagnostic prompt box appear next to the cell without needing to look away, greatly improving the accuracy and speed of diagnosis. When a doctor's gaze lingers on a cell for more than one second, the system automatically displays the cell's detailed morphological parameters (diameter, roundness, nucleocytoplasmic ratio), enabling gaze-point interaction. Furthermore, voice marking is implemented. A doctor can say "mark as abnormal," and the system will generate a virtual "flag" icon at the center of the current field of view and save the coordinates of that location for subsequent review.

[0044] In another practical application of this invention, precision industrial inspection can be performed. In the inspection of PCB circuit boards or semiconductor wafers, traditional methods require operators to refer to design drawings (CAD drawings). This system can convert CAD design drawings into line drawings, which are then directly superimposed onto the actual wafer image via a transparent screen. Operators can immediately see whether the actual circuitry is misaligned, broken, or redundant, achieving a "what you see is what you get" comparative inspection.

[0045] In another practical application of this invention, during ophthalmic or neurosurgical microsurgery, surgeons need to avoid critical blood vessels and nerves. A three-dimensional model of the blood vessels generated preoperatively by CT / MRI scans is projected and displayed on the transparent panel of the surgical microscope. During surgery, the surgeon essentially gains "X-ray vision," directly seeing the virtual projection path of subcutaneous blood vessels on the tissue surface, significantly reducing surgical risks.

[0046] In another practical application of this invention, in a teaching scenario, teachers and students use the same networked microscopes. The structure the teacher sees on their microscope can be circled and marked using a touchpad. This marking signal is transmitted over the network and displayed in real-time on the transparent screen of the student's microscope. Wherever the teacher points, that area illuminates in the student's field of vision, achieving "hands-on" teaching across space.

[0047] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention (such as changing the type of transparent screen, adjusting the parameters of the objective lens and eyepiece, introducing color display, etc.) should be included within the protection scope of the present invention.

Claims

1. A coaxial augmented reality microscope system, characterized in that, Along the optical axis from the object side to the image side, it includes, in sequence: an illumination source (106), a stage, a microscope objective (102), an image output unit, a transparent display panel (103), and an eyepiece (104). The microscope objective (102) is positioned above the stage and is used to receive the outgoing beam formed by the specimen (101) placed on the stage after the light from the illumination source (106) is transmitted or reflected, and to form an enlarged intermediate real image of the specimen (101) behind the image-side focal plane. The transparent display panel (103) is an active light-emitting device with a pixel array. It is disposed between the microscope objective (102) and the eyepiece (104) along the optical axis and is located on the imaging plane where the intermediate real image formed by the microscope objective (102) is located. The image output unit is directly connected to the transparent display panel (103) and is used to output a virtual image. The virtual image is a random two-dimensional image. It outputs a corresponding driving signal to the transparent display panel (103) to control the pixel array of the transparent display panel (103) to emit virtual image light carrying the virtual image. The transparent display panel (103) is configured to allow the light of the intermediate real image of the specimen to be physically transmitted through, and at the same time emits virtual image light according to the driving signal. The eyepiece (104) is located behind the transparent display panel (103) and is configured to simultaneously receive the real image light of the specimen transmitted through the transparent display panel (103) and the virtual image light emitted by the pixel array of the transparent display panel (103), and synchronously couple and magnify the real image and the virtual image into a superimposed virtual image, which is then projected to the exit pupil position for observation.

2. The coaxial augmented reality microscope system according to claim 1, characterized in that, The specific position of the transparent display panel (103) along the optical axis satisfies the following optical conditions: the light-emitting surface of the transparent display panel (103) is located within the object-side focal plane of the eyepiece (104), that is, the optical distance from the transparent display panel (103) to the first principal plane of the eyepiece (104) is less than the focal length f of the eyepiece (104). e This allows the virtual image to be magnified through the eyepiece to form an upright virtual image.

3. The coaxial augmented reality microscope system according to claim 2, characterized in that, The imaging plane where the intermediate real image of the transparent display panel (103) and the microscope objective (102) is located coincides with the optical axis or has a defocus amount on the order of micrometers, the defocus amount being less than the optical distance corresponding to the depth of field adjustment range of the human eye.

4. The coaxial augmented reality microscope system according to claim 1, characterized in that, The transparent display panel (103) includes an array of light-emitting units, wherein the light-emitting units are one or a combination of transparent organic light-emitting diodes, transparent micro light-emitting diodes, or transparent quantum dot light-emitting diodes; The overall visible light transmittance of the transparent display panel (103) is greater than 40%, and its pixel gaps are filled with transparent medium material.

5. The coaxial augmented reality microscope system according to claim 1, characterized in that, It also includes a tube lens located between the stage and the microscope objective (102), which converts the light from the specimen (101) into a parallel beam for emission, and the microscope objective (102) then converges the light from the specimen (101) to form an intermediate real image.

6. An augmented reality microscopy observation method based on the coaxial augmented reality microscope system according to any one of claims 1-5, characterized in that, Includes the following steps: S1: Turn on the illumination source (106), adjust the relative position of the specimen (101) and the microscope objective (102) so that the central real image of the specimen falls on the plane of the transparent display panel (103); S2: Observe through the eyepiece (104). At this time, the human eye simultaneously receives the transmitted light from the specimen (101) and the transmitted light from the background of the transparent display panel (103). S3: The image output unit acquires the auxiliary information that needs to be displayed and outputs it as a virtual image; S4: Drive the transparent display panel (103) to emit light and display the virtual image, and the light of the virtual image and the light of the specimen imaging are directly and physically superimposed in space; S5: The observer simultaneously observes the optical details of the specimen (101) and the digital information superimposed on or around the specimen surface through the eyepiece (104).

7. An augmented reality method, characterized in that, include: The wearer uses a coaxial augmented reality microscope system according to any one of claims 1-5 to form a complete and clear image; the image is superimposed on the real scene seen by the human eye to obtain an enhanced image.

Images