8K stereo fundus camera

Abstract

The 8K stereoscopic microscope comprises a super lens, a 4K stereoscopic fundus camera, a data transmission and storage unit, an AI artificial intelligence image processing unit and 8K binocular AR glasses. The super lens is used for a stereo fundus camera optical system, and the size of the optical system is reduced while high-quality images are obtained. The 4K stereo fundus camera is used for observing a fundus area and collecting high-definition fundus images according to needs. The data transmission and storage is used for image acquisition, transmission and storage; aI artificial intelligence image processing is used for image processing and enhancement to obtain a three-dimensional fundus image; the 8K binocular AR glasses are used for displaying high-definition stereo images. The system has the advantages of being compact in structure, wide in view field, high in definition, vivid in color and intelligent and convenient to operate, operation of AI artificial intelligence algorithms such as deep learning is supported, fundus image collection at any time, any place and any mode can be achieved, the requirements of the medical field for high-quality imaging and accurate diagnosis are met, and the system is suitable for popularization and application. And medical scientific research and clinical diagnosis development are promoted.

Description

Technical Field

[0001] The present invention relates to a fundus camera technology, and in particular to an 8K binocular stereo fundus camera that combines a super lens, a 4K stereo fundus camera, AI artificial intelligence processing, and 8K binocular AR glasses. Background Art

[0002] Fundus camera is a fundus camera device that uses optical imaging technology. It is specially used to take images of the retina of the eye and observe whether there are lesions in the retina, optic disc and blood vessel distribution. It can help doctors diagnose eye diseases and provide information to assist in the diagnosis of diseases of other organs. Based on binocular stereo vision, fundus cameras can obtain three-dimensional fundus images containing depth information, but the image quality is not clear and the operation is inconvenient, which may bring certain difficulties to doctors' diagnosis.

[0003] CN212415705U provides a fundus camera lighting system and a fundus camera, wherein the fundus lighting system comprises at least one light source, which is arranged inside a lens barrel and located on one side of the central axis inside the lens barrel; a reflective structure, which is arranged inside the lens barrel and opposite to the light source, and is used to reflect the light emitted by the light source, so that the light source forms an image outside the lens barrel after passing through an eyepiece group or an eyepiece objective lens arranged inside the lens barrel.

[0004] CN 115883941A discloses a 4K endoscopic camera system and a method of using the system, comprising an endoscopic collector for collecting optical signals, a camera for receiving and processing signals, and a monitor for imaging. The endoscopic collector comprises a camera for receiving and transmitting optical signals, an optical endoscope installed at the end of the camera and used for collecting optical signals. The camera is provided with a control unit, a plurality of cable sockets and a power socket electrically connected to the control unit, and a plurality of function buttons for inputting control instructions. The camera is also electrically connected to an input unit for inputting control instructions and an output unit for outputting instructions through the control unit. The control unit is also provided with basic functions and a function menu that can be called and selected by the function buttons and the input unit. The problem that the traditional endoscopic camera system cannot well control the color of the imaging during imaging, resulting in the difficulty in distinguishing the color difference between tissues, is solved.

[0005] With the development of optical imaging technology, image processing technology, AI artificial intelligence and other fields, stereo fundus cameras are gradually developing towards high definition, intelligence, miniaturization and convenience, but there are still some technical bottlenecks. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an 8K binocular stereo fundus camera that combines a super lens, 8K binocular AR glasses, a 4K stereo fundus camera and AI artificial intelligence image processing to improve the quality of fundus images and achieve high-definition stereo display, expanding the field of observation. The device has the characteristics of miniaturization, intelligence, and easy operation, meeting the needs of the medical field for high-quality imaging and accurate diagnosis, and promoting the progress of medical research and clinical diagnosis.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] An 8K stereo fundus camera comprises a super lens, a 4K stereo fundus camera, a data transmission and storage module, an AI artificial intelligence image processing unit and an 8K binocular AR glasses device.

[0009] The metalens is used in optical imaging systems to reduce the volume and weight of the optical system, while achieving high-definition imaging, a large field of view, and eliminating stray light and chromatic aberration.

[0010] The 4K stereo fundus camera uses an imaging optical system to observe the fundus area and collect high-definition fundus images as needed;

[0011] The data transmission and storage module is used to transmit, store and manage fundus image data;

[0012] The AI ​​artificial intelligence image processing unit performs enhancement, detection and three-dimensional imaging on the received fundus image data;

[0013] The 8K binocular AR glasses acquire the above 3D stereo fundus images for high-definition display to guide diagnosis and treatment.

[0014] Preferably, in the above-mentioned stereo fundus camera system, the metalens comprises a substrate and a nanostructure arranged on the substrate; the nanostructure is a polarization-dependent structure or a polarization-independent structure; the polarization-dependent structure comprises a nanofin or a nanoelliptical column, and the polarization-independent structure comprises a nanosquare column or a nanocylinder;

[0015] Preferably, a protective layer is provided on the nanostructure side;

[0016] Optionally, the optical system composed of the metalens can adopt a combination of one or more metalens, or a combination of a metalens and a refractive lens; the metalens or the metalens combination can be used in any suitable position of the optical system.

[0017] Preferably, the above-mentioned stereo fundus camera system, its 4K stereo fundus camera includes an illumination optical system, an imaging optical system and a photosensitive element; the illumination optical system is used to introduce appropriate light into the fundus; the imaging optical system is used to image the appearance of the fundus on the camera photosensitive element; the photosensitive element is used to convert the light signal collected by the imaging system into fundus image data for subsequent processing.

[0018] Preferably, the illumination optical system includes, from the eyeball side to the light source side, an eyepiece objective lens group, a hollow reflector, a first condenser lens group, a black dot plate, a second condenser lens group, an illumination aperture diaphragm, a dichroic mirror, a first light collecting lens group, an infrared light source, a second light collecting lens group and a visible light source in sequence; wherein, the first condenser lens group and the second condenser lens group are used to control and adjust the light collecting and focusing effects of the light, so that the light irradiated to the fundus can be accurately focused on the required area; the black dot plate is used to eliminate the ghost image formed by the illumination light beam and the retina objective lens; the illumination aperture diaphragm is used to control the incident angle of the light to avoid the interference of stray light; the dichroic mirror ensures that the white light and the infrared light are emitted coaxially; the first light collecting lens group is used to collimate and converge the light beam emitted by the infrared light source; the second light collecting lens is used to collimate and converge the light beam emitted by the visible light source; the infrared light source emits infrared light for observation and focusing to achieve mydriasis-free focusing; the visible light source is used to illuminate the fundus field of view and cooperate with the photosensitive element to collect the required fundus visible light pictures;

[0019] Preferably, the lighting system adopts internal lighting to achieve uniform fundus lighting brightness, high light energy utilization and clear imaging;

[0020] Preferably, the illumination system is a Kohler illumination optical path to improve the uniformity of the illumination beam;

[0021] Furthermore, the working state of the lighting system is: in the infrared light preview mode, the infrared light source is on and the visible light source is off; when taking pictures, the infrared light source is off and the visible light source flashes.

[0022] In particular, the dichroic mirror transmits white light and reflects infrared light.

[0023] Preferably, the imaging optical system comprises, from the eyeball side to the imaging side, an eyepiece objective lens group, a hollow reflector, a focusing lens group, a first plane reflector, a first imaging lens group, a second plane reflector, and a second imaging lens group in sequence; the imaging light of the human eye passes through the eyepiece objective lens group, the hollow reflector, the focusing lens group, the first plane reflector, and the first imaging lens group in sequence to form a first optical path; the imaging light of the human eye passes through the eyepiece objective lens group, the hollow reflector, the focusing lens group, the second plane reflector, and the second imaging lens group in sequence to form a second optical path;

[0024] Preferably, the photosensitive element is divided into a first photosensitive element and a second photosensitive element; the first photosensitive element and the second photosensitive element can both adopt 4K CMOS or CCD sensors, and in the future can be equipped with 8K or higher resolution sensors to generate higher resolution fundus images;

[0025] Furthermore, the first photosensitive element collects the reflected light signal of the first optical path; the second photosensitive element collects the reflected light signal of the second optical path;

[0026] Furthermore, the fundus data of the first light path collected by the first photosensitive element and the fundus data of the second light path collected by the second photosensitive element are transmitted independently.

[0027] Preferably, the imaging optical system and the illumination optical system share an eyepiece objective lens, and a hollow reflector is used as a beam splitter element for the illumination optical path and the imaging optical path;

[0028] Optionally, the 4K stereo fundus camera has no structural restrictions and can be either a desktop stereo fundus camera or a handheld stereo fundus camera;

[0029] In particular, the fundus imaging principle of the 4K stereo fundus camera is as follows: when working, the infrared light source of the illumination optical system sequentially passes through the first light collecting lens group, the dichroic mirror, the illumination aperture diaphragm, the second condenser lens group, the black dot plate, the first condenser lens group, the hollow reflector and the eye contact lens group to reach the pupil and enter the fundus, thereby illuminating the fundus. The illuminated fundus then passes through the first light path and the second light path, and finally forms an image on the photosensitive element. During the focusing process, the infrared light source is always on, and the refractive changes caused by different myopia and hyperopia are compensated by the focusing lens group. After the focusing is completed, the visible light source flash flashes instantaneously, and the photosensitive element obtains the fundus image.

[0030] Preferably, the data transmission and storage module of the above-mentioned stereo fundus camera system includes a transmission data cable, a storage module and data management software; the transmission data cable is used for connection and transmission between photosensitive elements, the AI ​​artificial intelligence image processing unit, a storage device, the 8K binocular AR glasses and other devices; the storage device is used to store the collected fundus image data, and the 3D fundus image after data enhancement; the data management software is used to manage, organize and analyze the data stored on the storage device, and provide data query, retrieval, backup and other functions.

[0031] Optionally, the transmission data cable includes but is not limited to HDMI signal cable, DP signal cable, DVI signal cable, SDI signal cable, USB signal cable and other data cables;

[0032] Optionally, the storage module includes but is not limited to a hard disk, a solid-state memory, a cloud storage, etc.;

[0033] Preferably, the data management software includes but is not limited to a database management system (DBMS), Hexagor DSP, and the like.

[0034] Preferably, in the above-mentioned stereo microscope system, its AI artificial intelligence image processing unit includes an image analysis module, an image processing module and a 3D image imaging module; the image analysis module is used to pre-process the fundus image data collected by the first optical path and the second optical path, filter out the fundus images with poor quality and low definition caused by factors such as vibration of the operating platform in advance, and mark the images respectively to reduce the data transmission load and ensure high-speed real-time transmission; the image processing module is used to process and enhance the fundus image to improve the image quality; the 3D image imaging module integrates the images of the first optical path and the second optical path after the data enhancement into a 3D stereo image;

[0035] Preferably, the image processing module includes an image geometry calibration module, an image defogging module, an image denoising module, an image chromaticity calibration module and an image visual adjustment module;

[0036] The image geometry calibration module determines the coordinates of the four corners of the first image with the first identifier and the second image with the second identifier in the RGB image data by using an edge search image algorithm, and then calibrates the first image and the second image into a first standard image and a second standard image respectively by using a trapezoidal calibration algorithm;

[0037] Furthermore, the first standard image and the second standard image are enlarged or reduced to have the same area;

[0038] The image defogging module uses a defogging algorithm to reduce the stray light phenomenon in fundus images and improve image clarity. The visual characteristics of stray light in fundus images have certain similarities with the common image fogging phenomenon and can be used as a reference for removing stray light;

[0039] The image denoising module removes the noise in the image through a denoising algorithm to improve the quality and clarity of the image. Fundus images may be affected by uneven lighting conditions, device sensor noise, motion blur, etc., resulting in various types of noise in the image, such as Gaussian noise, salt and pepper noise, etc. These noises will reduce the contrast, detail information and visual quality of the image, affecting doctors' judgment and diagnosis of fundus diseases;

[0040] The image chromaticity calibration module calibrates the chromaticity response and light intensity response of the color image data;

[0041] Optionally, the image chromaticity calibration module includes but is not limited to a color correction matrix and a gamma transformation algorithm, etc.;

[0042] The visual adjustment module is used to adjust the brightness, contrast, exposure time and white balance of color image data.

[0043] Preferably, the 3D image imaging module applies a stereoscopic vision parallel algorithm to extract an overlapping area image between the image of the first light path and the image of the second light path in the fundus RGB image data, and synthesizes a 3D image to obtain a stereoscopic image with spatial position information;

[0044] Optionally, the stereoscopic vision parallel algorithm includes but is not limited to web3D technology, etc.;

[0045] In particular, color image dehazing algorithms, image denoising algorithms, chromaticity calibration algorithms, visual adjustment algorithms, and stereoscopic vision algorithms are implemented with the help of artificial intelligence, such as deep learning or traditional machine learning techniques, to accelerate processing speed and improve accuracy.

[0046] Specifically, the deep learning neural network model is a computational model that imitates the neuron structure of the human brain, which abstracts and processes data layer by layer through multiple layers of neurons and weights. The neural network model includes convolutional neural network (CNN), generative adversarial network (GAN), recurrent neural network (RNN), Transformer and other models.

[0047] In particular, in some embodiments of the present invention, the color image dehazing algorithm uses a DFC-dehaze model. The DFC-dehaze model is improved based on the CycleGAN model. Specifically, the DFC-dehaze model uses a Dehazeformer-t model to replace the CNN-based generator of the CycleGAN model, and uses a local-global discriminator to process locally changing haze, so as to reduce the haze residue in the restored image. When the generated image is blurred or the color is not real, the discriminator gives a low score through a negative sample penalty mechanism. Among them, the Dehazeformer-t is improved from swin-transformer.

[0048] The image denoising algorithm uses the MambaIR model. The MambaIR model consists of a shallow feature extraction module, a deep feature extraction module, an upsampling module and a reconstruction module. Among them, the core of the entire model of the deep feature extraction module is mainly composed of multiple residual state space blocks (RSSB). Furthermore, the RSSB block mainly includes a convolutional layer, a Mamba module, a channel attention module, and a residual connection module. The chromaticity calibration algorithm and the visual adjustment algorithm can use a DNN model. The DNN model mainly includes an input layer, a hidden layer and an output layer. Among them, the number of hidden layers can be reasonably adjusted according to actual needs.

[0049] The stereo vision algorithm uses the ACVNet model. The ACVNet model includes four parts: feature extraction, attention weight body construction, cost aggregation, and disparity prediction. The feature extraction uses 2D CNN to extract feature maps for the left and right views respectively, and the size of the feature map is 1 / 4 of the original image. Then, the feature map is further extracted through a CNN with a stride of 1 to obtain feature maps with 64, 128, and 128 channels, respectively, and the size is kept at 1 / 4 of the original image. These feature maps are spliced ​​along the channel dimension to form an attention construction feature map. Finally, the spliced ​​feature map dimension is compressed to 32 through CNN to form a matching cost body construction feature map; the attention weight body construction uses the attention weight construction feature map of 320 channels to obtain the attention weight body, and then uses the matching cost body construction feature map of 32 channels to construct the initial matching cost body, and finally uses the attention weight to filter the initial matching cost body; the cost aggregation uses 4 3D convolutions and 2 stacked 3D U-Net networks for cost body regularization to obtain out1, out2, and out3; the disparity prediction calculates the disparity map based on the expected form.

[0050] In particular, the image processing and enhancement process: the first fundus image data and the second fundus image data are marked by the image analysis module, and then processed by the image geometry calibration, defogging, denoising, chromaticity calibration and visual adjustment modules respectively, and finally fused into a three-dimensional image output in the 3D image imaging module. It should be noted that the first fundus image data and the second fundus image data are processed and enhanced at the same time to ensure real-time performance.

[0051] Optionally, the data transmission and storage module is integrated with the AI ​​artificial intelligence image processing unit into an integrated structure - an image processing unit;

[0052] Preferably, the image processing unit may be any suitable hardware or a combination of hardware and software, such as a field programmable gate array.

[0053] Preferably, the above-mentioned stereo microscope system, its 8K binocular AR glasses use 8K binocular waveguide AR glasses;

[0054] The 8K binocular waveguide AR glasses system uses optical waveguide technology, has the advantages of high brightness and low power consumption, and provides clear and stable images;

[0055] Furthermore, the 8K binocular waveguide AR glasses system is equipped with 8K high-resolution and high-sensitivity sensors to track the user's head and eye movements in real time, achieving more accurate image positioning and display effects;

[0056] Furthermore, the 8K binocular waveguide AR glasses system supports multiple interaction modes such as voice recognition and gesture control, and displays fundus image information in the glasses to guide diagnosis and treatment;

[0057] In addition, the 8K binocular waveguide AR glasses system also includes modules such as a graphics processor and a GPU; the processor uses high-performance chips, such as MTK6765, etc., to provide powerful computing power and low power consumption performance; the GPU uses a high-performance GPU, such as IMG GE8320, etc., for 3D rendering and image processing.

[0058] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0059] (1) The present invention uses a meta-lens to construct an optical system, which is small, light, and has high degrees of freedom, and provides high definition and a large field of view to enhance optical stability and achieve better focusing quality;

[0060] (2) The 4K fundus camera uses Kohler illumination and internal illumination to improve the stability of the illumination beam; based on the stereoscopic vision method, it achieves high-definition stereoscopic imaging; it uses the non-mydriatic fundus camera mode to reduce eye discomfort; and it is equipped with a 4K sensor to obtain high-quality images;

[0061] (3) Use AI for image processing, provide intelligent image enhancement, realize automated image enhancement and obtain high-definition three-dimensional images;

[0062] (4) 8K binocular AR glasses can achieve high-definition stereoscopic display. The high-definition three-dimensional fundus images ensure that doctors can diagnose patients' eye diseases effectively, thereby improving the treatment effect on patients' eyes. Doctors can obtain case data and reference information in real time, which supports doctors to make remote diagnosis and multiple people to make diagnosis at the same time, thus achieving real-time and accurate treatment.

[0063] (5) Cases can be collected and stored to provide data support and reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a schematic diagram of an embodiment of an 8K stereo fundus camera system provided by the present invention;

[0065] Figure 2 is a schematic diagram of the structure of a super lens in an embodiment of an 8K stereo fundus camera system provided by the present invention;

[0066] Figure 3 It is a schematic diagram of an AI artificial intelligence image processing unit in an embodiment of an 8K stereo fundus camera system provided by the present invention;

[0067] Figure 4 This is a schematic diagram of the optical waveguide in the 8K binocular waveguide AR glasses in one embodiment of the 8K stereo fundus camera system provided by the present invention.

[0068] Reference numerals:

[0069] 1. Metalens; 11. Substrate; 12. Nanostructure; 2. 4K stereo fundus camera; 201. Eyepiece lens group; 202. Hollow reflector; 203. First condenser group; 204. Black dot plate; 205. Second condenser group; 206. Illumination aperture diaphragm; 207. Dichroic mirror; 208. First light collecting group; 209. Infrared light source; 210. Second light collecting group; 211. Visible light source; 212. Focusing group; 213. First plane reflector; 214. First imaging group; 215. First photosensitive element; 216. Second plane reflector; 217. Second imaging group ; 218. Second photosensitive element; 3. Data transmission and storage module; 31. Transmission data line; 32. Storage device; 33. Data management software; 4. AI artificial intelligence image processing unit; 41. Image analysis module; 42. Image processing module; 421. Image geometry calibration module; 422. Image defogging module; 423. Image denoising module; 424. Image chromaticity calibration module; 425. Image visual adjustment module; 43. 3D image imaging module; 5. 8K binocular AR glasses; 51. Mirror body; 52. Optical waveguide; 53. Incident grating; 54. Exit grating; 55. Incident light. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features involved in the embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0071] It should be noted that if there are directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention, the directional indications are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly. In addition, if there are descriptions of "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features.

[0072] It should be noted that the terms "installed", "connected" and "connected" in the embodiments of the present invention should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal connection of two components.

[0073] The embodiment of the present invention provides an 8K stereo fundus camera system, such as Figure 1 As shown, it specifically includes: a super lens 1, a 4K stereo fundus camera 2, a data transmission and storage module 3, an AI artificial intelligence image processing unit 4 and an 8K binocular AR glasses device 5.

[0074] The superlens 1 is applied to the optical system of the 4K stereo fundus camera 2, which can effectively reduce the volume and weight of the system, while achieving the advantages of high-definition imaging, large field of view, elimination of stray light and chromatic aberration, etc. The superlens 1 is a phase-type superlens, including a substrate 12 and a nanostructure 12 arranged on the substrate, such as Figure 2 shown.

[0075] Furthermore, the nanostructure 12 is a polarization-dependent structure or a polarization-independent structure; the polarization-dependent structure includes a nanofin or a nanoelliptical column, and the polarization-independent structure includes a nanosquare column or a nanocircular column.

[0076] In particular, a protective layer is provided on the side of the nanostructure 12, and the protective layer should be a material that is transparent relative to the working band, and can be a layered structure plated on the superlens 1, partially or completely covering the surface of the superlens to prevent damage to the superlens structure during assembly;

[0077] In particular, the optical system composed of the superlens 1 can be a combination of one or more superlenses, or a combination of a superlens and a refractive lens; the superlens or superlens combination can be placed in any suitable position of the optical system; the height and size of the superlens 1 should be determined in combination with the actual assembly. The nanostructure 12 is a phase change material with a subwavelength scale that can effectively couple with light, thereby causing the phase of the light to change rapidly. By adjusting the optical parameters of the nanostructure, such as refractive index, dielectric constant, shape and size, a full 360-degree phase change can be achieved, and the wavefront can be efficiently controlled.

[0078] The 4K stereo fundus camera 2 uses an optical system to observe the fundus area and collect high-definition fundus images as needed, including an illumination optical system, an imaging optical system and a photosensitive element; the illumination optical system is used to introduce appropriate light into the fundus; the imaging optical system is used to image the appearance of the fundus on the camera photosensitive element; the photosensitive element is used to convert the light signals collected by the imaging system into fundus image data for subsequent processing.

[0079] Furthermore, the illumination optical system includes, from the eyeball side to the light source side, an eyepiece objective lens group 201, a hollow reflector 202, a first condenser lens group 203, a black dot plate 204, a second condenser lens group 205, an illumination aperture diaphragm 206, a dichroic mirror 207, a first light collecting lens group 208, an infrared light source 209, a second light collecting lens group 210 and a visible light source 211 in sequence; the eyepiece objective lens group 201 is installed at the front end closest to the eyeball of the human eye; the hollow reflector 202 is installed behind the eyepiece objective lens group 201 and is coaxial with the eyepiece objective lens group 201, so that the illumination light path is folded 90°; the first condenser lens group 203 and the second condenser lens group 205 are installed on the lower side of the hollow reflector 202, coaxial with the above-mentioned folded light path, and are used to control and adjust the light condensing and focusing effect, so that the light irradiated to the fundus can be accurately focused on the required area; the black dot plate 20 4 is installed between the first condenser lens group 203 and the second condenser lens group 205 and is coaxial therewith, and is used to eliminate the ghost image formed by the action of the illumination light beam and the retina objective lens; the illumination aperture diaphragm 206 is installed at the lower side of the second condenser lens group 205 and is coaxial therewith, and is used to control the incident angle of the light and avoid the interference of stray light; the dichroic mirror 207 is installed at the lower side of the illumination aperture diaphragm 206 to separate the infrared light path and the visible light path, and at the same time ensure that the white light and the infrared light are emitted coaxially; the first light collecting lens group 208 is used to collimate and converge the light beam emitted by the infrared light source 209; the second light collecting lens group 210 is used to collimate and converge the light beam emitted by the visible light source 211; the infrared light source 209 emits infrared light for observation and focusing, and realizes non-mydriasis focusing; the visible light source 211 is used to illuminate the fundus field of view and cooperate with the photosensitive element to collect the required fundus visible light picture.

[0080] In particular, the lighting system adopts internal lighting to achieve uniform fundus lighting brightness, high light energy utilization and clear imaging;

[0081] Furthermore, the illumination system is a Kohler illumination optical path, which improves the uniformity of the illumination beam.

[0082] In particular, the infrared light source 209 includes but is not limited to a ring-shaped infrared LED lamp, etc.; the visible light source 211 includes but is not limited to a ring-shaped white light LED lamp or a xenon lamp, etc. The working state of the entire lighting system is: in the infrared light preview mode, the infrared light source 209 is on and the visible light source 211 is off; when taking a photo, the infrared light source 209 is off and the visible light source 211 flashes once.

[0083] Specifically, the dichroic mirror 207 transmits white light and reflects infrared light. The dichroic mirror is placed at a 45° angle to the optical axis of the illumination system, and an infrared light source 209 is placed on the right side to ensure that visible light and infrared light are emitted coaxially. The imaging optical system includes an eyepiece objective lens group 201, a hollow mirror 202, a focusing lens group 212, a first plane mirror 213, a first imaging objective lens group 214, a second plane mirror 216, and a second imaging objective lens group 217. The imaging light rays of the human eye sequentially pass through the eyepiece objective lens group 201, the hollow mirror 202, the focusing lens group 212, the first plane mirror 213, and the first imaging lens group 214 to form a first optical path. The imaging light rays of the human eye sequentially pass through the eyepiece objective lens group 201, the hollow mirror 202, the focusing lens group 212, the second plane mirror 216, and the second imaging lens group 217 to form a second optical path.

[0084] Specifically, the photosensitive elements are divided into a first photosensitive element 215 and a second photosensitive element 218. Both the first photosensitive element 215 and the second photosensitive element 218 can use 4K CMOS or CCD sensors, and in the future, sensors with 8K or higher resolution can be mounted to generate fundus images with higher resolution.

[0085] Furthermore, the first photosensitive element 215 collects the reflected light signals of the first optical path, and the second photosensitive element 218 collects the reflected light signals of the second optical path.

[0086] Furthermore, the imaging qualities of the two symmetric optical paths are exactly the same, and there is a viewing angle difference between the two images formed on the two photosensitive elements, so as to achieve the purpose of simultaneously photographing the same fundus retina image from two different angles.

[0087] Furthermore, the fundus data of the first optical path collected by the first photosensitive element 215 and the fundus data of the second optical path collected by the second photosensitive element 218 are independently transmitted.

[0088] Specifically, the imaging optical system and the illumination optical system share the eyepiece objective lens group 201, and a hollow mirror 202 is used as the beam splitter element for the illumination optical path and the imaging optical path. There is a hole in the middle of the hollow mirror 202 to allow the imaging light to pass through, while the mirror surface can reflect the illumination light. The angle between the hollow mirror 202 and the optical axis of the imaging optical path should be limited within a reasonable range to ensure the normal use of the imaging optical path and the illumination optical path. Specifically, the angle between the first plane mirror 213 and the optical axis of the first optical path is a, and 0° < a < 90°; the angle between the second plane mirror 216 and the optical axis of the second optical path is b, and 0° < b < 90°.

[0089] Specifically, the 4K stereoscopic fundus camera 2 has no structural limitations and can be either a desktop stereoscopic fundus camera or a handheld stereoscopic fundus camera.

[0090] In particular, the fundus imaging principle of the 4K stereo fundus camera 2 is as follows: when working, the infrared light source 209 of the illumination optical system emits an infrared light beam which passes through the first light collecting lens group 208, the dichroic mirror 207, the illumination aperture diaphragm 206, the second condenser lens group 205, the black dot plate 204, the first condenser lens group 203, the hollow reflector 202 and the eye contact lens group 201 in sequence to reach the pupil and enter the fundus, thereby illuminating the fundus, and observing and aligning the fundus using the imaging optical system at the same time. The illuminated fundus then passes through the first optical path and the second optical path, and is finally imaged on the photosensitive element. During the focusing process, the infrared light source 209 is always on, and the refractive changes caused by different myopia and hyperopia eyes are compensated by the focusing lens group 212. After the focusing is completed, the flash of the visible light source 211 flashes instantaneously, and the photosensitive element obtains the fundus image.

[0091] The data transmission and storage module 3 is used to transmit, store and manage fundus image data; it includes a transmission data line 31, a storage device 32 and a data management software 33; the transmission data line 31 is used for connection and transmission between the photosensitive element, the AI ​​artificial intelligence image processing unit 4, the storage device 32, the 8K binocular AR glasses 5 and other devices; the storage device 32 is used to store the collected fundus image data, and the 3D fundus image after data enhancement; the data management software 33 is used to manage, organize and analyze the data stored on the storage device, and provide data query, retrieval, backup and other functions.

[0092] Furthermore, the transmission data line 31 includes but is not limited to HDMI signal lines, DP signal lines, DVI signal lines, SDI signal lines, USB signal lines and other data lines, and a wireless connection method may also be adopted; it should be noted that the transmission data line 31 is only used to illustrate the connection line, and does not mean that there are only two transmission data lines.

[0093] Further, the storage module 32 includes but is not limited to a hard disk, a solid-state memory, a cloud storage, etc.;

[0094] Furthermore, the data management software 33 includes but is not limited to a database management system (DBMS), Hexagor DSP, and the like.

[0095] The AI ​​artificial intelligence image processing unit 4 performs enhancement, detection and three-dimensional imaging on the received fundus image data, including an image analysis module 41, an image processing module 42 and a 3D image imaging module 43. Figure 3As shown; the image analysis module 41 is used to pre-process the fundus image data collected by the first optical path and the second optical path, filter out the fundus images with poor quality and low definition caused by factors such as vibration of the operating platform in advance, and mark the images respectively to reduce the data transmission load and ensure high-speed real-time transmission; the image processing module 42 is used to process and enhance the fundus image to improve the image quality; the 3D image imaging module 43 integrates the images of the first optical path and the second optical path after the data enhancement into a 3D stereo image;

[0096] Further, the image processing module 42 includes an image geometry calibration module 421, an image defogging module 422, an image denoising module 423, an image chromaticity calibration module 424 and an image visual adjustment module 425;

[0097] Further, the image geometry calibration module 421 determines the coordinates of the four corners of the first image with the first identifier and the second image with the second identifier in the RGB image data by using an edge search image algorithm, and then calibrates the first image and the second image into the first standard image and the second standard image respectively by using a trapezoidal calibration algorithm;

[0098] Furthermore, the first standard image and the second standard image are enlarged or reduced to have the same area;

[0099] In particular, since the two optical imaging paths are different and are imaged through different lenses, the images formed will not be completely the same in terms of geometry, which is mainly manifested as trapezoidal distortion of a single-path image and inconsistency in the size of the two-path images, which will affect the accuracy of sample observation. The image geometry calibration unit 421 implements pixel-level calibration of the physical deviation between the two-path optical path systems, eliminating the geometric shape deviation and size deviation of the two-path images.

[0100] Furthermore, the image defogging module 422 uses a defogging algorithm to reduce the stray light phenomenon in the fundus image and improve the image clarity. The visual characteristics of the stray light in the fundus image have certain similarities with the common image fogging phenomenon, which can be used as a reference for removing the stray light; the image defogging algorithm is mainly divided into an algorithm based on image enhancement and an algorithm based on image restoration.

[0101] Furthermore, the image denoising module 423 removes the noise in the image through a denoising algorithm to improve the quality and clarity of the image. Fundus images may be affected by uneven lighting conditions, device sensor noise, motion blur, etc., resulting in various types of noise in the image, such as Gaussian noise, salt and pepper noise, etc. These noises will reduce the contrast, detail information and visual quality of the image, affecting the doctor's judgment and diagnosis of fundus diseases;

[0102] Furthermore, the image chromaticity calibration module 424 calibrates the chromaticity response and light intensity response of the color image data; the chromaticity calibration includes two processes: color calibration and gamma calibration. The color calibration obtains calibrated RGB data by performing linear transformation on the RGB data of the original image and processing the color correction matrix, and compensates for the spectral response of the photosensitive element so that the output RGB data color conforms to the spectral response curve of the human eye; the gamma calibration converts the linear response curve into a gamma index response through a gamma transformation algorithm, compensates for the light intensity response of the photosensitive element, and restores the color and brightness information of the real sample.

[0103] Furthermore, the image visual adjustment module 425 is used to adjust the brightness, contrast, exposure time and white balance of the color image data. Brightness adjustment is to increase or decrease the intensity of the three primary colors of RGB numerically, and is suitable for fundus images with insufficient light or high light reflectivity. Contrast adjustment can increase the dynamic range of color and make the light and dark areas clearer. Automatic white balance adjustment is intended to automatically adjust the RGB intensity ratio according to the light source environment of different tones to achieve the best reflection effect in an ideal white light source environment. Exposure time adjustment shortens or prolongs the image acquisition time according to different light source environments.

[0104] Furthermore, the 3D image imaging module 425 applies a stereoscopic vision parallel algorithm to extract the overlapping area image between the image of the first light path and the image of the second light path in the fundus RGB image data, and synthesize a 3D image to obtain a stereoscopic image with spatial position information; through the three-dimensional image, medical personnel can more accurately and clearly observe the size, spatial position, geometric shape and relationship between the fundus lesions and other surrounding tissue structures, and can observe the tissue structure of the patient's fundus from multiple angles and multiple levels;

[0105] In particular, the stereoscopic vision parallel algorithm includes but is not limited to web3D technology, etc.;

[0106] In particular, color image dehazing algorithms, image denoising algorithms, chromaticity calibration algorithms, visual adjustment algorithms, and stereoscopic vision algorithms are implemented with the help of artificial intelligence, such as deep learning or traditional machine learning techniques, to accelerate processing speed and improve accuracy. It should be noted that the above-mentioned deep learning or machine learning techniques should comply with the requirements of the "Key Points for the Review of Deep Learning Assisted Decision-Making Medical Device Software" to measure the algorithm's generalization ability and clinical use risks.

[0107] Specifically, the deep learning neural network model is a computational model that imitates the neuron structure of the human brain. It abstracts and processes data layer by layer through multiple layers of neurons and weights. These models can automatically learn and extract useful features from data to improve performance. The neural network model includes convolutional neural network (CNN), generative adversarial network (GAN), recurrent neural network (RNN), Transformer and other models. Different neural network models can be selected according to different usage requirements.

[0108] In particular, in some embodiments of the present invention, the color image dehazing algorithm uses a DFC-dehaze model. The DFC-dehaze model is improved based on the CycleGAN model. The CycleGAN model is a cycle-consistent adversarial network, which is a variant of the generative adversarial network and is widely used in image dehazing tasks, but the quality of the generated dehazed images is low. Specifically, the DFC-dehaze model uses the Dehazeformer-t model to replace the CNN-based generator of the CycleGAN model, and uses a local-global discriminator to process the locally changing haze, so as to reduce the haze residue in the restored image. When the generated image is blurred or the color is not real, the discriminator gives a low score through a negative sample penalty mechanism to suppress the generator from outputting an inappropriate dehazed image. Among them, the Dehazeformer-t is improved from the swin-transformer. Compared with the swin-transformer, the ReLU activation function is used to replace the GELU activation function in the Dehazeformer block. In addition, it applies RescaleNorm instead of LayerNorm to normalize the entire feature map to prevent the loss of correlation between blocks. Reflection filling is used to enhance the quality of the edges of the dehazed image.

[0109] The image denoising algorithm uses the MambaIR model. The MambaIR network performs low-level image restoration by adjusting the state space model, and is a simple but effective alternative to CNN and Transformer. Furthermore, the MambaIR model consists of a shallow feature extraction module, a deep feature extraction module, an upsampling module, and a reconstruction module, wherein the core of the entire model of the deep feature extraction module is mainly composed of multiple residual state space blocks (RSSB). Furthermore, the RSSB block mainly includes a convolutional layer, a Mamba module, a channel attention module, and a residual connection module. Among them, the convolutional layer is used to extract local features, the Mamba module is used for long-range dependency modeling, the channel attention is used to enhance the interaction between channels, and the residual connection is used for gradient propagation.

[0110] The chromaticity calibration algorithm and the visual adjustment algorithm can use a DNN model. The DNN model mainly includes an input layer, a hidden layer and an output layer. Among them, the number of hidden layers can be reasonably adjusted according to actual needs. The input layer receives the original image data, the hidden layer is responsible for extracting features and performing nonlinear transformations, and the output layer produces the final processing result. In the chromaticity calibration and visual adjustment tasks, the hidden layer of the DNN can be composed of multiple fully connected layers, and the number of these layers can be adjusted according to actual needs to adapt to the needs of scenes of different complexities.

[0111] The stereo vision algorithm uses the ACVNet model. The ACVNet model is a cost volume construction method based on attention weights, which can remove redundant information and retain information related to matching; in addition, the ACVNet model adopts a multi-level adaptive block matching strategy to obtain distinguishing matching features and improve the feature expression ability of weak (no) texture areas. Furthermore, the ACVNet model includes four parts: feature extraction, attention weight volume construction, cost aggregation, and disparity prediction. Among them, the feature extraction uses 2D CNN to extract feature maps for the left and right views respectively, and the size of the obtained feature map is 1 / 4 of the original image. Then, the feature map is further extracted through a CNN with a stride of 1 to obtain feature maps with 64, 128, and 128 channels, respectively, and the size is kept at 1 / 4 of the original image. These feature maps are spliced ​​along the channel dimension to form an attention construction feature map. Finally, the spliced ​​feature map dimension is compressed to 32 through CNN to form a matching cost volume construction feature map. The attention weight body construction uses the 320-channel attention weight to construct the feature map to obtain the attention weight body, then uses the 32-channel matching cost body construction feature map to construct the initial matching cost body, and finally uses the attention weight to filter the initial matching cost body. The cost aggregation uses 4 3D convolutions and 2 stacked 3D U-Net networks to perform cost body regularization to obtain out1, out2, and out3. The disparity prediction calculates the disparity map based on the expected form.

[0112] In particular, the image processing and enhancement process: after being marked by the image analysis module 41, the first fundus image data and the second fundus image data are processed by the image geometry calibration module 421, the image defogging module 422, the image denoising module 423, the image chromaticity calibration module 424 and the image visual adjustment module 425 respectively, and finally fused into a three-dimensional image output in the 3D image imaging module 43. It should be noted that the first fundus image data and the second fundus image data are processed and enhanced at the same time to ensure real-time performance.

[0113] In particular, the data transmission and storage module is integrated with the AI ​​artificial intelligence image processing unit into an integrated structure - the image processing unit; the image processing unit can be any suitable hardware such as a field programmable gate array or a combination of software and hardware. For example, in an embodiment, the image processing unit is a field programmable gate array (Field Programmable Gate Array, referred to as FPGA) or an embedded system, and can also be equipped with an AI computing platform such as NVIDIA Clara HoloscanSDK. At the same time, the image processing unit can be equipped with a Kryo quad CPU and an Adreno GPU, etc.

[0114] The 8K binocular AR glasses 5 display 3D stereo images in high definition to guide diagnosis and treatment. In an embodiment, 8K binocular waveguide AR glasses are used;

[0115] In particular, the 8K binocular waveguide AR glasses system uses optical waveguide technology, which has the advantages of high brightness and low power consumption, and provides clear and stable images. Figure 4 As shown, the optical waveguide 52 is installed inside the AR lens 51, and the incident light 55 enters the optical waveguide 52 through the input grating 53, is transmitted to the output grating 54, and enters the user's field of view through the output grating 54. The optical waveguide technology has the characteristics of large field of view and high light transmittance, which brings a natural and comfortable visual experience.

[0116] Furthermore, the 8K binocular waveguide AR glasses system is equipped with 8K high-resolution and high-sensitivity sensors to track the user's head and eye movements in real time, achieving more accurate image positioning and display effects;

[0117] Furthermore, the 8K binocular waveguide AR glasses system supports multiple interactive methods such as voice recognition and gesture control, and displays fundus image information in the glasses to guide diagnosis and treatment;

[0118] In particular, the 8K binocular waveguide AR glasses system also includes modules such as graphics processor and GPU; the processor uses high-performance chips, such as MTK6765, to provide powerful computing power and low power consumption performance; the GPU uses high-performance GPUs, such as IMGGE8320, for 3D rendering and image processing.

[0119] In summary, the present invention provides an 8K binocular stereo fundus camera system. The present invention combines a super lens, a 4K stereo fundus camera, AI artificial intelligence image processing and 8K binocular waveguide AR glasses to realize the world's first micro-nano optical 8K stereo fundus camera. The stereo fundus camera system has the characteristics of miniaturization, intelligence, convenient operation, high-definition imaging and stereoscopic display, supports the operation of AI artificial intelligence algorithms such as deep learning, and can realize fundus image acquisition at any time, any place and any way, providing a new solution for medical diagnosis and treatment, with important application value and broad market prospects.

[0120] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An 8K stereo fundus camera, characterized in that: The invention comprises a super lens (1), a 4K stereo fundus camera (2), a data transmission and storage module (3), an AI artificial intelligence image processing unit (4) and 8K binocular AR glasses (5); the super lens (1) is installed on the 4K stereo fundus camera (2) and is used for optical system imaging of the 4K stereo fundus camera (2), so as to obtain high-quality fundus images while reducing the volume of the optical system; the 4K stereo fundus camera (2) is connected to the AI ​​artificial intelligence image processing unit (4) through the data transmission and storage module (3), and the AI ​​artificial intelligence image processing unit (4) is connected to the 8K binocular AR glasses (5); The 4K stereo fundus camera (2) collects 4K high-definition fundus images; the data transmission and storage module (3) is used for transmitting and storing fundus image information; the AI ​​artificial intelligence image processing unit (4) is used for processing and enhancing fundus images to generate high-definition 3D images; and the 8K binocular AR glasses (5) are used for displaying high-definition stereo fundus images.

2. The 8K stereo fundus camera according to claim 1, characterized in that: The metalens (1) is one or more phase-type metalens, or a combination of a metalens and a refractive lens, which together constitute the optical system of the stereo fundus camera (2).

3. The 8K stereo fundus camera according to claim 1 or 2, characterized in that: The superlens comprises a substrate (11) and a plurality of nanostructures (12) arranged on the substrate (11).

4. The 8K stereo fundus camera according to claim 1, characterized in that: The 4K stereo fundus camera (2) comprises: The illumination optical system is used to introduce appropriate illumination into the fundus; the imaging optical system is used to image the appearance of the fundus on the photosensitive element of the 4K stereo fundus camera (2); the photosensitive element is used to convert the light signal collected by the imaging system into fundus image data for subsequent processing.

5. The 8K stereo fundus camera according to claim 4, characterized in that: The illumination optical system comprises, from the eyeball side to the light source side, an eyepiece objective lens group (201), a hollow reflector (202), a first condenser lens group (203), a black dot plate (204), a second condenser lens group (205), an illumination aperture diaphragm (206), a dichroic mirror (207), a first condenser lens group (208), an infrared light source (209), a second condenser lens group (210), and a visible light source (211); Visible light emitted by the visible light source (211) enters the hollow reflector (202) via the second light collecting lens group (210), the dichroic mirror (207), the illumination aperture diaphragm (206), the second condensing lens group (205), the black dot plate (204), and the first condensing lens group (203); The infrared light emitted by the infrared light source (209) enters the hollow reflector (202) via the first light collecting lens group (208), the dichroic mirror (207), the illumination aperture diaphragm (206), the second light collecting lens group (205), the black dot plate (204), and the first light collecting lens group (203); Visible light and infrared light reflected by the hollow reflector (202) are used to collect fundus images through an eyepiece objective lens group (201), and the fundus images are input into an imaging optical system; The dichroic mirror (207) transmits white light and reflects infrared light, ensuring that visible light and infrared light are emitted coaxially; the working state of the illumination optical system is: in the infrared light preview mode, the infrared light source (209) is turned on and the visible light source (211) is turned off; when taking pictures, the infrared light source (209) is turned off and the visible light source (211) flashes once; The illumination optical system adopts an internal illumination mode and a Kohler illumination optical path to improve the uniformity of the illumination beam; and the illumination optical system realizes a mydriasis-free working mode.

6. The 8K stereo fundus camera according to claim 4, characterized in that: The imaging optical system comprises, from the eyeball side to the image side, an eyepiece objective lens group (201), a hollow reflector (202), a focusing lens group (212), a first plane reflector (213), a first imaging objective lens group (214), a second plane reflector (216) and a second imaging objective lens group (217); the imaging light of the human eye passes through the eyepiece objective lens group (201), the hollow reflector (202), the focusing lens group (212), the first plane reflector (213) and the first imaging lens group (214) in sequence to form a first optical path; the imaging light of the human eye passes through the eyepiece objective lens group (201), the hollow reflector (202), the focusing lens group (212), the second plane reflector (216) and the second imaging lens group (217) in sequence to form a second optical path; the imaging optical system and the illumination optical system share the eyepiece objective lens group (201), and a hollow reflector (202) is used as a light splitting element for the illumination optical path and the imaging optical path; The photosensitive element is divided into a first photosensitive element (215) and a second photosensitive element (218); the first photosensitive element (215) and the second photosensitive element (218) adopt 4K CMOS or CCD sensors; the first photosensitive element (215) collects the reflected light signal of the first light path, and the second photosensitive element (218) collects the reflected light signal of the second light path; The fundus imaging principle of the 4K stereo fundus camera (2) is as follows: when in operation, an infrared light source (209) of the illumination optical system emits an infrared light beam which sequentially passes through a first light collecting lens group (208), a dichroic mirror (207), an illumination aperture diaphragm (206), a second light collecting lens group (205), a black dot plate (204), a first light collecting lens group (203), a hollow reflector (202) and an eyepiece lens group (201) to reach the pupil and enter the fundus, thereby illuminating the fundus and simultaneously using the imaging optical system for observation and alignment; the illuminated fundus then passes through the first light path and the second light path, and is finally imaged on a photosensitive element.

7. The 8K stereo fundus camera according to claim 1 or 4, characterized in that: The 4K stereo fundus camera (2) is a desktop stereo fundus camera or a handheld stereo fundus camera.

8. The 8K stereo fundus camera according to claim 1, characterized in that: The data transmission and storage module (3) comprises: a transmission data line (31) for connecting and transmitting fundus image information between a photosensitive element, the AI ​​artificial intelligence image processing unit (4), a storage device (32), and the 8K binocular AR glasses (5); a storage device (32) for storing collected fundus image data and data-enhanced 3D fundus images; and data management software (33) for managing, organizing and analyzing fundus image data stored on the storage device, and providing fundus image data query, retrieval, and backup.

9. The 8K stereo fundus camera according to claim 1, characterized in that: The AI ​​artificial intelligence image processing unit (4) comprises: an image analysis module (41) for preprocessing and marking fundus image data collected by the first light path and the second light path; an image processing module (42) for processing and enhancing the fundus image; and a 3D fundus image imaging module (43) for integrating the fundus images of the first light path and the second light path after the data enhancement into a 3D fundus image. The image processing module (42) comprises: an image geometry calibration module (421) for calibrating the pixel level of the physical deviation between the two optical path systems; an image defogging module (422) for reducing the stray light phenomenon in the fundus image and improving the image clarity; an image denoising module (423) for improving the image contrast and signal-to-noise ratio; an image chromaticity calibration module (424) for calibrating the chromaticity response and light intensity response of the color image data; and an image visual adjustment module (425) for adjusting the brightness, contrast, exposure time and white balance of the color image data. The color image defogging algorithm, image denoising algorithm, chromaticity calibration algorithm, visual adjustment algorithm and stereoscopic vision algorithm are implemented with the help of artificial intelligence to accelerate processing speed and improve accuracy.

10. The 8K stereo fundus camera according to claim 1, characterized in that: The 8K binocular AR glasses (5) adopt 8K optical waveguide AR glasses, including: Display, using a micro 8K high-resolution display; Light guide lenses use light guides to refract the light from the micro display and transmit it to the human eye; The processor is used to process the fundus information collected by the optical waveguide lens and input it to the GPU, which is used for 3D rendering and fundus image processing.

Images