8K stereoscopic display endoscope

Abstract

The invention discloses an 8K three-dimensional display endoscope which comprises a super lens, a 4K endoscope, an infrared detector, a data transmission and storage device, a super-computing AI image processing device and 8K binocular AR glasses. Wherein the super lens is used for optical system imaging; the 4K endoscope is used for realizing high-definition imaging so as to improve the image quality; the infrared detection is used for establishing a 2D or 3D image through non-minimally invasive scanning and determining the position of a diseased tissue; the data transmission and storage is used for image transmission and storage; the supercomputing AI image processing is used for image processing and enhancement; the 8K binocular AR glasses are used for displaying high-definition stereo images. The endoscope provided by the invention is the first micro-nano optical 8K three-dimensional display endoscope in the world, has the characteristics of high definition and vivid color, provides a brand new solution for medical diagnosis and treatment, and has important application value and wide market prospect.

Description

Technical Field

[0001] The present invention relates to an endoscope technology, and in particular to an 8K stereoscopic display endoscope that combines a super lens, 8K binocular AR glasses, a 4K endoscope, and supercomputing AI image processing. Background Art

[0002] As an important medical device, endoscope is used to observe the hard-to-observe areas in the human body. It is widely used in internal inspection and treatment. It is convenient, direct and efficient. However, traditional endoscopes have problems such as low image resolution, unclear image quality and inconvenient operation, which limit their application effect. With the development of optical technology and image processing technology, endoscope technology has gradually developed towards high-definition, three-dimensional display and intelligence, but there are still some technical bottlenecks. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an 8K stereoscopic display endoscope that combines a superlens, 8K binocular AR glasses, a 4K endoscope and supercomputing AI image processing to achieve high-definition stereoscopic display and improve the effect of endoscopic inspection and treatment.

[0004] In order to achieve the above technical objectives, the present invention adopts the following technical solutions: An 8K stereoscopic display endoscope includes a super lens, a 4K endoscope, an infrared detection auxiliary module, a data transmission and storage module, a supercomputing AI image processing unit and an 8K binocular AR glasses device.

[0005] The super lens is arranged at the end of the 4K endoscope for image acquisition, the infrared detection auxiliary module is arranged on the 4K endoscope, the 4K endoscope is connected to the supercomputing AI image processing unit through the data transmission and storage module, and the supercomputing AI image processing unit is connected to the 8K binocular AR glasses device; Among them, the superlens is used in the optical imaging system to provide high-definition image information; the 4K endoscope obtains images of internal human tissues through the imaging optical system; the infrared detection auxiliary module is used for non-minimally invasive scanning, and establishes 2D or 3D images, distinguishes normal tissues and diseased tissues inside the human body, and determines the location of diseased tissues; the data transmission and storage module is used to transmit and store image information collected by the 4K endoscope and infrared detection module, and transmit it to the supercomputing AI image processing unit at the same time; the supercomputing AI image processing unit performs image processing and enhancement on the above image information; the 8K binocular AR glasses realize high-definition stereoscopic display of the processed images, realizing immersive data interaction and operation.

[0006] Preferably, the optical system of the 4K endoscope based on the metalens can be one or more metalens, wherein the metalens is a phase-type metalens, comprising a substrate and a plurality of nanostructure units arranged on the substrate.

[0007] Preferably, the above-mentioned 4K endoscope includes an illumination light source, a visible light camera, a single-mode optical fiber and a hard tube for the endoscope body. The illumination light source, the visible light camera and the first single-mode optical fiber are all installed on the hard tube for the endoscope body; the illumination light source is connected to the first single-mode optical fiber, wherein the illumination light source is used to provide the visible light illumination conditions required for image acquisition; the visible light camera cooperates with the 4K endoscope to acquire visible light images; the first single-mode optical fiber is used to transmit visible light to the target detection area; the hard tube for the endoscope body is used to fix components such as the optical system and the single-mode optical fiber.

[0008] Preferably, the visible light camera adopts a 4K CMOS or CCD sensor, or is equipped with an 8K or higher resolution sensor to generate higher definition images.

[0009] Preferably, the infrared detection auxiliary module includes an infrared light source, an infrared camera, a second single-mode optical fiber and a dichroic mirror. The infrared light source is used to provide infrared lighting conditions for the target detection area; the infrared camera cooperates with the endoscope to collect infrared images; the second single-mode optical fiber is used to transmit the infrared light source to the target detection area; the dichroic mirror is used to divide the optical signal collected by the endoscope into two, one part of which is a visible light signal recognizable by the visible light camera, and the other part is an infrared light signal recognizable by the infrared camera; the optical signal is converted into image information by the sensor.

[0010] Preferably, the infrared camera must match the infrared light source.

[0011] Preferably, the first light emitted by the surface light source, the second light emitted by the infrared light source, and the optical signal are transmitted independently.

[0012] Preferably, the data transmission and storage module of the above-mentioned endoscope system includes a transmission data line and a storage module, wherein the transmission data line is used to transmit image information for subsequent processing; and the storage module is used to store and collect surgical cases.

[0013] Preferably, the above-mentioned endoscope system, its artificial intelligence AI image processing unit includes an image defogging module, an image denoising module, an image segmentation module, an image fusion module and a 3D image generation module. Among them, the image defogging module is used to perform defogging processing on visible light images to reduce image blur caused by fogging due to factors such as steam during surgery and other applications; the image denoising module is used to reduce the noise of the image processed by the image defogging module, reduce the image noise caused by factors such as light conditions, and improve the image contrast and clarity; the image segmentation module is used to segment the diseased tissue location area in the infrared image; the image fusion module is used to fuse the segmented diseased tissue location area with the visible light image to generate an image containing the diseased tissue location information; the 3D image generation module is used to generate a three-dimensional structure diagram from the fused image.

[0014] Preferably, the image defogging module, image denoising module, image segmentation module, image fusion module, and 3D image generation module can all be assisted by artificial intelligence algorithms such as deep learning. These artificial intelligence algorithms are currently relatively mature and can greatly improve image processing effects while reducing processing time and ensuring the real-time transmission of results.

[0015] Furthermore, the image defogging module, image denoising module, image segmentation module, image fusion module and 3D image generation module can all use any suitable neural network model such as a convolutional neural network (CNN), a generative adversarial network (GAN), a Transformer or a Mamba model; the neural network models can be combined with each other as long as there is no conflict.

[0016] The image defogging module uses the UVMNet defogging network; the UVMNet defogging network combines the advantages of the convolutional network and the Mamba model. As a newcomer in the field of image defogging, it has significant effects in the field of image defogging.

[0017] Specifically, the UVMNet includes a 4-layer network using a U-shaped structure; each layer of the network contains two consecutive convolution blocks and a UVM-net module; in the UVM-net module, after the image is convolved, remote modeling is performed on the State Space Sequence Models (SSMs) module.

[0018] The image denoising module uses the MambaIR network; the MambaIR network has superior performance in image denoising and has significant denoising effects on real images. The MambaIR network includes a shallow feature extraction module, a deep feature extraction module, an upsampling module and a reconstruction module. Among them, the Residue State Space Block (RSSB) is the core component of MambaIR, which specifically includes convolutional layers, Mamba modules, channel attention and residual connections; the convolutional layer is used for local feature extraction; the Mamba module is used to model long-range dependencies; the channel attention is used to enhance the interaction between channels; the residual connection facilitates gradient propagation.

[0019] The image segmentation module uses the UltraLight VM-Unet network; the UltraLight VM-Unet network has significant effects in the field of medical image segmentation; the UltraLight VM-Unet network has a total of 6 layers of networks, among which the first 3 layers are convolutional networks for extracting shallow features of the image, and each layer includes a standard convolution with a 3x3 convolution kernel and a maximum pooling operation; the last 3 layers are parallel Vision Mamba (PVM) layers for extracting deep features of the image; the PVM layer is improved based on the Vision Mamba layer; in addition, the jump connection path uses the channel attention bridge (CAB) module and the spatial attention bridge (SAB) module for multi-stage and multi-scale information fusion; specifically, the SAB module includes maximum pooling, average pooling and extended convolution with shared weights; the CAB module includes global average pooling, splicing operation, fully connected layer and sigmoid activation function.

[0020] Preferably, the diseased tissue image obtained by image segmentation can be marked by means of fluorescent marking or other means to improve the recognition of the diseased tissue.

[0021] The image fusion module uses a multi-mode image fusion method, mainly including neural network methods based on autoencoders (AE), CNNs and GANs. Preferably, in the embodiment of the present invention, a GAN-based network model is selected, such as in the embodiment of the present invention, a FusionGAN network is used; further, the FusionGAN network includes a generator and a discriminator; the generator network includes 5 convolution layers, the first and second layers use 5×5 convolution kernels, the third and fourth layers use 3×3 convolution kernels, and the fifth layer uses 1×1 convolution kernels. The convolution step size of each layer is 1, and no padding is used. The input is the spliced ​​infrared and visible light images; the discriminator network includes 5 convolution layers, the first four layers are all 3×3 convolution kernels, and the step size is set to 2. In order to avoid introducing noise, padding is added to the image only in the first layer of the input. Batch normalization is added to the second to fourth layers. The activation function of the first four layers is leaky ReLU, and the last layer is a linear layer for classification.

[0022] The 3D image generation module uses the Make-It-3D method; the Make-It-3D method uses deep learning to generate a high-fidelity 3D structure using only a single image; further, the Make-It-3D method includes two stages; the first stage optimizes the neural radiation field NeRF by combining the constraints of the front view of the reference image and the diffusion prior of the new view; the second stage converts the coarse model into a texture point cloud, and uses the high-quality texture of the reference image combined with the diffusion prior to further enhance the realism.

[0023] In particular, all of the above-mentioned deep learning models are pre-trained, and the trained model weights are loaded into the supercomputing AI image processing unit; the supercomputing AI image processing unit is equipped with high-performance image processing software and hardware to ensure high-speed and high-precision processing.

[0024] In particular, the above deep learning models can be optimized and integrated into a whole, simplifying the optimization steps; Preferably, in order to make the endoscope system compact and easy to use, the 4K endoscope is integrated with the infrared detection module into an integrated structure; the data processing and transmission module is integrated with the artificial intelligence AI image processing unit into an integrated structure - the image processing unit.

[0025] Preferably, the image processing unit may be any suitable hardware or a combination of hardware and software, such as a field programmable gate array.

[0026] Preferably, the 8K binocular AR glasses device of the above-mentioned endoscope system uses 8K resolution and optical waveguide technology; 8K resolution can display image data processed by the supercomputing AI processing unit, which helps to provide more accurate diagnosis; optical waveguide technology is thin and light, which can reduce the overall weight of AR glasses while providing a large field of view and high definition. When doctors wear 8K binocular AR glasses instead of traditional screen navigation, they have a new perspective of "walking into" the human body, and do not need to take their eyes away from the patient to observe the digital scan results on the split screen, which greatly shortens the diagnosis and treatment time.

[0027] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art: 1. Using superlens and 4K endoscopes can achieve high-definition imaging, significantly improving image quality, making the image clear and the color realistic. Superlens has the characteristics of small size and high degree of freedom, which can effectively reduce the volume of the imaging optical system while improving image quality; 2. Use infrared detection technology to achieve non-minimally invasive scanning and effectively reduce surgical risks. Infrared light can penetrate deeper tissues, which is helpful for determining the specific location of diseased tissues, improving the ability of medical staff to distinguish between normal tissues and diseased tissues, and is of great help in timely detection and treatment of diseased tissues; 3. Use supercomputing AI image processing to provide intelligent image enhancement and processing, realize automatic lesion detection and real-time auxiliary diagnosis and treatment; 4. 8K binocular AR glasses can achieve high-definition stereoscopic display to improve surgical results and accuracy. With 8K binocular AR glasses, medical staff can watch high-definition stereo mixed images (visible light images and infrared segmented images) in real time to improve surgical accuracy and treatment effects.

[0028] 5. The data storage module can collect and store surgical cases, providing data support and reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of an embodiment of an 8K stereoscopic display endoscope system provided by the present invention; Figure 2 It is a schematic diagram of an image displayed by 8K binocular AR glasses in one embodiment of the 8K stereoscopic display endoscope system provided by the present invention.

[0030] Reference numerals: 1. Super lens; 2. 4K endoscope; 21. Illumination source; 22. Visible light camera; 23. Single-mode optical fiber for transmitting illumination source; 24. Hard tube for mirror body; 3. Infrared detection module; 31. Infrared light source; 32. Infrared light camera; 33. Single-mode optical fiber for transmitting infrared light source; 34. Dichroic mirror; 4. Transmission and storage module; 41. Transmission line for visible light image data; 42. Transmission line for infrared light data; 43. Storage module; 5. Supercomputing AI image processing unit; 51. Image segmentation module; 52. Image defogging module; 53. Image denoising module; 54. Image fusion module; 55. 3D image generation module; 6. 8K optical waveguide binocular AR glasses; E1. Light source lamp head. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features involved in the embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0032] It should be noted that if there are directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention, the directional indications are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly. In addition, if there are descriptions of "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features.

[0033] The traditional endoscope system has low image resolution and unclear display, which seriously reduces its application effect. With the development of optical technology and image processing technology, endoscope technology is gradually developing towards high definition, three-dimensional display and intelligence, but there are still some technical bottlenecks.

[0034] Figure 1 As shown, it is a schematic diagram of an embodiment of the 8K stereoscopic display endoscope system of the present invention to solve the above-mentioned problems; the endoscope system provided in the embodiment includes a super lens 1, an endoscope 2, an infrared detection module 3, a data transmission and storage module 4, a supercomputing AI image processing unit 5, and 8K binocular AR glasses 6.

[0035] Among them, the superlens 1 is used in the optical imaging system, placed at the front end of the hard tube 24 of the endoscope body, including the substrate and the nanostructure. When the size of the endoscope is limited, the nanostructure can be used to directly regulate the distribution of the focused light field to improve the imaging quality of the optical system; the nanostructure is a sub-wavelength scale phase change material that can couple with light, causing the phase of the light to change rapidly. By adjusting the optical parameters of the nanostructure, such as refractive index, dielectric constant, shape and size, it is possible to achieve 360-degree phase mutation and efficiently control the light wavefront.

[0036] The endoscope 2 adopts a superlens optical system and acquires high-definition images through a high-resolution detector, including an illumination light source 21, a visible light camera 22, a single-mode optical fiber 23 and a hard tube 24 for the endoscope body; the illumination light source 21 is used to provide the visible light conditions required for image acquisition; the visible light camera 22 is equipped with a 4KCMOS or CCD sensor for capturing high-resolution images captured by the endoscope, and in the future it may be equipped with an 8K or higher resolution sensor to capture higher-definition images; the single-mode optical fiber 23 is used to transmit the first light emitted by the illumination light source 21 to the target detection area; the hard tube 24 is used to carry the optical fiber and fix the optical imaging system, etc.

[0037] The infrared detection module 3 is used to assist endoscopic detection and establish a 2D or 3D image through non-minimally invasive scanning, including an infrared light source 31, an infrared camera 32, a single-mode optical fiber 33, and a dichroic mirror 34; the infrared light source 31 is used to provide infrared lighting conditions for the target detection area; the infrared camera 32 is used to capture the signal transmitted by infrared radiation, and must match the wavelength emitted by the infrared light source 31 so that the infrared camera can accurately receive the optical signal; the single-mode optical fiber 33 is used to transmit the second light emitted by the infrared light source 31 to the target detection area; the dichroic mirror 34 is used to divide the optical signal collected by the endoscope 2 into two, one part of which is the infrared light signal recognizable by the infrared camera 32, and the other part is the visible light signal recognizable by the visible light camera 22; because the endoscope can obtain the infrared light image of the target detection area, and the infrared light image can penetrate the deeper tissue of the target detection area, it is helpful to distinguish normal tissue from diseased tissue and determine the specific location area of ​​the diseased tissue; In addition, the mirror body hard tube 24 includes at least three single-mode optical fibers. To avoid signal crosstalk, the first light emitted by the surface light source 21, the second light emitted by the infrared light source 31, and the optical signal are transmitted independently. In addition, the endoscope 2 and the infrared detection module 3 are integrated into an integrated structure, and the positions can be freely matched; Furthermore, the infrared light source 31 and the illumination light source 21 may be turned on at the same time, or may be turned on in sequence as required.

[0038] The data transmission and storage module 4 is used to transmit the visible light images captured by the visible light camera 22 and the infrared images captured by the infrared camera 32 to the supercomputing AI image processing unit 5 for subsequent processing, and includes data lines 41, 42 and a storage module 43; the storage module 43 is used to collect and store surgical cases to provide data support and reference.

[0039] The supercomputing AI image processing unit 5 includes an image segmentation module 51, an image defogging module 52, an image denoising module 53, an image fusion module 54 and a 3D image generation module 55, which are used to process and enhance the visible light image and the infrared image; the image segmentation module 51 segments the infrared image, distinguishes between normal tissue and diseased tissue, and obtains the specific location information of the diseased tissue; the image defogging module 52 is used to improve the clarity of the visible light image and reduce the image blur caused by fogging due to factors such as steam during surgery or other processes; the image denoising module 53 is used to reduce the noise of the visible light image and improve the image contrast and signal-to-noise ratio; the image fusion module 54 fuses the segmented diseased tissue location area with the visible light image to generate an image containing the diseased tissue location information and displays it in real time; in order to further highlight the coordinate position of the diseased tissue, the diseased tissue can be marked (such as fluorescent marking); the 3D image generation module 55 is used to generate a three-dimensional structure diagram from the fused image.

[0040] The above-mentioned supercomputing AI image processing unit can use artificial intelligence algorithms such as deep learning to process and enhance images, obtain images with higher clarity and more realistic colors, guide case treatment, and reduce human visual errors.

[0041] Furthermore, the image defogging module 51, the image denoising module 52, the image segmentation module 53, the image fusion module 54 and the 3D image generation module 55 can all use any suitable neural network model such as a convolutional neural network (CNN), a generative adversarial network (GAN), a Transformer or a Mamba model; the neural network models can be combined with each other as long as there is no conflict.

[0042] It is worth noting that the CNN network has a powerful feature extraction capability and a strong generalization capability, but CNN has a limited receptive field problem; Transformer has the ability to extract global features and supports parallel computing, but Transformer has large computing and memory consumption; the Mamba model can effectively solve the problems of Transformer. Therefore, in an embodiment of the present invention, the neural network model used is mainly a combination of CNN and Mamba models. In the future, with the continuous development of deep learning, a more efficient and superior neural network model can be used.

[0043] In an embodiment of the present invention, the image defogging module 51 uses the UVMNet defogging network. UVMNet combines the advantages of the convolutional network and the Mamba model. As a newcomer in the field of image defogging, it has a remarkable effect in the field of image defogging. Specifically, UVMNet includes a 4-layer network using a traditional U-shaped structure; each network layer contains two consecutive convolution blocks and a UVM-net module. In the UVM-net module, after the image is convolved, remote modeling is performed on the State Space Sequence Models (SSMs) module; wherein the convolution block uses a standard convolution layer (convolution kernel size is 3) and the activation function is LeakyReLU; the SSMs module uses a Bi-SSM module. Before the image is input into the network, it needs to be preprocessed to make it conform to the image input size of the neural network model.

[0044] The image denoising module 52 uses the MambaIR network. The MambaIR network has superior performance in image denoising and has a significant denoising effect on real images. Specifically, the MambaIR network includes a shallow feature extraction module, a deep feature extraction module, an upsampling module and a reconstruction module. The input image first undergoes shallow feature extraction, that is, a 3×3 convolution, and then the shallow features are subjected to a deep feature extraction stage to obtain deep features, and then the deep features are added to the input as input to the high-quality reconstruction stage; the deep feature extraction stage consists of multiple residual state space groups (Residue State Space Block, RSSB) and the last convolution. Among them, the residual state space block (RSSB) is the core component of MambaIR, specifically including convolutional layers, Mamba modules, channel attention and residual connections; the convolutional layers are used for local feature extraction; the Mamba modules are used to model long-range dependencies; the channel attention is used to enhance the interaction between channels; and the residual connection facilitates gradient propagation. When in use, the network can be trained in advance with the real image data taken.

[0045] The image segmentation module 53 uses the UltraLight VM-Unet network, which has significant effects in the field of medical image segmentation. The UltraLight VM-Unet network has a total of 6 layers of networks, of which the first 3 layers are convolutional networks for extracting shallow features of the image. Each layer includes a standard convolution with a 3x3 convolution kernel and a maximum pooling operation; the last 3 layers are parallelVision Mamba (PVM) layers for extracting deep features of the image. Among them, the PVM layer is improved based on the VisionMamba layer. In addition, the skip connection path uses the channel attention bridge (CAB) module and the spatial attention bridge (SAB) module for multi-stage and multi-scale information fusion; specifically, the CAB module enhances the information interaction between channels through global average pooling, splicing operations, fully connected layers, and Sigmoid activation functions; the SAB module includes operations such as maximum pooling, average pooling, and dilated convolution to capture important spatial features. By combining the advantages of the Mamba model with the efficiency of the UNet architecture, UltraLight VM-UNet significantly reduces the number of model parameters and computational requirements while maintaining high segmentation performance.

[0046] Preferably, the diseased tissue image obtained by image segmentation can be marked by means of fluorescent marking or other means to improve the recognition of the diseased tissue.

[0047] The image fusion module 54 uses a multi-mode image fusion method, mainly including neural network methods based on autoencoders (AE), CNNs, and GANs. Preferably, in the embodiment of the present invention, a network model based on GAN is selected. GAN can effectively model data distribution without supervised information, which is very consistent with the requirements of infrared and visible light image fusion; specifically, the generator is responsible for capturing the potential distribution in the source image samples and fully integrating these distribution characteristics into the fused image. The discriminator is responsible for distinguishing whether the input is a source image or a fused image from the distribution, thereby forcing the fusion result synthesized by the generator to contain as many distribution characteristics of the source image as possible. Furthermore, in an embodiment of the present invention, a FusionGAN network is used; the FusionGAN network includes a generator and a discriminator; the generator network includes 5 convolutional layers, the first and second layers use 5×5 convolution kernels, the third and fourth layers use 3×3 convolution kernels, and the fifth layer uses 1×1 convolution kernels. The convolution step size of each layer is 1, and no padding is used. The input is the spliced ​​infrared and visible light images; the discriminator network includes 5 convolutional layers, the first four layers are 3×3 convolution kernels, and the step size is set to 2. In order to avoid introducing noise, padding is added to the image only in the first layer of the input. Batch normalization is added to the second to fourth layers. The activation function of the first four layers is leaky ReLU, and the last layer is a linear layer for classification.

[0048] The 3D image generation module 55 uses the Make-It-3D method; the Make-It-3D method uses deep learning to generate high-fidelity 3D structures using only a single image; further, the Make-It-3D method includes two stages; the first stage optimizes the neural radiance field (NeRF) by combining the constraints of the frontal view of the reference image and the diffusion prior of the new view; the second stage converts the coarse model into a texture point cloud, and uses the high-quality texture of the reference image and the diffusion prior to further improve the realism. In particular, for NeRF rendering, the multi-scale hash coding of Instant NGP is used to implement the NeRF representation in the coarse optimization stage to achieve less computational cost; at the same time, an occupancy grid is maintained to achieve efficient ray sampling by skipping empty space; in addition, shadow enhancement, such as Lambertian and normal shadows, is used on the rendered image. For point cloud rendering, a 2D U-Net architecture with gated convolution is used; the dimension of the point descriptor is 19, of which the first 3 dimensions are initialized with RGB colors and the remaining dimensions are randomly initialized; in addition, a learnable descriptor is set for the background.

[0049] In particular, all of the above-mentioned deep learning models are pre-trained, and the trained model weights are loaded into the supercomputing AI image processing unit; the supercomputing AI image processing unit is equipped with high-performance image processing software and hardware to ensure high-speed and high-precision processing.

[0050] In particular, the above deep learning model can be optimized and integrated into a whole to simplify the optimization steps; specific usage scenarios require fine-tuning of the above neural network to achieve optimal performance; In addition, the data transmission and storage module 4 and the supercomputing AI image processing unit 5 can be integrated into an image processing unit; Furthermore, the image processing unit can select any suitable hardware or software and hardware combination structure according to the requirements. For example, in an embodiment, the image processing unit can be a field programmable gate array (FPGA) or an embedded system, and can also be equipped with an AI computing platform such as NVIDIA Clara HoloscanSDK to provide powerful data processing capabilities for the above-mentioned endoscope system and obtain clearer images.

[0051] 8K binocular AR glasses 6 use 8K optical waveguide AR glasses, which adopt optical waveguide technology and 8K resolution display to avoid the ghosting and field of view limitation problems that may exist in traditional AR devices, providing users with an excellent visual experience. Optical waveguide technology can make images clearer and more realistic while reducing glare and eye fatigue. 8K resolution display can present more delicate and vivid images, allowing users to experience more realistic augmented reality scenes. Figure 2 As shown, it is a schematic diagram of 8K binocular AR glasses 6 displaying images. 8K binocular AR glasses 6 display the images in the user's field of view in high-definition stereoscopic form to guide diagnosis and treatment.

[0052] Furthermore, the image processing unit and the 8K binocular AR glasses are connected via a data cable.

[0053] In some embodiments of the present invention, the endoscope 2 and the infrared detection module 3 can be integrated into one device, which is equipped with a sensor camera with a resolution of 4K or higher. The camera can capture the mixed image (including visible light image and infrared image) captured by the endoscope 2. Subsequently, the AI ​​image processing unit 5 of the supercomputer uses an artificial intelligence algorithm to identify and classify these images to obtain infrared images and visible light images. The infrared image is then segmented and marked, and fused with the visible light image, and finally displayed in high-definition stereo in binocular AR glasses. This integrated optimization method can improve the efficiency and accuracy of image processing and bring users a better augmented reality experience.

[0054] Furthermore, the image defogging module 52 and the image denoising module 53 can process and enhance the mixed image.

[0055] In summary, the present invention provides an 8K stereoscopic display endoscope. The endoscope uses a superlens as the optical system, obtains high-definition images through a high-resolution detector, and performs non-minimally invasive scanning through an infrared detection module to generate 2D or 3D images, distinguish normal tissue from diseased tissue, and determine the location information of the diseased tissue. The image data is transmitted to the supercomputing AI image processing unit through the data transmission and storage module for intelligent image processing and enhancement. The processed image is displayed through 8K binocular AR glasses to achieve high-definition stereoscopic display for guiding diagnosis and treatment. The system can also collect and store surgical cases at the same time to provide data support and reference. Through this overall optimized design, the imaging quality and surgical effect of the endoscope can be improved, bringing more possibilities to the medical field.

[0056] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An 8K stereoscopic display endoscope, characterized in that: It includes super lens, 4K endoscope, infrared detection auxiliary module, data transmission and storage module, supercomputing AI image processing unit and 8K binocular AR glasses device; The super lens is arranged at the end of the 4K endoscope for image acquisition, the infrared detection auxiliary module is arranged on the 4K endoscope, the 4K endoscope is connected to the supercomputing AI image processing unit through the data transmission and storage module, and the supercomputing AI image processing unit is connected to the 8K binocular AR glasses device; Among them, the superlens is used in the optical imaging system to provide high-definition image information; the 4K endoscope obtains images of internal human tissues through the imaging optical system; the infrared detection auxiliary module is used for non-minimally invasive scanning, and establishes 2D or 3D images, distinguishes normal tissues and diseased tissues inside the human body, and determines the location of diseased tissues; the data transmission and storage module is used to transmit and store image information collected by the 4K endoscope and infrared detection module, and transmit it to the supercomputing AI image processing unit at the same time; the supercomputing AI image processing unit performs image processing and enhancement on the above image information; the 8K binocular AR glasses realize high-definition stereoscopic display of the processed images, realizing immersive data interaction and operation.

2. The 8K stereoscopic display endoscope according to claim 1, characterized in that: The above-mentioned 4K endoscope is based on an optical system of a superlens, which is one or more superlenses; wherein the superlens is a phase-type superlens, including a substrate and a plurality of nanostructure units arranged on the substrate.

3. The 8K stereoscopic display endoscope according to claim 1, characterized in that: The above-mentioned 4K endoscope includes an illumination light source, a visible light camera, a single-mode optical fiber and a hard tube for the endoscope body; the illumination light source, the visible light camera and the first single-mode optical fiber are all installed on the hard tube for the endoscope body; the illumination light source is connected to the first single-mode optical fiber, wherein the illumination light source is used to provide the visible light illumination conditions required for image acquisition; the visible light camera cooperates with the 4K endoscope to acquire visible light images; the first single-mode optical fiber is used to transmit visible light to the target detection area; the hard tube for the endoscope body is used to fix the optical system and the single-mode optical fiber assembly; The visible light camera uses a 4K CMOS or CCD sensor, or is equipped with an 8K or higher resolution sensor to generate high-definition images.

4. The 8K stereoscopic display endoscope according to claim 1, characterized in that: The above-mentioned infrared detection auxiliary module includes an infrared light source, an infrared camera, a second single-mode optical fiber and a dichroic mirror; wherein the infrared light source is used to provide infrared lighting conditions for the target detection area; the infrared camera cooperates with the endoscope to collect infrared images; the second single-mode optical fiber is used to transmit the infrared light source to the target detection area; the dichroic mirror is used to divide the optical signal collected by the endoscope into two, one part of which is a visible light signal recognizable by the visible light camera, and the other part is an infrared light signal recognizable by the infrared camera; the optical signal is converted into image information by the sensor; the infrared camera must match the infrared light source; The first light emitted by the surface light source, the second light emitted by the infrared light source and the optical signal are transmitted independently.

5. The 8K stereoscopic display endoscope according to claim 1, characterized in that: The data transmission and storage module of the above-mentioned endoscope system includes a transmission data line and a storage module; wherein the transmission data line is used to transmit image information for subsequent processing; and the storage module is used to store and collect surgical cases.

6. The 8K stereoscopic display endoscope according to claim 1, characterized in that: The artificial intelligence AI image processing unit includes an image defogging module, an image denoising module, an image segmentation module, an image fusion module and a 3D image generation module; wherein the image defogging module is used to perform defogging processing on the visible light image; the image denoising module is used to reduce the noise of the image processed by the image defogging module; the image segmentation module is used to segment the diseased tissue location area in the infrared image; the image fusion module is used to fuse the segmented diseased tissue location area with the visible light image to generate an image containing the diseased tissue location information; the 3D image generation module is used to generate a three-dimensional structure diagram from the fused image.

7. The 8K stereoscopic display endoscope according to claim 1, characterized in that: The image defogging module, image denoising module, image segmentation module, image fusion module and 3D image generation module use convolutional neural network CNN, generative adversarial network GAN, Transformer or Mamba neural network model; The image defogging module uses the UVMNet defogging network; the UVMNet defogging network combines the convolutional network with the Mamba model; The UVMNet includes a 4-layer network using a U-shaped structure; each layer of the network includes two consecutive convolution blocks and a UVM-net module; after the convolution, the UVM-net module performs remote modeling on the SSMs module; The image denoising module uses the MambaIR network; the MambaIR network includes a shallow feature extraction module, a deep feature extraction module, an upsampling module and a reconstruction module; the residual state space block RSSB is the core of the MambaIR network, specifically including a convolutional layer, a Mamba module, a channel attention and a residual connection; the convolutional layer is used for local feature extraction; the Mamba module is used to model long-range dependencies; the channel attention is used to enhance the interaction between channels; the residual connection facilitates gradient propagation; The image segmentation module uses the UltraLight VM-Unet network; the UltraLight VM-Unet network has significant effects in the field of medical image segmentation; the UltraLight VM-Unet network has a total of 6 layers of networks, among which the first 3 layers are convolutional networks for extracting shallow features of the image, and each layer includes a standard convolution with a 3x3 convolution kernel and a maximum pooling operation; the last 3 layers are PVM layers for extracting deep features of the image; the PVM layer is improved based on the visual mamba layer; in addition, the jump connection path uses the channel attention bridge CAB module and the spatial attention bridge SAB module for multi-stage and multi-scale information fusion; specifically, the SAB module includes maximum pooling, average pooling and extended convolution with shared weights; the CAB module includes global average pooling, splicing operation, fully connected layer and sigmoid activation function.

8. The 8K stereoscopic display endoscope according to claim 1, characterized in that: The diseased tissue image obtained by image segmentation is marked with fluorescence to improve the recognition of the diseased tissue; The image fusion module uses a multi-mode image fusion method, including neural networks based on autoencoder AE, CNN and GAN.

9. The 8K stereoscopic display endoscope according to claim 1, characterized in that: A GAN-based network model is selected, and a FusionGAN network is used; the FusionGAN network includes a generator and a discriminator; the generator network includes 5 convolutional layers, the first and second layers use 5×5 convolution kernels, the third and fourth layers use 3×3 convolution kernels, and the fifth layer uses 1×1 convolution kernels. The convolution step size of each layer is 1, and no padding is used. The input is the spliced ​​infrared and visible light images; the discriminator network includes 5 convolutional layers, the first four layers all use 3×3 convolution kernels, and the step size is set to 2; to avoid introducing noise, padding is added to the image in the first layer of the input; batch normalization is added to the second to fourth layers; the activation function of the first four layers is leakyReLU, and the last layer is a linear layer for classification; The 3D image generation module uses the Make-It-3D method; the Make-It-3D method uses deep learning to generate a high-fidelity 3D structure using only a single image; the Make-It-3D method includes two stages; the first stage optimizes the neural radiation field NeRF by combining the constraints of the front view of the reference image and the diffusion prior of the new view; the second stage converts the rough model into a texture point cloud, and utilizes the high-quality texture of the reference image combined with the diffusion prior to enhance the realism.

10. The 8K stereoscopic display endoscope according to claim 1, characterized in that: The image defogging module, image denoising module, image segmentation module, image fusion module and 3D image generation module are pre-trained, and the trained model weights are loaded into the supercomputing AI image processing unit; the supercomputing AI image processing unit is equipped with high-performance image processing software and hardware; The 4K endoscope and the infrared detection module are integrated into an integrated structure; the data processing and transmission module and the artificial intelligence AI image processing unit are integrated into an integrated structure - the image processing unit; The image processing unit is a field programmable gate array; the 8K binocular AR glasses device uses 8K resolution and optical waveguide technology.

Images