Combined catadioptric optical system with 360 degree forward and panoramic vision for medical imaging

Abstract

The disclosed idea is based on the application of a catadioptric imaging system that includes a conventional perspective imaging system and a convex mirror with a specific curvature that can provide a 360-degree side view. Also, in this exposed idea, a hyperbolic mirror with a hole center is used to achieve both forward vision and 360 degree lateral vision. In this invention, the camera set (sensor and lenses) and hyperbolic mirror with a hole in the center are placed in a transparent tube made of glass or polymer. Using this system, the doctor can simultaneously see the front view and a 360-degree view of the intestinal wall and easily identify precancerous polyps that cannot be identified with the front view.

Description

Combined Catadioptric Optical System with 360 Degree Forward and Panoramic Vision for Medical Imaging

[0001] Using a wide view device in endoscopy or other similar procedures can solve many of the existing problems and increase the accuracy of diagnosis. During colonoscopy, some polyps may be hidden from the front or cannot be distinguished from the surrounding mucosa. Undiagnosed polyps can lead to colon cancer, therefore, imaging systems need to be improved.

[0002] The patent presented here is based on the use of a catadioptric imaging system that includes a conventional field of view camera system and a convex mirror with a special curvature that can provide a 360degree side view. Also in this proposed system, a pinhole mirror is used to achieve 360 degree forward and side vision.

[0003] In this invention, the camera unit (sensor and lens) and the frame mirror with a hole in the middle are placed inside a transparent tube of glass or polymer. Using this system, the doctor can see simultaneously from the front with a 360-degree view of the intestinal wall and easily identify precancerous polyps that could be dangerous if untreated.

[0004] G02B 17 / 00 – G02B 23 / 24

[0005] US20230181014

[0006] ENDOSCOPY DEVICE AND ENDOSCOPY SYSTEM WITH ANNULAR VIEW

[0007] An endoscopic device comprising a housing, first imaging means in the housing for detecting light emanating from objects from a predetermined first surrounding region annularly surrounding the endoscopic device and for producing a first image of the first surrounding region, a first image sensor for detecting the first image and for producing a first image signal, a first image sensor for sensing the first image and for generating a first image signal representing the first image, a second imaging means in the housing for sensing light emanating from objects in a predetermined second surrounding region and for generating a second image of the second surrounding region, and a second image sensor for sensing the second image and for generating a second image signal representing the second image. The first imaging device comprises a catadioptric imaging system with a first reflective surface, a second reflective surface, and a light refracting interface.

[0008] This mentioned invention relates to a an endoscopy system and shares aspects of our design in function and purpose. However, the comprising elements and process are completely different, for instance this one emphasizes a two-phased image producing method while ours uses a combined catadioptric system.

[0009] United States Patent 10536617

[0010] Dual-view probe for illumination and imaging, and use thereof

[0011] One embodiment of the invention is directed to an imaging probe which comprises a first element supplying electromagnetic radiation in a forward path for illuminating a forward field of view of space in front of the probe, a second element supplying electromagnetic radiation in a sideways and / or rearward path for illuminating a sideways and / or back field of view of space alongside the probe and an image sensor. The probe further includes an imaging device in imaging paths imaging the forward and sideways and / or back fields of view onto the image sensor. Preferably the forward path, the sideways and / or rearward path and the imaging paths are unobstructed by any component of the probe. Preferably the forward and sideways and / or back fields of view are registered in a fixed spatial relationship relative to one another on the image sensor. This probe can also be used with existing endoscopes or laparoscopes so that it will be easily merged into current instruments of different manufacturers, making it easier and cheaper to use this product without the expense of a totally new endoscopy or laparoscope platform. This probe can also be used to image objects in inanimate environments as well, such as in containers, buildings, rooms, engines and pipes.

[0012] This mentioned patent resembles our claimed design in the overall purpose of laparoscopy imaging and other similar medical procedures. But the method and comprising elements of the two systems differ, for instance, this one uses electromagnetic radiation to provide the wide view while ours relies on image processing algorithms and features convex mirrors.

[0013] United States Patent Application 20040254424

[0014] Integrated panoramic and forward view endoscope

[0015] The objective of the present invention is to provide a single endoscope that provides a field of view substantially greater than a hemisphere comprising a forward field of view and a panoramic field of view that are integrated on a single image plane. The invention is described with respect to a rigid endoscope, but the technology can be implemented on a flexible endoscope as well. The advantage of such an endoscope is that it would provide substantially more information to the physician than any single existing endoscope, and it can be used in place of multiple endoscopes with varying directions of view that are swapped throughout a procedure to provide different views. The invention can also be used in non-medical applications for inspection in closed or generally inaccessible spaces such as for example the interior of jet engines.

[0016] This mentioned invention is similar to ours in offering an improved panoramic as well as forward view feature in endoscopes. However, the methods and comprising elements are different, for instance, unlike this patent, ours includes a water spray nozzle for cleaning purposes and the resulting images can be displayed on multiple screen in real-time.

[0017] US8496580B2

[0018] Omnidirectional and forward-looking imaging device

[0019] Apparatus including an optical system (20) for use in an endoscope is provided. The optical system has distal and proximal ends, and includes an image sensor (32), positioned at the proximal end of the optical system. The optical system also includes an optical member (34) having distal (36) and proximal ends, and shaped so as to define: (a) a lateral surface, at least a distal portion of which is curved, configured to provide omnidirectional lateral viewing, (b) a distal indentation (44) in the distal end of the optical member, and (c) a proximal indentation (48) in the proximal end of the optical member. The optical system also includes a convex mirror (40), coupled to the distal end of the optical member, and shaped so as define an opening through which the distal indentation passes. A distal lens (52) is positioned distal to the mirror. The optical member, the mirror, and the distal lens have respective rotational shapes about a common rotation axis.

[0020] The above mentioned device is similar to our claimed one, since they both offer forward and panoramic 360 degree view imaging systems. They also have differences in their design and comprising elements, for instance, ours features a cleaning water spray and a biopsy channel whereas this one makes no mention of these parts.

[0021] EP3536217

[0022] CATADIOPTRIC MEDICAL IMAGING SYSTEM FOR OBSERVING THE INSIDE WALL OF A SURGICAL CAVITY

[0023] The invention relates to a catadioptric medical imaging system (1), in particular a surgical microscope (2). During surgery, it may be necessary to gain more information about a surgical cavity (6), in particular the type of tissue (29) at the inside walls (4) of the surgical cavity (6). To solve this problem, the catadioptric medical imaging system (1) according to the invention comprises a camera device (8) and a convex catoptric mirror (20) adapted to be inserted into the surgical cavity (6). The catoptric mirror (20) is mounted on an arm (22) and spaced apart from the camera device (8).

[0024] This mentioned invention also involves a catadioptric optical system that can be utilized in medical procedures. While the two patents share certain elements and processing methods, their overall purposes and scopes differ. This one is designed to be used primarily in surgery whereas ours can also be used in endoscopic or other similar procedures. Moreover, ours features a water spray element for cleansing purposes.

[0025] The invention presented here involves the utilization of a catadioptric imaging system, a combination of a traditional perspective imaging system and a convex mirror with a distinct curvature designed to offer a complete 360-degree side view. Furthermore, this patent design incorporates a hyperbolic mirror featuring a central aperture which enables the attainment of both frontal and peripheral vision.

[0026] To implement this idea, a camera set comprising sensors and lenses is integrated alongside the hyperbolic mirror within a transparent tube crafted from either glass or polymer. This setup facilitates a comprehensive visualization of the intestinal wall, allowing medical professionals to simultaneously observe the frontal perspective as well as a panoramic 360-degree view. This system allows doctors to efficiently identify precancerous polyps that could go unnoticed in instances where only a front view is available.

[0027] This invention marks a significant advancement in medical imaging, offering clarity and insight into the intricate workings of the human body. By enabling healthcare providers to conduct thorough examinations and detect anomalies with enhanced precision, this system enhances diagnostic accuracy and improves patient outcomes.

[0028] Colorectal cancer (CRC) is one of the most common gastrointestinal cancers in the world. According to global cancer statistics in 2018, colorectal cancer is the third most common cancer in the world, accounting for 11.2% of all cancers. In 2018, more than 1.1 million new cases and about 161,663 deaths from colorectal cancer were reported. This cancer is the fourth cause of death among cancers. The World Health Organization has predicted a 77% increase in the diagnosis of new cases and an 11% increase in deaths from colon cancer by 2030.

[0029] The economic burden of colorectal cancer across Europe in 2015 was €13.1 billion. The non-health cost including loss of productivity due to disability, early death and opportunity cost for caregivers accounts for about 61.6 of the total economic burden. Colonoscopy is the preferred method to detect and remove precancerous polyps. However, several studies have shown that a significant percentage of adenomas, up to 31%, can be conventionally missed.

[0030] In the endoscopy test, the doctor can view the video images received from the camera installed at the tip of the endoscope on the monitor. Although the front view of the prop and surgical instruments is suitable for placing the camera. But they are not always suitable for finding the affected area. Especially for the examination of the large intestine, considering its shape, it is necessary for the operator of the endoscopy and colonoscopy device to have special skills. Since the camera can only have a forward view, a lot of movements are needed to fully see the intestine. Therefore, it is difficult to see the entire bowel without losing a region of the bowel image. For example, in the bending area between the transverse and descending colon, as well as the inner side of the bend of the intestine, it is usually a blind spot and is ignored in the tests. To see such areas, the tip of the prop should be bent. This problem increases the difficulty and problems for the doctor and the patient.

[0031] As mentioned above, although the front view of the prop is suitable for placing the camera, they are not always suitable for finding the damaged area, instead, the 360-degree view can see the side and rear directions more appropriately and also It can observe the entire intestinal wall at the same time. Using a wide field of vision in endoscopy can solve many of the above problems and increase the accuracy of disease diagnosis. In forward colonoscopy, some polyps are hidden from forward view or cannot be distinguished from the surrounding mucosa. Undetected polyps can eventually lead to colon cancer. Therefore, there is a need for improved imaging systems.Solution of Problem

[0032] In this patent, the use of catadioptric imaging systems, including a special type of mirror and camera, which are located next to each other in a specific structure, is discussed to improve the performance of medical optical imaging devices. Using this system, the field of view (FOV) can be increased to about 350 x 90 degrees. In this structure, the camera and mirror are placed inside a cylindrical glass sheath at the end of the endoscopic and laparoscopic device. This new structure has a diameter of about 13 mm and a length of 20 mm.

[0033] In, you can see a simple view of the claimed device. In this invention, a hyperbolic convex mirror with a hole in the center is used to have both front and side vision. The optical sensor with 4K resolution is installed along the axis of the hyperbolic mirror and you look directly at it. A part of the light rays that enter the front view lens from the front is compressed and passes through the hole in the center of the mirror and hits the CCD of the camera. After hitting the surface of the mirror, the lateral light rays are directed towards the camera sensor, thus creating a 360degree and lateral image of the environment surrounding the endoscope prop and specifically the colonoscope. This imaging geometry can be seen in.

[0034] In this system, by piercing the central point of the hyperbolic mirror, which is the blind spot (the camera observes itself), we will achieve both front and side view. Images received with a hyperbolic mirror with a small curvature can be transformed into images with a natural perspective or cylindrical images, etc. In addition, if the curvature of the mirror surface is small, the astigmatism will be small. When the exact arrangement of the camera and the mirror is created and the camera's pin-hold matches the focal point of the mirror exactly, a wide-view vision sensor with only one effective point of view will be obtained. This type of mirror is the best choice for optical systems using normal cameras.

[0035] The material of the mirror can be metal or glass, however, in the proposed design, a metal mirror made of stainless steel called L316 is used. 316 steel has 16% chromium, 10% nickel and 2% molybdenum, which makes it completely resistant to rust and corrosion. The diameter of the recommended mirror is 10 mm and it can be made by CNC lathe and different polishing steps and layering of aluminum oxide metal.shows a view of a hyperbolic mirror with a small curvature and the center hole.

[0036] A 3D view of the claimed invention can be seen in. This is the front view of the laparoscopy prop. There is a protective glass in front of the device to protect the front vision lenses. There are 4 water spray nozzles around this lens, which are used to clean the protective glass of the lens. A small glass lens is also used to transmit light to the forward sight, the light is transmitted through the optical fiber to this aperture and illuminates the front view of the prop.

[0037] As displayed in, there are 2 optical diffusers around the transparent glass cylinder. These diffusers are placed in a ring around the body of the laparoscopy prop. Under these diffusers, there is a high efficiency LED ring, whose light, after passing through the diffuser, illuminates the intestinal wall in a completely homogeneous way and provides enough light to acheive the side view.

[0038] shows a side view of the designed system. In this picture, you can see how the hyperbolic mirror and the camera are placed. The transparent protective cover made of glass or polymer can be seen. Around this transparent cylindrical cover, 8 water sprays are installed on each side, so that it can be used to clean the cylindrical glass if needed. These nozzles spray water with high pressure on the surface of the coating and cause its surface to be cleaned.

[0039] In this view of the designed device, you can see a biopsy channel in the initial part of the prop. The diameter of this channel is about 3.3 mm, which is used for vacuum and movement of sampling tools. The exit angle of this channel is determined in such a way that the end of the sampling tool is placed in the middle of the lateral image and the doctor can place the sampling tool in front of the lesion according to the panoramic and 360-degree image that the monitor, observe and rotate the laparoscopy prop to perform sampling or treatment operations.

[0040] In the design, a small pressurized water channel is placed next to the biopsy channel. The doctor uses this pressurized water channel to clean the sampling tool or to clean the area around the lesion or to clean the bleeding caused by the sampling.

[0041] The images received from the CCD camera simultaneously include side and front view. They are designed in such a way that the center of the circular image corresponds to the front view and its border to the sides is the image related to the side view. To restore the image normally, the following steps are performed. By placing a mesh pattern called the calibration pattern around the mirror and taking several photos of the laparoscope camera image and by applying a mathematical algorithm to the corner points extracted from the mesh pattern by image processing algorithms, all the system parameters can be obtained.

[0042] The images received from the camera are circular and have a non-linear structure. It will be difficult for the doctor to understand and analyze these types of images. Therefore, by using image processing algorithms and geometric relationships governing the structure of the mirror and camera, the received images can be converted into images with a natural landscape or panoramic view, in which case It will be possible for the doctor to see an image with a large angular space.Advantage Effects of the Invention

[0043] • Combined 360 degree and forward image processing in real time

[0044] • Wide-range panoramic view of the colon with no blind spots

[0045] • Possible to use the biopsy channel according to the structure of the imaging system

[0046] • Fixing the distortion of the images and calibrating the imaging system

[0047] • Simultaneous display of the forward / side view and panoramic image on one or several monitors

[0048] • Measuring the size and shape of the lesion according to the possibility of removing the distortion

[0049] • Faster and more accurate diagnosis and minimizing the diagnostic test time

[0050] • Reducing pain and discomfort of the patient with optimized test method and minimal movements of the endoscope prop

[0051] • Cost-effective solution

[0052] Shows a general view of the combined catadioptric device.

[0053] Depicts the hyperbolic mirror used in the device.

[0054] Displays a 3D schematic view of the device.

[0055] Shows a side view of the laparoscopic device.

[0056] Captures a comprehensive side-cut view of the device.

[0057] Captures another side-cut view of the device.

[0058] Shows the details of the placement of certain elements.

[0059] Shows a general view of the combined catadioptric device components and the connection between its parts which include a 4k resolution perspective camera, a hyperbolic mirror at a specific angle and a center hole to achieve 360 degree, side / front view.

[0060] Depicts the hyperbolic mirror with slight curvature and a central hole

[0061] Displays a 3D schematic view of the front and outside of the device which includes these elements as numbered in the figures:

[0062] 4. Water spray nozzle

[0063] 5. Light diffusers

[0064] 7. Objective lens with lens cover

[0065] 8. Plastic distal end cap

[0066] 9. Light guide lens

[0067] Shows a side view of the laparoscopic device from the outside and inside wherein these parts are depicted:

[0068] 1. Round clear plastic or glass tube

[0069] 2. Biopsy or suction channel

[0070] 3. Forward water jet nozzle

[0071] 4. Water spray nozzles

[0072] 5. Light diffusers

[0073] 6. Center hole hyperbolic mirror

[0074] Captures a side-cut view of the device wherein the details of the comprising parts and their placements can be seen:

[0075] 5. Light diffusers

[0076] 6. Center hole hyperbolic mirror

[0077] 10. Lens cover

[0078] 11. Lens mount

[0079] 12. Glass optical lenses

[0080] 13. Lens mount

[0081] 14. LED

[0082] 15. Rount clear plastic / glass tube (same as no. 1 in previous figures)

[0083] 16. Water spray channel

[0084] 17. Light guide fibers

[0085] 18. CCD signal wires

[0086] 19. 4k CCD

[0087] 20. Biopsy or suction channel (same as no. 2 in previous figures)

[0088] Captures another side-cut view of the claimed device from a different angle.

[0089] Shows the details of the placement of certain elements such as the water spray channel and nozzles, light fibers, lens and sensors and LED tape positions.Examples

[0090] The invention is used in fibrescopy and medical procedures that need advanced image processing. Considering that the imaging system has an imaging equation that can be extracted with the help of the calibration of its parameters, the images received from the 4k camera can be converted into images with a natural perspective. By using image processing algorithms and digital signal processing (DSP) hardware, the received images can be real-time de-distorted and turned into panoramic images. During a standard colonoscopy visit, for instance, the doctor can simultaneously view frontal and 360-degree side images on two monitors, thus increasing both the accuracy and efficiency of diagnosing lesions.

[0091] The disclosed design is specifically intended for medical fibrescopes and specifically for colonoscopy procedures. However, since the principle of the patent idea is a new imaging system, it can be used in other industries and other applications including industrial endoscopy, such as those imaging endoscopes used for wells and tanks mapping, or any other industrial equipment that requires both panoramic and forward imaging systems.

Claims

A combined catadioptric optical system and device is designed for real-time 360 degree, forward and panoramic medical imaging to be used in endoscopic procedures such as colonoscopy and laparoscopy.According to claim 1, the designed system comprises a camera, light diffusers, light fibers, optical lenses, water spray and biopsy channel, center hole hyperbolic mirror, transparent tube, LED and CCD signal wires.According to claim 1, in the designed structure, the camera and mirror are placed inside a cylindrical glass sheath at the end of the endoscopic and laparoscopic device.According to claim 1, a hyperbolic convex mirror with a hole in the center is used to have both front and side vision.According to claim 1, the optical sensor with 4K resolution is installed along the axis of the hyperbolic mirror wherein a part of the light rays that enter the front view lens is compressed and passes through the hole in the center of the mirror and hits the CCD of the camera.According to claim 1, after hitting the surface of the mirror, the lateral light rays are directed towards the camera sensor, thus creating a 360-degree and lateral image of the environment surrounding the endoscope prop and specifically the colonoscope.According to claim 1, this optical system avoids having blind spots by placing the mirror and the camera at a specific angle so that the camera's pin-hold matches the focal point of the mirror exactly and natural or cylindrical perspective image is obtained.According to claim 1, the mirror has a small curvature and a center hole and can be made of metal or glass, preferably stainless steel that is resistant to rust and corrosion.According to claim 1, the optical lenses in the front feature protective glass covers and water spray nozzle that clean them or the surrounding area.According to claim 1, the second optical lense towards the end of the device is also used to transmit light to the forward sight, the light is transmitted through the optical fiber to this aperture and illuminates the front view of the prop.According to claim 1, there are 2 optical diffusers around the transparent glass cylinder, placed in a ring around the body of the laparoscopy prop. Under these diffusers, there is a high efficiency LED ring, whose light, after passing through the diffuser, illuminates the intestinal wall in a completely homogeneous way and provides enough light to acheive the side view.According to claim 1, a 3.3 mm biopsy channel is placed in the initial part of the prop with an exit angle which allows rotation and movement of vacuum or sampling tools.According to claim 1, the images from the CCD camera are processed by placing a mesh calibration pattern on the mirror and applying a mathematical processing algorithm to the extracted corner points in order to obtain panoramic and front / side views.According to claim 1, the obtained front / side and panoramic views can be displayed on multiple screens in real-time to aid in accuracy and efficiency of the medical procedure.

Images