Novel flexible polarization endoscopic imaging device

Abstract

The invention discloses a novel flexible polarization endoscopic imaging device which is used for tissue polarization imaging and is characterized in that a light source generates illumination light, and the illumination light is emitted through an illumination channel and illuminates an imaging target; the illumination channel is composed of an optical fiber bundle and a polarization state generator. The imaging channel is composed of an objective lens, a polarization state analyzer and a CMOS sensor. The system is integrated with multi-view imaging channels, calibration of internal and external parameters of poses is completed through the multiple imaging channels, corresponding image field depths are obtained through a multi-view stereoscopic vision algorithm, and pixel-by-pixel alignment of view fields of different imaging channels is completed through depth information. The target return light passes through the imaging channel to complete polarization imaging; the illumination channel and the imaging channel are jointly fixed in an outer sheath, and a sapphire protection window is fixed at the far end of the outer sheath to encapsulate the optical structure in the outer sheath. The optical structure is compact in design, full Stokes vector calculation can be completed through one-time shooting, and precise mechanical parts or polarization modulators are not needed.

Description

Technical Field

[0001] The present invention belongs to the technical field of endoscopic imaging, and particularly relates to a novel flexible polarization endoscopy imaging device. Background Art

[0002] Polarization imaging has attracted extensive attention in the field of tissue imaging because it can be used to reveal the morphology, structure, and compositional information of tissues. Tissue polarization imaging is achieved by measuring the Stokes vector (complete / partial) of the light radiated from tissues. Stokes polarization imaging generally combines linear polarization or circular polarization light sources to reveal the tissue structure information helpful for diagnosis and is also applied in the identification of cancerous tissues. With the continuous development of polarization imaging technologies and devices, it has promoted the application transformation of polarization imaging technologies in surgery and tissue diagnosis.

[0003] Existing polarization endoscope systems are based on rigid endoscopes. To achieve Stokes polarization imaging, it mainly relies on a mechanical device to rotate a wave plate or a linear polarizer. There is no flexible snapshot polarization endoscopy imaging device. In addition, previous polarization endoscopes and polarimeters with mechanical moving parts all have disadvantages such as relatively slow speed, vibration, and short lifespan. To reconstruct a Stokes image, at least four images acquired with four pairs of phase delays generated in chronological order are required, and it usually takes dozens of milliseconds for a mechanical rotation to switch the delay once, which limits the acquisition speed of such devices. Nowadays, a pixelated micro linear polarizer array can be directly integrated onto an image sensor to measure the polarization state in a single snapshot. However, the vast majority of such devices are only sensitive to linear polarization. Currently, there are few custom pixelated micro elliptical polarizer arrays, and it is difficult to achieve a complete Stokes polarization imaging measurement in one shot.

[0004] To address the above problems, the present invention provides a novel flexible polarization endoscopy imaging device, which can complete the calculation of the full Stokes vector in one shot, without the need for a mechanical component to precisely rotate the linear polarizer device, with fast and accurate polarization measurement and a simple implementation principle. Summary of the Invention

[0005] The purpose of the present invention is to provide a novel flexible polarization endoscopy imaging device for real-time polarization imaging of tissues in response to the requirements of high-quality intraoperative imaging, tumor detection, etc. The device is miniaturized and has a simple structure, with accurate polarization measurement imaging and high speed.

[0006] The purpose of the present invention is achieved through the following technical solutions: A novel flexible polarization endoscopy imaging device includes a light source, an illumination channel, a plurality of imaging channels, and an image control and acquisition system; The illumination channel consists of an optical fiber bundle and a polarization state generator, and is used to endow the illumination light with a polarization state; the imaging channel consists of an objective lens, a polarization state analyzer, and a CMOS sensor, and is used to complete the calibration of the internal and external parameters of the camera pose, obtain the corresponding image field depth through a multi-view stereo vision algorithm, and complete the pixel-by-pixel alignment of the fields of view of different imaging channels by using the image field depth information. The light source cooperates with the illumination channel to generate illumination light with a polarization state. The illumination light is emitted through the illumination channel and illuminates the imaging target. The target return light completes polarization imaging through the multi-view imaging channel, and the imaging data is transmitted into the image control and acquisition system to complete the calculation of the full Stokes vector.

[0007] Further, the illumination light with a polarization state is linearly polarized light or circularly polarized light.

[0008] Further, the illumination channel and the imaging channel are jointly fixed in an outer sheath, and a sapphire protective window is used to fix the distal end of the outer sheath to encapsulate the optical structure therein.

[0009] Further, when reconstructing the linear Stokes polarization measurement image, the number of imaging channels is not less than three.

[0010] Further, when reconstructing the full Stokes polarization measurement image, the number of imaging channels is not less than four.

[0011] Further, the image control and acquisition system is used to continuously capture and transmit the original polarization image data, and perform real-time processing and display on the captured data.

[0012] Further, the imaging device includes multiple imaging channels, and each imaging channel has a different polarization state analyzer; in front of each imaging channel, a linear polarizer with the same polarization direction and arbitrarily selected angle is jointly configured, and wave plates with different fast axis directions and arbitrarily combined angles are respectively configured to realize the reconstruction of the Stokes polarization image.

[0013] Further, the imaging channel is used to complete the calibration of the internal and external parameters of the camera pose, obtain the corresponding image field depth through a multi-view stereo vision algorithm, and complete the pixel-by-pixel alignment of the fields of view of different imaging channels by using the image field depth information; specifically: set one imaging channel as the reference channel, capture the reference polarization image, and the remaining imaging channels respectively capture polarization images and perform feature matching with the reference polarization image to obtain the corresponding image field depth. The depth map (X, Y, Z) is converted into a three-dimensional point cloud (X, Y, Z, R, G, B) through the internal parameters of each channel camera, and then the three-dimensional point cloud is converted to the perspective of the reference channel through the rotation and translation matrices of the external parameters of the camera, and then the three-dimensional point cloud is converted into a polarization image (X, Y, RGB) through the internal parameters of the reference channel camera to complete the pixel-by-pixel alignment of the fields of view of each imaging channel and the reference channel.

[0014] Further, the Zhang-Zhengyou calibration method is used for calibrating the internal and external parameters of the camera pose.

[0015] Further, the target return light undergoes polarization imaging through the multi-view imaging channels, and then the fields of view of each channel are aligned pixel by pixel to reconstruct a Stokes image in one shot.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention discloses a novel flexible polarization endoscope imaging device. The distal end of the endoscope integrates a polarization state generator, a polarization state analyzer, illumination optical fibers, and an image control and acquisition system. This compact optical structure design can meet the miniaturization requirements of the system, realize polarization imaging of the device in the endoscope environment, and is particularly suitable for the detection of shallow tissues (such as the skin surface and colorectal mucosa), having important clinical application prospects, especially in the early detection of early cancerous tissues and tumors. And there is no mechanical movement in the whole system, so the system only needs to be calibrated once and can be used for a long time; combined with the multi-view imaging channels, snapshot polarization imaging can be realized, and a complete Stokes vector can be calculated in one shot, and the polarization measurement is fast and accurate. Description of the Drawings

[0017] Figure 1 is a schematic structural diagram of the novel flexible polarization endoscope imaging device in the embodiment of the present invention; Figure 2 is a cross-sectional view of the distal end of the flexible polarization endoscope in the embodiment of the present invention.

[0018] In the figure, 1. Illumination channel; 2. -45° wave plate; 3. 0° wave plate; 4. Linear polarizer; 5. 30° wave plate; 6. 60° wave plate; 7. Imaging channel; A. Object end; B. Image end. Detailed Embodiments

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of the present invention. In addition, in the following description, the descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0020] This embodiment relates to a novel flexible polarization endoscope imaging device, as Figure 1As shown in the figure, it includes: a light source, an illumination channel, an imaging channel, and an image control and acquisition system; the light source cooperates with a Polarization State Generator (PSG) to generate system illumination light with linear polarization or circular polarization, which is emitted through a lens via the illumination channel and illuminates an imaging target (such as biological tissue). The returned light completes polarization imaging through a multi-view imaging channel, and the imaging data is transmitted to the image control and acquisition system to complete the calculation of the full Stokes vector. The illumination channel and multiple imaging channels of the imaging device are fixed in a metal outer sheath, and the internal optical system is encapsulated by a sapphire protective window at the distal end.

[0021] The illumination channel consists of an optical fiber bundle and a polarization state generator. The polarization state generator is placed at the end of the optical fiber bundle (i.e., the end of the endoscopic objective lens) to make the illumination light of the endoscopic imaging device have the required polarization state.

[0022] The imaging channel consists of an objective lens, a Phase Shift Analysis (PSA), and a CMOS sensor. The device has multiple imaging channels; among them, three or more imaging channels are required to reconstruct a linear Stokes polarization measurement image; four or more imaging channels are required to reconstruct a full Stokes polarization measurement image.

[0023] The image control and acquisition system is used to continuously capture and transmit the original polarization image, and perform real-time processing and display on the captured data.

[0024] In this embodiment, the imaging device includes multiple imaging channels, and each imaging channel has a different polarization state analyzer. A linear polarizer (optional at any angle) with the same polarization direction is respectively configured in front of each imaging channel, and wave plates with different fast axis directions (combinable at any angle) are respectively configured to realize the reconstruction of the Stokes polarization image.

[0025] Specifically, as Figure 2 shown, taking four imaging channels as an example, it is specifically described that a linear polarizer 4 (such as 0°) with the same polarization direction is respectively configured in front of each imaging channel 7, and a -45° wave plate 2, a 0° wave plate 3, a 30° wave plate 5, and a 60° wave plate 6 are configured to realize the reconstruction of the Stokes polarization image. Figure 2 In the figure, the object end A and the image end B are distributed on both sides of the imaging channel 7. The illumination channel 1 provides appropriate illumination light for the imaging target and creates basic conditions for the subsequent imaging process.

[0026] For the specific principle description, the propagation of polarized light in the imaging channel can be characterized by the following linear equation, where the polarization state (State of Polarization, SOP) of the incident light is represented by the input Stokes vector S inTo characterize, the polarization characteristics of the polarization state analyzer are described by the Mueller matrix M, and the SOP of the outgoing light is represented by the output Stokes vector S out (i.e., the image sensor).

[0027]

[0028] where The Mueller matrices characterizing different wave plates (-45°, 0°, 30° and 60°). The Mueller matrix characterizing the linear polarizer. In this design, the first row of M corresponding to different PSA states (here different angle wave plates) is taken to construct the PSA instrument matrix. This method is also applicable to PSA with other structures, as long as the matrix of the PSA instrument is full rank.

[0029] Therefore, the complete Stokes vector S can be obtained according to the following equation:

[0030]

[0031] The superscripts -1 and T represent the pseudo-inverse matrix and the transpose matrix respectively, and I n is a column vector, which is the intensity reading of each pixel of the image sensor. Where PSA refers to the PSA instrument matrix.

[0032] In this embodiment, the multiple imaging channels complete the calibration of the internal and external parameters of the camera pose, and obtain the corresponding image field depth through the multi-view stereo vision algorithm. Using this depth information, the pixel-by-pixel alignment of the fields of view of different imaging channels is completed. Specifically: One of the imaging channels is set as the reference channel, and the reference polarization image is captured. The remaining imaging channels capture polarization images respectively and perform feature matching with the reference polarization image to obtain the corresponding image field depth. The depth map (X, Y, Z) is converted into a three-dimensional point cloud (X, Y, Z, R, G, B) through the internal parameters of each channel camera, and then the three-dimensional point cloud is transformed to the perspective of the reference channel through the rotation and translation matrices of the external parameters of the camera. Then, the three-dimensional point cloud is converted into a polarization image (X, Y, RGB) through the internal parameters of the reference channel camera, completing the pixel-by-pixel alignment of the fields of view of each imaging channel and the reference channel.

[0033] In this embodiment, the Zhang-Zhengyou calibration method is used for the calibration of the internal and external parameters of the camera pose. The Zhang-Zhengyou calibration method encapsulated in the OpenCV library (cross-platform computer vision library) can be directly called. First, the internal parameters of the cameras of any two imaging channels are calibrated respectively, and then the relative rotation and translation parameters of the two cameras are calibrated.

[0034] In this embodiment, the depth of the image field can be obtained by binocular stereo vision matching technology. A technical route combining binocular matching and deep learning algorithms is adopted. When necessary (such as when feature matching cannot be performed in areas with too weak tissue texture), structured light can be introduced to assist in depth reconstruction to improve the accuracy of polarization image feature point recognition and matching. When constructing the sample data set, the endoscopic-related data is collected by a measurement system capable of obtaining the true depth value (referring to the actual distance from the object to the measurement point). Such measurement systems include but are not limited to depth sensors, 3D scanners, etc. Different types of tissue sample images are collected from different viewing directions (i.e., imaging angles) and different lighting conditions. At the same time, the corresponding fine three-dimensional surface information is collected by the depth measurement system as data labels for network training. The network structure mainly includes a feature enhancement network, a feature extraction network, a cost construction and aggregation network, a disparity calculation and optimization network. After the network is trained, end-to-end acquisition of the depth information of the entire tissue region can be achieved, providing data support for subsequent unified imaging fields.

[0035] In this embodiment, the target return light completes polarization imaging through the multi-view imaging channels. At this time, the fields of view of each channel have been pixel-aligned, so a complete Stokes image can be reconstructed with a single shot without the need for precision mechanical components or polarization modulators.

[0036] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various changes and modifications can be made to the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A novel flexible polarization endoscopy imaging device, characterized in that, It includes a light source, an illumination channel, multiple imaging channels, and an image control and acquisition system; The illumination channel consists of an optical fiber bundle and a polarization state generator, and is used to endow the illumination light with a polarization state; the imaging channel consists of an objective lens, a polarization state analyzer, and a CMOS sensor, and is used to complete the calibration of the internal and external parameters of the camera pose, obtain the corresponding image field depth through a multi-view stereo vision algorithm, and complete the pixel-by-pixel alignment of the fields of view of different imaging channels by using the image field depth information; The light source cooperates with the illumination channel to generate illumination light with a polarization state, which is emitted through the illumination channel and illuminates the imaging target. The target return light completes polarization imaging through the multi-view imaging channel, and the imaging data is transmitted into the image control and acquisition system to complete the calculation of the full Stokes vector.

2. The novel flexible polarization endoscope imaging device according to claim 1, wherein The illumination light with a polarization state is linearly polarized light or circularly polarized light.

3. The novel flexible polarization endoscopy imaging device according to claim 1, wherein The illumination channel and the imaging channel are jointly fixed in an outer sheath, and a sapphire protection window is used to fix the distal end of the outer sheath to encapsulate the optical structure therein.

4. The novel flexible polarization endoscopy imaging device according to claim 1, characterized in that, When reconstructing a linear Stokes polarization measurement image, the number of imaging channels is not less than three.

5. The novel flexible polarization endoscopy imaging device according to claim 1, characterized in that, When reconstructing a full Stokes polarization measurement image, the number of imaging channels is not less than four.

6. The novel flexible polarization endoscopy imaging device according to claim 1, wherein, The image control and acquisition system is used to continuously capture and transmit the original polarization image data, and perform real-time processing and display on the captured data.

7. The novel flexible polarization endoscopy imaging device according to claim 1, characterized in that, The imaging device includes multiple imaging channels, and each imaging channel has a different polarization state analyzer; in front of each imaging channel, a linear polarizer with the same polarization direction and an arbitrarily selected angle is configured in common, and wave plates with different fast axis directions and arbitrarily combined angles are respectively configured to realize the reconstruction of the Stokes polarization image.

8. The novel flexible polarization endoscopy imaging device according to claim 1, characterized in that, The imaging channel is used to complete the calibration of the internal and external parameters of the camera pose, obtain the corresponding image field depth through a multi-view stereo vision algorithm, and complete the pixel-by-pixel alignment of the fields of view of different imaging channels by using the image field depth information; specifically: set one imaging channel as the reference channel, capture the reference polarization image, and the remaining imaging channels respectively capture polarization images and perform feature matching with the reference polarization image to obtain the corresponding image field depth. The depth map (X, Y, Z) is converted into a three-dimensional point cloud (X, Y, Z, R, G, B) through the internal parameters of each channel camera, and then the three-dimensional point cloud is converted to the perspective of the reference channel through the rotation and translation matrices of the external parameters of the camera, and then the three-dimensional point cloud is converted into a polarization image (X, Y, RGB) through the internal parameters of the reference channel camera to complete the pixel-by-pixel alignment of the fields of view of each imaging channel and the reference channel.

9. The novel flexible polarization endoscopy imaging device according to claim 1, wherein The Zhang Zhengyou calibration method is adopted for the calibration of the internal and external parameters of the camera pose.

10. The novel flexible polarization endoscope imaging device according to claim 1, wherein, The target return light completes polarization imaging through the multi-view imaging channel, and then the fields of view of each channel are aligned pixel by pixel to realize the reconstruction of a Stokes image in one shot.

Images