A flexible polarization endoscopic imaging device

Abstract

The present invention discloses a flexible polarization endoscopic imaging device for tissue polarization imaging, comprising: a light source generating illumination light, which is emitted through an illumination channel and illuminates the imaging target; the illumination channel is composed of a fiber bundle and a polarization state generator; the imaging channel is composed of an objective lens, a polarization state analyzer, and a CMOS sensor; the system integrates multiple imaging channels, completes the calibration of internal and external parameters of the posture through multiple imaging channels, and obtains the corresponding image field depth through a multi-eye stereo vision algorithm, using this depth information to complete the pixel-by-pixel alignment of the fields of view of different imaging channels; the target return light passes through the imaging channel to complete polarization imaging; the illumination channel and imaging channel are jointly fixed in an outer sheath, and a sapphire protective window is fixed to the distal end of the outer sheath to encapsulate the optical structure therein. The optical structure of the present invention is compact and can complete the full Stokes vector calculation in a single shot, without the need for precision mechanical components or polarization modulators.

Description

Technical Field

[0001] The present invention belongs to the technical field of endoscopic imaging, and in particular relates to a flexible polarization endoscopic imaging device. Background Art

[0002] Polarization imaging has attracted widespread attention in the field of tissue imaging because it can be used to reveal information about tissue morphology, structure, and composition. Tissue polarization imaging is achieved by measuring the Stokes vector (full or partial) of light radiating from tissue. Stokes polarization imaging, typically combined with a linearly or circularly polarized light source, is used to reveal diagnostically useful tissue structural information and has also been applied to the identification of cancerous tissue. The continuous development of polarization imaging technology and equipment has driven the application and transformation of polarization imaging technology in surgery and tissue diagnosis.

[0003] Existing polarization endoscope systems are based on rigid tube endoscopes, and the realization of Stokes polarization imaging mainly relies on mechanical devices to rotate wave plates or linear polarizers. There is no flexible snapshot polarization endoscope imaging device. In addition, previous polarization endoscopes and polarimeters with mechanical moving parts have disadvantages such as relatively slow speed, vibration, and short life. Reconstructing a Stokes image requires at least four pairs of phase-delayed images to be generated in time sequence, and mechanical rotation switching of one delay usually takes tens of milliseconds, which limits the acquisition speed of such devices. Nowadays, pixelated micro-linear polarizer arrays can be directly integrated into image sensors to measure the polarization state in a single snapshot. However, the vast majority of such devices are only sensitive to linear polarization. Currently, there are few customized pixelated micro-elliptical polarizer arrays, making it difficult to achieve a complete Stokes polarization imaging measurement in one shot.

[0004] To address the above problems, the present invention provides a flexible polarization endoscopic imaging device that can complete full Stokes vector calculation in a single shot without the need for mechanical parts to precisely rotate the linear polarizer device. The polarization measurement is fast and accurate, and the implementation principle is simple. Summary of the Invention

[0005] The present invention aims to provide a flexible polarization endoscopic imaging device for real-time polarization imaging of tissues, in response to demands for high-quality intraoperative imaging and tumor detection. The device is compact and simple in structure, and provides accurate and fast polarization measurement imaging.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A flexible polarization endoscopic imaging device includes a light source, an illumination channel, multiple imaging channels, and an image control and acquisition system;

[0008] The illumination channel consists of an optical fiber bundle and a polarization state generator, which is used to make the illumination light have a polarization state. The imaging channel consists of an objective lens, a polarization state analyzer, and a CMOS sensor, which is used to complete the calibration of the camera's internal and external parameters, and obtain the corresponding image field depth through a multi-view stereo vision algorithm. The image field depth information is used to complete the pixel-by-pixel alignment of the fields of view of different imaging channels.

[0009] The light source cooperates with the illumination channel to generate illumination light with a polarization state, which is emitted through the illumination channel and illuminates the imaging target. The target return light completes polarization imaging through the multi-eye imaging channel, and the imaging data is transmitted to the image control and acquisition system to complete the full Stokes vector calculation.

[0010] Furthermore, the polarized illumination light is linearly polarized light or circularly polarized light.

[0011] Furthermore, the illumination channel and the imaging channel are fixed together in an outer sheath, and a sapphire protective window is fixed at the distal end of the outer sheath to encapsulate the optical structure therein.

[0012] Furthermore, when reconstructing the linear Stokes polarization measurement image, there are no less than three imaging channels.

[0013] Furthermore, when reconstructing the full Stokes polarization measurement image, the imaging channels are no less than four.

[0014] Furthermore, the image control and acquisition system is used to continuously capture and transmit raw polarization image data, and to process and display the captured data in real time.

[0015] Furthermore, the imaging device includes multiple imaging channels, each imaging channel has a different polarization state analyzer; in front of each imaging channel, a linear polarizer with the same polarization direction and an arbitrarily selected angle is commonly configured, and a wave plate with different fast axis directions and an arbitrary combination of angles is separately configured to achieve reconstruction of the Stokes polarization image.

[0016] Furthermore, the imaging channel is used to complete the calibration of the camera's internal and external parameters, and obtain the corresponding image field depth through a multi-eye stereo vision algorithm, and use the image field depth information to complete the pixel-by-pixel alignment of the fields of view of different imaging channels; specifically: one imaging channel is set as the reference channel, and a reference polarization image is captured. The remaining imaging channels respectively capture polarization images and perform feature matching with the reference polarization image to obtain the corresponding image field depth, and the depth map (X, Y, Z) is converted into a three-dimensional point cloud (X, Y, Z, R, G, B) through the internal parameters of each channel camera, and then the three-dimensional point cloud is converted to the reference channel perspective through the rotation and translation matrix of the camera's external parameters, and then the three-dimensional point cloud is converted into a polarization image (X, Y, RGB) through the internal parameters of the reference channel camera to complete the pixel-by-pixel alignment of the field of view of each imaging channel and the reference channel.

[0017] Furthermore, the camera pose internal and external parameter calibration adopts the Zhang Zhengyou calibration method.

[0018] Furthermore, the target return light passes through multiple imaging channels to complete polarization imaging, and then the fields of view of each channel are aligned pixel by pixel, so that a Stokes image can be reconstructed in one shot.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses a flexible polarization endoscopic imaging device, in which the distal end of the endoscope simultaneously integrates a polarization state generator, a polarization state analyzer, an illumination fiber, and an image control and acquisition system. This compact optical structure design can meet the system's miniaturization requirements and enable polarization imaging of the device in an endoscopic environment. It is particularly suitable for the detection of shallow tissues (such as the skin surface and colorectal mucosa) and has important clinical application prospects, especially in the early detection of early cancerous tissue and tumors. Furthermore, there is no mechanical movement in the entire system, so the system only needs to be calibrated once for long-term use. Combined with multiple imaging channels, snapshot polarization imaging can be achieved, and the complete Stokes vector can be calculated in a single shot, making polarization measurements fast and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 2 is a schematic structural diagram of a flexible polarization endoscopic imaging device in an embodiment of the present invention;

[0021] Figure 2 4 is a cross-sectional view of the distal end of the flexible polarization endoscope in an embodiment of the present invention.

[0022] In the figure, 1, illumination channel; 2, -45° wave plate; 3, 0° wave plate; 4, linear polarizer; 5, 30° wave plate; 6, 60° wave plate; 7, imaging channel; A, object end; B, image end. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. However, it should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the present invention.

[0024] This embodiment relates to a flexible polarization endoscopic imaging device, such as Figure 1As shown, it includes: a light source, an illumination channel, an imaging channel and an image control and acquisition system; the light source cooperates with a polarization state generator (PSG) to generate system illumination light of linearly polarized light or circularly polarized light, which is emitted through the illumination channel through a lens and illuminates the imaging target (such as biological tissue), and the return light completes polarization imaging through the multi-eye imaging channel. The imaging data is transmitted to the image control and acquisition system to complete the full Stokes vector calculation. The illumination channel and multiple imaging channels of the imaging device are fixed in a metal outer sheath, and the internal optical system is encapsulated by a sapphire protective window at the far end.

[0025] The illumination channel is composed of an optical fiber bundle and a polarization state generator. The polarization state generator is placed at the end of the optical fiber bundle (i.e., the end of the endoscope objective lens) to ensure that the illumination light of the endoscopic imaging device has the required polarization state.

[0026] The imaging channel consists of an objective lens, a polarization state analyzer (Phase Shift Analysis, PSA) and a CMOS sensor. The device has multiple imaging channels. Among them, reconstructing a linear Stokes polarization measurement image requires three or more imaging channels; reconstructing a full Stokes polarization measurement image requires four or more imaging channels.

[0027] The image control and acquisition system is used to continuously capture and transmit original polarization images, and to process and display the captured data in real time.

[0028] In this embodiment, the imaging device includes multiple imaging channels, each imaging channel has a different polarization state analyzer, and linear polarizers with the same polarization direction (any angle can be selected) and wave plates with different fast axis directions (any angle can be combined) are respectively configured in front of each imaging channel to achieve reconstruction of the Stokes polarization image.

[0029] Specifically, if Figure 2 As shown, taking four imaging channels as an example, it is specifically described that a linear polarizer 4 with the same polarization direction (such as 0°), a -45° wave plate 2, a 0° wave plate 3, a 30° wave plate 5 and a 60° wave plate 6 are respectively configured in front of each imaging channel 7 to realize the reconstruction of the Stokes polarization image. Figure 2 In the image processing apparatus, the object end A and the image end B are distributed on both sides of the imaging channel 7. The illumination channel 1 provides appropriate illumination light for the imaging target, creating basic conditions for the subsequent imaging process.

[0030] The specific principle description is that the polarized light propagation of the imaging channel can be characterized by the following linear equation, where the polarization state (SOP) of the incident light is expressed by the input Stokes vector S inTo characterize, the polarization characteristics of the polarization state analyzer are described by the Mueller matrix M, and the SOP of the output light is described by the output Stokes vector S out To represent (i.e. image sensor),

[0031]

[0032]

[0033] in Characterize the Mueller matrix for different wave plates (-45°, 0°, 30°, and 60°), The Mueller matrix representing the linear polarizer. In this design, the first row of M corresponding to different PSA states (here, waveplates at different angles) is used to construct the PSA instrument matrix. This method is also applicable to PSAs of other configurations, as long as the PSA instrument matrix is ​​full rank.

[0034] The complete Stokes vector S can thus be obtained according to the following equation:

[0035]

[0036]

[0037] The superscripts -1 and T represent the pseudo-inverse matrix and the transposed matrix, respectively. n is a column vector containing the intensity readings for each pixel of the image sensor. PSA refers to the PSA instrumentation matrix.

[0038] In this embodiment, the multiple imaging channels complete the calibration of the camera pose internal and external parameters, and obtain the corresponding image field depth through a multi-eye stereo vision algorithm, and use this depth information to complete the pixel-by-pixel alignment of the fields of view of different imaging channels. Specifically, one of the imaging channels is set as the reference channel, and a reference polarization image is captured. The remaining imaging channels respectively capture polarization images and perform feature matching with the reference polarization image to obtain the corresponding image field depth. The depth map (X, Y, Z) is converted into a three-dimensional point cloud (X, Y, Z, R, G, B) through the intrinsic parameters of each channel camera, and then the three-dimensional point cloud is converted to the reference channel perspective through the rotation and translation matrix of the camera extrinsic parameters. The three-dimensional point cloud is then converted into a polarization image (X, Y, RGB) through the intrinsic parameters of the reference channel camera, completing the pixel-by-pixel alignment of the field of view of each imaging channel with the reference channel.

[0039] In this embodiment, the camera pose internal and external parameter calibration adopts the Zhang Zhengyou calibration method, which can directly call the Zhang Zhengyou calibration method encapsulated in the OpenCV library (cross-platform computer vision library). First, the intrinsic parameters of the cameras of any two imaging channels are calibrated, and then the relative rotation and translation parameters of the two cameras are calibrated.

[0040] In this embodiment, the image field depth can be acquired using binocular stereo matching technology, employing a combination of binocular matching and deep learning algorithms. When necessary (e.g., when weak tissue texture precludes feature matching), structured light can be introduced to assist in depth reconstruction to improve the accuracy of polarization image feature point identification and matching. When constructing the sample dataset, endoscopic data is collected by a measurement system capable of obtaining true depth values ​​(the actual distance from the object to the measurement point). Such measurement systems include, but are not limited to, depth sensors and 3D scanners. Images of different tissue samples are collected from different viewing directions (i.e., imaging angles) and under different lighting conditions. The depth measurement system also collects corresponding detailed three-dimensional surface information as data labels for network training. The network architecture primarily comprises a feature enhancement network, a feature extraction network, a cost construction and aggregation network, and a disparity calculation and optimization network. Once the network is trained, end-to-end acquisition of full-area tissue depth information is achieved, providing unified data support for subsequent imaging fields.

[0041] In this embodiment, the target return light completes polarization imaging through multiple imaging channels. At this time, the fields of view of each channel have been aligned pixel by pixel, so a complete Stokes image can be reconstructed in one shot without the need for precision mechanical components or polarization modulators.

[0042] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A flexible polarization endoscopic imaging device, characterized in that: Includes light source, illumination channel, multiple imaging channels and image control and acquisition system; The illumination channel is composed of an optical fiber bundle and a polarization state generator, and the polarization state generator is placed at the end of the optical fiber bundle to provide the illumination light with a polarization state. The imaging channel is composed of an objective lens, a polarization state analyzer, and a CMOS sensor, and is used to complete the calibration of the camera's internal and external parameters, and obtain the corresponding image field depth through a multi-eye stereo vision algorithm, and use the image field depth information to complete the pixel-by-pixel alignment of the fields of view of different imaging channels. The imaging device includes multiple imaging channels, each of which has a different polarization state analyzer. In front of each imaging channel, a linear polarizer with the same polarization direction and an arbitrarily selected angle is commonly configured, and a wave plate with different fast axis directions and an arbitrary angle combination is separately configured to achieve reconstruction of the Stokes polarization image. The light source cooperates with the illumination channel to generate illumination light with a polarization state, which is emitted through the illumination channel and illuminates the imaging target. The target return light completes polarization imaging through the multi-eye imaging channel, and the imaging data is transmitted to the image control and acquisition system to complete the Stokes vector calculation.

2. The flexible polarization endoscopic imaging device according to claim 1, characterized in that: The polarized illumination light is linearly polarized light or circularly polarized light.

3. The flexible polarization endoscopic imaging device according to claim 1, characterized in that: The illumination channel and the imaging channel are fixed together in an outer sheath, and a sapphire protection window is fixed at the distal end of the outer sheath to encapsulate the optical structure therein.

4. The flexible polarization endoscopic imaging device according to claim 1, characterized in that: When reconstructing the linear Stokes polarimetry image, there are no less than three imaging channels.

5. The flexible polarization endoscopic imaging device according to claim 1, characterized in that: When reconstructing the full Stokes polarimetry image, the number of imaging channels is no less than four.

6. The flexible polarization endoscopic imaging device according to claim 1, characterized in that: The image control and acquisition system is used to continuously capture and transmit original polarization image data, and to process and display the captured data in real time.

7. The flexible polarization endoscopic imaging device according to claim 1, characterized in that: The imaging channel is used to complete the calibration of the camera's internal and external parameters, and obtain the corresponding image field depth through a multi-eye stereo vision algorithm, and use the image field depth information to complete the pixel-by-pixel alignment of the fields of view of different imaging channels; specifically, one imaging channel is set as the reference channel, and a reference polarization image is captured. The remaining imaging channels respectively capture polarization images and perform feature matching with the reference polarization image to obtain the corresponding image field depth. The depth map (X, Y, Z) is converted into a three-dimensional point cloud (X, Y, Z, R, G, B) through the intrinsic parameters of each channel camera. Subsequently, the three-dimensional point cloud is converted to the reference channel perspective through the rotation and translation matrix of the camera's external parameters. The three-dimensional point cloud is then converted into a polarization image (X, Y, RGB) through the intrinsic parameters of the reference channel camera to complete the pixel-by-pixel alignment of the fields of view of each imaging channel and the reference channel.

8. The flexible polarization endoscopic imaging device according to claim 1, characterized in that: The camera pose internal and external parameter calibration adopts the Zhang Zhengyou calibration method.

9. The flexible polarization endoscopic imaging device according to claim 1, characterized in that: The target return light passes through multiple imaging channels to complete polarization imaging, and then the fields of view of each channel are aligned pixel by pixel to reconstruct a Stokes image in one shot.

Images