Bent endoscopic imaging method and system based on few-mode fiber
By combining few-mode fiber and deep learning networks, the problems of limited endoscope probe size and image distortion in curved environments are solved, and stable imaging in narrow spaces is achieved, which is suitable for medical and industrial inspections.
Patent Information
- Application Number
- CN202511187736.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-25
- Publication Date
- 2025-10-14
AI Technical Summary
The existing endoscopes have limited probe size and image distortion caused by mode coupling of multimode optical fibers in a curved environment.
By combining few-mode fiber with image processing algorithms and deep learning networks, image information is transmitted using few-mode fiber through the image acquisition-processing module and the beam collimation module, and the model performance is optimized through the deep learning network to reconstruct a clear image.
It achieves stable imaging in narrow spaces, reduces transmission signal distortion caused by mode coupling, and is suitable for clear imaging in complex environments.
Smart Images
Figure CN120779585A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of optical fiber imaging technology, in particular to a bending endoscopic imaging method and system based on few-mode optical fiber. BACKGROUND
[0002] Endoscope technology is widely used in medical, industrial detection and other fields, and has shown great development potential. In the medical field, endoscopes provide important support for minimally invasive surgery and endoscopic examination, enabling doctors to clearly observe the internal tissue structure of the human body, improving diagnostic accuracy and surgical precision. In the industrial field, endoscope technology is applied to pipeline wall detection, internal inspection of aircraft engines and other scenarios, enabling non-destructive testing of difficult-to-access areas and ensuring equipment operation safety.
[0003] Currently commonly used endoscopes are mainly composed of single-mode fiber bundles or multi-lens systems, but the probes and pipe diameters of these two types of endoscopes are large, and the imaging resolution is limited by the aperture, making them unsuitable for use in narrow and complex environments. In contrast, single-fiber imaging technology only requires one optical fiber to complete image transmission, with advantages such as small size, low cost, and good flexibility. A single multi-mode optical fiber can carry a large number of modes and transmit a large amount of information in parallel, with a significantly reduced core diameter, making it suitable for examination in small environments such as blood vessels. However, there is a serious mode dispersion problem inside the multi-mode optical fiber, and the transmitted image will be severely distorted. The mode transmission in the optical fiber has a high degree of nonlinearity, and when the optical fiber is subjected to bending disturbance, complex coupling occurs between a large number of modes, causing the output speckle pattern to change dramatically. The randomness and unpredictability of mode coupling make it difficult for the optical fiber endoscopic imaging system to maintain stable imaging performance.
[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY
[0005] In view of the deficiencies of the prior art, the technical problem to be solved by the present application is to provide a bending endoscopic imaging method and system based on few-mode optical fiber, which combines image processing algorithms and few-mode optical fiber imaging to solve the problem of limited probe size caused by large aperture of multi-mode optical fiber endoscope and image distortion caused by mode coupling in a bending environment.
[0006] The technical solution of the present application to solve the technical problem of the method is to provide a bending endoscopic imaging method based on few-mode optical fiber, comprising the following steps:
[0007] Encode the MNIST handwritten digital image to a digital micromirror device (DMD) and irradiate it with a laser to generate an object light wave;
[0008] Couple the object light wave to a few-mode optical fiber to obtain an exit speckle pattern after transmission;
[0009] The speckle pattern is received by using the image acquisition-processing module and the speckle pattern is pretreated;
[0010] The deep learning network is trained by using the speckle dataset of the unbent optical fiber and the model performance is optimized;
[0011] The image reconstruction effect of the few-mode optical fiber under different bending conditions is tested.
[0012] Further, the method for coupling the object light wave to the few-mode optical fiber and obtaining the exit speckle pattern through transmission specifically comprises:
[0013] The light beam carrying the image information is greatly reduced in size after passing through the front-end microscopic objective lens, so that the light beam can be coupled to the core of the few-mode optical fiber, and after transmission through the few-mode optical fiber, the exit light beam is collimated and expanded by a microscopic objective lens, facilitating subsequent acquisition of the speckle image.
[0014] Further, the method for receiving the speckle pattern by using the image acquisition-processing module and pretreating the speckle pattern specifically comprises:
[0015] The DMD is used to cyclically play different MNIST handwritten digital images, and the CCD camera is synchronously triggered to respectively acquire the exit speckle pattern of the unbent optical fiber and the exit speckle pattern of the optical fiber under different bending conditions. The DMD is used as an active trigger source, and the CCD camera is used as a passive response device.
[0016] When the exit speckle pattern of the optical fiber under bending is acquired, a flat plate is additionally arranged in the middle part of the few-mode optical fiber, so that the few-mode optical fiber is bent in a fixed horizontal direction, thereby facilitating unification of the bending direction of different bending curvatures, and the optical fiber is moved in a fixed step in the horizontal and vertical directions of the optical fiber, so as to change the bending curvature of the optical fiber in the platform part.
[0017] According to the pixel size of the MNIST handwritten digital image, the acquired speckle pattern is down-sampled, so that the pixel number of the handwritten digital image is consistent with the pixel number of the speckle image, facilitating subsequent deep learning training and reconstruction.
[0018] Further, the method for training the deep learning network by using the speckle dataset of the unbent optical fiber and optimizing the model performance specifically comprises:
[0019] In the training stage, the training set speckle pattern of the speckle dataset of the unbent optical fiber and the corresponding MNIST handwritten digital label pattern are used as the input of the deep learning network, and the deep learning network is trained.
[0020] In the test phase, the test set speckle pattern of the unbent optical fiber speckle data set is input into the pre-trained network model to obtain a predicted handwritten digital picture; the predicted handwritten digital picture is compared with the original MNIST handwritten digital image to calculate the image reconstruction accuracy; the model performance is continuously optimized by adjusting the learning rate, weight, batch size and other network parameters; and the best model is saved.
[0021] Further, the method for testing the image reconstruction effect of the few-mode optical fiber under different bending conditions specifically comprises:
[0022] The collected optical fiber exit speckles under different bending conditions are input into the image acquisition-processing module and used as the input of the pre-trained best deep learning model to obtain a reconstructed image and calculate the average Dice coefficient between the reconstructed image and the label image.
[0023] The technical scheme for solving the technical problem of the system is to provide a bending endoscopic imaging system based on a few-mode optical fiber, which comprises a light beam collimation module, a pattern loading module, an image transmission module and an image acquisition-processing module.
[0024] The light beam collimation module comprises a laser and a laser collimator, and is used for collimating the laser beam emitted by the laser;
[0025] The pattern loading module comprises a DMD, a mirror M1 and a mirror M2, the DMD is used for loading an image and modulating a light wave to carry image information, and the mirrors M1 and M2 are used for guiding the light wave into the optical fiber;
[0026] The image transmission module comprises a front-end microscopic objective lens, a few-mode optical fiber and a rear-end microscopic objective lens connected in sequence, and is used for coupling the light signal into the few-mode optical fiber for transmission and coupling out of the optical fiber;
[0027] The image acquisition-processing module comprises a filter, a CCD camera and a computer, is used for receiving and storing a speckle pattern, and reconstructs and restores the speckle by using an image processing algorithm in the computer.
[0028] Further, the few-mode optical fiber in the image transmission module belongs to a common step-index optical fiber, has a fiber core diameter of 9 mu m, can transmit 6 modes, has a large effective refractive index difference between modes, can significantly reduce mode coupling when the optical fiber is bent, and thus avoids the problem of transmission signal distortion caused by mode coupling.
[0029] Further, the front-end microscopic objective lens in the image transmission module is used for reducing the size of the light beam to match the size of the few-mode optical fiber core, and the rear-end microscopic objective lens is used for enlarging and collimating the size of the exit light beam, thereby facilitating subsequent speckle image reception.
[0030] Further, the CCD camera in the image acquisition-processing module transmits the obtained speckle pattern to a computer, the computer is provided with a deep learning model, the deep learning model outputs the input speckle pattern as a restored image, and the structure of the deep learning model is a shallow U-shaped convolutional neural network, and the structure design aims to balance the complexity and calculation efficiency of the model, so as to realize the reconstruction of the speckle pattern of limited mode.
[0031] Compared with the prior art, the present application has the following beneficial effects:
[0032] 1. A few-mode fiber-based curved endoscopic imaging method and system, the few-mode fiber has the characteristics of few modes and large effective refractive index difference between modes, which can significantly reduce mode coupling when the fiber is bent, thereby avoiding the problem of transmission signal distortion caused by mode coupling.
[0033] 2. A few-mode fiber-based curved endoscopic imaging method and system, the few-mode fiber has a thin core, which can greatly reduce the size of the endoscope in actual application, and is suitable for imaging in narrow space.
[0034] 3. A few-mode fiber-based curved endoscopic imaging method and system, in view of the problems of few modes of the few-mode fiber and small size of the speckle image, a shallow U-shaped convolutional neural network is used for speckle image training and reconstruction, and the speckle image of the fiber with different bending degrees has good reconstruction effect. BRIEF DESCRIPTION OF DRAWINGS
[0035] Figure 1 It is a flowchart of the few-mode fiber-based curved endoscopic imaging method of the embodiment of the present application;
[0036] Figure 2 It is a schematic diagram of different bending degrees of the few-mode fiber of the embodiment of the present application;
[0037] Figure 3 It is a structure diagram of the shallow U-shaped convolutional neural network of the embodiment of the present application;
[0038] Figure 4 It is a speckle reconstruction effect diagram of different bending degrees of the embodiment of the present application;
[0039] Figure 5 It is a structure diagram of the few-mode fiber-based curved endoscopic imaging system of the embodiment of the present application. DETAILED DESCRIPTION
[0040] The specific embodiments of the present application are given below. The specific embodiments are only used to further illustrate the present application, and do not limit the protection scope of the present application.
[0041] Figure 1A flowchart of a few-mode fiber-based bending endoscopic imaging method of an embodiment of the present application.
[0042] At step S100, MNIST handwritten digital images are encoded to a digital micromirror device (DMD) and irradiated with a laser to generate an object light wave;
[0043] In this embodiment, the MNIST handwritten digital images have a size of 28x28 pixels, and the laser light source is a 671nm solid-state laser, with a collimated beam diameter of about 3mm.
[0044] At step S200, the object light wave is coupled to a few-mode fiber, and a transmitted speckle pattern is obtained;
[0045] Specifically, the light beam carrying image information is greatly reduced in size after passing through the front-end microscope objective, so that the light beam can be coupled to the few-mode fiber core. After transmission through the few-mode fiber, the exiting light beam is collimated and expanded by a rear-end microscope objective, facilitating subsequent speckle image acquisition.
[0046] In this embodiment, the few-mode fiber is a common step-index fiber, with a length of 60cm, a core diameter of 9μm, a cladding diameter of 125μm, and a numerical aperture (NA) of 0.1. In this fiber system, the normalized frequency is V=4.21, which can excite up to four modes (LP01, LP11, LP21, LP02) for transmission (including six degenerate modes), thereby reducing the impact of mode coupling on imaging quality by limiting the number of modes. The front-end microscope objective has a magnification of 20x, and the rear-end microscope objective has a magnification of 40x.
[0047] At step S300, the speckle pattern is acquired and pre-processed using an image acquisition and processing module.
[0048] Specifically, different MNIST handwritten digital images are played back in a loop using the DMD, and a CCD camera is triggered synchronously to acquire the un-bent fiber exiting speckle pattern and the fiber exiting speckle pattern under different bending conditions. The DMD is the active trigger source, and the CCD camera is the passive response device.
[0049] Further, when acquiring the fiber exiting speckle pattern under bending conditions, a flat plate is added to the middle part of the fiber to bend the fiber in a fixed horizontal direction, thereby facilitating the unification of the bending direction of different bending curvatures. The bending curvature of the fiber is changed in steps of 1cm in each group in the horizontal and vertical directions of the fiber, from slight bending with small curvature to obvious bending with large curvature. A schematic diagram of different bending degrees is shown in Figure 2 .
[0050] Further, according to the pixel size of the MNIST handwritten digital image, the collected speckle patterns are down-sampled so that the pixel number of the handwritten digital image is consistent with the pixel number of the speckle, facilitating subsequent deep learning training and reconstruction.
[0051] In this embodiment, the synchronization trigger frequency is 50Hz; the imaging area of the CCD camera is cut into 56x56 pixels, and after one down-sampling, it is reduced to 28x28 pixels.
[0052] In this embodiment, the number of collected speckle patterns of the un-bent optical fiber is 10000, and the number of speckle patterns of the optical fiber under different bending conditions is 500.
[0053] Step S400, training the deep learning network with the un-bent optical fiber speckle dataset and optimizing the model performance.
[0054] Specifically, the collected un-bent optical fiber speckle patterns are divided into training set speckle patterns and test set speckle patterns. In the training stage, the training set speckle patterns and the corresponding label patterns are used as the input of the deep learning network to train the network; in the test stage, the test set speckle patterns are input into the pre-trained network model to obtain the predicted handwritten digital image, and the predicted handwritten digital image is compared with the original image to calculate the accuracy of the reconstruction, and the network parameters such as learning rate, weight, batch size are adjusted to continuously optimize the model performance, and the best model is saved.
[0055] Further, since the few-mode optical fiber used in the present application only supports a limited number of modes, its transmission capacity is limited, and different input images will experience significant information compression during transmission. This compression effect leads to a lack of diversity in the feature distribution of the generated speckle patterns, increasing the difficulty of image reconstruction. Therefore, a shallow U-shaped convolutional neural network model is built, and the structure design aims to balance the complexity and computational efficiency of the model, so as to realize the reconstruction of the limited mode speckle pattern.
[0056] Figure 3 The structure diagram of the shallow U-shaped convolutional neural network of the embodiment of the present application; the training process of the shallow U-shaped convolutional neural network includes two stages of encoding and decoding. In the encoding stage, the input speckle pattern is processed by two times of convolution with a size of 3x3, a step of 1 and a padding method of "Same", and the activation function after convolution adopts Relu, which is used to realize nonlinear transformation and keep the signal smooth. In the decoding stage, the feature map is first up-sampled to increase its size to twice the original size. At the same time, the feature map corresponding to the encoding stage is spliced into the decoding feature map through the skip connection (Skip Connection), and the spliced feature map is sequentially subjected to one deconvolution operation and two convolution operations with a size of 3x3, and the activation function still adopts Relu.
[0057] Further, a self-attention mechanism module is introduced in the encoding stage, and the attention mechanism module is respectively placed after the activation function of the second convolutional layer and after the activation function of the bottleneck layer, so that the network can learn the information of the speckle invariant under different bending conditions, and the reconstruction effect of the curved optical fiber is improved.
[0058] In this embodiment, the shallow U-shaped convolutional neural network adopts two activation functions of ReLU and Sigmoid, aiming to improve the performance of the network in the optical fiber imaging image reconstruction task; and a binary cross-entropy loss function (Binary Cross-Entropy, BCE) is selected as the loss function of the model. The BCE loss function is usually used to process the predicted value normalized to the range of [0, 1], and the difference is calculated with the label image pixel by pixel, and the formula is as follows:
[0059]
[0060] Wherein, y i and respectively represent the true value and the predicted value of the i-th pixel, and N represents the total number of pixels in the image.
[0061] Step S500, test the image reconstruction effect of the few-mode optical fiber under different bending conditions;
[0062] Specifically, the collected optical fiber exit speckle images under different bending conditions are input into the image acquisition and processing module, and are used as the input of the pre-trained best deep learning model to obtain the reconstructed image, and the average Dice coefficient between the reconstructed image and the label image is calculated, that is, the overlap degree between the two groups of images, and the calculation formula is as follows:
[0063]
[0064] Wherein, X represents the reconstructed image after binarization, and Y represents the label image. The value of the Dice coefficient is between 0 and 1, and the closer the Dice value is to 1, the higher the pixel overlap degree between the reconstructed image and the label image is.
[0065] In this embodiment, the average Dice coefficient of the reconstructed image under different bending degrees reaches 0.74 or more, and part of the results are shown in Figure 4 , which proves that the present application has good imaging performance under the condition of fiber bending.
[0066] Figure 5 The structure diagram of the few-mode optical fiber based curved endoscopy imaging system of the embodiment of the present application, the few-mode optical fiber based curved endoscopy imaging system can but not limited to include sequentially connected beam collimation module 1, pattern loading module 2, image transmission module 3, image acquisition and processing module 4;
[0067] The light beam collimation module 1 comprises a laser 101 and a laser collimator 102, the laser 101 is connected with the laser collimator 102, and the collimation of the laser beam is realized.
[0068] The pattern loading module 2 comprises a digital micromirror array (DMD) 201, a mirror M1 202 and a mirror M2 203, the digital micromirror array (DMD) 201 is located between the laser collimator 102 and the mirror M1 202, the mirror M1 202 is connected with the mirror M2, the digital micromirror array (DMD) 201 is used for loading an image and modulating a light wave, the light beam reflected by the DMD carries image information, and the mirrors M1 and M2 are used for guiding the light wave into an optical fiber.
[0069] The image transmission module 3 comprises a front-end microscopic objective 301, a few-mode optical fiber 302 and a rear-end microscopic objective 303 which are sequentially connected, the input end of the front-end microscopic objective is connected with the mirror M2, the front-end microscopic objective 301 is used for reducing the size of the light beam so as to match the size of the light beam with the size of the core of the few-mode optical fiber, and the rear-end microscopic objective 303 is used for enlarging and collimating the size of the outgoing light beam, thereby facilitating the subsequent reception of a speckle pattern image.
[0070] The image acquisition and processing module 4 comprises a filter 401, a CCD camera 402 and a computer 403 which are sequentially connected, the CCD camera 402 in the image acquisition and processing module 4 transmits the obtained speckle pattern to the computer 403, the computer 403 is provided with a pre-trained deep learning model, the deep learning model outputs the input speckle pattern as a restored image, and the structure of the deep learning model is a shallow U-shaped convolutional neural network, so as to realize the reconstruction of the limited mode speckle pattern.
[0071] In summary, the present application provides a few-mode optical fiber-based bending endoscopic imaging method and system, a single few-mode optical fiber is used as an image transmission optical fiber, and the size of the image transmission optical fiber is greatly reduced; the present application can overcome the problem that the image transmission quality of a multi-mode optical fiber is greatly degraded under bending, clear image transmission can be realized under different bending degrees, and more accurate and reliable imaging results can be provided for the fields of medical diagnosis, industrial detection and biological imaging.
[0072] The above specific description further details the purpose, technical scheme and beneficial effects of the present application, and it should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
[0073] The present application is not described in the prior art.
Claims
1. A curved endoscopic imaging method based on few-mode optical fiber, characterized in that: include: The MNIST handwritten digit image is encoded into a digital micromirror device (DMD) and illuminated with a laser to generate a light wave. The object light wave is coupled into the few-mode fiber and the output speckle pattern is obtained after transmission; Using an image acquisition-processing module to receive the speckle pattern and pre-process the speckle pattern; Use unbent fiber speckle datasets to train deep learning networks and optimize model performance; Test the image reconstruction effect of few-mode fiber under different bending conditions.
2. The method for curved endoscopic imaging based on few-mode fiber according to claim 1, characterized in that: The method of coupling the object light wave to the few-mode optical fiber and obtaining the output speckle pattern through transmission specifically includes: The light beam carrying image information is greatly reduced in size after passing through the front-end microscope objective, allowing the beam to be precisely coupled to the core of the few-mode fiber. After being transmitted through the few-mode fiber, the outgoing light beam is collimated and expanded by a microscope objective to facilitate the subsequent acquisition of speckle images.
3. The method for curved endoscopic imaging based on few-mode fiber according to claim 1, characterized in that: The method of using the image acquisition-processing module to receive the speckle pattern and pre-process the speckle pattern specifically includes: The DMD was used to loop different MNIST handwritten digit images and synchronously trigger the CCD camera to collect speckle patterns emitted from an unbent fiber and from fibers with different bending conditions. The DMD served as an active trigger source, while the CCD camera was a passive response device. When collecting speckle patterns emitted from a bent fiber, a flat plate is added to the middle portion of the few-mode fiber to bend the fiber in a fixed horizontal direction, thereby unifying the bending directions of fibers with different curvatures. The flat plate is moved in a fixed step length in the horizontal and vertical directions of the fiber to change the curvature of the fiber on the platform. According to the pixel size of the MNIST handwritten digit image, the acquired speckle pattern is downsampled so that the number of pixels of the handwritten digit image is consistent with the number of pixels of the speckle pattern, which facilitates subsequent deep learning training and reconstruction.
4. The method for curved endoscopic imaging based on few-mode fiber according to claim 1, characterized in that: The method of using an unbent optical fiber speckle dataset to train a deep learning network and optimize model performance specifically includes: In the training phase: using the training set speckle pattern of the unbent optical fiber speckle dataset and the corresponding MNIST handwritten digit label pattern as inputs of the deep learning network to train the deep learning network; During the testing phase, the test set speckle pattern of the unbent fiber speckle dataset was input into the pre-trained network model to obtain a predicted handwritten digit image. The predicted handwritten digit image was compared with the original MNIST handwritten digit image, and the image reconstruction accuracy was calculated. The model performance was continuously optimized by adjusting network parameters such as the learning rate, weight, and batch size, and the best deep learning model was saved.
5. The method for curved endoscopic imaging based on few-mode fiber according to claim 1, characterized in that: The method for testing the image reconstruction effect of the few-mode fiber under different bending conditions specifically includes: The acquired optical fiber outgoing speckle patterns under different bending conditions are input into an image acquisition-processing module, and then used as input to the optimal deep learning model described in claim 4 to obtain a reconstructed image; the average Dice coefficient between the reconstructed image and the corresponding MNIST handwritten digit label image is calculated to evaluate the image reconstruction effect.
6. A curved endoscopic imaging system based on few-mode fiber for implementing the method according to any one of claims 1 to 5, characterized in that: include: Beam collimation module, pattern loading module, image transmission module, image acquisition-processing module; The beam collimation module includes a laser and a laser collimator, and is used to collimate the laser beam emitted by the laser; The pattern loading module includes a DMD, a reflector M1, and a reflector M2. The DMD is used to load the image and modulate the light wave to carry the image information. The reflectors M1 and M2 are used to guide the object light wave into the optical fiber. The image transmission module includes a front-end microscope objective lens, a few-mode optical fiber, and a rear-end microscope objective lens, and is used to couple the optical signal into the few-mode optical fiber for transmission and couple it out of the optical fiber; The image acquisition and processing module includes a filter, a CCD camera, and a computer, and is used to receive and store the speckle pattern and reconstruct and restore the speckle pattern using an image processing algorithm in the computer.
7. The curved endoscopic imaging system based on few-mode fiber according to claim 6, characterized in that: The few-mode optical fiber in the image transmission module is an ordinary step-index optical fiber with a core diameter of 9 μm. It can transmit 6 modes. The effective refractive index difference between the modes is large, which can significantly reduce the mode coupling when the optical fiber is bent, thereby avoiding the problem of transmission signal distortion caused by mode coupling.
8. The curved endoscopic imaging system based on few-mode fiber according to claim 6, characterized in that: The front-end microscope objective lens in the image transmission module is used to reduce the beam size so that the beam size matches the core diameter of the few-mode optical fiber, and the rear-end microscope objective lens is used to magnify and collimate the output beam size to facilitate subsequent speckle image reception.
9. The curved endoscopic imaging system based on few-mode fiber according to claim 6, characterized in that: The CCD camera in the image acquisition and processing module transmits the acquired speckle pattern to a computer. The computer is equipped with a deep learning model that outputs the input speckle pattern as a restored image. The structure of the deep learning model is a shallow U-shaped convolutional neural network. Its structural design aims to balance the complexity of the model and computational efficiency to reconstruct the speckle pattern of a limited pattern.