Endoscopic bending microscopic imaging loss recovery method and device for step refractive index optical fiber

Abstract

The invention discloses a step refractive index fiber endoscopic bending microscopic imaging loss recovery method and device. The method comprises the following steps: s1, searching and removing an image loss part in a step refractive index fiber endoscope scanning image; s2, image restoration: carrying out feature matching on the two displacement pictures processed in the step s1 to calculate x-direction and y-direction displacements of the two images, and then aligning and fusing the two images in a Poisson fusion mode to repair image loss parts; and s3, background denoising: estimating transmissivity by calculating a dark channel and a background illumination value, and denoising the image according to the transmissivity to obtain a denoised image. According to the method, through frequency domain low-pass filtering and feature matching fusion, lost image center information is successfully recovered, image center details are effectively recovered, and image recovery can be realized through processing of an imaged image.

Description

Technical Field

[0001] The present invention belongs to the technical field of microscopic imaging, and particularly relates to a method and device for restoring the loss of step-index fiber endoscope bending microscopic imaging. Background Art

[0002] Doctors in the field of endoscopy have been eager to achieve real-time high-definition endoscopy imaging technology, which will combine in-vivo imaging with therapeutic intervention to reach a diagnostic level comparable to ex-vivo pathological diagnosis. In recent years, with the development of technologies such as cell endoscopy and confocal laser endoscopy, in-vivo cell-level imaging has gradually become possible. In addition, super-resolution microscopy technology, with its resolution beyond the subcellular level, has brought significant breakthroughs to the fields of biology and life sciences. However, due to the complex optical system, the application of super-resolution microscopes in narrow spaces has encountered challenges. Currently, a key technical problem is how to achieve stable in-vivo super-resolution endoscopy imaging. To solve this problem, a highly anticipated method is to use multi-mode optical fibers as thin as a hair (micrometer scale) as minimally invasive probes and achieve sub-millimeter-scale detection through wavefront shaping technology. Compared with traditional endoscopes, multi-mode fiber endoscopes have the advantages of being thin, having high resolution, and low cost. This technology currently faces two main limitations: First, once the optical fiber is bent, its light propagation mode changes, which affects its imaging ability during flexible operation; second, the resolution of the optical fiber is limited by the numerical aperture and it is difficult to break through the diffraction limit.

[0003] Regarding the problem of light propagation near the bending of the optical fiber, the Yang Qing research group at Zhejiang University proposed a spatial frequency domain coding tracking adaptive beacon light field coding method. In the spatial frequency domain coding tracking adaptive beacon light field coding method, the Yang Qing research team developed a full vector modulation method to control the incident wavefront, enabling the Fresnel reflected light at the distal end to be accurately focused on a single-pixel detector in the spatial frequency domain, that is, on the Fourier plane at the proximal end of the optical fiber, thereby creating a spatial frequency beacon. This method changes the state tracking of multi-mode optical fibers from traditional spatial speckle tracking to spatial frequency beacon tracking of a single pixel. The intensity change of the beacon reflects the degree of correlation between the current bending state of the optical fiber and the pre-calibrated transmission matrix. By deeply analyzing the light propagation mode in the optical fiber and the relationship between these modes and the reflected light, the team revealed the direct connection between the intensity of the spatial frequency beacon captured by the single-pixel detector and the random phase, polarization, and amplitude changes caused by the deformation of the optical fiber. In addition, considering the radial cylindrical symmetry of multi-mode optical fibers, the team adopted a data dimensionality reduction technique to simplify the state search problem to a lower-dimensional problem. These innovations ultimately enabled the search speed of the transmission matrix to reach the kHz level.

[0004] "Single Multimode Fiber for In Vivo Light Field Encoded Endoscopic Imaging", published in the August 2023 issue of *Nature Photonics*, Volume 17, Issue 8, Pages 679 - 687, performs real - time processing and calculation to adjust the corresponding transmission matrix for the optical propagation mode transformation during the imaging process of multimode fibers. However, the computational load is large, making it difficult to perform real - time image restoration.

[0005] "Observation of Distant Objects at Fiber Resolution Using a Multimode - Fiber - Based Holographic Endoscope", published in the March 2021 edition of *Progress in Photonics*, Volume 6, Issue 3, Article ID: 036112, calculates the transmission matrix using the linear relationship between the input field and the output field of the multimode fiber, and designs a series of holograms based on the transmission matrix to be loaded onto the spatial light modulator. Each hologram corresponds to a specific wavefront shape. Due to the optical transmission characteristics in the fiber, especially in step - index fibers, there are propagation - invariant modes that do not change their field distribution when propagating in the fiber. Utilizing this property, when these wavefronts are transmitted through the fiber to the far - end, they can form a focus in the far - field. By displaying a series of calculated holograms on the spatial light modulator, raster - scan imaging can be achieved in the far - field. These holograms cause the light beam at the far - end of the fiber to scan in space, thus covering the entire field of view. By collecting the photoelectric conversion of the fiber and the photodiode, the potential of achieving high - resolution, high - speed, and high - signal - to - noise ratio imaging while keeping the device size small is realized, providing a new tool for clinical diagnosis and minimally invasive surgery. However, the optical field control of the fiber in this scheme only controls a single polarization state, while the present application realizes the optical field control of the input fiber for two polarization states, so that the focused spot formed at the far - end of the fiber output has a higher power ratio and will have a higher image contrast in the finally obtained image. In addition, the fiber in this scheme cannot be bent or deformed. Once deformed, the imaging quality rapidly deteriorates to the point where no image can be obtained, while the present application utilizes the imaging characteristics of step - index fibers and, according to the image restoration method proposed in the present application under image loss, realizes the reconstruction of the lost image, that is, imaging under fiber bending deformation.

[0006] Therefore, how to solve the problem of image loss due to bending during point - shot scanning caused by the offset of the output focus position and the deformation of the focus shape in multimode fiber bending endoscopy imaging, and provide a method and device for loss recovery of step - index fiber endoscopy bending microscopy imaging are technical problems that need to be urgently solved by those skilled in the art. Summary of the Invention

[0007] The first object of the present invention is to provide a method for loss recovery of step - index fiber endoscopy bending microscopy imaging in view of the problems in the prior art.

[0008] To achieve this, the above object of the present invention is realized by the following technical solutions:

[0009] Method for recovering step-index fiber optic endoscope bending microscopic imaging loss, comprising the following steps:

[0010] S1: Search for and remove the image loss part in the step-index fiber optic endoscope scan image: Use the fast Fourier transform to convert the image information into frequency domain information, identify the high-frequency variables in the frequency domain information, capture the image information corresponding to the frequency domain information at the bending part of the step-index fiber optic endoscope scan image, locate the image loss part, use a mask to remove the redundant part of the image, restore the image center information in the step-index fiber optic scan image, and remove the high-frequency speckle information caused by bending imaging;

[0011] S2: Image restoration, select two displacement pictures that have completed the processing of step s1 for feature matching and displacement calculation, and then align and fuse the two images by Poisson fusion to obtain the fused image;

[0012] Perform feature matching and displacement calculation on the fused image and a new image, then align and fuse the two images by Poisson fusion to update the fused image, repeat this step until there is no new image, and output the final fused image;

[0013] To repair the image loss part;

[0014] S3: Background denoising, estimate the transmittance by calculating the dark channel and background illumination value, and perform image denoising according to the transmittance to obtain the denoised image.

[0015] While adopting the above technical solution, the present invention can also adopt or combine the following technical solutions:

[0016] As a preferred technical solution of the present invention: In step S1, perform Fourier transform on the picture data collected by the step-index fiber optic scan to obtain the frequency domain information corresponding to the picture data, and identify the image loss part through frequency domain analysis.

[0017] As a preferred technical solution of the present invention: Use a circular mask to filter out the high-frequency information in the picture to remove the redundant information generated during the bending process of the step-index fiber optic.

[0018] As a preferred technical solution of the present invention: Step S2 includes the following steps:

[0019] S2.1, Key point extraction and descriptor generation: Use the sIFT algorithm to extract key points from two images that have been bent and collected by the bending scanning device and have been pre-processed in step s1 to remove the loss information in the image, and generate descriptors for each key point;

[0020] S2.2, Feature point matching: Calculate the Euclidean distance of all descriptors in the two images, and find the pair of descriptors with the smallest distance as the matching points;

[0021] S2.3, Calculate the displacement: By calculating the displacement of the coordinates of the successfully matched feature points in the X and Y dimensions, and determining the X displacement and Y displacement between the two images through multi-point redundancy;

[0022] S2.4, Set the source image and the target image: Set one of the two pre-processed images as the source image Is, and the other as the target image I t , and the fusion area is the effective area Ω of the source image;

[0023] S2.5, Calculate the gradient and boundary conditions: Calculate the gradient of the source image I s of Calculate the region boundary of the objective function I t on the transformation X, Y

[0024] S2.6, Solve the Poisson equation for image fusion: In the source image region Ω, solve the Poisson equation where Δ is the Laplace operator, is the divergence of the source image gradient, and apply the boundary condition I = I on the target image boundary t , and the fused image I can be obtained by solving this equation,

[0025] As a preferred technical solution of the present invention: In step S2.3, convert the motor rotation angle into the actual image transformation pixels to obtain the XY values of the two image transformations:

[0026] The optical fiber moving device is composed of a gear set consisting of a spur gear with an outer diameter L of 62 mm and a module of 22 mm, driven by an incremental motor with a Hall sensor. The angle r of the motor rotation is obtained from formula (2),

[0027] where t is the number of Hall sensor AB phase signal pulses, and T is the number of motor lines;

[0028] The coordinate transformation in the real state can be converted into the coordinate transformation in the image coordinates through formula (3),

[0029] where (x2, y2) are the image coordinates, (x1, y1) are the actual image coordinates, a, b, c, d are the internal parameters of the image, and e, f are the external parameters of the image.

[0030] As a preferred technical solution of the present invention: Step S3 specifically includes the following steps:

[0031] S3.1, Calculate the dark channel of the fused image through formula (4):

[0032] Among them, J represents the fused image, c represents the color channel, r, g, and b respectively correspond to different channels, and Ω(x) represents the local pixel block centered on the pixel point x;

[0033] S3.2, the dark channel J dark After converting to the LAB color space, the median value is taken as the estimated atmospheric light value A of the fused image;

[0034] S3.3, the estimated transmittance t of the image can be obtained through formula (5),

[0035] Among them, is the normalized Jdark, ω is a parameter, taking 0.95;

[0036] S3.4, the denoised image can be obtained through the calculation of formula (6):

[0037] The second object of the present invention is to provide a step-index fiber endoscope bending microscopic imaging loss recovery device for the problems in the prior art.

[0038] To this end, the above object of the present invention is achieved through the following technical solutions:

[0039] The step-index fiber endoscope bending microscopic imaging loss recovery device includes:

[0040] A laser that provides a light source, and the light beam output from the laser is transmitted into an isolator, and the isolator prevents the reflected light from returning to the laser.

[0041] The first lens and the second lens perform beam expansion processing for expanding the laser beam to provide sufficient light intensity for imaging;

[0042] The first mirror and the second mirror adjust the coupling position and direction of the light beam so that the light beam can accurately enter the optical splitter;

[0043] The optical splitter divides the expanded light into two arms, one arm is the reference light, and the other arm is the signal light, which is transmitted through the optical fiber;

[0044] The part of the reference arm includes a second variable optical attenuator, an eighth lens, a first polarization-maintaining optical fiber, and a fourth half-wave plate: the reference light is controlled in intensity by the second variable optical attenuator, and the output light of the first polarization-maintaining optical fiber is collimated by the eighth lens, and enters the beam combiner through the control of the fourth half-wave plate to be combined with the signal light;

[0045] The signal arm part involves a fourth lens, a second half-wave plate, a spatial light modulator, a polarizer, a third half-wave plate, a fifth lens, a beam deflector, a sixth lens, a seventh lens for aggregation, a quarter-wave plate for changing the polarization state, and a first microscope objective lens: The fourth lens is used to collimate the output beam of the first polarization-maintaining fiber, and after the polarization state is controlled by the second half-wave plate, it is irradiated onto the spatial light modulator; after the beam passes through the spatial light modulator, it is filtered by the polarizer, the polarization state is controlled by the third half-wave plate, and then it passes through the fifth lens, the beam deflector for beam combination, the sixth lens, the seventh lens for aggregation, the quarter-wave plate for changing the polarization state, and the first microscope objective lens, and finally is coupled into the step-index fiber;

[0046] The beam output from the step-index fiber is expanded by the second microscope objective lens and the ninth lens, and is combined with the beam of the reference arm in the combiner. The combined beam is finally combined by the tenth lens and captured and recorded by the camera;

[0047] The fiber bending device is a gear structure driven by two gears with outer diameters of 62 mm and 22 mm respectively, so that the step is smaller when the step-index fiber bends and scans for imaging;

[0048] The collection part includes a collection multimode fiber and a photomultiplier tube. The collection multimode fiber is responsible for collecting the optical signal emitted by the step-index fiber and irradiating on the surface of the imaging object, and transmitting it to the photomultiplier tube through the collection multimode fiber. The photomultiplier tube is responsible for converting the collected optical signal into an electrical signal.

[0049] Compared with the prior art, a method and device for recovering the loss of step-index fiber endoscope bending microscopic imaging of the present invention have the following beneficial effects: The present invention provides an image post-processing scheme to realize the recovery of the loss of step-index fiber endoscope bending microscopic imaging. Utilizing the inherent characteristics of fiber bending, the lost image information is recovered through the post-processing of the imaging image. By using a low-pass filter mask to eliminate redundant information, feature extraction and matching, and multi-point redundancy for moving image coordinate transformation, Poisson fusion to eliminate stitching traces, and dark channel prior method for denoising and other means working together, the image information can be recovered more comprehensively, solving the image recovery problem comprehensively from multiple angles, improving the quality and visual effect of the image, and changing the problem that the computational amount of other technologies for real-time transformation of the transmission matrix is large when dealing with high-resolution images or fast-moving objects and cannot meet the real-time requirement.

[0050] Through the fusion of frequency-domain low-pass filtering and feature matching, the lost central information of the image is successfully restored by fusing multiple preprocessed images. The central details of the image are effectively restored, and the image contrast is significantly improved. The high-frequency speckle information generated during the fiber bending process is removed, and the image details are clearer. Through the dark channel dehazing algorithm, the background blur is effectively removed, the image clarity and signal-to-noise ratio are significantly improved, and the overall image quality is comprehensively optimized. Image restoration can be achieved by processing the captured images without changing the existing correction system, without additional hardware devices and complex calibration systems, reducing the system complexity and cost, while also improving the reliability and stability, and having great application prospects in the field of high-precision imaging technology. Description of the Drawings

[0051] Figure 1 It is a schematic structural diagram of a device for loss recovery of endoscopic microscopic imaging under the bending deformation of a step-index fiber according to the present invention;

[0052] Figure 2 It is a schematic flow diagram of removing the foggy background noise brought by fiber imaging through the dark channel dehazing algorithm according to the present invention;

[0053] Figure 3 In Figure a, it is a curve graph of bending imaging of a fixed grid pattern using a step-index refraction fiber by a device for loss recovery of endoscopic microscopic imaging under the bending deformation of a step-index fiber according to the present invention;

[0054] Figure 3 In Figure b, it is a curve graph of the feature matching degree before and after the restoration and reconstruction of the optical character using the method for loss recovery of step-index fiber endoscopic bending microscopic imaging according to the present invention;

[0055] Figure 3 In Figure c (i), it is an imaging picture of a fixed grid collected at different angles; (ii) and (iii) are the images before and after reconstruction and restoration at different bending angles respectively;

[0056] In the accompanying drawings: 1 - laser; 2 - isolator; 3 - first lens; 4 - first mirror; 5 - second lens; 6 - second mirror; 7 - first half-wave plate; 8 - third lens; 9 - optical splitter; 10 - first variable optical attenuator; 11 - first polarization-maintaining fiber; 12 - fourth lens; 13 - second half-wave plate; 14 - spatial light modulator; 15 - first polarizer; 16 - third half-wave plate; 17 - fifth lens; 18 - beam deflector; 19 - sixth lens; 20 - seventh lens; 21 - quarter-wave plate; 22 - first microscopic objective lens; 23 - step-index fiber; 24 - fiber bending device; 25 - second variable optical attenuator; 26 - second polarization-maintaining fiber; 27 - eighth lens; 28 - fourth half-wave plate; 29 - second microscopic objective lens; 30 - ninth lens; 31 - beam combiner; 32 - tenth lens; 33 - camera; 34 - collecting multimode fiber; 35 - photomultiplier tube. Detailed implementation manners

[0057] The present invention will be further described in detail with reference to the accompanying drawings and specific embodiments.

[0058] The step-index fiber endoscope bending microscopic imaging loss recovery device provided by the present invention is a device for bending-scanning step-index fiber microscopic imaging, including:

[0059] A laser 1 configured to provide a light source for the system. The light beam output from the laser is transmitted into the isolator 2. The function of this isolator is to prevent the reflected light in the system from returning to the laser.

[0060] The first lens 3 and the second lens 5, which mainly perform beam expansion processing for expanding the laser beam.

[0061] The first mirror 4 and the second mirror 6, whose functions are to adjust the coupling position and direction of the light beam.

[0062] A first half-wave plate 7 for adjusting the splitting ratio of the light coupled into the optical splitter.

[0063] A third lens 8 for focusing the light beam of the first half-wave plate 7 onto the optical splitter 9.

[0064] An optical splitter 9 designed to split the expanded light into two paths: a reference arm and a signal arm. The optical splitter divides the expanded light into two arms, one arm is the reference light, and the other arm is the signal light, which is transmitted through the optical fiber.

[0065] The part of the reference arm includes:

[0066] The reference light is controlled in intensity by the second variable optical attenuator 25, enters the reference arm optical path while maintaining the polarization state of the light unchanged through the second polarization-maintaining fiber 26, and the output light of the first polarization-maintaining fiber 11 is collimated by the eighth lens 27 and enters the beam combiner after being controlled by the fourth half-wave plate 28;

[0067] The signal arm part involves:

[0068] The first variable optical attenuator 10 is used to adjust the light intensity output from the optical splitter to ensure that the light intensity entering the first polarization-maintaining fiber 11 is appropriate.

[0069] The fourth lens 12 is used to collimate the output light beam of the first polarization-maintaining fiber 11, and after the polarization state is controlled by the second half-wave plate 13, it irradiates the spatial light modulator 14;

[0070] After the light beam passes through the spatial light modulator 14, it is filtered by the first polarizer 15 to control the polarization state of the light entering the third half-wave plate 16, and then passes through the fifth lens 17, the beam shifter 18 for beam combination, the sixth lens 19, the seventh lens 20 for focusing, the quarter-wave plate 21 to change the polarization state, and the first microscopic objective lens 22, and finally is coupled into the step-index fiber 23;

[0071] The light beam output from the step-index fiber is expanded by the second microscopic objective lens 29 and the ninth lens 30, and is combined with the light beam of the reference arm in the beam combiner 31. The combined light beam is finally combined by the tenth lens 32 and captured and recorded by the camera 33.

[0072] The fiber bending device 24 is a gear structure driven by two gears with outer diameters of 62 mm and 22 mm respectively, and this structure can make the step of the step-index fiber bending and scanning imaging smaller.

[0073] The collection part consists of a multimode collection fiber 34 and a photomultiplier tube 35. The multimode collection fiber is responsible for collecting the optical signal emitted by the step-index fiber and irradiating on the surface of the imaging object, and transmitting it to the photomultiplier tube through the multimode collection fiber. The photomultiplier tube is responsible for converting the collected optical signal into an electrical signal.

[0074] Based on the above device, the present invention provides a method for recovering the loss of step-index fiber endoscope bending microscopic imaging. Specifically, it includes:

[0075] The present invention proposes a method for recovering the loss of step-index fiber endoscope bending microscopic imaging. By using the characteristic that the image loss of the step-index fiber spreads from the inside to the outside gradually during endoscope microscopic imaging, the external image that is not easily lost during the bending of the step-index fiber endoscope imaging is scanned, stitched, and fused to be restored to the non-loss state. The specific solution is divided into three parts:

[0076] The first part is pre - processing. The purpose of the pre - processing in this solution is to find the lost part of the image and use a mask to remove the lost part from the image.

[0077] The second part is image restoration. This part calculates the x - direction and y - direction positions of the changes between the two images by feature - matching displacement of the two displacement pictures, and then stitches and fuses the two images together through Poisson fusion to repair the lost part of the image.

[0078] The third part is background denoising. This solution uses the dark - channel defogging algorithm to remove the fog - like background noise brought by fiber - optic imaging.

[0079] As Figure 2 shown, before image acquisition, the optical path of the signal arm needs to be collimated. In the present invention, the 9 - beam splitter divides the expanded light into two beams, where the reference arm is 2% and the signal arm is 98%. The optical signal is combined by the 31 - beam combiner and recorded by the 33 - camera. The optical signals of the signal arm and the reference arm are combined and then recorded. Among them, the optical transmission control device in the optical fiber includes a step - index fiber and a spatial light modulator. The spatial light modulator adjusts the amplitude, phase, and polarization state of the light beam entering the step - index fiber; the optical signal combination and recording device records the input and output information at both ends of the step - index fiber, establishes the optical field correspondence between the input and output of the step - index fiber, and constructs the transmission matrix of the step - index fiber accordingly; uses the transmission matrix of the step - index fiber to precisely control the spatial light modulator to make the amplitude and phase of the output light of the step - index fiber, so as to generate a focal point at the fiber output end; by generating multiple focal points in the far - field of the fiber and performing point - by - point scanning in sequence, the endoscope imaging picture of the corresponding object can be generated according to the returned light intensity collected by the photomultiplier tube.

[0080] In step S1 of the present invention, specifically, the picture data collected by scanning the step - index fiber is Fourier - transformed to obtain the frequency - domain information corresponding to the picture data. Since the image center information will be gradually lost and covered by high - frequency speckle information when the step - index fiber is bent for imaging, redundant information generated during the bending process of the step - index fiber can be removed by using a circular mask to filter out the high - frequency information in the picture.

[0081] Step S2 for image restoration is specifically as follows: For the two pictures after pre - processing, the SIFT algorithm is used to extract feature points (keypoints), and a descriptor is generated for each keypoint. After the keypoints in the two images are extracted, the Euclidean distance between the descriptors of the two images is calculated for feature - point matching. Finally, the X and Y - dimension displacements of the two images are calculated by calculating the coordinates of the feature points with successful feature - point matching, and the X displacement and Y displacement of the transformation between the two images are determined through multi - point redundancy. One of the two pre - processed images is set as the source image Is, and the other is set as the target image I t, the fusion region is the effective region Ω of the source image. Calculate the gradient of the source image I s of Calculate the regional boundary of the objective function I t on the transformations X, Y Within the source image region Ω, solve the Poisson equation where Δ is the Laplacian operator, is the divergence of the source image gradient. Impose the boundary condition I = I t on the boundary of the target image. By solving this equation, the fused image I can be obtained.

[0082] Perform feature matching and displacement calculation on the fused image and the new image, then align and fuse these two images by Poisson fusion, update the fused image, and repeat this step until there is no new image, and output the final fused image to repair the lost part of the image.

[0083] Specifically for step S3 background denoising, perform dark channel calculation on the fused image through formula (4), where J represents the fused image, c represents the color channel, r, g, b respectively correspond to different channels, and Ω(x) represents the local pixel block centered on pixel point x. The dark channel J dark After being converted to the LAB color space, take the median as the estimated atmospheric light value A of the fused image. The estimated transmittance t of the image can be obtained through formula (5), where is the normalized J dark , ω is a parameter, taking 0.95. Finally, the denoised image I can be obtained through the calculation of formula (6).

[0084] Figure 3 In figure a, it is the curve graph of the bending imaging of the grid fixed pattern by using the step refractive index optical fiber with the device of the present invention. Among them, through the optical fiber bending device, the step refractive index optical fiber is bent from 0 to 90 degrees, and its feature matching degree shows an almost exponential decay. Figure 3 In figure c (i), it is the imaging pictures collected for the fixed grid at different angles. Figure 3 In figure b, it is the curve graph of the feature matching degree before and after the restoration and reconstruction of the optical characters by using the restoration method of the present invention. It can be seen from the image that the feature matching degree of the restored image by using the solution of the present invention is generally increased by 40% compared with the original image. Figure 3 In figure c (ii) and (iii), they are the images before reconstruction and restoration and the images after reconstruction and restoration at different bending angles respectively.

[0085] The step-index fiber endoscope bending microscopic imaging loss recovery method and device of the present invention adopt an image post-processing scheme to recover the step-index fiber endoscope bending microscopic imaging loss. Utilizing the phenomenon that the loss information expands from the center to the outside when the step-index fiber bends, the collected image is subjected to frequency-domain low-pass filtering using the fast Fourier transform, removing the redundant information lost due to the bending of the step-index fiber, removing the high-frequency speckle information generated during the fiber bending process, making the image details clearer and the resolution improved. Using multiple images collected by bending and scanning with a fiber bending device, through the method of feature matching, the transformations in the X and Y dimensions of the corresponding images are found. The multiple images are fused and stitched using the Poisson fusion method, realizing the recovery of the loss information in the center of the image.

[0086] Example 1

[0087] The second part of obtaining the XY transformation of two images through feature matching can be replaced by the following part:

[0088] The present invention uses a fiber optic moving device composed of a gear set consisting of a spur gear with an outer diameter L of 62 mm and a module of 22 mm, which is driven by an incremental motor with a Hall sensor. Therefore, the rotation angle r of the motor can be obtained from formula (2), where t is the number of Hall sensor AB-phase signal pulses and T is the number of motor lines. The coordinate transformation in the real state can be converted into the coordinate transformation in the image coordinates through formula (3), where (x2, y2) are the image coordinates, (x1, y1) are the image coordinates, a, b, c, d are the internal parameters of the image, and ef are the external parameters of the image. Therefore, the rotation angle of the motor can be calculated and converted into the actual image transformation pixels. Thus, the XY values of the two image transformations are obtained.

[0089] (1) The present invention utilizes the imaging loss characteristics when the step-index fiber bends for endoscope imaging, and uses the effective edge information to perform real-time imaging to recover the lost part of the collected image during movement.

[0090] (2) The present invention uses the means of eliminating loss information by using a low-pass filter mask for the loss part of the step-index fiber.

[0091] (3) The present invention calculates the moving image coordinate transformation by using feature extraction and matching and the method of multi-point redundant decision.

[0092] (4) The present invention uses the Poisson fusion method for image stitching and fusion to eliminate the image stitching traces.

[0093] (5) The present invention uses the dark channel prior method for post-processing to denoise the image.

[0094] The above specific embodiments are used to explain the present invention and are only the preferred embodiments of the present invention, rather than limiting the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and scope of the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A method for recovering the loss of step-refractive-index optical fiber endoscope bending microscopic imaging, comprising the following steps: S1: Find and remove the image loss part in the step-index fiber endoscope scanning image: Use fast Fourier transform to convert the image information into frequency domain information, identify the high-frequency variables in the frequency domain information, capture the image information of the corresponding frequency domain information in the step-index fiber endoscope scanning image at the bending part, locate the image loss part, use mask to remove the image redundant part, restore the image center information in the step-index fiber scanning image, and remove the high-frequency speckle information caused by bending imaging; S2: Image restoration, selecting the two displacement images processed in step S1 for feature matching and displacement calculation, and then aligning and fusing the two images by Poisson fusion to obtain a fused image; The fused image is matched with the new image for features and displacement calculation, and the two images are aligned and fused by Poisson fusion. The fused image is updated and repeated until there is no new image. The final fused image is output to repair the lost part of the image. S3: Background denoising: the transmittance is estimated by calculating the dark channel and the background illumination value, and the image is denoised according to the transmittance to obtain the denoised image.

2. The step-index optical fiber endoscopic bending microscopic imaging loss recovery method according to claim 1, characterized in that: In step S1, the image data collected by scanning the step-index optical fiber is subjected to Fourier transformation to obtain frequency domain information corresponding to the image data, and the image loss portion is identified through frequency domain analysis.

3. The step-index optical fiber endoscopic bending microscopic imaging loss recovery method according to claim 1, characterized in that: A circular mask is used to filter out high-frequency information in the image to remove redundant information generated during the bending process of the step-index fiber.

4. The step-index optical fiber endoscopic bending microscopic imaging loss recovery method according to claim 1, characterized in that: Step S2 includes the following steps: S2.1, key point extraction and descriptor generation: using the SIFT algorithm, key point extraction is performed on the two images that are collected by bending through the bending scanning device and have been pre-processed in step S1 to remove the loss information in the images, and a descriptor of each key point is generated; S2.2, feature point matching: calculate the Euclidean distance of all descriptors in the two images and find the pair of descriptors with the smallest distance as the matching point; S2.3, calculating the displacement: by calculating the displacement in the X and Y dimensions of the coordinates of the successfully matched feature points, the X displacement and Y displacement between the two images are determined through multi-point redundancy; S2.4, set the source image and the target image: set one of the two pre-processed images as the source image I s , and the other one is set as the target image I t , the fusion area is the valid area Ω of the source image; S2.5, Calculate gradients and boundary conditions: Calculate source image I s Gradient Calculate the objective function I t Region boundaries on transform X, Y S2.6, solve Poisson's equation and perform image fusion: Solve Poisson's equation in the source image region Ω where Δ is the Laplace operator, is the divergence of the source image gradient, and imposes a boundary condition I = I within the target image boundary. t , by solving this equation, we can get the fused image I, 5. The step-index optical fiber endoscopic bending microscopic imaging loss recovery method according to claim 4, characterized in that: In step S2.3, the motor rotation angle is converted into actual image transformation pixels to obtain the XY values ​​of the two image transformations: The optical fiber moving device is composed of a gear set with an outer diameter L of 62mm and a single-mode gear L of 22mm. The drive is an incremental motor with a Hall sensor. The motor rotation angle r is obtained by formula (2). Among them, t is the number of AB phase signal pulses of the Hall sensor, and T is the number of motor lines; The coordinate transformation in the real state can be converted into the coordinate transformation in the image coordinates by formula (3): Among them, (x2, y2) is the image coordinate, (x1, y1) is the actual image coordinate, a, b, c, d are the image intrinsic parameters, and ef is the image extrinsic parameter.

6. The step-index optical fiber endoscopic bending microscopic imaging loss recovery method according to claim 1, characterized in that: Step S3 specifically includes the following steps: S3.1, calculate the dark channel of the fused image using formula (4): I dark (x)=min y∈Ω(x) min c∈{r，g，b} I c (y) (4) Where J represents the fused image, c represents the color channel, r, g, and b correspond to different channels respectively, and Ω(x) represents the local pixel block centered on the pixel point x; S3.2, Dark Channel J dark After converting to LAB color space, the median is taken as the estimated atmospheric light value A of the fused image; S3.3, the estimated transmittance t of the image can be obtained by formula (5), in, is the normalized J dark , ω is a parameter, taking 0.95; S3.4, the denoised image can be obtained by calculating formula (6):

7. A device for recovering the loss of step-index optical fiber endoscope bending microscopic imaging, characterized in that: include: The laser provides the light source. The light beam output from the laser is transmitted to the isolator, which prevents the reflected light from returning to the laser. The first lens and the second lens perform beam expansion processing for expanding the laser beam to provide sufficient light intensity for imaging; The first reflector and the second reflector adjust the coupling position and direction of the light beam so that the light beam can accurately enter the beam splitter; The beam splitter divides the expanded light into two arms, one arm is the reference light, and the other arm is the signal light, which are transmitted through the optical fiber; the reference arm includes the second variable optical attenuator, the eighth lens, the first polarization-maintaining fiber and the fourth half-wave plate: the reference light is intensity-controlled by the second variable optical attenuator, and the output light of the first polarization-maintaining fiber is collimated by the eighth lens, and is controlled by the fourth half-wave plate to enter the beam combiner and be combined with the signal light; The signal arm part involves a fourth lens, a second half-wave plate, a spatial light modulator, a polarizer, a third half-wave plate, a fifth lens, a beam shifter, a sixth lens, a seventh lens, a quarter-wave plate to change the polarization state, and a first microscope objective lens: the fourth lens is used to collimate the output light beam of the first polarization-maintaining optical fiber, and irradiate the output light beam to the spatial light modulator after the output light beam passes through the second half-wave plate to control the polarization state; after the output light beam passes through the spatial light modulator, the output light beam is filtered by the polarizer, the polarization state is controlled by the third half-wave plate, and then the output light beam passes through the fifth lens, a beam shifter to combine the beam, the sixth lens, the seventh lens to aggregate the beam, the quarter-wave plate to change the polarization state, and the first microscope objective lens, and finally couples the output light beam into the step-refractive-index optical fiber; The light beam output from the step-refractive-index optical fiber is expanded by the second microscope objective lens and the ninth lens, and is combined with the light beam from the reference arm in the beam combiner. The combined light beam is finally combined by the tenth lens and captured and recorded by the camera. The fiber bending device is a gear structure driven by two gears with outer diameters of 62mm and 22mm respectively, which makes the step size smaller when bending and scanning the step-index fiber. The collecting part includes a collecting multimode optical fiber and a photomultiplier tube, wherein the collecting multimode optical fiber is responsible for collecting the light signal emitted by the step refractive index optical fiber and irradiating the surface of the imaging object, and transmitting it to the photomultiplier tube through the collecting multimode optical fiber. The photomultiplier tube is responsible for converting the collected light signal into an electrical signal.

Images