A method for improving the imaging quality of a multimode fiber endoscope based on correlated imaging

Abstract

The present invention discloses a method for improving the imaging quality of a multimode fiber endoscope based on correlated imaging, belonging to the technical field of correlated imaging. First, a multimode fiber endoscope is used to emit a modulated light field to irradiate a target area, and a bucket value of the light field after passing through the target area is collected by a detector; then the modulated light field and the bucket value are input into a reconstruction algorithm, and the alternating direction method of multipliers is used to reconstruct the image in combination with the total variation denoising algorithm and low-rank constraint, and finally a reconstructed image is obtained. The present invention has clearer imaging of the target area, and the application has generalization, and can be used for imaging processing under various undersampling conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of correlated imaging, and particularly to a method for improving the imaging quality of a multimode fiber endoscope based on correlated imaging. Background Art

[0002] As a representative of medical device visualization, endoscope imaging technology is widely used in the medical field. Doctors can observe the pathological conditions of patients' tissues and organs through endoscopes and make diagnoses of patients' conditions in a timely manner.

[0003] Based on the characteristic of parallel transmission of multiple guided wave modes in multimode fibers, a single multimode fiber has been widely used to transmit two-dimensional image information, and endoscope technology based on multimode fibers has also received extensive attention.

[0004] Improving the imaging quality of endoscopes has always been a key research content. In recent years, the booming correlated imaging technology has brought a new imaging method to endoscope imaging technology. In the correlated imaging process, the reference light path does not pass through the object, and a detector with spatial resolution is not used in the light path for illuminating the object. It is impossible to directly image with any single light field alone, but the object information can be reconstructed by calculating the correlation characteristics between light fields. Therefore, correlated imaging has the advantage of non-local imaging and has important application potential in the biomedical field, especially in scenarios where the imaging light path is relatively complex and tortuous, such as using endoscopes to image and analyze tissues in living organisms.

[0005] Traditional second-order ghost imaging technology reconstructs the information of an object by calculating the second-order correlation between light fields. However, limited by the Nyquist sampling theorem, traditional second-order ghost imaging has certain requirements for the sampling rate. However, when using an endoscope for detection, it is generally not possible to directly align with the pathological site, and at the same time, it is necessary to inspect the pathological site as comprehensively as possible. The endoscope examination generally takes 15 to 30 minutes. Assuming that the size of the image to be reconstructed is 1000*1000 and the sampling frequency is 1 kHz, at least 1000 s is required to completely restore the target information. Therefore, to ensure a more comprehensive examination of the pathological site within a limited time, it is necessary to reduce the sampling rate for each imaging, and it is difficult to meet the requirements of traditional second-order ghost imaging for the sampling rate. Therefore, in such an undersampling scenario, the image quality calculated using traditional second-order ghost imaging technology is often poor. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for improving the imaging quality of a multimode fiber endoscope based on correlated imaging. Based on ghost imaging technology, using the alternating direction method of multipliers, combined with the total variation denoising algorithm and low-rank constraint, it can effectively remove the noise in the image, and at the same time, it can also reconstruct a better image in the scenario where it is difficult to achieve a high sampling rate in endoscope imaging.

[0007] A method for improving the imaging quality of a multimode fiber endoscope based on correlated imaging, comprising the following steps:

[0008] Step 1, use the multimode fiber endoscope to emit a modulated light field to irradiate the target area, and use a detector to collect the bucket value of the light field after passing through the target area;

[0009] ① The calculation process of the bucket value is as follows:

[0010] The size of the speckle image obtained by the light field transmitted through the multimode fiber is f*f, and the predetermined number of samples of the modulated light field is g, that is, the light field information contains g speckle images;

[0011] Read in the object information matrix and all light field information matrices with a size of f*f, perform dot multiplication on the object information matrix and each light field information matrix to obtain g new matrices of f*f.

[0012] Add up all the values on each new matrix to obtain g values, combine these values into a column vector A of size g*1, and subtract the average value of A from each value on A. The resulting column vector is the bucket value input b of the reconstruction algorithm.

[0013] ② The calculation process of the modulated light field is as follows:

[0014] Change each light field information matrix into a row vector of 1*(f*f), and recombine these row vectors into a matrix B of size g*(f*f); calculate the mean value of each column of B to obtain a row vector containing these average values, and subtract this row vector from each row of B. The resulting matrix is the light field input R of the reconstruction algorithm.

[0015] Step 2, input the modulated light field and the bucket value into the reconstruction algorithm, optimize the reconstruction algorithm, and reconstruct the image based on the optimized reconstruction algorithm to finally obtain the reconstructed image;

[0016] Use the alternating direction multiplier method, combine the total variation denoising algorithm and the low-rank constraint to reconstruct the image. The specific process is as follows:

[0017] Step 201, the optimization equation of the reconstruction algorithm is:

[0018]

[0019] Rewrite the above formula as:

[0020]

[0021] Subsequently, introduce the constraint condition Then the augmented Lagrangian function of the optimization equation is:

[0022]

[0023] Let \(p = \mu / \delta\), and simplify the above function to:

[0024]

[0025] where, \(\|\cdot\|_1\) is the \(L_1\) norm, \(\|\cdot\|\) * is the nuclear norm, \(\|\cdot\|_2\) is the \(L_2\) norm, \(x\) is the image to be reconstructed, \(\lambda\) is the regularization parameter, is the gradient operator, \(\mu\) is the Lagrange multiplier, \(\delta>0\) is the Lagrange penalty term, \(R\) and \(b\) are the light field and bucket value obtained by preprocessing respectively.

[0026] Step 202, update the image \(x\) to be reconstructed through low-rank constraint. The specific process is as follows:

[0027] Take the partial derivative of the image \(x\) to be reconstructed in the function, temporarily remove the derivative of the nuclear norm in the derivative, and find \(x\); then perform singular value decomposition on \(x\), perform low-rank constraint on the decomposed matrix, and then update \(x\);

[0028] Step 203, update the parameters \(d\) and \(p\) in turn based on the updated \(x\);

[0029] Update the parameter \(d\): Keep \(x\) and \(p\) unchanged, and let to update \(d\);

[0030] Update the parameter \(p\): Keep the updated \(x\) and \(d\) unchanged, and let to update \(p\).

[0031] Step 204, each time \(x\), \(d\), and \(p\) are all updated, the iteration count is incremented by 1, and it is judged whether the iteration count reaches the upper limit. If so, the optimization process ends and the reconstructed image is output; otherwise, return to Step 202 and continue the iteration until the iteration count reaches the upper limit.

[0032] The advantages and beneficial effects of the present invention are as follows:

[0033] (1) The present invention is an innovative algorithm for correlated imaging. By means of the total variation denoising algorithm and low-rank regularization, it can effectively remove the noise of the image and make the imaging of the target area clearer.

[0034] (2) Under the condition of undersampling, sometimes it is difficult for the endoscope to directly aim at the lesion area and a certain calibration time is required; at the same time, the detection of the lesion area is multi-angle and multiple imaging is required; however, the inspection time of the endoscope is limited. Therefore, it is necessary to limit the sampling rate of each imaging to ensure that the lesion site can be inspected as comprehensively as possible within a limited time. The high efficiency of the alternating direction multiplier method also makes the imaging under the condition of undersampling more feasible.

[0035] (3) The application of the present invention has generalization, not limited to the imaging processing of diseased tissues, but also capable of clarifying the imaging of various targets under undersampling conditions. Description of the Drawings

[0036] Figure 1 is the overall process of multimode fiber endoscope imaging based on correlation imaging of the present invention;

[0037] Figure 2 is the specific process of the reconstruction algorithm in the present invention;

[0038] Figure 3 is the original image of the polyp tissue in Example 1;

[0039] Figure 4 is the restored image when applying the traditional second-order correlation ghost imaging algorithm at a sampling rate of 1 / 16 in Example 1;

[0040] Figure 5 is the restored image when applying the method of the present invention at a sampling rate of 1 / 16 in Example 1;

[0041] Figure 6 is the original image of the house in Example 2;

[0042] Figure 7 is the restored image when applying the traditional second-order correlation ghost imaging algorithm at a sampling rate of 1 / 16 in Example 2;

[0043] Figure 8 is the restored image when applying the method of the present invention at a sampling rate of 1 / 16 in Example 2. Detailed Embodiment

[0044] The present invention will be further described in detail below in conjunction with the drawings and embodiments.

[0045] As Figure 1 shown, the specific process is to first use the endoscope to emit a modulated light field to irradiate the lesion area to be examined, then use a bucket detector to collect the light field containing the target information, and then preprocess the light field and the bucket value obtained by the detector and use it as the input of the algorithm, and then use the algorithm in the present invention to restore the image.

[0046] As Figure 2 shown, the optimization equation of the reconstruction algorithm can be written as follows:

[0047] arg min{λ||Dx||1+||x|| *}s.t.Rx = b

[0048] Rewrite the above formula as:

[0049]

[0050] Subsequently, the constraint condition Dx = d is introduced, and the augmented Lagrangian function of the optimization equation is written:

[0051]

[0052] Let p = μ / δ, and simplify the above function to:

[0053]

[0054] where, ||*||1 is the L1 norm, ||*|| * is the nuclear norm, ||*||2 is the L2 norm, x is the image to be reconstructed, λ is the regularization parameter, D is the total variation operator, μ is the Lagrange multiplier, δ>0 is the Lagrange penalty term, R and b are the light field and bucket value obtained by preprocessing respectively.

[0055] Optimization process: Take the partial derivative of the image x to be reconstructed in the function, temporarily remove the derivative of the nuclear norm in the derivative, find x, perform singular value decomposition on x, perform low-rank constraint on the decomposed matrix and then update x, then keep x and p unchanged, let to update d, and finally keep the updated x and d unchanged, and let p = p+(Dx - d) to update p. Repeat the above steps before reaching the number of iterations.

[0056] Example 1:

[0057] This example mainly studies the performance of the present invention in restoring the lesion site. In this example, a polyp tissue image with a resolution of 64*64 is used as the object to be measured. The light field uses the speckle image obtained by multi-mode fiber transmission, with the same size of 64*64. When recording and pre-calibrating the modulated light field, the set predetermined sampling number is 256, that is, the light field information contains 256 speckle images. That is, the sampling rate is 1 / 16.

[0058] Read the polyp tissue image and each speckle image to obtain an object information matrix and a light field information matrix with a size of 64*64. Multiply the object information matrix with each light field matrix to obtain 256 new matrices of 64*64. Add up all the values on each new matrix to obtain 256 values, combine these values into a column vector A of 256*1 size, and subtract the average value of A from each value on A. The obtained column vector is the bucket value input b of the reconstruction algorithm.

[0059] Subsequently, change each light field matrix to a size of 1*4096, and recombine these row vectors into a matrix B of 256*4096 size; calculate the mean value of each column of B to obtain a row vector containing these mean values, and subtract this row vector from each row of B. The obtained matrix is the light field input R of the reconstruction algorithm.

[0060] Subsequently, the traditional second-order correlation imaging algorithm and the algorithm in the present invention are respectively used to restore the image. The signal-to-noise ratio (SNR) is used as an index to evaluate the noise reduction ability of the algorithm. The higher the SNR, the less noise is generated; the structural similarity (SSIM) is used as an index to evaluate the ability of the algorithm to restore the image under undersampling conditions. The higher the structural similarity, the more information is restored.

[0061] The formula for the signal-to-noise ratio is

[0062]

[0063] where (x, y) represents the pixel coordinates in the image, O(x, y) is the pixel value of the original image of the object to be measured at (x, y), and S(x, y) is the pixel value of the image restored by the algorithm at (x, y).

[0064] The formula for the structural similarity of the image is

[0065]

[0066] where the subscripts x and y respectively represent the images to be compared, is the average brightness of the image; is the standard deviation of the pixel values; is the covariance of the pixel values of the two images. C1 = (K1L) 2 , C2 = (K2L) 2 , K1 is usually taken as 0.01, K2 is usually taken as 0.03, and L is the dynamic range of the gray level, which is taken as 255 in the embodiments of the present invention.

[0067] As Figure 3 shown, it is the original image of the polyp tissue adopted in Example 1. As Figure 4 shown, it is the restored image when applying the traditional second-order correlation ghost imaging algorithm at a sampling rate of 1 / 16 in Example 1, and its signal-to-noise ratio is 4.246. As Figure 5 shown, it is the restored image when applying the algorithm of the present invention at a sampling rate of 1 / 16 in Example 1, and its signal-to-noise ratio is 16.8843. Figure 3 and Figure 4 The structural similarity is 0.081; Figure 3 and Figure 5 The structural similarity is 0.625. It can be seen that under undersampling conditions, the traditional ghost imaging technology can no longer restore the image, and there is more noise in the restored image; while the algorithm of the present invention can still restore a better image, and its noise reduction ability is significantly improved compared with the ghost imaging technology.

[0068] Example 2:

[0069] This embodiment mainly studies the generalization of the algorithm in the present invention. In this embodiment, a house image with a resolution of 64*64 is used as the object to be measured. The light field uses a speckle image obtained by multimode fiber transmission, with the same size of 64*64. When recording and pre-calibrating the modulated light field, the set predetermined sampling number is 256, that is, the light field information includes 256 speckle images. That is, the sampling rate is 1 / 16.

[0070] The input of the algorithm is obtained using the same steps as in Embodiment 1, and the traditional second-order correlation imaging algorithm and the algorithm in the present invention are used to restore the image. The signal-to-noise ratio (SNR) and structural similarity (SSIM) are used as indicators to evaluate the algorithm.

[0071] As Figure 6 shown, it is the original house image used in Example 2. As Figure 7 shown, it is the restored image when applying the traditional second-order correlation ghost imaging algorithm at a sampling rate of 1 / 16 in Example 2, and its signal-to-noise ratio is 6.3851. As Figure 8 shown, it is the restored image when applying the present invention at a sampling rate of 1 / 16 in Example 2, and its signal-to-noise ratio is 14.2874. Figure 6 and Figure 7 The structural similarity is 0.103; Figure 6 and Figure 8 The structural similarity is 0.503. It can be seen that even if it is not biological tissue, the algorithm of the present invention can still restore the image, and the denoising ability is still improved, having a certain generalization.

Claims

1. A method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging, characterized in that: The steps include: Step 1: Use a multimode fiber endoscope to emit a modulated light field to illuminate the target area, and use a detector to collect the bucket value of the light field after passing through the target area; Step 2: input the modulated light field and the bucket value into the reconstruction algorithm, optimize the reconstruction algorithm, reconstruct the image based on the optimized reconstruction algorithm, and finally obtain the reconstructed image; The image is reconstructed using the alternating direction multiplier method, combined with the total variation denoising algorithm and low-rank constraints. The specific process is as follows: Step 201, the optimization equation of the reconstruction algorithm is: Rewrite the above formula as: Then the constraints are introduced , then the augmented Lagrangian function of the optimization equation is: make , simplifying the above function to: in, is the L1 norm, is the nuclear norm, is the L2 norm, is the image to be reconstructed, λ is the regularization parameter, is the gradient operator, μ is the Lagrangian multiplier, δ>0 is the Lagrangian penalty term, R and b are the preprocessed light field and bucket value respectively; Step 202, the image x to be reconstructed is updated by using the low-rank constraint. The specific process is as follows: Calculate the partial derivative of the image x to be reconstructed in the function, temporarily remove the derivative of the nuclear norm in the derivative, and calculate x; then perform singular value decomposition on x, and update x after applying low-rank constraints to the decomposed matrix; Step 203, updating parameters d and p in sequence based on the updated x; Update parameter d: Keep x and p unchanged, let To update d; Update parameter p: Keep the updated x and d unchanged, let To update p; In step 204, each time x, d, and p are all updated, the number of iterations is increased by 1, and it is determined whether the number of iterations reaches the upper limit. If so, the optimization process ends and the reconstructed image is output; otherwise, return to step 202 and continue to iterate until the upper limit of the number of iterations is reached.

2. According to the method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging according to claim 1, it is characterized in that: The calculation process of the bucket value is: The size of the speckle image obtained by transmitting the light field through the multimode optical fiber is f*f, and the predetermined sampling number of the modulated light field is g, that is, the light field information contains g speckle images; Read in the object information matrix and all light field information matrices of size f*f, perform dot multiplication between the object information matrix and each light field information matrix, and obtain g new matrices of f*f; Add up all the values ​​on each new matrix to get g values, combine these values ​​into a column vector A of size g*1, subtract the average value of A from each value on A, and the resulting column vector is the bucket value input b of the reconstruction algorithm.

3. According to the method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging according to claim 2, it is characterized in that: The calculation process of the modulated light field is: Convert each light field information matrix into a row vector of 1*(f*f), and recombine these row vectors into a matrix B of size g*(f*f); calculate the average of each column of B to obtain a row vector containing these average values, and subtract each row of B from this row vector. The resulting matrix is ​​the light field input R of the reconstruction algorithm.

4. According to the method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging according to claim 1, it is characterized in that: Undersampled endoscopic imaging image processing for generalization.

Images