Method for improving imaging quality of multimode fiber endoscope based on correlated imaging

Abstract

The invention discloses a multi-mode fiber endoscope imaging quality improving method based on correlated imaging, and belongs to the technical field of correlated imaging. The method comprises the following steps of: emitting a modulated light field to irradiate a target area by using a multimode fiber endoscope, and collecting a barrel value of the light field passing through the target area by using a detector; and then inputting the modulation light field and the barrel value into a reconstruction algorithm, reconstructing the image by using an alternating multiplier method in combination with a total variation denoising algorithm and low-rank constraint, and finally obtaining a reconstructed image. According to the method, imaging of the target area is clearer, application has generalization, and the method can be used for imaging processing under various under-sampling conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of correlation imaging, and in particular to a method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging. Background Art

[0002] Endoscopic imaging technology, as a representative of medical device visualization, is widely used in the medical field. Doctors can observe the pathological conditions of the patient's tissues and organs through an endoscope and make a timely diagnosis of the patient's condition.

[0003] Based on the characteristics of parallel transmission of multiple waveguide modes in multimode optical fiber, a single multimode optical fiber has been widely used to transmit two-dimensional image information, and endoscope technology based on multimode optical fiber has also received widespread attention.

[0004] Improving the quality of endoscopic imaging has always been a key research topic. The booming correlation imaging technology in recent years has brought a new imaging method to endoscopic imaging technology. The reference light path in the correlation imaging process does not pass through the object, and the light path used to illuminate the object does not use a detector with spatial resolution. Any light field alone cannot directly form an image, but the object information can be reconstructed by calculating the correlation features between the light fields. Therefore, correlation imaging has the advantage of non-localized imaging, and has important application potential in the biomedical field, especially in scenarios where the imaging light path is complex and tortuous, such as using an endoscope to perform imaging analysis on tissues in vivo.

[0005] Traditional second-order ghost imaging technology reconstructs the information of an object by calculating the second-order correlation between light fields. However, due to the limitations of the Nyquist sampling theorem, traditional second-order ghost imaging has certain requirements for the sampling rate. However, when using an endoscope for detection, it is generally not possible to directly aim at the lesion, and it is also necessary to examine the lesion as comprehensively as possible. Endoscopic examinations generally take 15 to 30 minutes. Assuming that the size of the image to be reconstructed is 1000*1000 and the sampling frequency is 1khz, it takes at least 1000s to fully restore the target information. Therefore, in order to ensure a more comprehensive inspection of the lesion within a limited time, it is necessary to reduce the sampling rate of each imaging, which makes it difficult to ensure that traditional second-order ghost imaging has certain requirements for the sampling rate. Therefore, in this under-sampling scenario, the image quality calculated using traditional second-order ghost imaging technology is often poor. Summary of the invention

[0006] The purpose of the present invention is to provide a method for improving the imaging quality of a multimode fiber optic endoscope based on correlation imaging. Based on ghost imaging technology, it utilizes the alternating multiplier method, combines the total variation denoising algorithm and the low-rank constraint, and can effectively remove the noise in the image. At the same time, it can also reconstruct a better image in the scenario of endoscopic imaging where it is difficult to achieve a high sampling rate.

[0007] A method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging comprises the following steps:

[0008] Step 1: Use a multimode fiber endoscope to emit a modulated light field to illuminate the target area, and use a detector to collect the bucket value of the light field after passing through the target area;

[0009] ①The calculation process of bucket value is:

[0010] The size of the speckle image obtained by transmitting the light field through the multimode optical fiber is f*f, and the predetermined sampling number of the modulated light field is g, that is, the light field information contains g speckle images;

[0011] Read in the object information matrix and all light field information matrices, both of which are of size f*f, and perform dot multiplication between the object information matrix and each light field information matrix to obtain g new matrices of f*f.

[0012] Add up all the values ​​on each new matrix to get g values, combine these values ​​into a column vector A of size g*1, subtract the average value of A from each value on A, and the resulting column vector is the bucket value input b of the reconstruction algorithm.

[0013] ②The calculation process of the modulated light field is:

[0014] Convert each light field information matrix into a row vector of 1*(f*f), and recombine these row vectors into a matrix B of size g*(f*f); calculate the average of each column of B to obtain a row vector containing these average values, and subtract each row of B from this row vector. The resulting matrix is ​​the light field input R of the reconstruction algorithm.

[0015] Step 2: input the modulated light field and the bucket value into the reconstruction algorithm, optimize the reconstruction algorithm, reconstruct the image based on the optimized reconstruction algorithm, and finally obtain the reconstructed image;

[0016] The image is reconstructed using the alternating multiplier method, combined with the total variation denoising algorithm and low-rank constraints. The specific process is as follows:

[0017] Step 201, the optimization equation of the reconstruction algorithm is:

[0018]

[0019] Rewrite the above formula as:

[0020]

[0021] Then the constraints are introduced Then the augmented Lagrangian function of the optimization equation is:

[0022]

[0023] Let p = μ / δ, and simplify the above function to:

[0024]

[0025] Among them, ||*||1 is the L1 norm, ||*|| * is the nuclear norm, ||*||2 is the L2 norm, x is the image to be reconstructed, λ is the regularization parameter, is the gradient operator, μ is the Lagrangian multiplier, δ>0 is the Lagrangian penalty term, R and b are the preprocessed light field and bucket value respectively.

[0026] Step 202, the image x to be reconstructed is updated by using the low-rank constraint. The specific process is as follows:

[0027] Calculate the partial derivative of the image x to be reconstructed in the function, temporarily remove the derivative of the nuclear norm in the derivative, and calculate x; then perform singular value decomposition on x, and update x after applying low-rank constraints to the decomposed matrix;

[0028] Step 203, updating parameters d and p in sequence based on the updated x;

[0029] Update parameter d: Keep x and p unchanged, let To update d;

[0030] Update parameter p: Keep the updated x and d unchanged, let To update p.

[0031] In step 204, each time x, d, and p are all updated, the number of iterations is increased by 1, and it is determined whether the number of iterations reaches the upper limit. If so, the optimization process ends and the reconstructed image is output; otherwise, return to step 202 and continue to iterate until the upper limit of the number of iterations is reached.

[0032] The advantages and beneficial effects of the present invention are:

[0033] (1) The present invention is an innovative algorithm for correlation imaging. With the help of total variation denoising algorithm and low-rank regularization, it can effectively remove image noise and make the imaging of the target area clearer.

[0034] (2) Under undersampling conditions, it is sometimes difficult for the endoscope to be directly aimed at the lesion area, which requires a certain amount of calibration time; at the same time, the detection of the lesion area is multi-angle, which requires multiple imaging; but the endoscope inspection time is limited, so it is necessary to limit the sampling rate of each imaging to ensure that the lesion area is inspected as comprehensively as possible within the limited time. The high efficiency of the alternating multiplier method also makes imaging under undersampling conditions more feasible.

[0035] (3) The application of the present invention is generalized and is not limited to the imaging processing of diseased tissues, but can also clarify the imaging of various targets under under-sampling conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is the overall process of multimode fiber optic endoscope imaging based on correlation imaging of the present invention;

[0037] Figure 2 It is the specific process of the reconstruction algorithm in the present invention;

[0038] Figure 3 This is the original image of the polyp tissue in Example 1;

[0039] Figure 4 This is the restored image of Example 1 using the traditional second-order correlation ghost imaging algorithm when the sampling rate is 1 / 16;

[0040] Figure 5 is the restored image of the method of the present invention when the sampling rate is 1 / 16 in Example 1;

[0041] Figure 6 is the original image of the house in Example 2;

[0042] Figure 7 This is the restored image of Example 2 when the sampling rate is 1 / 16 and the traditional second-order correlation ghost imaging algorithm is applied;

[0043] Figure 8 This is the restored image obtained by applying the method of the present invention when the sampling rate in Example 2 is 1 / 16. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0045] like Figure 1 As shown, the specific process is to first use an endoscope to emit a modulated light field to illuminate the lesion area to be examined, then use a bucket detector to collect the light field containing the target information, and then pre-process the light field used and the bucket value obtained by the detector and use it as the input of the algorithm, and then use the algorithm of the present invention to restore the image.

[0046] like Figure 2 As shown, the optimization equation of the reconstruction algorithm can be written as follows:

[0047] arg min{λ||Dx||1+||x|| *}stRx=b

[0048] Rewrite the above formula as:

[0049]

[0050] Then we introduce the constraint condition Dx=d and write the augmented Lagrangian function of the optimization equation:

[0051]

[0052] Let p = μ / δ, and simplify the above function to:

[0053]

[0054] Among them, ||*||1 is the L1 norm, ||*|| * is the nuclear norm, ||*||2 is the L2 norm, x is the image to be reconstructed, λ is the regularization parameter, D is the total variation operator, μ is the Lagrangian multiplier, δ>0 is the Lagrangian penalty term, R and b are the preprocessed light field and bucket values ​​respectively.

[0055] Optimization process: Calculate the partial derivative of the image to be reconstructed x in the function, temporarily remove the derivative of the nuclear norm in the derivative, and after calculating x, perform singular value decomposition on x, and update x after applying low-rank constraints to the decomposed matrix, then keep x and p unchanged, and let To update d, finally keep the updated x and d unchanged, and set p = p + (Dx-d) to update p. Repeat the above steps until the number of iterations is reached.

[0056] Embodiment 1:

[0057] This embodiment mainly studies the performance of the present invention in restoring the lesion site. In this embodiment, a polyp tissue image with a resolution of 64*64 is used as the object to be tested. The light field uses a speckle image obtained by multimode optical fiber transmission, and the size is also 64*64. When recording and pre-calibrating the modulated light field, the predetermined sampling number is set to 256, that is, the light field information contains 256 speckle images. That is, the sampling rate is 1 / 16.

[0058] Read in the polyp tissue image and each speckle image to obtain an object information matrix and a light field information matrix of size 64*64. Perform a dot multiplication between the object information matrix and each light field matrix to obtain 256 new matrices of 64*64. Add up all the values ​​on each new matrix to get 256 values, and combine these values ​​into a column vector A of size 256*1. Subtract the average value of A from each value on A, and the resulting column vector is the bucket value input b of the reconstruction algorithm.

[0059] Then, each light field matrix is ​​converted to a size of 1*4096, and these row vectors are recombined into a matrix B of size 256*4096; the average of each column of B is calculated to obtain a row vector containing these average values, and each row of B is subtracted from this row vector. The resulting matrix is ​​the light field input R of the reconstruction algorithm.

[0060] The traditional second-order correlation imaging algorithm and the algorithm of the present invention are then used to restore the image. The signal-to-noise ratio (SNR) is used as an indicator to evaluate the denoising ability of the algorithm. The higher the signal-to-noise ratio, the less noise is generated. The structural similarity (SSIM) is used to evaluate the ability of the algorithm to restore the image under undersampling conditions. The higher the structural similarity, the more information is restored.

[0061] The signal-to-noise ratio formula is

[0062]

[0063] Among them, (x, y) represents the coordinates of the pixel point in the image, O(x, y) is the pixel value of the original image of the object to be tested at (x, y), and S(x, y) is the pixel value of the image restored by the algorithm at (x, y).

[0064] The image structure similarity formula is:

[0065]

[0066] The subscripts x and y represent the images to be compared, is the average brightness of the image; is the standard deviation of pixel values; is the covariance of the pixel values ​​of the two images. C1=(K1L) 2 , C2=(K2L) 2 , K1 is usually 0.01, K2 is usually 0.03, L is the dynamic range of grayscale, and in the embodiment of the present invention, it is 255.

[0067] like Figure 3 As shown in Figure 1, this is the original image of the polyp tissue used in Example 1. Figure 4 As shown in Figure 1, the restored image using the traditional second-order correlation ghost imaging algorithm when the sampling rate is 1 / 16 in Example 1 has a signal-to-noise ratio of 4.246. Figure 5 As shown, the restored image using the algorithm of the present invention when the sampling rate is 1 / 16 in Example 1 has a signal-to-noise ratio of 16.8843. Figure 3 and Figure 4 The structural similarity is 0.081; Figure 3 and Figure 5 The structural similarity is 0.625. It can be seen that under the condition of undersampling, the traditional ghost imaging technology can no longer restore the image, and the restored image contains more noise; however, the algorithm of the present invention can still restore a better image, and the denoising ability is significantly improved compared with the ghost imaging technology.

[0068] Embodiment 2:

[0069] This embodiment mainly studies the generalization of the algorithm in the present invention. In this embodiment, a house image with a resolution of 64*64 is used as the object to be measured. The light field uses a speckle image obtained by multimode optical fiber transmission, and the size is also 64*64. When recording and pre-calibrating the modulated light field, the preset number of samples is set to 256, that is, the light field information contains 256 speckle images. That is, the sampling rate is 1 / 16.

[0070] The same steps as in Example 1 are used to obtain the input of the algorithm and the traditional second-order correlation imaging algorithm and the algorithm of the present invention are used to restore the image. The signal-to-noise ratio (SNR) and structural similarity (SSIM) are used as indicators to evaluate the algorithm.

[0071] like Figure 6 As shown in , this is the original image of the house used in Example 2. Figure 7 As shown in Figure 2, the restored image using the traditional second-order correlation ghost imaging algorithm when the sampling rate is 1 / 16 in Example 2 has a signal-to-noise ratio of 6.3851. Figure 8 As shown, it is the restored image of the present invention when the sampling rate is 1 / 16 in Example 2, and its signal-to-noise ratio is 14.2874. Figure 6 and Figure 7 The structural similarity is 0.103; Figure 6 and Figure 8 The structural similarity is 0.503. It can be seen that even if it is not a biological tissue, the algorithm of the present invention can still restore the image, and the denoising ability is still improved, which has a certain generalization.

Claims

1. A method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging, characterized in that: The steps include: Step 1: Use a multimode fiber endoscope to emit a modulated light field to illuminate the target area, and use a detector to collect the bucket value of the light field after passing through the target area; Step 2: input the modulated light field and the bucket value into the reconstruction algorithm, optimize the reconstruction algorithm, reconstruct the image based on the optimized reconstruction algorithm, and finally obtain the reconstructed image; The image is reconstructed using the alternating multiplier method, combined with the total variation denoising algorithm and low-rank constraints. The specific process is as follows: Step 201, the optimization equation of the reconstruction algorithm is: Rewrite the above formula as: Then the constraints are introduced Then the augmented Lagrangian function of the optimization equation is: Let p = μ / δ, and simplify the above function to: Among them, ||*||1 is the L1 norm, ||*|| * is the nuclear norm, ||*||2 is the L2 norm, x is the image to be reconstructed, λ is the regularization parameter, is the gradient operator, μ is the Lagrangian multiplier, δ>0 is the Lagrangian penalty term, R and b are the preprocessed light field and bucket value respectively; Step 202, the image x to be reconstructed is updated by using the low-rank constraint. The specific process is as follows: Calculate the partial derivative of the image x to be reconstructed in the function, temporarily remove the derivative of the nuclear norm in the derivative, and calculate x; then perform singular value decomposition on x, and update x after applying low-rank constraints to the decomposed matrix; Step 203, updating parameters d and p in sequence based on the updated x; Update parameter d: Keep x and p unchanged, let To update d; Update parameter p: Keep the updated x and d unchanged, let To update p; In step 204, each time x, d, and p are all updated, the number of iterations is increased by 1, and it is determined whether the number of iterations reaches the upper limit. If so, the optimization process ends and the reconstructed image is output; otherwise, return to step 202 and continue to iterate until the upper limit of the number of iterations is reached.

2. According to the method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging according to claim 1, it is characterized in that: The calculation process of the bucket value is: The size of the speckle image obtained by transmitting the light field through the multimode optical fiber is f*f, and the predetermined sampling number of the modulated light field is g, that is, the light field information contains g speckle images; Read in the object information matrix and all light field information matrices of size f*f, perform dot multiplication between the object information matrix and each light field information matrix, and obtain g new matrices of f*f; Add up all the values ​​on each new matrix to get g values, combine these values ​​into a column vector A of size g*1, subtract the average value of A from each value on A, and the resulting column vector is the bucket value input b of the reconstruction algorithm.

3. According to the method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging according to claim 2, it is characterized in that: The calculation process of the modulated light field is: Convert each light field information matrix into a row vector of 1*(f*f), and recombine these row vectors into a matrix B of size g*(f*f); calculate the average of each column of B to obtain a row vector containing these average values, and subtract each row of B from this row vector. The resulting matrix is ​​the light field input R of the reconstruction algorithm.

4. According to the method for improving the imaging quality of a multimode optical fiber endoscope based on correlation imaging according to claim 1, it is characterized in that: Undersampled endoscopic imaging image processing for generalization.

Images