A multi-depth-of-field full-field optical image fusion method based on NSCT and contrast gradient
By using NSCT and contrast gradient image fusion methods, the problem of full-field optical angiography images was solved, generating high-quality full-focus fused images and overcoming the difficulties of uneven thickness of biological samples and lens depth of field limitations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FOSHAN UNIVERSITY
- Filing Date
- 2022-06-14
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to achieve full-focus images in optical angiography, especially due to the uneven thickness of biological samples and the limitations of optical lens depth of field, which make it difficult to acquire clear images.
A multi-depth full-field optical image fusion method based on NSCT and contrast gradient is adopted. The image is decomposed into low-pass and band-pass sub-band images through non-subsampled contour wave transform, and fused using sparse representation and contrast gradient. Finally, a clear fused image is generated through inverse transform.
It achieves robustness improvements in visual effects and objective evaluation, effectively preserves the focus information of multi-focus source images, avoids undesirable effects such as blockiness and slits, and generates high-quality full-focus fusion images.
Smart Images
Figure CN114926384B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing data fusion technology, and in particular to a multi-depth full-field optical image fusion method based on NSCT and contrast gradient. Background Technology
[0002] Utilizing full-field and high-resolution imaging instruments, combined with full-field optical imaging methods, can stably reflect the global blood flow status of organisms, improving the reliability of biological research. Full-field optical angiography is an emerging vascular imaging method with high spatiotemporal resolution, suitable for real-time in vivo imaging, and has promising clinical application prospects. Biological optical imaging has wide applications in biological science research and clinical diagnosis. In recent years, full-field optical coherence tomography (FF-OCT) and two-dimensional visualization full-field laser Doppler imaging using laser speckle contrast imaging have emerged. However, when the sample imaging position is within the focal depth of the camera system, sample surface information is transmitted to the camera sensor plane. Typical biological samples, such as mouse ears, rabbit ears, and zebrafish, are generally 1mm to 5mm thick. The non-uniform surface and thickness of these biological samples can easily lead to the problem of not being able to obtain clear, long-depth-of-field angiographic images in lenses with high magnification.
[0003] Computer vision technology has been widely applied in various fields. Multi-depth-of-field full-field optical image fusion can also be regarded as multi-focus image fusion. It combines multiple source images with complementary focus areas based on a certain fusion algorithm to generate a fully focused fused image in which all objects are clear. During the fusion process, important focus information from all input images must be transferred to the fused image. The key to the fusion process is to accurately identify the clear regions or pixels within the focus range. Fusion methods can be broadly divided into two categories: transform domain-based methods and spatial domain-based methods. Spatial domain-based methods directly construct the fusion result by selecting pixels, blocks, or regions from the focus areas of the source images. However, the fusion result depends on the accuracy of activity detection, and may suffer from block artifacts due to inherent ambiguity caused by textureless and edgeless regions. In addition, when pixels in transition regions are not properly processed, undesirable effects such as slits may appear along the boundaries of the focus areas. Compared to spatial domain methods, transform domain-based methods have almost none of these drawbacks. Transform domain-based methods first transform the source images to the transform domain, then merge the transform coefficients according to certain fusion rules, and finally perform an inverse transform on the fusion coefficients to obtain the fused image. In recent years, researchers both domestically and internationally have recognized the significant role of transform domain methods in multi-focus image fusion. Among these methods, the selection of multi-scale transform (MST) tools and fusion rules is crucial for MST-based image fusion methods. Since MST decomposes images into different scales in a manner similar to the human visual system's (HVS) coarse-to-fine image processing, transform domain-based methods offer higher contrast and better signal-to-noise ratios. The most commonly used multi-scale decomposition methods are pyramid transform and wavelet transform. Pyramid-based fusion methods include Laplacian pyramid (LP), ratio low-pass pyramid (RP), contrast pyramid (CP), and gradient pyramid (GP). Wavelet transform-based methods include discrete wavelet transform (DWT), stationary wavelet transform, and dual-tree complex wavelet transform. Summary of the Invention
[0004] To address the aforementioned shortcomings, this invention proposes a multi-depth-of-field full-field optical image fusion method based on NSCT and contrast gradient. The aim is to solve the problem that the obtained full-field optical angiography images cannot be fully focused due to the poor optical characteristics of biological samples and the depth-of-field limitations of optical lenses.
[0005] To achieve this objective, the present invention adopts the following technical solution:
[0006] A multi-depth full-field optical image fusion method based on NSCT and contrast gradient includes the following steps:
[0007] Step S1: Construct an optical system and use the intensity wave modulation effect to obtain multiple full-field optical angiography images with depth of field;
[0008] Step S2: Input two full-field optical angiography images with different depths of field. Perform NSCT decomposition on the two different full-field optical angiography images with different depths of field. Each full-field optical angiography image with different depths of field is decomposed into a low-pass sub-band image and multiple bandpass sub-band images.
[0009] Step S3: Use sparse representation to fuse the two low-pass subband images to obtain a fused low-pass subband image;
[0010] Step S4: Based on contrast gradient and spatial frequency, fuse multiple bandpass sub-band images to obtain a fused bandpass sub-band image;
[0011] Step S5: Perform optimal linear reconstruction on the low-pass subband fused image and the bandpass subband fused image, and then perform inverse NSCT transformation to obtain the final fused image.
[0012] Preferably, in step S3, fusing the two low-pass subband maps using sparse representation specifically includes the following steps:
[0013] Step S31: Using a sliding window technique, divide the two low-pass subband images into 8×8 blocks with the same size as a single atom in the dictionary, and overlap them by λ pixels to obtain W blocks in each of the two low-pass subband images. The image block set is represented as... and
[0014] Step S32: Rearrange the image blocks in the two low-pass sub-band images at each position l into a column vector, resulting in... and
[0015] Step S33: Use the Orthogonal Matching Pursuit Algorithm (OMP) to pair and Optimize the solution to obtain and Corresponding sparse representation coefficients and The sparse representation coefficients of the low-pass subbands of the fused image are obtained by comparing the L1 norm values and selecting the sparse representation coefficients of the low-pass subbands.
[0016]
[0017] Used a complete dictionary D Linear representation yields a vector.
[0018]
[0019] Where D is the learned dictionary, and v represents the sparsity coefficients of image q;
[0020] Step S34: Repeat steps S31-S33 for all image blocks in the two low-pass subband images to obtain the low-pass subband vector of the fused low-pass subband image. Each vector is converted into an 8×8 image patch and placed sequentially in its original position to obtain the low-pass subband fusion component L. f .
[0021] Preferably, in step S4, fusing multiple bandpass sub-band maps based on contrast gradient and spatial frequency specifically includes the following steps:
[0022] Step S41: Calculate the decision map based on the spatial frequency and local contrast of the bandpass subband map. The calculation formula is as follows:
[0023]
[0024] in, Let A represent the decision map of source image A in the k-th direction at scale u. and Let represent the spatial frequencies of the bandpass subband map of source image A in the k-th direction at scale u and the spatial frequencies of the bandpass subband map of source image B in the k-th direction at scale u, respectively. It refers to Local contrast;
[0025] Step S42: Perform morphological filtering on the decision graph to obtain the initial decision graph. The calculation formula is as follows:
[0026]
[0027] in, The area of source image A is represented by r, and the scaling factor is represented by r. This represents the initial decision graph after morphological filtering.
[0028] Step S43: Consistency check optimizes the initial decision graph to generate the final decision graph. The calculation formula is as follows:
[0029]
[0030] in, The final decision map, representing the focused region obtained after consistency verification of source image A, is Θ, centered at position (i,j) and of size [missing information]. The neighborhood of the center point, where a and b represent the horizontal and vertical pixel distances from the center point, respectively;
[0031] Step S44: Based on the final decision map, calculate the bandpass subband fusion image using the following formula:
[0032]
[0033] in, This represents a bandpass / subband fusion image. This represents the bandpass subband map of source image A. This represents the bandpass subband map of the B source image. This represents the final decision graph of source image A. This represents the final decision graph of the B source image.
[0034] Preferably, in step S5, the formula for generating the final fused image is as follows:
[0035]
[0036] Where F represents the final fused image, L represents a bandpass / subband fused image. f This represents a low-pass subband fused image.
[0037] Preferably, in step S32, for The optimization solution is as follows:
[0038] Using OMP sparse representation vector Optimize the solution:
[0039]
[0040]
[0041] Where δ is the sparse reconstruction error.
[0042] Preferably, in step S41, the method for calculating the spatial frequency includes the following steps:
[0043] Step S411: Traverse the rows of the pixel matrix in a bandpass sub-band image, calculate the sum of squares of the differences between adjacent pixels in the row direction, and take the square root of the average of the sums of squares to obtain the row frequency value; the calculation formula is as follows:
[0044]
[0045] Step S412: Traverse the columns of the pixel matrix in a bandpass sub-band image, calculate the sum of squares of the differences between adjacent pixels in the column direction, and take the square root of the average of the sums of squares to obtain the column frequency value; the calculation formula is as follows:
[0046]
[0047] Step S413: Based on the row frequency values and column frequency values, obtain the spatial frequency values of the entire bandpass sub-band diagram; the calculation formula is as follows:
[0048]
[0049] Among them, the bandpass subband map of source image A The spatial frequency value at point (i,j) is expressed as follows: express; Represents the bandpass subband map of source image A. Row frequency values of the pixel matrix Represents the bandpass subband map of source image A. The column frequency values of the pixel matrix.
[0050] Preferably, in step S41, the formula for calculating local contrast is as follows:
[0051]
[0052] in, Let represent the spatial frequency of the bandpass subband map in the k-th direction of the B source image at scale u. It refers to The local contrast is p×q, where p×q represents the size of the neighborhood Ω.
[0053] The technical solutions provided in this application embodiment may include the following beneficial effects:
[0054] This invention utilizes non-subsampled contour wave transform and contrast gradient methods to fuse full-field optical angiography images with different focal regions. This fusion method is more robust and outperforms some commonly used and advanced methods in terms of visual effects and objective evaluation. It can effectively preserve the focal information in multi-focal source images without introducing erroneous information. Attached Figure Description
[0055] Figure 1 This is a flowchart of a multi-depth full-field optical image fusion method based on NSCT and contrast gradient. Detailed Implementation
[0056] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0057] A multi-depth full-field optical image fusion method based on NSCT and contrast gradient includes the following steps:
[0058] Step S1: Construct an optical system and use the intensity wave modulation effect to obtain multiple full-field optical angiography images with depth of field;
[0059] Step S2: Input two full-field optical angiography images with different depths of field. Perform NSCT decomposition on the two different full-field optical angiography images with different depths of field. Each full-field optical angiography image with different depths of field is decomposed into a low-pass sub-band image and multiple bandpass sub-band images.
[0060] Step S3: Use sparse representation to fuse the two low-pass subband images to obtain a fused low-pass subband image;
[0061] Step S4: Based on contrast gradient and spatial frequency, fuse multiple bandpass sub-band images to obtain a fused bandpass sub-band image;
[0062] Step S5: Perform optimal linear reconstruction on the low-pass subband fused image and the bandpass subband fused image, and then perform inverse NSCT transformation to obtain the final fused image.
[0063] This application presents a multi-depth full-field optical image fusion method based on NSCT and contrast gradient. NSCT is a non-subsampled contourlet transform. First, multiple full-field optical angiography images with different depths are obtained using intensity wave modulation. Then, using the full-field optical angiography images with different depths as source images, the two input source images are decomposed into two low-pass sub-band images and multiple band-pass sub-band images using NSCT. Next, the two low-pass sub-band images are fused using sparse representation to obtain a low-pass sub-band fused image. The multiple band-pass sub-band images are fused based on contrast gradient and spatial frequency to obtain a band-pass sub-band fused image. Finally, the low-pass sub-band fused image and the band-pass sub-band fused image are subjected to inverse NSCT to obtain the final clear fused image.
[0064] This invention utilizes non-subsampled contour wave transform and contrast gradient methods to fuse full-field optical angiography images with different focal regions. This fusion method is more robust and outperforms some commonly used and advanced methods in terms of visual effects and objective evaluation. It can effectively preserve the focal information in multi-focal source images without introducing erroneous information.
[0065] Preferably, in step S3, fusing the two low-pass subband maps using sparse representation specifically includes the following steps:
[0066] Step S31: Using a sliding window technique, divide the two low-pass subband images into 8×8 blocks with the same size as a single atom in the dictionary, and overlap them by λ pixels to obtain W blocks in each of the two low-pass subband images. The image block set is represented as... and
[0067] Step S32: Rearrange the image blocks in the two low-pass sub-band images at each position l into a column vector, resulting in... and
[0068] Step S33: Use softmax-L1 regularization to apply the following to the target area. and After optimization and By linking these factors together, and selecting the sparse representation coefficients of the low-pass subbands, we obtain the sparse representation coefficients of the low-pass subband fused image:
[0069]
[0070] Used a complete dictionary D Linear representation yields a vector.
[0071]
[0072] Where D is the learned dictionary, and v represents the sparsity coefficients of image q;
[0073] Step S34: Repeat steps S31-S33 for the image blocks in the two low-pass subband images to obtain the low-pass subband vector of the fused low-pass subband image. Each vector is converted into an 8×8 image patch and placed sequentially in its original position to obtain the low-pass subband fusion component L. f .
[0074] Specifically, sparse representation is used to fuse the two low-pass subband maps to obtain the low-pass subband fusion component L. f This refers to the low-pass subband fusion image. Further explanation: in this embodiment, the dictionary D is obtained by selecting multiple high-quality multi-focus full-field optical angiography images as training data and learning them using the K-SVD method.
[0075] Preferably, in step S4, fusing multiple bandpass sub-band maps based on contrast gradient and spatial frequency specifically includes the following steps:
[0076] Step S41: Calculate the decision map based on the spatial frequency and local contrast of the bandpass subband map. The calculation formula is as follows:
[0077]
[0078] in, Let A represent the decision map of source image A in the k-th direction at scale u. and Let represent the spatial frequencies of the bandpass subband map of source image A in the k-th direction at scale u and the spatial frequencies of the bandpass subband map of source image B in the k-th direction at scale u, respectively. It refers to Local contrast;
[0079] Step S42: Perform morphological filtering on the decision graph to obtain the initial decision graph. The calculation formula is as follows:
[0080]
[0081] in, The area of source image A is represented by r, and the scaling factor is represented by r. This represents the initial decision graph after morphological filtering.
[0082] Step S43: Consistency check optimizes the initial decision graph to generate the final decision graph. The calculation formula is as follows:
[0083]
[0084] in, The final decision map of source image A, representing the detection result of the focused region obtained after consistency verification, is Θ, which is centered at position (i,j) and has a size of The neighborhood of the center point, where a and b represent the horizontal and vertical pixel distances from the center point, respectively;
[0085] Step S44: Based on the final decision map, calculate the bandpass subband fusion image using the following formula:
[0086]
[0087] in, This represents a bandpass / subband fusion image. This represents the bandpass subband map of source image A. This represents the bandpass subband map of the B source image. This represents the final decision graph of source image A. This represents the final decision graph of the B source image.
[0088] In this embodiment, multiple bandpass sub-band images are fused based on contrast gradient and spatial frequency to obtain a fused bandpass sub-band image. This is beneficial for the subsequent generation of the final fused image. Specifically, the final decision map is the image needed when forming the bandpass subband fused image. The decision map is a binary image, where white pixels represent the focused area and black pixels represent the non-focused area. The decision maps of the two source images are complementary.
[0089] Preferably, in step S5, the formula for generating the final fused image is as follows:
[0090]
[0091] Where F represents the final fused image, L represents a bandpass / subband fused image. f This represents a low-pass subband fused image.
[0092] Specifically, the final fused image is obtained by performing a non-downsampled contour wave inverse transform on the fused bandpass subband image and the fused lowpass subband image.
[0093] Preferably, in step S32, for The optimization solution is as follows:
[0094] Using OMP sparse representation vector Optimize the solution:
[0095]
[0096]
[0097] Where δ is the sparse reconstruction error.
[0098] Specifically, OMP refers to the Orthogonal Matching Pursuit algorithm, which has a fast convergence speed. It further improves the... The optimization degree of solving the sparse representation vector.
[0099] Preferably, in step S41, the method for calculating the spatial frequency includes the following steps:
[0100] Step S411: Traverse the rows of the pixel matrix in a bandpass sub-band image, calculate the sum of squares of the differences between adjacent pixels in the row direction, and take the square root of the average of the sums of squares to obtain the row frequency value; the calculation formula is as follows:
[0101]
[0102] Step S412: Traverse the columns of the pixel matrix in a bandpass sub-band image, calculate the sum of squares of the differences between adjacent pixels in the column direction, and take the square root of the average of the sums of squares to obtain the column frequency value; the calculation formula is as follows:
[0103]
[0104] Step S413: Based on the row frequency values and column frequency values, obtain the spatial frequency values of the entire bandpass sub-band diagram; the calculation formula is as follows:
[0105]
[0106] Among them, the bandpass subband map of source image A The spatial frequency value at point (i,j) is expressed as follows: express; Represents the bandpass subband map of source image A. Row frequency values of the pixel matrix Represents the bandpass subband map of source image A. The column frequency values of the pixel matrix.
[0107] In this embodiment, by calculating the row frequency and column frequency values of the bandpass sub-bandmap of source image A, the spatial frequency value of the bandpass sub-bandmap of source image A is further calculated; similarly, by calculating the row frequency and column frequency values of the bandpass sub-bandmap of source image B, the spatial frequency value of the bandpass sub-bandmap of source image B is further calculated.
[0108] Preferably, in step S41, the formula for calculating local contrast is as follows:
[0109]
[0110] in, Let represent the spatial frequency of the bandpass subband map in the k-th direction of the B source image at scale u. It refers to The local contrast is p×q, where p×q represents the size of the neighborhood Ω.
[0111] In this embodiment, the decision map of source image A is further obtained by calculating the local contrast of the spatial frequency of the bandpass sub-band map of source image B and comparing it with the spatial frequency of the bandpass sub-band map of source image A.
[0112] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0113] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A multi-depth-of-field full-field optical image fusion method based on NSCT and contrast gradient, characterized in that: Includes the following steps: Step S1: Construct an optical system and use the intensity wave modulation effect to obtain multiple full-field optical angiography images with depth of field; Step S2: Input two full-field optical angiography images with different depths of field, and perform NSCT decomposition on the two multi-depth full-field optical angiography images. Each full-field optical angiography image is decomposed into a low-pass sub-band image and multiple bandpass sub-band images. Step S3: Use sparse representation to fuse the two low-pass subband images to obtain a fused low-pass subband image; Step S4: Based on contrast gradient and spatial frequency, fuse multiple bandpass sub-band images to obtain a fused bandpass sub-band image; Step S5: Perform optimal linear reconstruction on the low-pass subband fused image and the bandpass subband fused image, and then perform inverse NSCT transform to obtain the final fused image; In step S4, the fusion of multiple bandpass sub-band maps based on contrast gradient and spatial frequency specifically includes the following steps: Step S41: Calculate the decision map based on the spatial frequency and local contrast of the bandpass subband map. The calculation formula is as follows: in, Indicates that the source image A is at a certain scale. Next Decision diagram for each direction, and These represent the scale of source image A. Next Spatial frequencies of bandpass subband maps in each direction and B-source images at scale Next Spatial frequencies of bandpass subband maps in each direction, It refers to Local contrast; Step S42: Perform morphological filtering on the decision graph to obtain the initial decision graph. The calculation formula is as follows: in, Represents the area of source image A. r Represents the scaling factor. This represents the initial decision graph after morphological filtering. Step S43: Consistency check optimizes the initial decision graph to generate the final decision graph. The calculation formula is as follows: in, The final decision map representing the focused region obtained after consistency verification of source image A. Based on position ( i , j Centered on, size is The neighborhood, a and b These represent the horizontal and vertical pixel distances from the center point, respectively. Step S44: Based on the final decision map, calculate the bandpass subband fusion image using the following formula: in, This represents a bandpass / subband fusion image. This represents the bandpass subband map of source image A. This represents the bandpass subband map of the B source image. This represents the final decision graph of source image A. This represents the final decision graph of the B source image.
2. The multi-depth full-field optical image fusion method based on NSCT and contrast gradient according to claim 1, characterized in that: In step S3, the fusion of the two low-pass subband maps using sparse representation specifically includes the following steps: Step S31: Use the sliding window technique to divide the two low-pass subband maps into sections with the same size as individual atoms in the dictionary. The blocks, and overlapping. From 100 pixels, W blocks are obtained in two low-pass sub-band images, where the set of image blocks is represented as follows: and ; Step S32: Place each position ， The image patches in the two low-pass subband images are rearranged into column vectors to obtain... and ; Step S33: Use the Orthogonal Matching Pursuit Algorithm (OMP) to pair and Optimize the solution to obtain and Corresponding sparse representation coefficients and The sparse representation coefficients of the low-pass subbands of the fused image are obtained by comparing the L1 norm values and selecting the sparse representation coefficients of the low-pass subbands. Used a complete dictionary D Linear representation yields a vector. ; Where D is the learned dictionary, and v represents the sparsity coefficients of image q; Step S34: Repeat steps S31-S33 for all image blocks in the two low-pass subband images to obtain the low-pass subband vector of the fused low-pass subband image. Transform each vector into Image blocks are placed sequentially in their original positions to obtain low-pass subband fusion components. .
3. The multi-depth full-field optical image fusion method based on NSCT and contrast gradient according to claim 1, characterized in that: In step S5, the final formula for generating the fused image is as follows: in, F This represents the final fused image. This represents a bandpass / subband fusion image. This represents a low-pass subband fused image.
4. The multi-depth full-field optical image fusion method based on NSCT and contrast gradient according to claim 2, characterized in that: In step S32, for The optimization solution is as follows: Use OMP to { sparse representation vector Optimize the solution: in, It is a sparse reconstruction error.
5. The multi-depth full-field optical image fusion method based on NSCT and contrast gradient according to claim 1, characterized in that: In step S41, the method for calculating spatial frequency includes the following steps: Step S411: Traverse the rows of the pixel matrix in a bandpass sub-band image, calculate the sum of squares of the differences between adjacent pixels in the row direction, and take the square root of the average of the sums of squares to obtain the row frequency value; the calculation formula is as follows: Step S412: Traverse the columns of the pixel matrix in a bandpass sub-band image, calculate the sum of squares of the differences between adjacent pixels in the column direction, and take the square root of the average of the sums of squares to obtain the column frequency value; the calculation formula is as follows: Step S413: Based on the row frequency values and column frequency values, obtain the spatial frequency values of the entire bandpass sub-band diagram; the calculation formula is as follows: Among them, the bandpass subband map of source image A The spatial frequency value at point (i,j) is expressed as follows: express; Represents the bandpass subband map of source image A. Row frequency values of the pixel matrix Represents the bandpass subband map of source image A. The column frequency values of the pixel matrix.
6. The multi-depth full-field optical image fusion method based on NSCT and contrast gradient according to claim 1, characterized in that: In step S41, the formula for calculating local contrast is as follows: in, Indicates the scale of source image B. Next Spatial frequencies of bandpass subband maps in each direction, It refers to Local contrast, Representing the neighborhood The size.