Image fusion method and system based on multi-scale transformation and convolution sparse representation

Abstract

The present invention discloses an image fusion method based on multi-scale transformation and convolution sparse representation, including: decomposing medical images of different modalities by rolling guided filtering to obtain basic sub-images and detail sub-images, decomposing the basic sub-images into high-frequency sub-bands and low-frequency sub-bands in multi-scale directions by NSST, encoding them into high-frequency sparse coefficients and low-frequency sparse coefficients by convolution sparse representation, fusing the sparse coefficients by corresponding different fusion rules for the high-frequency and low-frequency coefficients, reconstructing the fusion coefficients by convolution sparse representation, reconstructing the fusion sub-bands by inverse NSST to obtain the fused basic sub-image, and simultaneously using the weighted improved Laplace energy and fused detail sub-images, adding the fused basic sub-images and detail sub-images to obtain the final fused image. The present invention solves the problems existing in the existing medical image fusion methods, provides a better solution for the field of medical image fusion, and achieves the acquisition of high-quality fused images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and in particular relates to an image fusion method and system based on multi-scale transformation and convolution sparse representation. Background Art

[0002] With the continuous development of medical imaging engineering and computer technology, many advanced imaging devices have emerged, which provide clinical diagnosis with multiple modal medical images, which reflect different information about human organs and diseased tissues from different angles. For example: CT images have strong spatial resolution and geometric characteristics, and the imaging of dense structures such as bones is very clear, but the contrast of soft tissues is low; MRI images can clearly reflect anatomical structures such as soft tissues, organs, and blood vessels, but are insensitive to calcification points, and will be geometrically distorted by magnetic interference; SPECT and PET images can reflect the metabolic level and blood flow of tissues and organs, and present "hot spots" for tumor lesions, but their resolution is low, and it is difficult to obtain accurate anatomical structure morphology. It can be seen that the information transmitted by each modality of medical images has its own unique advantages, as well as limitations and shortcomings. Therefore, when facing complex diseases, a single modality of medical images is not enough to provide sufficient auxiliary information for clinicians to refer to, so as to give an accurate diagnosis of the disease or formulate a suitable surgical treatment plan.

[0003] In recent years, computer technology has developed rapidly, and multimodal medical image fusion technology has emerged and has been widely studied. Image fusion technology can organically combine multimodal medical imaging information, provide more useful information to assist disease diagnosis, and reduce the randomness and redundancy of information. Multimodal medical image fusion improves the diagnostic efficiency of clinicians for certain complex diseases, helping clinicians make more accurate diagnoses and develop more appropriate surgical treatment plans.

[0004] Existing image fusion methods are usually divided into three levels: pixel level, feature level, and decision level. Compared with pixel-level image fusion, feature-level and decision-level image fusion are prone to information loss. Therefore, the most basic pixel-level image fusion method is usually used in the prior art. Pixel-level image fusion directly acts on the pixels in the original data layer to obtain an image that conforms to human visual perception and is easy for computer processing. This method can more completely retain the original information in the image, and thus obtain a fusion result with clear edge structure and richer texture details.

[0005] Pixel-level image fusion methods can usually be divided into fusion methods based on spatial domain, transform domain, and sparse representation.

[0006] Among them, the fusion method based on the spatial domain directly processes the pixels of the source image, selects appropriate pixels or regions according to the saliency measurement results, and obtains the fused image by simple merging. Although this type of method has high computational efficiency and is easy to implement, the fusion results often have problems such as low brightness contrast and spectral distortion.

[0007] The image fusion method based on transform domain can effectively make up for the shortcomings of the method based on spatial domain. It decomposes the source image into sub-images of multiple scales. Each sub-image of each scale is fused using different fusion rules, which can retain information of different scales. After fusion, the fusion coefficients are reconstructed into a fused image through the corresponding inverse transformation. This method specifically includes pyramid transformation, discrete wavelet transformation, dual-tree complex wavelet transformation, shearlet transformation, non-subsampled shearlet transformation, curvelet transformation, contourlet transformation, non-subsampled contourlet transformation, etc. The above-mentioned image fusion method based on transform domain, that is, the fusion method based on multi-scale transformation, can well preserve the detailed information such as the internal texture of the input image, but it will also introduce noise in the fused image.

[0008] Among the fusion methods based on multi-scale transformation, the discrete wavelet transform-based method can realize the multi-scale representation of the image and better extract the feature information of the image. However, the decomposition basis directions of this type of method are relatively few, and the selection of the number of directions is not flexible, and the curves and edges of the image cannot be captured well. Non-subsampled contourlet transform (NSCT) and non-subsampled shearlet transform (NSST) can solve the problems existing in wavelet transform. NSCT and NSST have translation invariance and anisotropy. While increasing the number of image decomposition directions and the flexibility of decomposition scale, they can better retain the edge information of the source image, effectively improve the perceptual quality of the fused image, and have a better image visual effect. However, the decomposition methods of NSCT and NSST are relatively time-consuming and computationally inefficient.

[0009] The sparse representation (SR) based method is completely different from the transform domain based method. It learns an over-complete dictionary from a set of training images so that the dictionary can stably and reasonably sparsely represent the source image: first, the input image is sparsely decomposed through the dictionary, then the decomposed sparse coefficients are fused based on the designed fusion rules, and finally the fused image is reconstructed through sparse coefficient reconstruction. The sparse representation based method has relatively better performance than the multi-scale transform based method, but there are still problems such as limited detail retention and sensitivity to mismatched areas. At the same time, the learning of over-complete dictionaries is usually time-consuming.

[0010] That is, among the existing pixel-level image fusion methods, the spatial domain-based methods have problems of insufficient contrast and spectral distortion, the multi-scale transformation-based methods have problems of noise introduction and low computational efficiency, and the sparse representation-based methods have problems of insufficient detail preservation and sensitivity to mismatched areas, all of which will affect the quality of the fusion results to a certain extent. Therefore, the existing image fusion methods still have some problems in the multimodal medical image fusion task and lack effective solutions. Summary of the invention

[0011] In view of the deficiencies in the prior art, the present invention provides an image fusion method and system based on multi-scale transformation and convolution sparse representation, which solves the problems of low contrast, energy loss and insufficient texture details in the weighted average fusion method based on multi-scale transformation, as well as the problems of limited detail retention ability, sensitivity to mismatched areas and weak expression ability of a single dictionary in the fusion method based on sparse representation. The present invention combines multi-scale transformation with convolution sparse representation, sets fusion rules with different adaptabilities, and constructs a multimodal medical image fusion framework with better robustness. The system can not only well preserve the basic structure and texture details of the image, but also preserve the brightness and contrast in the image, thereby realizing the rapid acquisition of high-quality fused images, which helps clinicians make more accurate medical decisions and formulate more effective surgical treatment plans.

[0012] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0013] A first aspect of the present invention provides an image fusion method based on multi-scale transformation and convolution sparse representation.

[0014] An image fusion method based on multi-scale transformation and convolution sparse representation, comprising:

[0015] The source images of different modalities are decomposed into basic sub-images and detail sub-images respectively through rolling guided filtering;

[0016] The basis sub-images of different source images are decomposed into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through non-subsampled shearlet transform;

[0017] Convolution sparse representation encoding is performed on the high-frequency subband and the low-frequency subband of each scale and direction respectively, and high-frequency sparse coefficients and low-frequency sparse coefficients of corresponding multiple scales and directions are obtained respectively;

[0018] Different fusion rules are used to fuse the high-frequency sparse coefficients and low-frequency sparse coefficients of each scale and direction of different source images, so as to obtain high-frequency fused sparse coefficients and low-frequency fused sparse coefficients of multiple scales and directions, and then reconstruct them into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through convolution sparse representation;

[0019] The high-frequency subbands and low-frequency subbands of multiple scales and directions are fused and reconstructed by inverse non-subsampled shearlet transform to obtain a fused basic sub-image;

[0020] By fusing the detail sub-images based on the weighted improved Laplace energy, a fused detail sub-image is obtained;

[0021] The fused basic sub-image and the fused detail sub-image are added together to obtain the final fused image.

[0022] A further technical solution, before performing convolutional sparse representation encoding, further includes:

[0023] Based on the obtained high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions, training and learning are performed to obtain corresponding high-frequency convolutional sparse dictionaries and low-frequency convolutional sparse dictionaries of multiple scales and directions.

[0024] A further technical solution is to train and learn to obtain a low-frequency convolutional sparse dictionary, including: using low-frequency sub-band images as training data sets, and using alternating direction multiplier method to optimally solve the convolutional sparse dictionary learning model to obtain a low-frequency convolutional sparse dictionary.

[0025] A further technical solution is to train and learn to obtain high-frequency convolution sparse dictionaries of corresponding multiple scales and directions, including: using high-frequency sub-band images of different scales and directions as training data sets, using alternating direction multiplier method to optimally solve the convolution sparse dictionary learning model, and learning a high-frequency convolution sparse dictionary group at each scale, each high-frequency convolution sparse dictionary group includes high-frequency convolution sparse dictionaries corresponding to multiple directions at the scale.

[0026] A further technical solution is to perform convolution sparse representation encoding on the high-frequency subband and low-frequency subband of each scale and direction, respectively, to obtain the corresponding high-frequency sparse coefficients and low-frequency sparse coefficients of multiple scales and directions, specifically: using the learned high-frequency convolution sparse dictionary and low-frequency convolution sparse dictionary of multiple scales and directions, perform convolution sparse encoding on the corresponding high-frequency subband and low-frequency subband, respectively, to obtain the corresponding high-frequency sparse coefficients and low-frequency sparse coefficients, respectively.

[0027] A further technical solution is to reconstruct high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through convolutional sparse representation. Specifically, using the learned high-frequency convolutional sparse dictionary and low-frequency convolutional sparse dictionary of multiple scales and directions, convolution operations are performed on the corresponding high-frequency sparse coefficients and low-frequency sparse coefficients to obtain the corresponding high-frequency sub-bands and low-frequency sub-bands.

[0028] A further technical solution is to fuse the high-frequency sparse coefficients and the low-frequency sparse coefficients of each scale and direction using different fusion rules, including:

[0029] The low-frequency sparse coefficients are fused using a fusion rule based on the combination of weighted local energy and average L1 norm;

[0030] A fusion rule based on improved spatial frequency and average L1 norm is used to fuse high-frequency sparse coefficients.

[0031] A second aspect of the present invention provides an image fusion system based on multi-scale transformation and convolution sparse representation.

[0032] An image fusion system based on multi-scale transformation and convolution sparse representation, comprising:

[0033] An image decomposition module is used to decompose source images of different modalities into basic sub-images and detail sub-images respectively through rolling guided filtering;

[0034] The basic sub-image fusion module is used to decompose the basic sub-images of different source images into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through non-subsampled shearlet transform; perform convolution sparse representation encoding on the high-frequency sub-bands and low-frequency sub-bands of each scale and direction, respectively, to obtain the corresponding high-frequency sparse coefficients and low-frequency sparse coefficients of multiple scales and directions; use different fusion rules to fuse the high-frequency sparse coefficients and low-frequency sparse coefficients of each scale and direction of different source images, to obtain high-frequency fused sparse coefficients and low-frequency fused sparse coefficients of multiple scales and directions, and then reconstruct them into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through convolution sparse representation; fuse and reconstruct the high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through inverse non-subsampled shearlet transform to obtain the fused basic sub-image;

[0035] A detail sub-image fusion module is used to fuse the detail sub-images based on the weighted improved Laplace energy to obtain a fused detail sub-image;

[0036] The image fusion module is used to add the fused basic sub-image and the fused detail sub-image to obtain the final fused image.

[0037] A third aspect of the present invention provides a computer-readable storage medium.

[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the image fusion method based on multi-scale transformation and convolution sparse representation as described above.

[0039] A fourth aspect of the present invention provides an electronic device.

[0040] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the image fusion method based on multi-scale transformation and convolution sparse representation as described above are completed.

[0041] One or more of the above technical solutions have the following beneficial effects:

[0042] (1) The present invention provides an image fusion method based on multi-scale transformation and convolution sparse representation, which solves the problem of the influence of noise introduced in the multi-scale transformation method on the fusion result. The source image is decomposed by a rolling guided filtering method. During the decomposition process, the Gaussian filter can remove or smooth the complex small areas in the image and suppress the noise introduced by the multi-scale transformation. At the same time, since the joint bilateral filter has good edge preservation characteristics, the original structure can be retained to the greatest extent.

[0043] (2) The method provided by the present invention adopts non-subsampled shearlet transform to realize multi-scale decomposition of sub-images. Since non-subsampled shearlet transform has multi-scale and translation invariance, it can better preserve the edge and texture details of the image; sparse representation is used to encode the sub-band image, and two adaptive fusion rules are designed for high-frequency coefficients and low-frequency coefficients. The low-frequency fusion rule is measured according to the regional energy, which can well preserve the energy and contour in the image; the high-frequency fusion rule is measured according to the spatial frequency. The gradient of the image can well preserve the texture details in the image, which solves the problems of low contrast, energy loss and insufficient texture details in the traditional weighted average fusion method based on multi-scale transform and the maximum selection fusion method.

[0044] (3) The method provided by the present invention uses convolutional sparse representation (CSR) to encode the entire image. Convolutional sparse representation is a convolution form of traditional sparse representation. Unlike the sparse coding of image blocks in traditional sparse representation, convolutional sparse coding encodes the entire image, so it has better detail retention capabilities and solves the problem of poor fusion effect in mismatched areas. At the same time, multiple dictionaries are used to encode high-frequency subbands of different scales, which solves the problem of insufficient expression ability of a single dictionary, that is, it solves the problems of limited detail retention ability, sensitivity to mismatched areas, and weak expression ability of a single dictionary in traditional fusion methods based on sparse representation.

[0045] (4) The method provided by the present invention can largely solve the problems existing in the existing medical image fusion methods, provide a better solution for the field of medical image fusion, achieve the acquisition of high-quality fused images, and help clinicians make more accurate medical decisions and formulate more effective surgical treatment plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 This is an overall framework diagram of the identification method described in Embodiment 1 of the present invention;

[0048] Figure 2 This is an overall flow chart of the identification method described in Example 1 of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] Embodiment 1

[0052] In the existing pixel-level image fusion technology, the weighted average fusion or maximum fusion selection methods based on multi-scale transformation have the problems of low contrast, energy loss and insufficient texture details, while the fusion method based on sparse representation has the problems of limited detail retention ability, sensitivity to mismatched areas and weak expression ability of a single dictionary. In order to solve the problems that still exist in the above two commonly used fusion methods, this embodiment proposes a multimodal medical image fusion method based on multi-scale transformation and convolutional sparse representation, which combines multi-scale transformation and convolutional sparse representation, sets fusion rules with different adaptability, and constructs a multimodal medical image fusion framework with better robustness, thereby achieving the acquisition of high-quality fused images, which helps clinicians make more accurate medical decisions and formulate more effective surgical treatment plans.

[0053] This embodiment provides an image fusion method based on multi-scale transformation and convolution sparse representation, such as Figure 1 As shown, the method includes:

[0054] Step 1: Decompose the source images of different modes into basic sub-images and detail sub-images respectively through rolling guided filtering;

[0055] Step 2, decomposing the basic sub-images of different source images into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions by non-subsampled shearlet transform;

[0056] Step 3: Perform convolution sparse representation encoding on the high-frequency subband and the low-frequency subband of each scale and direction, respectively, to obtain high-frequency sparse coefficients and low-frequency sparse coefficients of corresponding multiple scales and directions;

[0057] Step 4: Use different fusion rules to fuse the high-frequency sparse coefficients and low-frequency sparse coefficients of each scale and direction of different source images, obtain high-frequency fused sparse coefficients and low-frequency fused sparse coefficients of multiple scales and directions, and then reconstruct them into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through convolution sparse representation;

[0058] Step 5: Reconstruct the high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions by inverse non-subsampled shearlet transform to obtain a fused basic sub-image;

[0059] Step 6: The detail sub-images are fused based on the weighted improved Laplace energy to obtain a fused detail sub-image;

[0060] Step 7: Add the fused basic sub-image and the fused detail sub-image to obtain the final fused image.

[0061] The method described in this embodiment is a multimodal medical image fusion implemented at the pixel level. Pixel-level images are easily affected by noise, which in turn affects the quality of the final fused image. Specifically, combined with the existing image fusion methods, it can be seen that the fusion quality of multimodal medical images mainly depends on two key parts: image decomposition and image fusion rules. Among them, whether the decomposition of the basic information and texture details of the source image is thorough directly affects the quality of image fusion. At the same time, the different fusion rules adopted by the decomposed different sub-images will also affect the final quality of the fused image.

[0062] First, noise is easily introduced in the process of image decomposition. For example, the image fusion method based on multi-scale transformation decomposes the image into multi-scale and multi-directional representations, which can well retain the texture detail information of multiple scales. However, this decomposition process will introduce noise, and medical images themselves have more meaningless noisy texture details. Therefore, this method greatly affects the quality of image fusion.

[0063] In order to solve the influence of noise on the fusion results in the multi-scale transformation method, this embodiment uses a rolling guided filtering method (RGF) to decompose the source images of different modalities into basic sub-images and detail sub-images. The decomposition process can not only keep the clarity and accuracy of the anatomical structure contours in the image, but also remove or smooth the complex small areas in the image, thereby suppressing the influence of noise introduced by multi-scale transformation.

[0064] In step 1 of the scheme described in this embodiment, source images of different modes are decomposed into basic sub-images and detail sub-images respectively through rolling guided filtering, that is, source images A and source images B of different modes are decomposed respectively to obtain basic sub-images and detail sub-images of source image A, and basic sub-images and detail sub-images of source image B.

[0065] like Figure 2 As shown, the rolling guided filter decomposition specifically includes two parts: small structure removal and edge restoration.

[0066] First, small structures are removed by using a Gaussian filter. Specifically, the source image is input into the Gaussian filter to remove structures whose scale is smaller than the standard deviation, and the image G(x) after small structures are removed is obtained. The Gaussian filter is defined as follows:

[0067]

[0068]

[0069] In the above formula, I V represents the input source image A or B, x and y represent the coordinates of the input image, represents the standard deviation, and N(x) represents the set of pixels adjacent to x.

[0070] Secondly, due to the good edge preservation characteristics of the joint bilateral filter, after Gaussian filtering, the edge of the image is restored by the joint bilateral filter. 1 It represents the output G(x) after Gaussian filtering, and J t+1 It is represented as the result of edge recovery after the tth iteration, J t+1 It can be calculated by the following formula:

[0071]

[0072]

[0073] In the above formula, σ s and σ r They represent the spatial weight and distance weight respectively, and in this embodiment are set to 3.0 and 0.1 respectively.

[0074] The above steps of removing small structures and restoring edges are iterated repeatedly until the set number of iterations is met, so as to smooth out irrelevant details in the source image while retaining large-scale structural contours. Therefore, through the above scheme, the source image can be decomposed into a basic sub-image that retains most structural features and a detail sub-image that retains a small amount of texture details, where the basic sub-image b after the i-th iteration is i and detail sub-graph di It can be expressed as:

[0075]

[0076] d i =b i-1 -b i

[0077] In the above formula, RGF represents the rolling guided filter, i is the number of iterations. In this embodiment, the number of iterations of the rolling guided filter is set to 3. After 3 iterations, the two input source images A and B are decomposed into base sub-images (base A , base B ) and detail sub-graph (detail A , detail B ).

[0078] Furthermore, in step 2, the base sub-image is decomposed into high-frequency sub-band images and low-frequency sub-band images of multiple scales and directions by non-subsampled shearlet transform (NSST), wherein the non-subsampled shearlet transform includes two parts: non-subsampled pyramid filter (NSPF) and shift-invariant shearing filter bank (SFB).

[0079] First, the image is decomposed at multiple scales by NSPF, which decomposes the basic sub-image according to many different scales from fine to coarse, and obtains multi-scale high-frequency sub-bands and low-frequency sub-bands; then, the high-frequency sub-bands are decomposed in multiple directions by SFB. For each scale of high-frequency sub-band, SFB can be used to obtain sub-images in multiple different directions within the corresponding frequency band.

[0080] In this embodiment, the basic subgraph base A and the base subgraph base B Three-level NSST scale decomposition is performed to obtain four subbands, including one low-frequency subband and three high-frequency subbands whose size is equal to the input image. For these high-frequency subbands of the three scales, SFB directional decomposition is performed in different directions, and the number of decomposition directions is set to 4, 8, and 8 respectively on the three scales.

[0081] That is, the non-subsampled shearlet transform (NSST) is used to decompose the base sub-graphs base and A and the base subgraph base B The decomposed sub-bands and Among them, s represents the scale of the high-frequency subband, k represents the number of decomposition directions, and Represent the high-frequency subbands of A and B in the s scale and k direction, L A and L BRepresent the low frequency subbands of A and B respectively.

[0082] Furthermore, after obtaining the multi-scale and multi-directional high-frequency subbands and low-frequency subbands of the basic sub-images of the source image A and the source image B, before performing the convolutional sparse representation encoding in step 3, it also includes training and learning to obtain the corresponding high-frequency convolutional sparse dictionaries and low-frequency convolutional sparse dictionaries of multiple scales and directions based on the obtained high-frequency subbands and low-frequency subbands of multiple scales and directions.

[0083] The low-frequency convolution sparse dictionary is obtained by training and learning, including: using the low-frequency sub-band image as a training data set, using the alternating direction multiplier method (ADMM) to optimally solve the convolution sparse dictionary learning model, and obtaining the low-frequency convolution sparse dictionary D0.

[0084] The training and learning obtains high-frequency convolution sparse dictionaries of corresponding multiple scales and directions, including: using high-frequency sub-band images of different scales and directions as training data sets, using alternating direction multiplier method (ADMM) to optimally solve the convolution sparse dictionary learning model, and learning a high-frequency convolution sparse dictionary group at each scale, each high-frequency convolution sparse dictionary group includes high-frequency convolution sparse dictionaries corresponding to multiple directions at the scale. In this embodiment, a total of three high-frequency convolution sparse dictionary groups D1, D2 and D3 can be obtained, and different convolution sparse sub-dictionary groups can capture information corresponding to high-frequency sub-band images of different scales.

[0085] The convolutional sparse dictionary learning model is an optimization problem, which can be expressed as follows:

[0086]

[0087]

[0088] In the above formula, S v，m is the mth high-frequency or low-frequency subband image of image A or B used as training data during training, C k，m is the high-frequency or low-frequency sparse coefficient obtained by encoding the m-th subband image in the k-th direction, d k is the high-frequency dictionary or low-frequency dictionary of multiple scales and directions, and k represents K different decomposition directions. For example, when training the high-frequency subband image at the first scale, a total of K high-frequency sparse dictionaries d in different directions can be obtained. k , k∈[1,K], K high-frequency dictionaries in different directions constitute a set of dictionaries D1, which are used to encode the high-frequency sub-band images of the first scale; when training the low-frequency sub-band images, the low-frequency sub-band images have only one direction, so only one low-frequency convolutional sparse dictionary D0 can be obtained.

[0089] The above-mentioned use of the alternating direction multiplier method (ADMM) to optimally solve the convolutional sparse dictionary learning model specifically refers to the use of the Fourier-based alternating direction multiplier method (ADMM) to effectively solve the biconvex optimization problem in convolutional dictionary learning. Among them, ADMM includes two steps: dictionary update and sparse coefficient update. These two parts are iteratively performed to achieve iterative updates of the dictionary and sparse coefficients to optimize the solution to obtain the final learned dictionary. After the iterative update is completed, the high-frequency dictionary groups at different scales are formed into a set, that is, the high-frequency dictionary D H , the low-frequency dictionary is formed into a set, that is, the low-frequency dictionary D L In this embodiment, the high-frequency dictionary D H is the set of D1, D2 and D3, the low-frequency dictionary D L Equal to D0.

[0090] Furthermore, in step 3, convolution sparse representation encoding is performed on the high-frequency subband and low-frequency subband of each scale and direction, respectively, to obtain high-frequency sparse coefficients and low-frequency sparse coefficients of corresponding multiple scales and directions. That is, the learned high-frequency dictionary D B and the low frequency dictionary D L Multiple high and low frequency sub-bands of source medical images A and B and Convolutional sparse coding is performed to obtain the corresponding high-frequency sparse coefficients and low-frequency sparse coefficients respectively.

[0091] In order to better preserve texture details and eliminate artifacts in the encoding process, we introduce total variation regularization (TV regularization) into the convolutional sparse coding model. Therefore, the sparse coding model can be expressed as follows:

[0092]

[0093] In the above formula, TV(x)=||g x *S V |1+||g y *S v ||1, g x and g y are gradient operators used to calculate the gradients in the row and column directions of the image, respectively. V Represents the subband image of A or B, V∈(A,B).

[0094] The above sparse coding model is quickly optimized and solved by the ADMM algorithm, and the sparse coefficients of the source images A and B can be obtained respectively. and Where L and H represent the low-frequency subband and high-frequency subband corresponding to the coefficient, respectively, and s and k represent the scale and direction of the corresponding subband, respectively.

[0095] Furthermore, in step 4, after obtaining high-frequency sparse coefficients and low-frequency sparse coefficients at multiple scales and directions, different fusion rules are used to fuse the high-frequency sparse coefficients and low-frequency sparse coefficients at each scale and direction to obtain high-frequency fused sparse coefficients and low-frequency fused sparse coefficients at multiple scales and directions.

[0096] Among them, the fusion of low-frequency sparse coefficients is specifically as follows: low-frequency subband L A and L B contains basic information such as the brightness and contour of the image. Therefore, this embodiment adopts a fusion rule based on weighted local energy combined with the average L1 norm to fuse the low-frequency sparse coefficients. The local energy strategy can better reflect the energy and significant features of the image, while the average L1 norm strategy of the convolution sparse coefficients can effectively reduce the impact of misregistration.

[0097] Step 4.1.1 First, calculate the regional energy, the calculation formula is as follows:

[0098]

[0099]

[0100] In the above formula, L V represents the low-frequency subband image of A or B, w is the weight matrix corresponding to the sliding window area, the closer the distance point (i, j), the greater the weight, p and q are the size of the window area. In this embodiment, p=1, q=1 is set, that is, the window size is 3*3, to avoid losing feature information when the window is too large.

[0101] Step 4.1.2 Secondly, calculate the low-frequency sparse coefficient C AL and C BL The average L1 norm of A and B is obtained to obtain the activity level map α of A and B in the low-frequency subband. AL and α BL , the formula is as follows:

[0102] α AL (i, j) = ||C AL (i, j)||1

[0103] α BL (i, j) = ||C BL (i, j)||1

[0104] In the above formula, C AL (i, j) and C BL (i, j) is the low-frequency sparse vector of A and B at position (i, j). Since the low-frequency subband has only one direction, its dimension is 1.

[0105] Step 4.1.3 Then, the final activity level map is obtained through the sliding window based averaging strategy As shown below:

[0106]

[0107] In the above formula, V represents different source images, r represents the radius of the window. In this embodiment, V represents source image A or source image B, and the radius is set to 1, that is, the window size is 3*3.

[0108] Step 4.1.4 Finally, the fused low-frequency sparse coefficient C is calculated according to the following strategy: FL :

[0109]

[0110]

[0111]

[0112] For the fusion of high-frequency sparse coefficients, specifically: considering that a large amount of texture details of the image are contained in the high-frequency sub-bands of the source images A and B and Therefore, a strategy based on improved spatial frequency and average L1 norm is adopted to fuse the high-frequency sparse coefficients. The improved spatial frequency and gradient can well reflect the image texture.

[0113] Step 4.2.1 First, according to the high frequency sub-band Calculate the respective improved spatial frequency and The calculation formula is as follows:

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] In the above formula, Represents the high-frequency subband in the k direction of the s scale, V represents the source image A or the source image B, and M and N represent the size of the high-frequency subband.

[0121] Step 4.2.2 Secondly, calculate the average L1 norm of each high-frequency sparse coefficient to obtain the activity level map and The formula is as follows:

[0122]

[0123]

[0124] In the above formula, and It is the high-frequency sparse vector of A and B at the (i, j) position on the s scale. Since the high-frequency subband has K directions, the dimension of the vector is K.

[0125] Step 4.2.3 Then, the final activity level map is obtained through the sliding window based averaging strategy As shown below:

[0126]

[0127] In the above formula, V represents different source images, r represents the radius of the window, and we set the radius to 1, that is, the window size is 3*3.

[0128] Step 4.2.4 Finally, calculate the fused high-frequency sparse coefficient according to the following formula

[0129]

[0130]

[0131]

[0132] Furthermore, after obtaining the high-frequency fusion sparse coefficients and low-frequency fusion sparse coefficients of multiple scales and directions through the above calculation, they are reconstructed into corresponding high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through convolution sparse representation.

[0133] Specifically, for the low-frequency fusion sparse coefficient, the fused low-frequency sparse coefficient C FL With low frequency sparse dictionary Perform convolution operation and reconstruct the fused low-frequency subband L F , the formula is as follows:

[0134]

[0135] In the above formula, Equal to the low-frequency coefficient dictionary D0.

[0136] For high-frequency sparse coefficients, the fused high-frequency sparse coefficients of different scales are With high frequency sparse dictionary Perform convolution operation to reconstruct the fused high-frequency sub-bands at s scales The formula is as follows:

[0137]

[0138] In the above formula, Represents a sparse dictionary in k directions at s high-frequency scale. The high-frequency sparse dictionary in all K directions at a certain scale constitutes

[0139] Furthermore, in step 5, the high-frequency subbands and low-frequency subbands of multiple scales and directions are fused and reconstructed by inverse non-subsampled shearlet transform to obtain a fused basic sub-image. and low frequency subband L F Perform inverse NSST transformation to obtain the fused base sub-graph base F .

[0140] In step 6, the detail sub-images are fused based on the weighted improved Laplace energy to obtain a fused detail sub-image.

[0141] The modified Laplace energy sum (SML) is suitable for measuring the focus of image details. Therefore, the energy of the detail sub-image is estimated by weighted modified Laplace energy sum (WSML), and the WSML formula is as follows:

[0142]

[0143] ML V (i, j) = |2D V (i, j)-D V (i-1, j)-D V (i+1, j)|+|2D V (i, j)-D V (i, j-1)-D v (i, j+1)|

[0144] In the above formula, D V Represents the input detail subgraph detail A or detail B , p and q define the size of the window. In this embodiment, the window size is set to 3*3, so the weight matrix W is:

[0145]

[0146] P V Defined as detail subgraph detail A or detail BThe energy of the fused detail subgraph F It can be obtained according to the following formula:

[0147]

[0148]

[0149] In the above formula, D V Represents the detail sub-graph detail A or detail sub-graph detail B , P V Represents the energy of the corresponding detail sub-graph.

[0150] In step 7, the fused base sub-image and the fused detail sub-image are added together to obtain the final fused image. F And the fused detail sub-image detail F After that, the two parts are added together to obtain the final fused image I F , the final fused image is calculated as follows:

[0151] I F =base F +detail F

[0152] Through the above-mentioned multimodal medical image fusion method based on non-subsampled shearlet transform and convolution sparse representation, the rapid and effective fusion of multimodal medical images is achieved. Among them, the rolling guided filter (RGF) decomposition solves the influence of the noise introduced in the fusion process on the quality of the fused image; the non-subsampled shearlet (NSST) decomposition and different adaptive fusion rules solve the problems of insufficient contrast, loss of texture details, edge artifacts, etc. that may appear in the fused image; the convolution sparse representation (CSR) method further enhances the retention of texture details and improves the fusion effect of the mismatched area. Therefore, compared with other methods, the scheme described in this embodiment has higher fusion image quality and better robustness of the fusion framework. It is suitable for the fusion of medical images of multiple different modalities, can improve the efficiency of clinicians in diagnosing complex diseases, and helps clinicians make more accurate diagnoses and formulate more appropriate surgical treatment plans.

[0153] In addition, this embodiment verifies the effectiveness of multimodal medical image fusion through experiments. In the verification experiment, 60 different pairs of CT and MR images were prepared. The size of the test images was 256*256, and they had been strictly pre-registered. The test images were all from the Harvard Medical School brain dataset (http: / / www.med.harvard.edu / AANLIB / home.html). The above scheme was used for testing, and the test results were quantitatively compared with three related algorithms (based on non-subsampled shearlet transform, based on sparse representation, and based on convolutional sparse representation). The quantitative evaluation indicators include EN, Q AB / F The results are shown in Table 1 below, which further demonstrates that the method described in this embodiment has better fusion quality.

[0154] Table 1 Experimental quantitative comparison results

[0155]

[0156] Embodiment 2

[0157] This embodiment provides an image fusion system based on multi-scale transformation and convolution sparse representation, the system comprising:

[0158] An image decomposition module is used to decompose source images of different modalities into basic sub-images and detail sub-images respectively through rolling guided filtering;

[0159] The basic sub-image fusion module is used to decompose the basic sub-images of different source images into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through non-subsampled shearlet transform; perform convolution sparse representation encoding on the high-frequency sub-bands and low-frequency sub-bands of each scale and direction, respectively, to obtain the corresponding high-frequency sparse coefficients and low-frequency sparse coefficients of multiple scales and directions; use different fusion rules to fuse the high-frequency sparse coefficients and low-frequency sparse coefficients of each scale and direction of different source images, to obtain high-frequency fused sparse coefficients and low-frequency fused sparse coefficients of multiple scales and directions, and then reconstruct them into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through convolution sparse representation; fuse and reconstruct the high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through inverse non-subsampled shearlet transform to obtain the fused basic sub-image;

[0160] A detail sub-image fusion module is used to fuse the detail sub-images based on the weighted improved Laplace energy to obtain a fused detail sub-image;

[0161] The image fusion module is used to add the fused basic sub-image and the fused detail sub-image to obtain the final fused image.

[0162] Embodiment 3

[0163] This embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the image fusion method based on multi-scale transformation and convolution sparse representation as described above are completed.

[0164] Embodiment 4

[0165] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps in the image fusion method based on multi-scale transformation and convolution sparse representation as described above are completed.

[0166] The steps involved in the above embodiments 2 to 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0167] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0168] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0169] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An image fusion method based on multi-scale transformation and convolution sparse representation, characterized in that: include: The source images of different modalities are decomposed into basic sub-images and detail sub-images respectively through rolling guided filtering; The basis sub-images of different source images are decomposed into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through non-subsampled shearlet transform; Convolution sparse representation encoding is performed on the high-frequency subband and the low-frequency subband of each scale and direction respectively, and high-frequency sparse coefficients and low-frequency sparse coefficients of corresponding multiple scales and directions are obtained respectively; Different fusion rules are used to fuse the high-frequency sparse coefficients and low-frequency sparse coefficients of each scale and direction of different source images, so as to obtain high-frequency fused sparse coefficients and low-frequency fused sparse coefficients of multiple scales and directions, and then reconstruct them into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through convolution sparse representation; The high-frequency subbands and low-frequency subbands of multiple scales and directions are fused and reconstructed by inverse non-subsampled shearlet transform to obtain a fused basic sub-image; By fusing the detail sub-images based on the weighted improved Laplace energy, a fused detail sub-image is obtained; The fused basic sub-image and the fused detail sub-image are added together to obtain the final fused image.

2. The image fusion method based on multi-scale transformation and convolution sparse representation as claimed in claim 1, characterized in that: Before convolutional sparse representation encoding, it also includes: Based on the obtained high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions, training and learning are performed to obtain corresponding high-frequency convolutional sparse dictionaries and low-frequency convolutional sparse dictionaries of multiple scales and directions.

3. The image fusion method based on multi-scale transformation and convolution sparse representation as claimed in claim 2, characterized in that: The low-frequency convolution sparse dictionary is obtained by training and learning, including: using the low-frequency sub-band image as a training data set, using the alternating direction multiplier method to optimally solve the convolution sparse dictionary learning model, and obtaining the low-frequency convolution sparse dictionary.

4. The image fusion method based on multi-scale transformation and convolution sparse representation as claimed in claim 2, characterized in that: The training and learning obtains high-frequency convolution sparse dictionaries of corresponding multiple scales and directions, including: using high-frequency sub-band images of different scales and directions as training data sets, using alternating direction multiplier method to optimally solve the convolution sparse dictionary learning model, and learning a high-frequency convolution sparse dictionary group at each scale, each high-frequency convolution sparse dictionary group includes high-frequency convolution sparse dictionaries corresponding to multiple directions at the scale.

5. The image fusion method based on multi-scale transformation and convolution sparse representation as claimed in claim 1, characterized in that: Convolution sparse representation encoding is performed on the high-frequency subband and low-frequency subband of each scale and direction, respectively, to obtain the corresponding high-frequency sparse coefficients and low-frequency sparse coefficients of multiple scales and directions, specifically: using the learned high-frequency convolution sparse dictionary and low-frequency convolution sparse dictionary of multiple scales and directions, convolution sparse encoding is performed on the corresponding high-frequency subband and low-frequency subband, respectively, to obtain the corresponding high-frequency sparse coefficients and low-frequency sparse coefficients.

6. The image fusion method based on multi-scale transformation and convolution sparse representation as claimed in claim 1, characterized in that: Different fusion rules are used to fuse the high-frequency sparse coefficients and low-frequency sparse coefficients of each scale and direction, including: The low-frequency sparse coefficients are fused using a fusion rule based on the combination of weighted local energy and average L1 norm; A fusion rule based on improved spatial frequency and average L1 norm is used to fuse high-frequency sparse coefficients.

7. The image fusion method based on multi-scale transformation and convolutional sparse representation as claimed in claim 1, characterized in that: The high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions are reconstructed through convolutional sparse representation. Specifically, the high-frequency convolutional sparse dictionaries and low-frequency convolutional sparse dictionaries of multiple scales and directions are learned to perform convolution operations on the corresponding high-frequency sparse coefficients and low-frequency sparse coefficients to obtain the corresponding high-frequency sub-bands and low-frequency sub-bands.

8. An image fusion system based on multi-scale transformation and convolution sparse representation, characterized in that: include: An image decomposition module is used to decompose source images of different modalities into basic sub-images and detail sub-images respectively through rolling guided filtering; The basic sub-image fusion module is used to decompose the basic sub-images of different source images into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through non-subsampled shearlet transform; convolution sparse representation encoding is performed on the high-frequency sub-band and low-frequency sub-band of each scale and direction, and high-frequency sparse coefficients and low-frequency sparse coefficients of corresponding multiple scales and directions are obtained respectively; Different fusion rules are used to fuse the high-frequency sparse coefficients and low-frequency sparse coefficients of each scale and direction of different source images, so as to obtain high-frequency fused sparse coefficients and low-frequency fused sparse coefficients of multiple scales and directions, and then reconstruct them into high-frequency sub-bands and low-frequency sub-bands of multiple scales and directions through convolution sparse representation; The high-frequency subbands and low-frequency subbands of multiple scales and directions are fused and reconstructed by inverse non-subsampled shearlet transform to obtain a fused basic sub-image; A detail sub-image fusion module is used to fuse the detail sub-images based on the weighted improved Laplace energy to obtain a fused detail sub-image; The image fusion module is used to add the fused basic sub-image and the fused detail sub-image to obtain the final fused image.

9. A computer-readable storage medium, characterized in that: There are a plurality of instructions, which are suitable for being loaded by a processor of a terminal device and executing the steps of the image fusion method based on multi-scale transformation and convolution sparse representation according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the image fusion method based on multi-scale transformation and convolution sparse representation as described in any one of claims 1 to 7 are completed.

Images