Multimodal medical image fusion method, device and terminal equipment

Abstract

The present invention proposes a multimodal medical image fusion method, apparatus, and terminal device. The method comprises the following steps: optimizing and reconstructing the original medical image to obtain a reconstructed medical image; fusing the reconstructed medical image to obtain a pre-fused medical image; selecting the original medical image with the highest detail richness as a guide image; and performing detail-guided fusion based on the pre-fused medical image and the guide image. By optimizing and reconstructing the original medical image to obtain a reconstructed medical image, the present invention can reduce the impact of noise that may exist in the original medical image on fusion performance while ensuring the visual fidelity of the reconstructed medical image. By performing detail-guided fusion based on the pre-fused medical image and the guide image, the present invention can integrate as much detail and texture information as possible from the original medical image into the final fusion result.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and in particular relates to a multimodal medical image fusion method, apparatus and terminal device. Background Art

[0002] At present, with the development of image fusion technology, many multimodal image fusion methods have been proposed. Existing fusion methods can be roughly divided into spatial domain-based fusion methods and frequency domain-based fusion methods. Spatial domain-based fusion methods directly fuse the original multimodal medical images and achieve a certain fusion effect by designing different fusion rules, including simple weighted average fusion, principal component analysis-based fusion methods, and filter operator-based fusion methods. Such methods will lose some details and texture information. In frequency domain-based fusion methods, some transformation algorithms are often used to transform the original image into a multi-scale transformation space, and different fusion rules are designed for different components. Such methods have high time complexity, and the setting of decomposition parameters directly affects the fusion performance.

[0003] Most existing fusion methods fail to consider the vulnerability of medical images to interference from magnetic fields and external environments during the imaging process, resulting in multimodal medical images containing noisy information. Furthermore, most fusion methods fail to fully preserve the detailed information in the original multimodal images, which is crucial information for medical diagnosis and treatment.

[0004] Therefore, traditional technical solutions have the problem that multimodal medical images contain noise information and cannot fully preserve the detail information in the original multimodal images. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal medical image fusion method, apparatus and terminal device, aiming to solve the problem that traditional technical solutions have multimodal medical images containing noise information and cannot fully retain the detail information in the original multimodal images.

[0006] A first aspect of an embodiment of the present invention provides a multimodal medical image fusion method, the method comprising the following steps:

[0007] Optimizing and reconstructing the original medical image to obtain a reconstructed medical image;

[0008] fusing the reconstructed medical images to obtain a pre-fused medical image;

[0009] Selecting the original medical image with the highest detail richness as a guide image;

[0010] Detail loop guided fusion is performed based on the pre-fused medical image and the guide image.

[0011] By optimizing and reconstructing the original medical image to obtain a reconstructed medical image, the impact of possible noise in the original medical image on the fusion performance can be reduced while ensuring the visual fidelity of the reconstructed medical image. By performing detail loop guided fusion based on the pre-fused medical image and the guide image, as much detail and texture information as possible can be integrated from the original medical image into the final fusion result.

[0012] In one embodiment, optimizing and reconstructing the original medical image to obtain the reconstructed medical image comprises the following steps:

[0013] performing singular value decomposition on the original medical image;

[0014] An optimal reconstruction rank is selected to optimize and reconstruct the original medical image to obtain the reconstructed medical image.

[0015] By optimizing and reconstructing the original medical image to obtain a reconstructed medical image, the impact of possible noise in the original medical image on the fusion performance can be reduced while ensuring the visual fidelity of the reconstructed medical image.

[0016] In one embodiment, selecting the optimal reconstruction rank to optimize and reconstruct the original medical image to obtain the reconstructed medical image comprises the following steps:

[0017] Constructing a visual fidelity loss function based on the reconstruction result of the original medical image under the reconstruction rank;

[0018] Obtaining the optimal reconstruction rank through the visual fidelity loss function;

[0019] The original medical image is optimized and reconstructed using the optimal reconstruction rank to obtain the reconstructed medical image.

[0020] By selecting the optimal reconstruction rank to optimize and reconstruct the original medical image, the reconstructed medical image is obtained, which can maximize the denoising of the original medical image while ensuring visual fidelity and retain the details and texture information in the original medical image.

[0021] In one embodiment, fusing the reconstructed medical images to obtain a pre-fused medical image includes: fusing the reconstructed medical images through a guided filtering fusion operator to obtain the pre-fused medical image.

[0022] By fusing the reconstructed medical images, a pre-fused medical image is obtained, and the pre-fused medical image can be used to perform detail loop-guided fusion with a guide image, thereby obtaining a final fusion result.

[0023] In one embodiment, selecting the original medical image with the highest detail richness as the guide image comprises the following steps:

[0024] Construct detail richness decision function based on edge strength operator;

[0025] The original medical image is input into the detail richness decision function to obtain the guide image.

[0026] By selecting the original medical image with the highest detail richness as the guide image, the guide image can be used to perform detail-loop guided fusion with the pre-fused medical image, so that the final fusion result retains as much detail and texture information as possible in the original medical image.

[0027] In one embodiment, performing detail loop guided fusion based on the pre-fused medical image and the guide image comprises the following steps:

[0028] Fusing the pre-fused medical image with the guide image to obtain a first cycle fusion result;

[0029] The nth cycle fusion result is fused with the guide image to obtain the n+1th cycle fusion result, where n≥1, until the best cycle fusion result is obtained.

[0030] By performing detail-loop guided fusion based on the pre-fused medical image and the guide image, as much detail and texture information as possible can be integrated from the original medical image into the final fusion result.

[0031] In one embodiment, obtaining the best loop fusion result includes: constructing a fusion quality evaluation function based on the nth loop fusion result and the (n+1)th loop fusion result, and obtaining the best loop fusion result when the fusion quality evaluation function converges.

[0032] By constructing a fusion quality evaluation function, the optimal loop fusion result is obtained when the fusion quality evaluation function converges, so that the optimal loop fusion result can retain as much detail and texture information as possible in the original medical image.

[0033] A second aspect of an embodiment of the present invention provides a multimodal medical image fusion device, comprising:

[0034] A reconstruction module, used to optimize and reconstruct the original medical image to obtain a reconstructed medical image;

[0035] a pre-fusion module, configured to fuse the reconstructed medical images to obtain a pre-fused medical image;

[0036] a selection module, configured to select the original medical image with the highest detail richness as a guide image;

[0037] A loop fusion module is used to perform detail loop guided fusion based on the pre-fused medical image and the guide image.

[0038] A third aspect of an embodiment of the present invention provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the computer program.

[0039] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0040] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: the above-mentioned multimodal medical image fusion method obtains a reconstructed medical image by optimizing and reconstructing the original medical image, and can reduce the impact of noise that may exist in the original medical image on the fusion performance while ensuring the visual fidelity of the reconstructed medical image. By performing detail loop guided fusion based on the pre-fused medical image and the guide image, as much detail and texture information as possible can be integrated from the original medical image into the final fusion result. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A schematic diagram of a multimodal medical image fusion method provided in an embodiment of the present invention;

[0043] Figure 2 A schematic flow chart of step S101 of a multimodal medical image fusion method provided in an embodiment of the present invention;

[0044] Figure 3 A flowchart of steps S101 to S1012 of a multimodal medical image fusion method provided in an embodiment of the present invention;

[0045] Figure 4 A schematic flow chart of step S103 of a multimodal medical image fusion method provided in an embodiment of the present invention;

[0046] Figure 5 A schematic flow chart of step S104 of a multimodal medical image fusion method provided in an embodiment of the present invention;

[0047] Figure 6 An overall flow chart of a multimodal medical image fusion method provided by an embodiment of the present invention;

[0048] Figure 7 A schematic structural diagram of a multimodal medical image fusion device provided by an embodiment of the present invention;

[0049] Figure 8 A schematic structural diagram of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0052] The first aspect of this embodiment provides a multimodal medical image fusion method, such as Figure 1 As shown, the method includes the following steps:

[0053] S101. Optimize and reconstruct the original medical image to obtain a reconstructed medical image.

[0054] In this embodiment, it should be noted that the optimization and reconstruction of the original medical image to obtain the reconstructed medical image can be performed by optimizing and reconstructing multiple original medical images separately through a fidelity-driven medical image reconstruction algorithm. Based on this algorithm, the original medical image can be denoised while ensuring visual fidelity.

[0055] S102: Fusing the reconstructed medical images to obtain a pre-fused medical image.

[0056] In this embodiment, it should be noted that the fusing of the reconstructed medical images to obtain the pre-fused medical image may be a fusion of multiple reconstructed medical images obtained by optimizing and reconstructing multiple original medical images.

[0057] S103: Select the original medical image with the highest detail richness as a guide image.

[0058] In this embodiment, it should be noted that the selection of the original medical image with the highest detail richness as the guide image can be achieved by measuring which image among multiple original medical images contains the most texture and detail information, and then using the original medical image as the guide image.

[0059] S104: Perform detail circular guided fusion based on the pre-fused medical image and the guide image.

[0060] In this embodiment, it should be noted that the detail loop guided fusion based on the pre-fused medical image and the guide image can be performed by a fusion algorithm based on detail loop guided retention to obtain a final fusion result by performing detail loop guided fusion on the pre-fused medical image and the guide image.

[0061] By optimizing and reconstructing the original medical image to obtain a reconstructed medical image, the impact of possible noise in the original medical image on the fusion performance can be reduced while ensuring the visual fidelity of the reconstructed medical image. By performing detail loop guided fusion based on the pre-fused medical image and the guide image, as much detail and texture information as possible can be integrated from the original medical image into the final fusion result.

[0062] Alternatively, as Figure 2 As shown, step S101 optimizes and reconstructs the original medical image to obtain the reconstructed medical image, including the following steps:

[0063] S1011. Perform singular value decomposition on the original medical image.

[0064] In this embodiment, it should be noted that the singular value decomposition of the original medical image can be expressed as: X=U∑V T , where X∈i m×n Represents the original medical image, U is an m×m matrix, V is an n×n matrix, ∑ is a matrix in which all elements except the main diagonal are 0, and the elements on the main diagonal are called singular values.

[0065] S1012: Select an optimal reconstruction rank to optimize and reconstruct the original medical image to obtain the reconstructed medical image.

[0066] In this embodiment, it should be noted that the original medical image can be restored by singular value reconstruction: Where σ is the singular value in the matrix ∑, u and v are the singular matrices in the matrices U and V, and r is the rank obtained by the singular value decomposition of X. After selecting the reconstruction rank, the above formula can be used to reconstruct the original medical image. When the reconstruction rank is equal to r, the reconstructed medical image is consistent with the original medical image. Selecting different reconstruction ranks will result in different reconstructed medical images.

[0067] By optimizing and reconstructing the original medical image to obtain a reconstructed medical image, the impact of possible noise in the original medical image on the fusion performance can be reduced while ensuring the visual fidelity of the reconstructed medical image.

[0068] Alternatively, as Figure 3 As shown, step S1012 selects the optimal reconstruction rank to optimize and reconstruct the original medical image, and obtaining the reconstructed medical image includes the following steps:

[0069] S10121. Construct a visual fidelity loss function based on the reconstruction result of the original medical image under the reconstruction rank.

[0070] In this embodiment, it should be noted that the visual fidelity loss function constructed based on the reconstruction result of the original medical image under the reconstruction rank is: I (p) = X p (I)-U t (X p (I)), where X p (I) represents the reconstruction result of the original medical image I under the reconstruction rank p, 1≤p≤r, U t (·) represents the weighted mean curvature denoising filter, which is calculated as follows: U t +1 =U t +δH w (U t ), this formula can be used to perform strong edge and detail texture preserving filtering on the image, where t represents the number of iterations, δ represents the weighted average filter step size, and H w (U t ) indicates about U t The weighted average curvature filter is calculated as follows:

[0071]

[0072] Among them, H(U) can be calculated as:

[0073]

[0074] in represents the gradient operator, represents the divergence operator, n represents the dimension, and when the original medical image I is a 2D image, n=2.

[0075] S10122. Obtain the optimal reconstruction rank using the visual fidelity loss function.

[0076] In this embodiment, it should be noted that when T I (p) = X p (I)-U t (X p (I)) remains unchanged as the reconstruction rank p increases, i.e. When it is 0, it means that the increase of reconstruction rank p cannot further improve the strong edge, detail and texture information in the reconstruction result. At this time, the reconstruction rank p is the optimal reconstruction rank.

[0077] S10123. Optimize and reconstruct the original medical image using the optimal reconstruction rank to obtain the reconstructed medical image.

[0078] In this embodiment, it should be noted that the calculation formula for obtaining the reconstructed medical image by optimizing and reconstructing the original medical image through the optimal reconstruction rank is:

[0079] X′=σ1u1v1 T +σ2u2v2 T +...+σ p u p v p T .

[0080] By selecting the optimal reconstruction rank to optimize and reconstruct the original medical image, the reconstructed medical image is obtained, which can maximize the denoising of the original medical image while ensuring visual fidelity and retain the details and texture information in the original medical image.

[0081] Optionally, step S102 of fusing the reconstructed medical images to obtain a pre-fused medical image includes: fusing the reconstructed medical images through a guided filter fusion operator to obtain the pre-fused medical image.

[0082] In this embodiment, it should be noted that the original medical image I A with I B The reconstructed medical images obtained after optimization and reconstruction are I AR and I BR , using guided filtering fusion operator to I AR and I BR Fusion is performed to obtain pre-fused medical image I FR :

[0083]

[0084] Among them G r,ε (P,I) represents the guided filtering operation, r and ε represent the filter size and blur parameter of the guided filter respectively, and B n With D n Represent the original medical image I n The base layer and detail layer are calculated as B n =I n* Z, Z represents the mean filter, D n =I n -B n , when n is 1, I n For I A , when n is 2, I n For I B , N is 2, the above formula can be recorded as I FR =GF(I AR ,I BR ), that is, GF(·) is used to represent the computational process of image fusion through guided filter fusion operator.

[0085] By fusing the reconstructed medical images, a pre-fused medical image is obtained, and the pre-fused medical image can be used to perform detail loop-guided fusion with a guide image, thereby obtaining a final fusion result.

[0086] Alternatively, as Figure 4 As shown, step S103 of selecting the original medical image with the highest detail richness as the guide image includes the following steps:

[0087] S1031. Construct a detail richness decision function based on an edge strength operator.

[0088] In this embodiment, it should be noted that the detail richness decision function constructed based on the edge strength operator is:

[0089]

[0090] Among them, I D represents the original medical image with the highest detail richness, I A with I B represents the original medical image, EI(·) represents the edge intensity operator, and is calculated as follows:

[0091]

[0092] Among them, m and n are the local window sizes centered at pixel (i, j), and w x and w y is the weight matrix used to extract the horizontal and vertical texture of the image.

[0093] S1032: Input the original medical image into the detail richness decision function to obtain the guide image.

[0094] In this embodiment, it should be noted that the original medical image I A with I B By inputting the detail richness judgment function, the original medical image I with the highest detail richness can be obtained. D , and use it as the guide image.

[0095] By selecting the original medical image with the highest detail richness as the guide image, the guide image can be used to perform detail-loop guided fusion with the pre-fused medical image, so that the final fusion result retains as much detail and texture information as possible in the original medical image.

[0096] Alternatively, as Figure 5 As shown, step S104 of performing detail loop guided fusion based on the pre-fused medical image and the guide image includes the following steps:

[0097] S1041: Fuse the pre-fused medical image with the guide image to obtain a first cycle fusion result.

[0098] In this embodiment, it should be noted that the fusion of the pre-fused medical image and the guide image to obtain the first cycle fusion result can be expressed as I FR 1 =GF(I D ,I FR ).

[0099] S1042: Fusing the nth cyclic fusion result with the guide image to obtain the n+1th cyclic fusion result, where n≥1, until the best cyclic fusion result is obtained.

[0100] In this embodiment, it should be noted that the nth cycle fusion result is fused with the guide image to obtain the n+1th cycle fusion result, which can be expressed as

[0101] By performing detail-loop guided fusion based on the pre-fused medical image and the guide image, as much detail and texture information as possible can be integrated from the original medical image into the final fusion result.

[0102] Optionally, obtaining the best loop fusion result includes: constructing a fusion quality evaluation function based on the nth loop fusion result and the (n+1)th loop fusion result, and obtaining the best loop fusion result when the fusion quality evaluation function converges.

[0103] In this embodiment, it should be noted that mutual information (MIN), visual information fidelity (VIF), signal-to-noise ratio (PSNR), etc. can be used to construct a fusion quality evaluation function, which is expressed as Q(·). The condition for the convergence of the fusion quality evaluation function is Or Q′(n)=0, when the convergence condition is met, the fusion result I cannot be further improved by increasing the number of cycles n. FR n This means that the fusion result obtained under the current number of cycles is the best cycle fusion result I F The detailed loop guided fusion based on the pre-fused medical image and the guide image can be expressed as I F =MSGF(I D ,I FR ).

[0104] By constructing a fusion quality evaluation function, the optimal loop fusion result is obtained when the fusion quality evaluation function converges, so that the optimal loop fusion result can retain as much detail and texture information as possible in the original medical image.

[0105] This embodiment provides a general flow chart of the multimodal medical image fusion method. Figure 6 As shown, by optimizing and reconstructing the original medical image to obtain a reconstructed medical image, the impact of possible noise in the original medical image on the fusion performance can be reduced while ensuring the visual fidelity of the reconstructed medical image. By performing detail loop guided fusion based on the pre-fused medical image and the guide image, as much detail and texture information as possible can be integrated from the original medical image into the final fusion result.

[0106] The second aspect of this embodiment provides a multimodal medical image fusion device, such as Figure 7 Shown, including:

[0107] Reconstruction module, used to execute or implement the above Figure 1 Corresponding to step S101 in each embodiment;

[0108] Pre-fusion module, used to execute or implement the above Figure 1 Corresponding to step S102 in each embodiment;

[0109] Select a module to execute or implement the above Figure 1 Corresponding to step S103 in each embodiment;

[0110] Loop fusion module, used to perform or implement the above Figure 1 This corresponds to step S104 in each embodiment.

[0111] A third aspect of this embodiment provides a terminal device, such as Figure 8 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described above is implemented.

[0112] A fourth aspect of this embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0113] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A multimodal medical image fusion method, characterized in that: The method comprises the following steps: Optimizing and reconstructing the original medical image to obtain a reconstructed medical image; fusing the reconstructed medical images to obtain a pre-fused medical image; Selecting the original medical image with the highest detail richness as a guide image; performing detail loop guided fusion based on the pre-fused medical image and the guide image; The step of optimizing and reconstructing the original medical image to obtain the reconstructed medical image comprises the following steps: performing singular value decomposition on the original medical image; Selecting the optimal reconstruction rank to optimize and reconstruct the original medical image to obtain the reconstructed medical image specifically includes the following steps: The visual fidelity loss function is constructed based on the reconstruction result of the original medical image under the reconstruction rank: I (p) = X p (I)-U t (X p (I)), where X p (I) represents the reconstruction result of the original medical image I under the reconstruction rank p, 1≤p≤r, U t (·) represents weighted mean curvature denoising filter; Obtaining the optimal reconstruction rank through the visual fidelity loss function; The original medical image is optimized and reconstructed using the optimal reconstruction rank to obtain the reconstructed medical image.

2. The multimodal medical image fusion method according to claim 1, wherein: The fusing the reconstructed medical images to obtain the pre-fused medical image includes: fusing the reconstructed medical images through a guided filtering fusion operator to obtain the pre-fused medical image.

3. The multimodal medical image fusion method according to claim 1, wherein: The selecting the original medical image with the highest detail richness as the guide image comprises the following steps: Construct detail richness decision function based on edge strength operator; The original medical image is input into the detail richness decision function to obtain the guide image.

4. The multimodal medical image fusion method according to claim 1, wherein: The performing of detail loop guided fusion based on the pre-fused medical image and the guide image comprises the following steps: Fusing the pre-fused medical image with the guide image to obtain a first cycle fusion result; The nth cycle fusion result is fused with the guide image to obtain the n+1th cycle fusion result, where n≥1, until the best cycle fusion result is obtained.

5. The multimodal medical image fusion method according to claim 4, wherein: Obtaining the best loop fusion result includes: constructing a fusion quality evaluation function based on the nth loop fusion result and the (n+1)th loop fusion result, and obtaining the best loop fusion result when the fusion quality evaluation function converges.

6. A multimodal medical image fusion device, characterized in that: include: A reconstruction module, used to optimize and reconstruct the original medical image to obtain a reconstructed medical image; a pre-fusion module, configured to fuse the reconstructed medical images to obtain a pre-fused medical image; a selection module, configured to select the original medical image with the highest detail richness as a guide image; a loop fusion module, configured to perform detail loop guided fusion based on the pre-fused medical image and the guide image; The reconstruction module is specifically used to: perform singular value decomposition on the original medical image; Selecting the optimal reconstruction rank to optimize and reconstruct the original medical image to obtain the reconstructed medical image specifically includes the following steps: The visual fidelity loss function is constructed based on the reconstruction result of the original medical image under the reconstruction rank: I (p) = X p (I)-U t (X p (I)), where X p (I) represents the reconstruction result of the original medical image I under the reconstruction rank p, 1≤p≤r, U t (·) represents weighted mean curvature denoising filter; Obtaining the optimal reconstruction rank through the visual fidelity loss function; The original medical image is optimized and reconstructed using the optimal reconstruction rank to obtain the reconstructed medical image.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images