Multimodal medical image fusion method and system based on edge region segmentation detection

Abstract

The present invention discloses a multimodal medical image fusion method and system based on edge region segmentation detection, the method comprising: obtaining a source image and performing image decomposition processing respectively to obtain an energy layer and a detail layer of the source image; introducing a fuzzy region active contour operator and a heat conduction matrix algorithm to perform fusion processing to obtain a pre-fused edge detail layer; performing non-edge feature detection and fusion processing on the detail layer of the source image to obtain a pre-fused significant detail layer; performing fusion processing through an absolute maximum operation algorithm to obtain a pre-fused base layer; combining the pre-fused edge detail layer, the pre-fused significant detail layer and the pre-fused base layer to construct a multimodal medical image fusion result. The present invention can accurately extract the anatomical or functional information of the source image. As a multimodal medical image fusion method and system based on edge region segmentation detection, the present invention can be widely used in the field of multimodal image fusion technology.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal image fusion, and in particular to a multimodal medical image fusion method and system based on edge region segmentation detection. Background Art

[0002] Multimodal medical images cover anatomical and functional images, providing a comprehensive view of human organs from different perspectives. The current multimodal medical image fusion methods are mainly divided into four categories: deep learning-based methods, multiscale transformation-based methods, sparse representation-based methods, and edge-preserving-based methods. Deep learning-based methods mainly use autoencoders, convolutional neural networks, generative adversarial networks, transformers, and diffusion models, which have excellent feature extraction capabilities. Generative model-based methods aim to make the source image and the fused image have similar distributions, but these methods lack interpretability and controllability, and training is also more complicated. Multiscale transformation-based methods include non-subsampled contourlet transform, non-subsampled shearlet transform, and neural P system. Although the methods based on multi-scale transformation are flexible, they may still lose basic features when adaptively selecting decomposition and reconstruction parameters. The methods based on sparse representation still face unresolved challenges, including the need for high-discriminant dictionaries, efficient training in sparse domains, and high computational requirements. In addition, the edge-preserving methods emphasize maintaining the edge details of the image and can be combined with a variety of fusion strategies. However, their performance depends on the design of the decomposition and fusion algorithms. In summary, the related algorithms have the problem that the fused images lack key diagnostic features, so that the final generated fused images incorrectly express the anatomical or functional information of the original images. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a multimodal medical image fusion method and system based on edge region segmentation detection, which can effectively retain the edge features, significant structural details and energy information in the source image at the same time, and then accurately extract the anatomical or functional information of the source image.

[0004] The first technical solution adopted by the present invention is: a multimodal medical image fusion method based on edge region segmentation detection, comprising the following steps:

[0005] Acquire a first source image and a second source image and perform image decomposition processing on them respectively to obtain an energy layer of the first source image, a detail layer of the first source image, an energy layer of the second source image, and a detail layer of the second source image;

[0006] The fuzzy region active contour operator and heat conduction matrix algorithm are introduced to fuse the detail layer of the first source image with the detail layer of the second source image to obtain a pre-fused edge detail layer.

[0007] Performing non-edge feature detection and fusion processing on the detail layer of the first source image and the detail layer of the second source image to obtain a pre-fused significant detail layer;

[0008] Acquire the insignificant detail layer of the first source image and the insignificant detail layer of the second source image, combine the energy layer of the first source image and the energy layer of the second source image, perform fusion processing through an absolute maximum operation algorithm, and obtain a pre-fused base layer;

[0009] The pre-fused edge detail layer, the pre-fused significant detail layer and the pre-fused base layer are combined to construct a multimodal medical image fusion result.

[0010] Furthermore, the first source image and the second source image are subjected to image decomposition processing by an average filter operation, and the expression is:

[0011] ;

[0012] In the above formula, Respectively represent The energy layer and detail layer of an image, represents pixel coordinates, Indicates images, represents the averaging filter, Represents a convolution operation.

[0013] Furthermore, the step of introducing the fuzzy region active contour operator and the heat conduction matrix algorithm to fuse the detail layer of the first source image with the detail layer of the second source image to obtain a pre-fused edge detail layer specifically includes:

[0014] A threshold is set, and a fuzzy region active contour operator is introduced to perform salient region extraction processing on the first source image and the second source image respectively, so as to obtain an initial salient layer of the first source image and an initial salient layer of the second source image;

[0015] Performing edge information detection processing on the initial saliency layer of the first source image and the initial saliency layer of the second source image respectively by using a heat conduction matrix algorithm to obtain an edge detail layer of the first source image and an edge detail layer of the second source image;

[0016] An edge detail layer fusion algorithm is constructed by combining the gradient amplitude and the pixel value, and the edge detail layer of the first source image is fused with the edge detail layer of the second source image to obtain an edge fusion decision graph;

[0017] The detail layer of the first source image, the detail layer of the second source image and the edge fusion decision graph are fused to obtain a pre-fused edge detail layer.

[0018] Furthermore, it also includes introducing the fuzzy region active contour operator and the heat conduction matrix algorithm to construct an active contour-heat conduction model based on the fuzzy region, and its specific expression is as follows:

[0019] ;

[0020] In the above formula, represents the active contour based on fuzzy region in the active contour based on fuzzy region-heat conduction model, represents the fuzzy region term with fuzzy sets, represents the edge energy term, represents the edge detector, represents the pseudo level set membership function.

[0021] Furthermore, the step of performing non-edge feature detection and fusion processing on the detail layer of the first source image and the detail layer of the second source image to obtain a pre-fused significant detail layer specifically includes:

[0022] Performing edge detail feature elimination processing on the detail layer of the first source image and the detail layer of the second source image respectively to obtain a non-edge detail layer of the first source image and a non-edge detail layer of the second source image;

[0023] Performing salient region extraction processing on the first source image and the second source image respectively by using a fuzzy region active contour operator to obtain a salient decision map of the first source image and a salient decision map of the second source image;

[0024] Combining the local phase coherence strength algorithm and the pseudo level set function, a salient detail layer fusion algorithm is constructed;

[0025] Extracting and processing the non-edge detail layer of the first source image and the salient decision graph of the first source image by using a salient detail layer fusion algorithm to obtain a salient detail layer of the first source image;

[0026] Extracting and processing the non-edge detail layer of the second source image and the salient decision graph of the second source image by using a salient detail layer fusion algorithm to obtain a salient detail layer of the second source image;

[0027] The salient detail layer of the first source image is fused with the salient detail layer of the second source image to obtain a pre-fused salient detail layer.

[0028] Further, the step of obtaining the insignificant detail layer of the first source image and the insignificant detail layer of the second source image and combining the energy layer of the first source image and the energy layer of the second source image, and performing fusion processing through an absolute maximum operation algorithm to obtain a pre-fused base layer specifically includes:

[0029] Performing significant detail layer elimination processing on the non-edge detail layer of the first source image and the non-edge detail layer of the second source image respectively to obtain the non-significant detail layer of the first source image and the non-significant detail layer of the second source image;

[0030] The non-significant detail layer of the first source image is fused with the energy layer of the first source image to obtain a base layer of the first source image;

[0031] The non-significant detail layer of the second source image is fused with the energy layer of the second source image to obtain a base layer of the second source image;

[0032] The base layer of the first source image and the base layer of the second source image are fused by an absolute maximum operation algorithm to obtain a pre-fused base layer.

[0033] Furthermore, the expression for constructing the multimodal medical image fusion result is specifically as follows:

[0034] ;

[0035] In the above formula, represents the multimodal medical image fusion result, represents the pre-fused edge detail layer, represents the pre-fused salient detail layer, represents the pre-fused base layer, Represents pixel coordinates.

[0036] The second technical solution adopted by the present invention is: a multimodal medical image fusion system based on edge region segmentation detection, comprising:

[0037] The first module is used to obtain the first source image and the second source image and perform image decomposition processing respectively to obtain an energy layer of the first source image, a detail layer of the first source image, an energy layer of the second source image, and a detail layer of the second source image;

[0038] The second module is used to introduce the fuzzy region active contour operator and the heat conduction matrix algorithm to fuse the detail layer of the first source image with the detail layer of the second source image to obtain a pre-fused edge detail layer;

[0039] The third module is used to perform non-edge feature detection and fusion processing on the detail layer of the first source image and the detail layer of the second source image to obtain a pre-fused significant detail layer;

[0040] The fourth module is used to obtain the non-significant detail layer of the first source image and the non-significant detail layer of the second source image and combine the energy layer of the first source image and the energy layer of the second source image, and perform fusion processing through an absolute maximum operation algorithm to obtain a pre-fused base layer;

[0041] The fifth module is used to combine the pre-fused edge detail layer, the pre-fused significant detail layer and the pre-fused base layer to construct a multimodal medical image fusion result.

[0042] The beneficial effects of the method and system of the present invention are as follows: the present invention obtains a first source image and a second source image and performs image decomposition processing respectively, and then introduces a fuzzy region active contour operator and a heat conduction matrix algorithm to fuse the detail layer of the first source image with the detail layer of the second source image, and highlights the structural edges in the fused image by introducing the fuzzy region active contour operator and the heat conduction matrix algorithm, thereby improving the accuracy of edge extraction, and further performs non-edge feature detection and fusion processing on the detail layer of the first source image and the detail layer of the second source image, effectively obtaining significant structural detail features in the source image, and then fuses the energy layer of the first source image with the energy layer of the second source image through an absolute maximum operation algorithm, better from the brightness information in the sub-energy layer of the source image, and finally combines the pre-fused edge detail layer, the pre-fused significant detail layer and the pre-fused base layer to construct a multimodal medical image fusion result, which can effectively retain the edge features, significant structural details and energy information in the source image at the same time, and thus accurately extract the anatomical or functional information of the source image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of the steps of the multimodal medical image fusion method based on edge region segmentation detection of the present invention;

[0044] Figure 2 It is a structural block diagram of a multimodal medical image fusion system based on edge region segmentation detection of the present invention;

[0045] Figure 3 It is a schematic diagram of a framework of multimodal medical image fusion provided by a specific embodiment of the present invention;

[0046] Figure 4 It is a schematic diagram of comparative analysis of experimental results with other fusion algorithms provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0048] First, the implementation background of the embodiment of the present invention is explained. Multimodal medical images cover anatomical and functional images, providing a comprehensive view of human organs from different perspectives. Anatomical images can visualize organ structures in detail, such as computed tomography images (CT) and magnetic resonance imaging images (MRI). For example, CT images clearly depict bone structures and tumor features, including their size and location. Images of different MRI modes have their own functions: MR-T1 images can display specific anatomical structures, MR-T2 images highlight tissue lesions, and MR-GAD images provide high contrast and clear tissue lesions and their edges. In contrast, functional images such as positron emission tomography (PET) images use radioactive isotope-labeled tracers to reflect physiological processes and provide important insights into the location of lesions. Although anatomical images are excellent in resolution and structural details, they cannot convey functional information; while functional images, although with lower resolution, provide valuable information about the metabolic function of organs. The limitations of these single-modality images often hinder comprehensive clinical diagnosis. To address these problems, image fusion technology has become a key tool that integrates complementary information from different imaging technologies into a single image and minimizes redundancy. For example, fusion of PET and MRI images can combine the high contrast and high resolution soft tissue details of MRI with the functional lesion information of PET.With the increasing demand for medical safety monitoring and prognosis, multimodal medical image fusion technology has received increasing attention.

[0049] The disadvantage of existing technologies is that there are still challenges in distinguishing basic information from complex information. In addition, medical images of different modalities have different edge characteristics, and it is also critical to select an edge detection operator that performs well in detection accuracy, edge localization, and noise suppression. At the same time, it is still a major challenge to effectively fuse the detected edge information into the result image and ensure adaptability to medical images of various modalities. Existing methods have inherent limitations, especially in simultaneously retaining edge features, significant structural details, and energy information in medical images, which may cause the fused image to lack key diagnostic features, so that the final generated fused image incorrectly expresses the anatomical or functional information of the original image.

[0050] Based on this, the embodiment of the present invention designs a multimodal medical image fusion method that effectively maintains the energy of different modal images while retaining valuable edges and key target areas. In order to achieve the fusion of functional images and anatomical images, especially when processing functional image pairs such as MR-T1 / PET and MR-T2 / PET, the "RGB→YUV→RGB" strategy is adopted. This method involves converting the image from RGB to YUV color space and then converting it back to RGB for the fusion process. This technology is widely used in various medical image fusion methods. The FA-HC model is designed to detect details presented in the form of edges in the detail layer, which can effectively extract edge features. In addition, the FRAC operator is introduced to generate a significant detail layer. The designed LIP fusion rule extracts important structural information from the significant detail layer. Compared with the existing fusion method, the present invention better extracts and maintains energy information by fusing the base layer constructed with the energy layer and the non-significant detail layer. Finally, the pre-fused edge detail layer, the significant detail layer and the base layer are reconstructed to obtain the fusion result.

[0051] Reference Figure 1 and Figure 3 The present invention provides a multimodal medical image fusion method based on edge region segmentation detection, the method comprising the following steps:

[0052] S100, acquiring a first source image and a second source image and performing image decomposition processing on each of them to obtain an energy layer of the first source image, a detail layer of the first source image, an energy layer of the second source image, and a detail layer of the second source image;

[0053] In this embodiment, the medical source image is subjected to average filtering decomposition processing to obtain respective energy layers and detail layers, which are expressed as follows:

[0054] ;

[0055] In the above formula, Respectively represent The energy layer and detail layer of an image, represents pixel coordinates, Indicates images, represents the average filter, and the size of the average filter is , Represents a convolution operation.

[0056] S200, introducing a fuzzy region active contour operator and a heat conduction matrix algorithm, fusing the detail layer of the first source image with the detail layer of the second source image to obtain a pre-fused edge detail layer;

[0057] S210, setting a threshold, introducing a fuzzy region active contour operator to perform salient region extraction processing on the first source image and the second source image, respectively, to obtain an initial salient layer of the first source image and an initial salient layer of the second source image;

[0058] Specifically, the salient information of the detail layer is mainly presented in the form of edge features. In order to extract edge features from the detail layer, an edge detection model FA-HC is designed. The specific process of FA-HC is as follows:

[0059] First, the FRAC operator is introduced to accurately obtain the salient area of ​​the source image. The initial saliency detection provides a basis for edge detection. The expression for obtaining the salient area of ​​the source image is as follows:

[0060] ;

[0061] In the above formula, represents the salient area of ​​the source image, represents the source image, Represents the FRAC operator.

[0062] image The salient features are enhanced, emphasizing the main structure and boundaries of the object. , the decision diagram with significant features can be obtained by the following formula , whose expression is:

[0063] ;

[0064] In the above formula, represents a significant decision graph, represents the salient area of ​​the source image extracted initially, Indicates the preset threshold.

[0065] The initial saliency layer can be obtained by the following formula , the expression for obtaining the initial significant layer is:

[0066] ;

[0067] In the above formula, represents the initial salient layer, represents the source image, represents a significant decision graph, Indicates width, Indicates the total number of images.

[0068] S220, performing edge information detection processing on the initial saliency layer of the first source image and the initial saliency layer of the second source image respectively by using a heat conduction matrix algorithm to obtain an edge detail layer of the first source image and an edge detail layer of the second source image;

[0069] Specifically, a heat transfer matrix algorithm is introduced to further detect edge information from the initial saliency layer. In medical image processing, the complex tissue structure and subtle grayscale gradients of the source image have a great impact on the processing results, and existing edge detection methods are often insufficient to overcome the above problems. The diffusion characteristics of the heat transfer matrix operator can help solve these challenges by retaining key edge information while suppressing noise more effectively. This method opens up new possibilities for enhancing medical image analysis by improving edge detection. Therefore, the present invention introduces HCM into edge detection of medical images as a key operation in the FA-HC model.

[0070] To obtain the initial saliency layer Thermal Conductivity Matrix , a 3×3 mask is applied so that Move on each pixel of The initial salient layer obtained Thermal Conductivity Matrix It can be expressed as:

[0071] ;

[0072] In the above formula, represents the thermal conductivity, represents the average surface area of ​​the heat conduction path within the mask, Represents the path length from the maximum gray value position to the minimum gray value position in the mask, represents the maximum gray value within the mask, represents the minimum gray value within the mask, Represents the parameters in the two calculation processes, Indicates that the mask is arrive The average surface area of ​​the heat conduction path at the location, Indicates that the mask is arrive The average surface area of ​​the heat conduction path at the location, Indicates the mask and The distance to the location, Indicates the mask and The distance to the location.

[0073] if and (or ) are adjacent pixels, then set (or is 20, set (or is 1. However, if the two pixels are diagonally related, then set (or is 10, set (or for .

[0074] Therefore, the edge detail layer can be obtained by the following formula , the expression for obtaining edge details is:

[0075] ;

[0076] In the above formula, represents the threshold value, represents the edge detail layer, Represents an edge decision graph.

[0077] Furthermore, it should be noted that the definition of the FRAC (Fuzzy Regional Active Contour driven by weighted global and local fitting energy) operator in the embodiment of the present invention is as follows:

[0078] Active contour models are able to adapt to changes in contour topology and achieve effective image segmentation. These models represent the changing contours as zero level sets and drive them to the target boundary by minimizing the energy functional. Active contour models are generally divided into two categories: edge-based and region-based. Region-based models rely on global image features (such as intensity, color, and texture) to guide contour motion and incorporate these features into the energy function. In the region-based active contour model, the contour is described by a zero level set function, often called a signed distance function. The fuzzy energy-based active contour operator integrates fuzzy sets into the active contour model framework and uses a pseudo level set function to represent the contour curve. , which is defined as:

[0079] ;

[0080] In the above formula, represents the source image, Represents pixel coordinates. Represent the object area and background area respectively, defined as contour curves Inner and outer areas.

[0081] The fuzzy energy active contour model has no regularization term, which leads to the non-smoothness of the evolution curve and cannot maintain the distance characteristics of the pseudo level set function. In order to achieve better performance and less running time, a fuzzy region-based active contour model with weighted global and local fitting energy, referred to as FRAC, is proposed on this basis. FRAC drives the contour to develop towards the target boundary by weighting the global and local fitting energy, thereby effectively extracting salient areas.

[0082] Contains fuzzy area energy and edge energy FRAC model Defined as:

[0083] ;

[0084] In the above formula, Represents an edge detector, which is used to reduce image noise and smooth image edges.

[0085] Fuzzy region terms with fuzzy sets Defined as:

[0086] ;

[0087] The constant All ≥ 0 are fixed weight parameters, and are two weighting constants, , is the weighted index of each fuzzy membership, defining the profile Average strength inside and outside and The expression is:

[0088] ;

[0089] Assume a local image domain ,make for The fuzzy membership function in the local domain is the average intensity inside (object) and outside (background). and The expression is:

[0090] ;

[0091] in Pixel With pixels The spatial distance between them, the size of the local window is , is the radius of the local window. Through local spatial window filtering, the two constants can be and As contours The local average intensity inside and outside.

[0092] Edge Energy It is composed of regularization term and penalty term, which helps to obtain accurate positioning of the target boundary and smooth pseudo level set function. The edge energy is defined as:

[0093] ;

[0094] in and is a positive parameter. The first term represents the length of the evolving contour, which ensures the smoothness of the pseudo level set function. The second term maintains the consistency between the signed distance function and the pseudo level set function.

[0095] Therefore, the basic steps of the FRAC operator are as follows:

[0096] 1) Initialize the pseudo level set function: a portion of the source image pixels is set , the other part sets .

[0097] 2) Initial constant , Average strength and The expression calculation of , Average strength and The expression is evaluated.

[0098] 3) Update membership , so that the fuzzy region term with fuzzy set The energy in the definition expression is minimized. At the same time, the average intensity and The expression and average intensity and The expression updates the new constant , , and Then, using the edge energy The edge energy in the definition expression is used to regularize and smooth the pseudo level set function. Repeat the above operation until the iteration is completed. Therefore, the output can be obtained .

[0099] Given an input image , which will be obtained from the FRAC operation The operation is recorded as:

[0100] ;

[0101] In the above formula, To get FRAC operator.

[0102] S230, constructing an edge detail layer fusion algorithm by combining the gradient amplitude and the pixel value, fusing the edge detail layer of the first source image with the edge detail layer of the second source image to obtain an edge fusion decision graph;

[0103] S240 , fusing the detail layer of the first source image, the detail layer of the second source image, and the edge fusion decision graph to obtain a pre-fused edge detail layer.

[0104] Specifically, regions with significant gradient changes usually correspond to clear details in the image. In order to accurately fuse edges, the present invention designs a new fusion rule based on gradient magnitude and pixel value, called the GP rule, to capture clear edge information and details, as shown below:

[0105] ;

[0106] In the above formula, Represents the features extracted using GP rules, express gradient.

[0107] Then we can get the edge fusion decision graph , whose expression is:

[0108] ;

[0109] use Blending edge detail layers , whose expression is:

[0110] ;

[0111] In the above formula, Represents the fused edge detail layer.

[0112] S300, performing non-edge feature detection and fusion processing on the detail layer of the first source image and the detail layer of the second source image to obtain a pre-fused significant detail layer;

[0113] S310, performing edge detail feature elimination processing on the detail layer of the first source image and the detail layer of the second source image respectively, to obtain a non-edge detail layer of the first source image and a non-edge detail layer of the second source image;

[0114] Specifically, in addition to prominent edge features, the detail layer also includes a large number of non-edge features. This is represented as weak structural details that reflect the difference between the center pixel and the corresponding surrounding pixels. The non-edge detail layer is extracted by removing the fused edge image pixel values ​​from the detail layer. , whose expression is:

[0115] ;

[0116] In the above formula, represents the non-edge detail layer, represents the edge detail layer after fusion, Represents the detail layer.

[0117] S320, performing salient region extraction processing on the first source image and the second source image respectively by using a fuzzy region active contour operator to obtain a salient decision map of the first source image and a salient decision map of the second source image;

[0118] Specifically, in addition, the salient decision graph generated by FRAC can be used To get the significant detail layer , as shown below:

[0119] ;

[0120] In the above formula, represents the salient detail layer, Represents a salient decision graph.

[0121] S330, combining the local phase coherence strength algorithm and the pseudo level set function to construct a significant detail layer fusion algorithm;

[0122] S340, extracting and processing the non-edge detail layer of the first source image and the salient decision graph of the first source image by using a salient detail layer fusion algorithm to obtain a salient detail layer of the first source image;

[0123] S350, extracting and processing the non-edge detail layer of the second source image and the salient decision graph of the second source image by using a salient detail layer fusion algorithm to obtain a salient detail layer of the second source image;

[0124] Specifically, in order to effectively fuse the significant detail layer, the embodiment of the present invention designs a significant detail layer fusion rule based on local phase coherence intensity and pseudo level set function, namely the LIP rule, which introduces the LPC operator to detect the large sharpness changes of the detail layer. By applying the LPC operator, the LPC intensity image can be obtained. , whose expression is:

[0125] ;

[0126] In the above formula, represents the intensity image, Represents the LPC operator.

[0127] The LPC intensity image mainly represents the prominent textures, edges and other structural elements in the image. However, in areas with complex details, important structures may be masked by other smaller structures or noise, resulting in incomplete extraction of important structures during the fusion process. To solve this problem, the FRAC operator is introduced to extract important structural features. The FRAC model improves the description and details of important structures by optimizing the pseudo level set function; the resulting image indirectly reflects the energy characteristics of the input image, especially in areas with edges and significant structures. Use the LIP rule to obtain of The intensity image is expressed as:

[0128] ;

[0129] In the above formula, Represents the detail layer image generated by the FRAC operator based on the pseudo level set function.

[0130] Then we get the saliency fusion decision graph , whose expression is:

[0131] ;

[0132] In the above formula, represents the saliency fusion decision graph, express of Intensity image.

[0133] S360: Fusing the salient detail layer of the first source image with the salient detail layer of the second source image to obtain a pre-fused salient detail layer.

[0134] Specifically, you can use Get the fused significant detail layer , whose expression is:

[0135] ;

[0136] In the above formula, Represents the fused edge detail layer.

[0137] S400, obtaining a non-significant detail layer of the first source image and a non-significant detail layer of the second source image, combining the energy layer of the first source image and the energy layer of the second source image, and performing fusion processing through an absolute maximum operation algorithm to obtain a pre-fused base layer;

[0138] S410, performing significant detail layer elimination processing on the non-edge detail layer of the first source image and the non-edge detail layer of the second source image respectively, to obtain the non-significant detail layer of the first source image and the non-significant detail layer of the second source image;

[0139] S420, fusing the non-significant detail layer of the first source image with the energy layer of the first source image to obtain a base layer of the first source image;

[0140] S430, fusing the non-significant detail layer of the second source image with the energy layer of the second source image to obtain a base layer of the second source image;

[0141] Specifically, the energy information of the image, such as contrast and brightness, is usually concentrated in the energy layer. In addition, the non-significant details in the detail layer contain a large number of non-significant detail features, and also contain some residual energy information. In order to better fuse the energy information, we combine the energy layer and the non-significant detail layer to form the base layer, as shown below:

[0142] ;

[0143] In the above formula, It is the non-salient detail layer obtained by removing the fused significant image pixel values ​​from the non-edge detail layer.

[0144] S440 , fusing the base layer of the first source image and the base layer of the second source image using an absolute maximum operation algorithm to obtain a pre-fused base layer.

[0145] Specifically, anatomical medical images have higher tissue density or resolution, which can accurately describe small lesions and their extent through brightness information in sub-energy layers. Therefore, we apply the Abs-max operation (absolute maximum operation) to fuse the base layers as follows:

[0146] ;

[0147] in Based on the fusion decision diagram.

[0148] S500, combining the pre-fused edge detail layer, the pre-fused significant detail layer and the pre-fused base layer to construct a multimodal medical image fusion result.

[0149] In this embodiment, The pre-fused edge detail layer containing a lot of boundary information is obtained by fusing the edge detail layer expression ,Depend on The expression of the fused salient detail layer is obtained by: , and the pre-fused base layer containing a lot of energy information obtained by fusing the base layer expression with the Abs-max operation Combined, the final fusion result is obtained, which is expressed as:

[0150] ;

[0151] In the above formula, represents the multimodal medical image fusion result, represents the pre-fused edge detail layer, represents the pre-fused salient detail layer, represents the pre-fused base layer, Represents pixel coordinates.

[0152] In summary, the embodiment of the present invention converts the functional image from the RGB color space to the YUV color space, and generates a separate image for each channel. This conversion is particularly suitable for the fusion of functional images, such as MR / PET and MR / SPECT image fusion. Then, the Y channel image and the anatomical medical image are decomposed and fused according to the designed fusion method. Then, the U and V channels are recombined with the fused Y channel, and the image is converted back to the RGB color space to produce the final fusion result.

[0153] Therefore, compared with the prior art, the embodiments of the present invention have the following advantages:

[0154] 1) The embodiment of the present invention designs a four-layer decomposition model and fusion scheme of multimodal medical images based on FRAC and HCM. Compared with existing edge detection algorithms, such as Sobel operator, Canny edge detection, Laplace Gaussian algorithm and anisotropic diffusion algorithm, they have been used to identify prominent edge features in various imaging applications. However, when applied to complex medical imaging, these algorithms often find it difficult to effectively balance noise suppression and edge preservation. In order to extract edge information from the detail layer of medical images and effectively maintain the edges and contrast of the source image, the present invention designs a model that can extract edge features in the detail layer - the fuzzy region-based active contour - heat conduction (FA-HC) model. The model uses the FRAC operator to process the source image before using the heat conduction matrix operator to extract the edge. This operation highlights the structural edges in the fused image and improves the accuracy of edge extraction.

[0155] 2) The embodiment of the present invention also designs a fusion rule based on gradient magnitude and pixel value, called GP rule, to pre-fuse the edge detail layer. For the non-edge detail layer, the FRAC operator is used to extract the significant detail layer; to fuse these significant detail layers, a LIP rule based on local phase coherence intensity and pseudo level set function is designed; for the energy layer, it is added to the non-significant detail layer, and the Abs-max rule is applied to obtain the pre-fused base layer. Finally, the final fusion result is obtained by combining the pre-fused edge detail layer, the significant detail layer and the base layer.

[0156] In summary, the embodiments of the present invention can effectively retain the edge features, significant structural details and energy information in the source image at the same time, design a new edge detection model FA-HC, which is used to extract the edge detail layer from the detail layer of the source image, design the GP rule, which is used to pre-fuse the edge detail layer, and can effectively retain the edge information of the source image, and design the LIP rule, which is used to pre-fuse the significant detail layer, and can effectively retain the significant structure of the source image.

[0157] Finally, if Figure 4 As shown, in order to further demonstrate the advantages and effectiveness of the embodiments of the present invention, a set of comparative experiments was conducted with 7 most advanced image fusion algorithms to analyze the advantages and disadvantages of each algorithm from the perspective of subjective visual evaluation. Figure 4 (a) and Figure 4 (b) Source images MR-T2 and PET images (both of which have a size of 256×256), Figure 4 (c) Figure 4 (d) Figure 4 (e) Figure 4 (f) Figure 4 (g) Figure 4 (h) Figure 4 (i) and Figure 4(j) in the figure are: medical image fusion algorithm based on sparse representation and neighborhood energy activity (EBLS), multimodal medical image fusion algorithm based on multi-dictionary and truncated Huber filter (MDHU), universal unsupervised image fusion algorithm based on memory unit (U2Fusion), multimodal image fusion algorithm based on correlation driven dual branch feature decomposition (CDDFuse), multimodal medical image fusion algorithm based on multi-channel aggregation network (MCAFusion), equivariant multimodal image fusion algorithm (EMMA), three-modal medical image fusion generative adversarial network based on primitive relational reasoning (PRRGAN) and fusion results under the fusion algorithm of this scheme (Proposed). It can be seen from the result graph that, except for the fusion results of the embodiment of the present invention, the fusion results of other methods have incomplete retention of white tissue in MR-T2, unclear edges or large noise. In summary, the fusion performance of the method proposed in the embodiment of the present invention is better than other fusion methods.

[0158] Reference Figure 2 , a multimodal medical image fusion system based on edge region segmentation detection, including:

[0159] The first module 201 is used to obtain a first source image and a second source image and perform image decomposition processing respectively to obtain an energy layer of the first source image, a detail layer of the first source image, an energy layer of the second source image, and a detail layer of the second source image;

[0160] The second module 202 is used to introduce a fuzzy region active contour operator and a heat conduction matrix algorithm to fuse the detail layer of the first source image and the detail layer of the second source image to obtain a pre-fused edge detail layer;

[0161] The third module 203 is used to perform non-edge feature detection and fusion processing on the detail layer of the first source image and the detail layer of the second source image to obtain a pre-fused significant detail layer;

[0162] The fourth module 204 is used to obtain the non-significant detail layer of the first source image and the non-significant detail layer of the second source image and combine the energy layer of the first source image and the energy layer of the second source image, and perform fusion processing through an absolute maximum operation algorithm to obtain a pre-fused base layer;

[0163] The fifth module 205 is used to combine the pre-fused edge detail layer, the pre-fused significant detail layer and the pre-fused base layer to construct a multimodal medical image fusion result.

[0164] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0165] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A multimodal medical image fusion method based on edge region segmentation detection, characterized in that: The following steps are involved: Acquire a first source image and a second source image and perform image decomposition processing on them respectively to obtain an energy layer of the first source image, a detail layer of the first source image, an energy layer of the second source image, and a detail layer of the second source image; A threshold is set, and a fuzzy region active contour operator is introduced to perform salient region extraction processing on the first source image and the second source image respectively, so as to obtain an initial salient layer of the first source image and an initial salient layer of the second source image; Performing edge information detection processing on the initial saliency layer of the first source image and the initial saliency layer of the second source image respectively by using a heat conduction matrix algorithm to obtain an edge detail layer of the first source image and an edge detail layer of the second source image; An edge detail layer fusion algorithm is constructed by combining the gradient amplitude and the pixel value, and the edge detail layer of the first source image is fused with the edge detail layer of the second source image to obtain an edge fusion decision graph; Fusing the detail layer of the first source image, the detail layer of the second source image, and the edge fusion decision graph to obtain a pre-fused edge detail layer; Performing edge detail feature elimination processing on the detail layer of the first source image and the detail layer of the second source image respectively to obtain a non-edge detail layer of the first source image and a non-edge detail layer of the second source image; Performing salient region extraction processing on the first source image and the second source image respectively by using a fuzzy region active contour operator to obtain a salient decision map of the first source image and a salient decision map of the second source image; Combining the local phase coherence strength algorithm and the pseudo level set function, a salient detail layer fusion algorithm is constructed; Extracting and processing the non-edge detail layer of the first source image and the salient decision graph of the first source image by using a salient detail layer fusion algorithm to obtain a salient detail layer of the first source image; Extracting and processing the non-edge detail layer of the second source image and the salient decision graph of the second source image by using a salient detail layer fusion algorithm to obtain a salient detail layer of the second source image; Fusing the salient detail layer of the first source image with the salient detail layer of the second source image to obtain a pre-fused salient detail layer; Acquire the insignificant detail layer of the first source image and the insignificant detail layer of the second source image, combine the energy layer of the first source image and the energy layer of the second source image, perform fusion processing through an absolute maximum operation algorithm, and obtain a pre-fused base layer; The pre-fused edge detail layer, the pre-fused significant detail layer and the pre-fused base layer are combined to construct a multimodal medical image fusion result.

2. The multimodal medical image fusion method based on edge region segmentation detection according to claim 1, characterized in that: The first source image and the second source image are decomposed by averaging filtering, and the expression is: ； In the above formula, Respectively represent The energy layer and detail layer of an image, represents pixel coordinates, Indicates images, represents the averaging filter, Represents a convolution operation.

3. The multimodal medical image fusion method based on edge region segmentation detection according to claim 2 is characterized in that: It also includes the introduction of the fuzzy region active contour operator and the heat conduction matrix algorithm to construct an active contour-heat conduction model based on the fuzzy region, and its specific expression is as follows: ； In the above formula, represents the active contour based on fuzzy region in the active contour based on fuzzy region-heat conduction model, represents the fuzzy region term with fuzzy sets, represents the edge energy term, represents the edge detector, represents the pseudo level set membership function.

4. The multimodal medical image fusion method based on edge region segmentation detection according to claim 3 is characterized in that: The step of obtaining the insignificant detail layer of the first source image and the insignificant detail layer of the second source image and combining the energy layer of the first source image and the energy layer of the second source image, and performing fusion processing through an absolute maximum operation algorithm to obtain a pre-fused base layer specifically includes: Performing significant detail layer elimination processing on the non-edge detail layer of the first source image and the non-edge detail layer of the second source image respectively to obtain the non-significant detail layer of the first source image and the non-significant detail layer of the second source image; The non-significant detail layer of the first source image is fused with the energy layer of the first source image to obtain a base layer of the first source image; The non-significant detail layer of the second source image is fused with the energy layer of the second source image to obtain a base layer of the second source image; The base layer of the first source image and the base layer of the second source image are fused by an absolute maximum operation algorithm to obtain a pre-fused base layer.

5. The multimodal medical image fusion method based on edge region segmentation detection according to claim 4 is characterized in that: The expression for constructing the multimodal medical image fusion result is specifically as follows: ； In the above formula, represents the multimodal medical image fusion result, represents the pre-fused edge detail layer, represents the pre-fused salient detail layer, represents the pre-fused base layer, Represents pixel coordinates.

6. A multimodal medical image fusion system based on edge region segmentation detection, characterized in that: Includes the following modules: The first module is used to obtain the first source image and the second source image and perform image decomposition processing respectively to obtain an energy layer of the first source image, a detail layer of the first source image, an energy layer of the second source image, and a detail layer of the second source image; The second module is used to set a threshold value, introduce a fuzzy region active contour operator to perform salient region extraction processing on the first source image and the second source image respectively, and obtain an initial salient layer of the first source image and an initial salient layer of the second source image; Performing edge information detection processing on the initial saliency layer of the first source image and the initial saliency layer of the second source image respectively by using a heat conduction matrix algorithm to obtain an edge detail layer of the first source image and an edge detail layer of the second source image; An edge detail layer fusion algorithm is constructed by combining the gradient amplitude and the pixel value, and the edge detail layer of the first source image is fused with the edge detail layer of the second source image to obtain an edge fusion decision graph; Fusing the detail layer of the first source image, the detail layer of the second source image, and the edge fusion decision graph to obtain a pre-fused edge detail layer; The third module is used to perform edge detail feature elimination processing on the detail layer of the first source image and the detail layer of the second source image respectively, so as to obtain the non-edge detail layer of the first source image and the non-edge detail layer of the second source image; Performing salient region extraction processing on the first source image and the second source image respectively by using a fuzzy region active contour operator to obtain a salient decision map of the first source image and a salient decision map of the second source image; Combining the local phase coherence strength algorithm and the pseudo level set function, a salient detail layer fusion algorithm is constructed; Extracting and processing the non-edge detail layer of the first source image and the salient decision graph of the first source image by using a salient detail layer fusion algorithm to obtain a salient detail layer of the first source image; Extracting and processing the non-edge detail layer of the second source image and the salient decision graph of the second source image by using a salient detail layer fusion algorithm to obtain a salient detail layer of the second source image; Fusing the salient detail layer of the first source image with the salient detail layer of the second source image to obtain a pre-fused salient detail layer; The fourth module is used to obtain the non-significant detail layer of the first source image and the non-significant detail layer of the second source image and combine the energy layer of the first source image and the energy layer of the second source image, and perform fusion processing through an absolute maximum operation algorithm to obtain a pre-fused base layer; The fifth module is used to combine the pre-fused edge detail layer, the pre-fused significant detail layer and the pre-fused base layer to construct a multimodal medical image fusion result.

Images