Multi-degradation medical image unified fusion method based on degradation prototype learning

Abstract

The invention relates to a multi-degradation medical image unified fusion method based on degradation prototype learning, and belongs to the field of medical image fusion. The method comprises the steps that low-dose PET data, CT metal artifact data and MRI data with motion artifacts are generated through the imaging principle; learning a degradation prototype by using a feature selection mechanism; the basic fusion model is decomposed into a plurality of branches through a low-rank decomposition strategy, and processing can be performed through different branches when different degradation data are fused; a prompt module based on a learnable feature prototype is designed, and fusion is promoted by injecting degradation-related invariant features into different LoRa branches; and constructing a fused image through an output layer by integrating the degradation elimination fusion features in different scales. According to the method, the medical images containing degradation can be effectively fused, and the robustness and practicability in reality are improved.

Description

Technical Field

[0001] The present invention relates to a unified fusion method of multiple degraded medical images based on degraded prototype learning, and belongs to the technical field of medical image fusion. Background Art

[0002] Existing multimodal medical image fusion techniques primarily integrate data from different imaging modalities to generate images containing more comprehensive lesion information. Most of these methods require high-quality, non-degraded source images as input to ensure good fusion results. However, this high-quality requirement for source images is often not met in real-world applications. In reality, medical image acquisition is subject to interference from objective factors, resulting in a variety of degradation information within the image content. This degradation information necessitates that, before medical image fusion can proceed, the degraded source image data must first be restored and reconstructed before being fed into the fusion network for feature fusion. However, this two-stage processing approach presents numerous limitations in clinical applications. In particular, when dealing with degradation, a single restoration network is often designed and trained for a specific degradation. Consequently, when faced with diverse real-world data input, a single network cannot effectively recover and process multiple, or even unknown, degradations. Furthermore, this two-stage feature processing approach results in redundant and complex network structures, hindering widespread deployment. Therefore, the present invention proposes a unified fusion method for multi-degraded medical images based on degradation prototype learning. Summary of the Invention

[0003] In order to address the shortcomings of existing methods, the present invention provides a unified fusion method for multiple degraded medical images based on degradation prototype learning to address the problem of poor quality of fused images caused by multiple degradations of source images and unknown degraded images in existing multimodal medical image fusion. The method of the present invention can effectively fuse medical images containing degradation, thereby improving its robustness and practicality in reality.

[0004] The technical solution of the present invention is: a unified fusion method of multiple degraded medical images based on degraded prototype learning, the method comprising:

[0005] Step 1. Obtain aligned multimodal medical image datasets: Obtain aligned multimodal medical image data from public datasets to generate image degradation to provide training samples;

[0006] Step 2: Construct degraded data based on real imaging principles: Generate low-dose PET data, CT metal artifact data, and MRI data with motion artifacts through imaging principles;

[0007] Step 3: Use feature selection mechanism to learn degenerate prototypes;

[0008] Step 4. Use LoRa branches to handle multiple degradations: Through the low-rank decomposition strategy, the basic fusion model is decomposed into multiple branches. When fusing different degradation data, different branches can be used for processing.

[0009] Step 5. Promote degradation elimination and feature fusion through degradation prototype prompts: Design a prompt module based on learnable feature prototypes to help LoRa branches eliminate degradation and promote fusion by injecting degradation-related invariant features into different LoRa branches;

[0010] Step 6. Integrate the features of each branch and construct a fused image: By integrating the degraded fusion features at different scales, a fused image is constructed through the output layer.

[0011] Furthermore, the Step 1 includes:

[0012] Each multimodal medical image pair in the aligned multimodal medical image data is composed of a group of medical images with complementary features, including: MRI-T1, MRI-T2; MRI, CT; PET, CT;

[0013] First, the multimodal medical image dataset is preprocessed. The specific preprocessing method is as follows: the resolution of the three-dimensional image is uniformly resampled to 256×256×256; the data is randomly inverted for data augmentation, and the pixel values ​​of the image are normalized to between [0,1].

[0014] Furthermore, the Step 2 includes:

[0015] (1) For MRI-T1 and MRI-T2 fusion scenarios, MRI data with motion artifacts were constructed by simulating the imaging characteristics of the patient's movement in K space during T2 impression imaging;

[0016] (2) For the MRI and CT fusion scenario, metal artifact masks from clinical acquisition were collected, and metal artifact imaging was simulated by adding noise and beam hardening to construct CT metal artifact data;

[0017] (3) For the PET and CT fusion scenario, the imaging effect of low-dose PET is simulated by downsampling the standard-dose PET data and adding Poisson noise, thus constructing low-dose PET data.

[0018] Furthermore, the Step 3 includes:

[0019] Step 3.1, degrade the image and reference images Input to the convolution-based feature encoder to extract shallow features , and deep features , ;

[0020] Step 3.2, through deep features , The image is divided into blocks and encoded to obtain sequence information, which is then input into the Transformer for further feature extraction. The extracted features are recorded as , ; In order to aggregate information related to image degradation categories, a learnable degradation category token is also spliced , ;

[0021] Step 3.3, degenerate category tokens containing global information , As the degenerate representation of each set of features to be fused:

[0022] , Classification prediction results obtained through the fully connected layer Degenerate category label with the current feature consistent, making , Guide the learning of degenerate prototypes;

[0023] Step 3.5: Introducing a degenerate prototype consisting of learnable parameters ; When the network performs feature fusion, the prototype is degraded Through the feature selection mechanism, the prompt information related to the current degradation is screened out from the degradation prototype; among them, the information screening depends on the degradation-related feature degradation category token 、 The degradation information contained in

[0024] Step 3.6, degenerate the category tokens by degenerating the relevant features 、 Send it into the linear layer and Softmax function to get the same as the degraded prototype Information filtering heads of the same shape ; At this time, the information filtering header Degenerate prototype Dot product, from the degenerate prototype Select the degradation-related feature representation, that is, the prompt information , This will serve as a hint to help the subsequent network eliminate degradation.

[0025] Furthermore, the Step 4 includes:

[0026] Step 4.1. Using low-rank decomposition technology, replace the multi-channel convolution in the basic fusion model with low-rank convolution to construct three different LoRa branches;

[0027] The shallow features obtained in Step 4.2 and Step 3 are cascaded and used as features to be fused. The degraded fused features are sent to the basic fusion model and the constructed LoRa branch for de-degradation and fusion respectively. The respective results are integrated and sent to the output layer to obtain the final fusion result.

[0028] At the beginning of training, the parameters of the basic fusion model and the LoRa branch are updated together; in the second half of training, the parameters of the basic fusion model will be frozen, and the entire network framework will be fine-tuned by updating only the LoRa branch.

[0029] Furthermore, the Step 5 includes:

[0030] In the basic fusion model And the LoRa branch has added a prompt module that can learn feature prototypes;

[0031] Shallow features , After cascading, it is encoded into multi-scale features , (i=1, 2, 3, 4), are input into the basic network of corresponding scales and LoRa network 、 、 The features are restored and integrated in , where i represents the scale of the network;

[0032] Basic Network and LoRA network 、 、 In processing features When you need the prompt information you got earlier Assistance is required to obtain fusion features ; Fusion features The calculation formula is:

[0033] ;

[0034] in 、 、 The classification score representing the degradation type of the source image obtained by the classifier.

[0035] Furthermore, the Step 6 includes:

[0036] Step 6.1, design 4-scale fusion branches in the multi-scale feature integration architecture; , Cascade and convolution downsampling operations to obtain multi-scale features ;

[0037] Step 6.2: Multi-scale features are fed into the skip link embedding module to obtain the fused features after eliminating degradation. ;

[0038] Step 6.3: Fusion Features The upsampling operation is integrated together and finally sent to the output layer output() composed of convolution and ReLU activation function to obtain the fused image with degradation eliminated: .

[0039] Furthermore, the Step 6 further includes:

[0040] Introducing pixel intensity loss Used to ensure that the fusion result highlights the high contrast information in the source image in terms of visual effect; pixel intensity loss Expressed as:

[0041]

[0042] Introducing gradient loss Used to preserve edge contours and detail textures in the source image:

[0043]

[0044] in, Indicates that there is no degradation in the image to be fused. represents the clear image label corresponding to the degraded image, Indicates the maximum value operation. represents the gradient operator.

[0045] The present invention also provides a unified fusion system for multiple degraded medical images based on degradation prototype learning. The system includes a module for executing the unified fusion method for multiple degraded medical images based on degradation prototype learning.

[0046] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the unified fusion method of multiple degraded medical images based on degraded prototype learning is implemented.

[0047] The beneficial effects of the present invention are:

[0048] 1. This invention uses a single image feature encoding module to simultaneously meet the dual requirements of degradation elimination and feature fusion. During the fusion process, it fully exploits the complementary information between modalities, eliminating degradation information while simultaneously performing feature fusion. This solves the problem of degradation of the source image leading to degradation of the fusion result within the same framework.

[0049] 2. This paper proposes a degradation prototype learning mechanism that can flexibly model and represent degradation information when faced with multiple degradations or even unknown degradations. Through a learnable degradation prototype prompt module, the degradation information is input as a prompt into the fusion network, enabling the fusion network to robustly eliminate the degradation information and complete feature fusion.

[0050] 3. Based on the basic fusion model, this invention introduces multiple LoRa branches. Each branch network has a significantly reduced number of parameters, lower training overhead, and is easier to deploy. Through the LoRa branches, features containing different degradations are handed over to the corresponding branch networks for processing, and through feature integration, a more robust degraded fusion feature is obtained. In the second half of training, by fixing the network parameters of the basic model, training is further accelerated, allowing the LoRa branches to play a more effective role while reducing training overhead.

[0051] 4. This invention addresses the problem of multimodal medical image fusion by constructing a unified fusion framework capable of handling various types of degraded data. Through degradation prototype learning and a branching architecture, it can flexibly handle a variety of known and even unknown data. This overcomes the drawback of current fusion methods that require the invocation of multiple degradation elimination networks when processing degraded medical images. The simplicity of the architecture and ease of deployment of this method make it significantly more practical than existing methods.

[0052] 5. Based on the sample indicators collected through experiments on public datasets, the method proposed in this invention can effectively fuse medical images containing degradation, and the effect is better than the existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the process structure of the present invention;

[0054] Figure 2 A schematic diagram of the degraded medical image structure of the present invention;

[0055] Figure 3 This is a schematic diagram of the degradation prototype learning module of the present invention;

[0056] Figure 4 This is a schematic diagram of the LoRa branch architecture of the basic fusion architecture of the present invention;

[0057] Figure 5A schematic diagram of the multi-scale feature integration architecture of the present invention;

[0058] Figure 6 The figure is a comparison chart of the experimental effects of the method of the present invention and the existing method. DETAILED DESCRIPTION

[0059] Example 1: Figures 1-6 As shown, a unified fusion method for multiple degraded medical images based on degraded prototype learning includes:

[0060] Step 1. Obtain an aligned multimodal medical image dataset: Obtain aligned multimodal medical image data from a public dataset to generate image degradation to provide training samples; each multimodal medical image pair in the aligned multimodal medical image data consists of a set of medical images with complementary features, such as MRI-T1, MRI-T2; MRI, CT; PET, CT;

[0061] First, the multimodal medical image dataset is preprocessed. The specific preprocessing method is as follows: the resolution of the three-dimensional image is uniformly resampled to 256×256×256; the data is randomly inverted for data augmentation, and the pixel values ​​of the image are normalized to between [0,1].

[0062] Step 2: Construct degraded data based on real imaging principles: Generate low-dose PET data, CT metal artifact data, and MRI data with motion artifacts through imaging principles;

[0063] Furthermore, the Step 2 includes:

[0064] (1) For MRI-T1 and MRI-T2 fusion scenarios, since the T2 modality has a longer imaging time during the imaging process, the movement of the patient during imaging may cause motion artifacts. Based on this feature, by simulating the imaging characteristics of the patient's movement in the K space during T2 imaging, such as Figure 2 As shown, MRI data with motion artifacts is constructed; specifically expressed as:

[0065]

[0066] in represents a random affine transformation operation, represents the Fourier transform operation, represents the inverse Fourier transform operation, represents the degraded image produced, represents the original T2 modality image; this operation randomly deflects the original image into three angles, then combines the deflected images in k-space, and finally obtains the degraded image through inverse Fourier transform;

[0067] (2) For MRI and CT fusion scenarios, due to the imaging characteristics of CT images, when a patient has metal devices in his body, it will reflect the CT imaging rays. This is reflected in the imaging results, as metal devices in the patient's body will cause radial metal artifacts. Based on this characteristic, such as Figure 2 As shown in Figure 1, metal artifact masks from clinical acquisition are collected, and metal artifact imaging is simulated by adding noise and beam hardening to construct CT metal artifact data; specifically, it is expressed as:

[0068]

[0069] in represents the beam hardening function, represents the generated CT degradation data, represents the original CT data, Metal instrument mask representing clinical collection, represents the added Poisson noise;

[0070] (3) For the PET and CT fusion scenario, PET imaging often has noise in low-dose radiological imaging due to the impact of radioactive contrast agents on the patient's body. By downsampling the standard-dose PET data and adding Poisson noise, the imaging effect of low-dose PET is simulated and low-dose PET data is constructed. Specifically, it is expressed as:

[0071]

[0072] in represents the downsampling operation, represents the upsampling operation, represents the generated degradation data, represents the raw PET data, Represents the added Poisson noise. By adopting and adding noise data, the imaging effect of PET under low dose contrast agent is simulated;

[0073] Step 3: Use feature selection mechanism to learn degenerate prototypes;

[0074] Furthermore, the Step 3 includes:

[0075] Step 3.1, such as Figure 3 As shown, the degraded image and reference images Input to the convolution-based feature encoder to extract shallow features , and deep features , The shallow features contain rich image details, which will be fed into the subsequent network as detailed information for degradation elimination and feature fusion. The deep features are further obtained from the shallow features through several convolutional coding layers. They contain higher-dimensional image information. We will use the high-dimensional information of the deep features to promote the learning of degradation prototypes.

[0076] Step 3.2, through deep features , The image is divided into blocks and encoded to obtain sequence information, which is then input into the Transformer for further feature extraction. The extracted features are recorded as , ; In order to aggregate information related to image degradation categories, a learnable degradation category token is also spliced , ;

[0077] Step 3.3, degenerate category tokens containing global information , As the degenerate representation of each set of features to be fused, in order to achieve this:

[0078] , Classification prediction results obtained through the fully connected layer , should be consistent with the degenerate category label of the current feature Consistent, that is, the prediction results of the classification The following losses should be minimized:

[0079] ;

[0080] Where CE is represented by cross entropy loss. Thus, , It can guide the learning of degenerate prototypes;

[0081] Step 3.5, such as Figure 3 As shown, a degenerate prototype consisting of learnable parameters is introduced ; When the network performs feature fusion, the prototype is degraded Through the feature selection mechanism, the prompt information related to the current degradation is screened out from the degradation prototype; among them, the information screening depends on the degradation-related feature degradation category token 、 The degradation information contained in

[0082] Step 3.6, degenerate the category tokens by degenerating the relevant features 、 Send it into the linear layer and Softmax function to get the same as the degraded prototype Information filtering heads of the same shape ; At this time, the information filtering header Degenerate prototype Dot product, from the degenerate prototype Select the degradation-related feature representation, that is, the prompt information , It will serve as a prompt to help the subsequent network eliminate degradation; the prompt feature selection is expressed as:

[0083] ,

[0084] ;

[0085] Through the feature selection mechanism, the corresponding prompt information can be dynamically extracted from the degraded prototype according to the degradation situation in the current feature. At the same time, the degraded prototype will also complete the gradient transfer through the extracted prompt features and perform differential updates. When encountering unknown degradation, the information screening head It is possible to combine the input data to extract information from known degradation cues to characterize unknown degradations, thereby increasing the robustness of the network in processing unknown data.

[0086] Step 4. Use LoRa branches to handle multiple degradations: Use low-rank decomposition strategy to combine the basic fusion model Decomposed into multiple branches, when fusing different degradation data, different branches can be used for processing;

[0087] Basic fusion model in this invention , a multi-scale convolutional architecture in the form of U-net is adopted, and the skip link part of the architecture is designed as follows Figure 3 The Transformer-like module shown is built based on convolution; it consists of a multi-head attention module and a feedforward module. The multi-head attention network expands the feature channels multiple times and calculates self-attention for each group of channel features, thereby flexibly processing and strengthening features. The feedforward module feeds the features into a convolution with a kernel size of 1×1×1, and then obtains the processed features through an activation function. It can effectively integrate the output results of the multi-head attention and facilitate gradient transfer.

[0088] Furthermore, the Step 4 includes:

[0089] Step 4.1, using low-rank decomposition technology, such as Figure 3 As shown, the basic fusion model Multi-channel convolution in Replaced with low-rank convolution ,in, Indicates the number of input channels, Indicates the number of output channels, Represents the rank of low-rank convolution, thereby constructing 3 different LoRa branches 、 、 ;

[0090] Step 4.2, Step 3 to obtain shallow features , After being cascaded, the features to be fused are used, and the fused features with degradation are sent to the basic fusion model respectively. And the constructed LoRa branch 、 、 De-degradation and fusion are performed in the process, and the respective results are integrated and sent to the output layer to obtain the final fusion result;

[0091] For the LoRa branch, we set r to 4, which will significantly reduce the number of LoRa parameters and make it easier to train than the basic network.

[0092] At the beginning of training, the basic fusion model Update parameters together with the LoRa branch; in the second half of training, the basic fusion model The parameters of the LoRa branch will be frozen, and the entire network framework will be fine-tuned by updating only the LoRa branch, further reducing the training overhead and improving the practicality of the network.

[0093] Step 5. Promote degradation elimination and feature fusion through degradation prototype prompts: Design a prompt module based on learnable feature prototypes to help LoRa branches eliminate degradation and promote fusion by injecting degradation-related invariant features into different LoRa branches;

[0094] Furthermore, the Step 5 includes:

[0095] In the basic fusion model And the LoRa branch has added a prompt module that can learn feature prototypes;

[0096] Shallow features , After cascading, it is encoded into multi-scale features , (i=1, 2, 3, 4), are input into the basic network of corresponding scales and LoRa network 、 、 The features are restored and integrated in , where i represents the scale of the network;

[0097] Basic Network and LoRA network 、 、 In processing features When you need the prompt information you got earlier Assistance is required to obtain fusion features ,like Figure 4 As shown; fusion features The calculation formula is:

[0098] ;

[0099] in 、 、 The classification score representing the degradation type of the source image obtained by the classifier.

[0100] Prompt information It is mapped to the spatial dimension of the feature within the network, and the information is injected into the fusion feature in the form of spatial attention to help the network better eliminate degradation and perform feature fusion.

[0101] in The probability of a specific category containing degradation in the current feature is multiplied by its weights on the output features of each network branch. The network output features are then represented as linear combinations of the branches. This allows the network to call the corresponding branch to process the feature when dealing with known degradations. Furthermore, when encountering unknown degradations, the network can treat the unknown degradation information as a linear combination of known degradations. By modulating the feature outputs of each network branch, features containing unknown degradation information can be repaired.

[0102] Step 6. Integrate the features of each branch and construct a fused image: By integrating the degraded fusion features at different scales, a fused image is constructed through the output layer.

[0103] Furthermore, the Step 6 includes:

[0104] Step 6.1, design 4-scale fusion branches in the multi-scale feature integration architecture; , Cascade and convolution downsampling operations to obtain multi-scale features ;like Figure 4 As shown;

[0105] Step 6.2: Multi-scale features are fed into the skip link embedding module to obtain the fused features after eliminating degradation. ;

[0106] Step 6.3: Fusion Features The upsampling operation is integrated together and finally sent to the output layer output() composed of convolution and ReLU activation function to obtain the fused image with degradation eliminated: .

[0107] Furthermore, the Step 6 further includes:

[0108] Introducing pixel intensity loss Used to ensure that the fusion result highlights the high contrast information in the source image in terms of visual effect; pixel intensity loss Expressed as:

[0109]

[0110] Introducing gradient loss Used to preserve edge contours and detail textures in the source image:

[0111]

[0112] in, Indicates that there is no degradation in the image to be fused. represents the clear image label corresponding to the degraded image, Indicates the maximum value operation. represents the gradient operator.

[0113] The present invention also provides a unified fusion system for multiple degraded medical images based on degradation prototype learning. The system includes a module for executing the unified fusion method for multiple degraded medical images based on degradation prototype learning.

[0114] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the unified fusion method of multiple degraded medical images based on degraded prototype learning is implemented.

[0115] In order to verify the effectiveness of the method of the present invention, the present invention was trained and tested on the BraTs2020, TCIA_FDG-PET-CT-Lesions, and SynthRAD2023 public data sets. These three data sets each contain 2500 sets of strictly aligned image data. The present invention selected 100 sample pairs each for testing, and the rest were used for model training. The model proposed in the present invention was trained on a server platform equipped with RTX 4090 and equipped with a Pytorch environment. During the training process, the present invention set the number of epochs to 200, the learning rate to 2e-5, the batch size to 3, and used the AdamW optimizer to update the network parameters.

[0116] Furthermore, the present invention organizes and displays the fusion effect diagram after the test. The fusion results of the method of the present invention are as follows Figure 6 As shown, the method proposed in the present invention can effectively solve the negative impact of the degradation information contained in the source image on the final fusion result during medical image fusion, and obtain a high-fidelity fused image without degradation noise.

[0117] The method described in this paper characterizes various degradation information through the learning of degradation prototypes, even capable of characterizing unknown degradations. By using prompt learning, this degradation information is injected into the LoRa branch of the fusion network. This allows the network to exploit complementary information between modalities based on actual conditions, eliminating degradation while simultaneously integrating features, resulting in high-quality fusion results unaffected by degradation. This improves the robustness and practicality of the fusion network in real-world applications.

[0118] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A unified fusion method for multiple degraded medical images based on degraded prototype learning, characterized by: The method comprises: Step 1. Obtain aligned multimodal medical image datasets: Obtain aligned multimodal medical image data from public datasets to generate image degradation to provide training samples; Step 2: Construct degraded data based on real imaging principles: Generate low-dose PET data, CT metal artifact data, and MRI data with motion artifacts through imaging principles; Step 3: Use feature selection mechanism to learn degenerate prototypes; Step 4. Use LoRa branches to handle multiple degradations: Through the low-rank decomposition strategy, the basic fusion model is decomposed into multiple branches. When fusing different degradation data, different branches can be used for processing. Step 5. Promote degradation elimination and feature fusion through degradation prototype prompts: Design a prompt module based on learnable feature prototypes to help LoRa branches eliminate degradation and promote fusion by injecting degradation-related invariant features into different LoRa branches; Step 6. Integrate the features of each branch and construct a fused image: By integrating the degraded fusion features at different scales, a fused image is constructed through the output layer.

2. The unified fusion method for multiple degraded medical images based on degraded prototype learning according to claim 1 is characterized by: Step 1 includes: Each multimodal medical image pair in the aligned multimodal medical image data is composed of a group of medical images with complementary features, including: MRI-T1, MRI-T2; MRI, CT; PET, CT; First, the multimodal medical image dataset is preprocessed. The specific preprocessing method is as follows: the resolution of the three-dimensional image is uniformly resampled to 256×256×256; the data is randomly inverted for data augmentation, and the pixel values ​​of the image are normalized to between [0,1].

3. The unified fusion method of multiple degraded medical images based on degradation prototype learning according to claim 1 is characterized by: Step 2 includes: (1) For MRI-T1 and MRI-T2 fusion scenarios, MRI data with motion artifacts were constructed by simulating the imaging characteristics of the patient's movement in K space during T2 impression imaging; (2) For the MRI and CT fusion scenario, metal artifact masks from clinical acquisition were collected, and metal artifact imaging was simulated by adding noise and beam hardening to construct CT metal artifact data; (3) For the PET and CT fusion scenario, the imaging effect of low-dose PET is simulated by downsampling the standard-dose PET data and adding Poisson noise, thus constructing low-dose PET data.

4. The unified fusion method for multiple degraded medical images based on degradation prototype learning according to claim 1 is characterized by: Step 3 includes: Step 3.1, degrade the image and reference images Input to the convolution-based feature encoder to extract shallow features , and deep features , ; Step 3.2, through deep features , The image is divided into blocks and encoded to obtain sequence information, which is then input into the Transformer for further feature extraction. The extracted features are recorded as , ; In order to aggregate information related to image degradation categories, a learnable degradation category token is also spliced , ; Step 3.3, degenerate category tokens containing global information , As the degenerate representation of each set of features to be fused: , Classification prediction results obtained through the fully connected layer Degenerate category label with the current feature consistent, making , Guide the learning of degenerate prototypes; Step 3.5: Introducing a degenerate prototype consisting of learnable parameters ; When the network performs feature fusion, the prototype is degraded Through the feature selection mechanism, the prompt information related to the current degradation is screened out from the degradation prototype; among them, the information screening depends on the degradation-related feature degradation category token 、 The degradation information contained in Step 3.6, degenerate the category tokens by degenerating the relevant features 、 Send it into the linear layer and Softmax function to get the same as the degraded prototype Information filtering heads of the same shape ; At this time, the information filtering header Degenerate prototype Dot product, from the degenerate prototype Select the degradation-related feature representation, that is, the prompt information , This will serve as a hint to help the subsequent network eliminate degradation.

5. The unified fusion method of multiple degraded medical images based on degradation prototype learning according to claim 1 is characterized by: Step 4 includes: Step 4.

1. Using low-rank decomposition technology, replace the multi-channel convolution in the basic fusion model with low-rank convolution to construct three different LoRa branches; The shallow features obtained in Step 4.2 and Step 3 are cascaded and used as features to be fused. The degraded fused features are sent to the basic fusion model and the constructed LoRa branch for de-degradation and fusion respectively. The respective results are integrated and sent to the output layer to obtain the final fusion result. At the beginning of training, the parameters of the basic fusion model and the LoRa branch are updated together; in the second half of training, the parameters of the basic fusion model will be frozen, and the entire network framework will be fine-tuned by updating only the LoRa branch.

6. The unified fusion method of multiple degraded medical images based on degradation prototype learning according to claim 1, characterized in that: Step 5 includes: In the basic fusion model And the LoRa branch has added a prompt module that can learn feature prototypes; Shallow features , After cascading, it is encoded into multi-scale features , (i=1, 2, 3, 4), are input into the basic network of corresponding scales and LoRa network 、 、 The features are restored and integrated in , where i represents the scale of the network; Basic Network and LoRA network 、 、 In processing features When you need the prompt information you got earlier Assistance is required to obtain fusion features ; Fusion features The calculation formula is: ; in 、 、 The classification score representing the degradation type of the source image obtained by the classifier.

7. The unified fusion method of multiple degraded medical images based on degraded prototype learning according to claim 1, characterized in that: Step 6 includes: Step 6.1, design 4-scale fusion branches in the multi-scale feature integration architecture; , Cascade and convolution downsampling operations to obtain multi-scale features ; Step 6.2: Multi-scale features are fed into the skip link embedding module to obtain the fused features after eliminating degradation. ; Step 6.3: Fusion Features The upsampling operation is integrated together and finally sent to the output layer output() composed of convolution and ReLU activation function to obtain the fused image with degradation eliminated: .

8. The unified fusion method of multiple degraded medical images based on degraded prototype learning according to claim 1, characterized in that: The Step 6 also includes: Introducing pixel intensity loss Used to ensure that the fusion result highlights the high contrast information in the source image in terms of visual effect; pixel intensity loss Expressed as: ； Introducing gradient loss Used to preserve edge contours and detail textures in the source image: ； in, Indicates that there is no degradation in the image to be fused. represents the clear image label corresponding to the degraded image, Indicates the maximum value operation. represents the gradient operator.

9. A unified fusion system for multiple degraded medical images based on degraded prototype learning, characterized by: The system includes: a module for executing the unified fusion method of multiple degraded medical images based on degradation prototype learning according to any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the unified fusion method of multiple degraded medical images based on degraded prototype learning according to any one of claims 1 to 8 is implemented.

Images