Progressive fusion method and system for functional medical images

Abstract

The invention discloses a functional medical image progressive fusion method and system, and relates to the technical field of image processing. Comprising the following steps: acquiring a PET image and a CT image to be processed, and taking an improved 3D U-Net structure as a generator of a generative adversarial network GANs; obtaining a PET feature combined image and a CT feature combined image, and training the generator to obtain an image fusion model; inputting a to-be-processed PET image and a to-be-processed CT image into the image fusion model to obtain a PET feature combined image and a CT feature combined image, and then performing down-sampling to obtain a PET feature map and a CT feature map; carrying out splicing operation on the added image and the up-sampling image to obtain a spliced image; and performing up-sampling on the spliced image, and outputting a fused image through the last layer of up-sampling module. According to the method, gradual change details of focus metabolism information and anatomical structure information can be observed, and the method is suitable for multi-modal fusion requirements of various functional medical images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a functional medical image progressive fusion method and system. Background Art

[0002] Medical imaging technology plays a key role in modern clinical diagnosis and treatment, and can provide tissue and functional information through a variety of imaging modes. In recent years, fusion imaging of positron emission tomography (PET) and computed tomography (CT) has gradually become a powerful tool. PET images focus on reflecting metabolic activity and the functional status of the lesion area, while CT images provide precise anatomical details. Therefore, fusion of PET / CT images can combine the advantages of the two modes and more comprehensively display the functional and structural characteristics of the lesion.

[0003] In the existing field of medical image fusion, traditional methods are mainly based on fixed-ratio image fusion schemes. A common method is to directly superimpose different imaging modes (such as PET / CT) according to set weights. This scheme can combine the metabolic information of PET and the anatomical details of CT to provide doctors with a more comprehensive view of lesions. In recent years, GANs have been widely used in medical image fusion, but most schemes still use the fusion method of half PET / CT, and do not effectively solve the problem of dynamically adjusting the ratio. These GANs-based schemes are mainly trained with fixed-ratio PET / CT samples, and generate images of the ratio used for training during generation.

[0004] The defects of the existing technology are: although these GANs solutions can generate high-quality fused images at a single ratio, they lack the ability to achieve continuous transition between multiple ratio values ​​and cannot generate gradually changing image effects between PET / CT; they cannot flexibly adjust the fusion ratio between PET / CT features according to specific clinical needs, which limits the diversity and flexibility of image analysis. For doctors who want to observe the dynamic changes between the metabolic characteristics and anatomical structural characteristics of lesions, fused images at a single ratio cannot intuitively show the gradual fusion effect from PET to CT, which affects the doctor's in-depth analysis and interpretation of lesions. Summary of the invention

[0005] Based on this, it is necessary to provide a progressive fusion method and system for functional medical images to address the above technical issues.

[0006] The embodiment of the present invention provides a method for progressive fusion of functional medical images, including:

[0007] Acquire PET images and CT images to be processed;

[0008] A low-frequency feature extraction module for processing PET images and a high-frequency feature extraction module for processing CT images are added to the input end of the three-dimensional convolutional neural network 3D U-Net respectively; the three-dimensional convolutional neural network 3D U-Net contains multiple processing layers composed of down-sampling modules and up-sampling modules with corresponding relationships, and a controllable adjustment module is added between every two down-sampling modules and the up-sampling modules corresponding to the two down-sampling modules in the processing layer, and then a splicing operation Cat is added between adjacent up-sampling modules; finally, the convolution operation after each up-sampling module is removed to form an improved 3D U-Net structure, and the improved 3DU-Net structure is used as a generator of the generative adversarial network GANs;

[0009] The low-frequency feature extraction module of the generator is used to extract features from the PET image to be processed, and the extracted low-frequency features are spliced ​​with the original PET image to obtain a PET feature combination image; the high-frequency feature extraction module of the generator is used to extract features from the CT image to be processed, and the extracted high-frequency features are spliced ​​with the original CT image to obtain a CT feature combination image;

[0010] The generator is trained by using PET feature combination images and CT feature combination images, so that the generator learns to adjust the information weight of PET / CT images, and obtains an image fusion model for fusing medical images;

[0011] The PET image and CT image to be processed are input into the image fusion model, and the PET feature combination image is obtained through the low-frequency feature extraction module, and the CT feature combination image is obtained through the high-frequency feature extraction module; the PET feature combination image and the CT feature combination image are downsampled through the downsampling module in each processing layer to obtain a PET feature map and a CT feature map; the fusion ratio MFR between the PET feature map and the CT feature map is adjusted through the controllable adjustment module, the PET feature map and the CT feature map are added according to the fusion ratio MFR, and the added image and the upsampled image obtained by the upsampling module of the previous layer are spliced ​​to obtain a spliced ​​image containing PET features and CT features at different levels; the spliced ​​image is upsampled through the upsampling module of this layer, and the fused image representing the fusion process of the PET image to be processed and the CT image to be processed at different fusion ratios MFR is output through the upsampling module of the last layer.

[0012] Optionally, the generator is connected to a discriminator of a generative adversarial network GANs including a concatenation operation Cat and multiple 4×4 convolutional layers;

[0013] The fused image generated by the generator, the original PET image and the original CT image are spliced ​​through the splicing operation Cat to obtain a multimodal fused image; the multimodal fused image is jointly judged through multiple convolutional layers to obtain the authenticity of the fused image.

[0014] Optionally, the low-frequency feature extraction module and the high-frequency feature extraction module both include a Fourier transform module and a splicing operation Cat;

[0015] Performing Fourier transform on the PET image through the Fourier transform module in the low-frequency feature extraction module to obtain the low-frequency features of the PET image, and splicing the low-frequency features and the original PET image through the splicing operation Cat in the low-frequency feature extraction module to obtain a PET feature combined image;

[0016] The CT image is Fourier transformed through the Fourier transform module in the high-frequency feature extraction module to obtain the high-frequency features of the CT image. The high-frequency features and the original CT image are spliced ​​through the splicing operation Cat in the high-frequency feature extraction module to obtain a CT feature combined image.

[0017] Optionally, a preprocessing operation of size standardization and intensity normalization is performed on the PET image and the CT image to be processed before inputting them into the image fusion model;

[0018] Size normalization means resampling and normalizing the PET image and CT image to be processed to a uniform resolution and size, and intensity normalization means scaling the pixel values ​​of the PET image and CT image to be processed to [0, 1].

[0019] Optionally, obtaining a fused image of the to-be-processed PET image and the to-be-processed CT image specifically includes:

[0020] The PET feature map and the CT feature map are added according to the fusion ratio MFR to obtain a spliced ​​image of the PET feature map and the CT feature map; the formula of the addition process is: [PET feature map*MFR+CT feature map*(1-MFR)];

[0021] The added image and the upsampled image obtained by the upsampling module of the previous layer are stitched together to obtain a stitched image containing PET features and CT features at different levels;

[0022] The stitched image is then upsampled through the upsampling module of this layer to implement the convolution operation, and a fused image of the PET image to be processed and the CT image to be processed is obtained.

[0023] Optionally, a loss function is also included, including:

[0024] The overall loss of the generator is L G The formula is:

[0025]

[0026] Progressive fusion loss L PF The formula is:

[0027]

[0028] Discriminator loss L D The formula is:

[0029] L D =E f [logD(f)]+E x,y,MFR [log(1-D(G(x,y,MFR)))];

[0030] Among them, L adv Represents the discriminant loss of the fused image, x represents the PET image, y represents the CT image, f represents the expected progressive fusion image controlled by the controllable adjustment module, λ represents the weight of the loss function, MFR represents the fusion ratio, G represents the generator model, and D represents the discriminator model.

[0031] The embodiment of the present invention further provides a functional medical image progressive fusion system, including:

[0032] An image acquisition module, used for acquiring PET images and CT images to be processed;

[0033] A generator construction module is used to add a low-frequency feature extraction module for processing PET images and a high-frequency feature extraction module for processing CT images to the input end of a three-dimensional convolutional neural network 3D U-Net; the three-dimensional convolutional neural network 3D U-Net includes multiple processing layers composed of down-sampling modules and up-sampling modules with corresponding relationships, a controllable adjustment module is added between every two down-sampling modules and the up-sampling modules corresponding to the two down-sampling modules in the processing layer, and a splicing operation Cat is added between adjacent up-sampling modules; finally, the convolution operation after each up-sampling module is removed to form an improved 3D U-Net structure, and the improved 3D U-Net structure is used as a generator of a generative adversarial network GANs;

[0034] A feature combination module is used to extract features from the PET image to be processed through the low-frequency feature extraction module of the generator, and to splice the extracted low-frequency features with the original PET image to obtain a PET feature combination image; to extract features from the CT image to be processed through the high-frequency feature extraction module of the generator, and to splice the extracted high-frequency features with the original CT image to obtain a CT feature combination image;

[0035] A model training module is used to train the generator through PET feature combination images and CT feature combination images, so that the generator learns to adjust the information weight of the PET / CT images and obtains an image fusion model for fusing medical images;

[0036] The image fusion module is used to input the PET image and CT image to be processed into the image fusion model, obtain the PET feature combination image through the low-frequency feature extraction module, and obtain the CT feature combination image through the high-frequency feature extraction module; down-sample the PET feature combination image and the CT feature combination image through the down-sampling module in each processing layer to obtain the PET feature map and the CT feature map; adjust the fusion ratio MFR between the PET feature map and the CT feature map through the controllable adjustment module, add the PET feature map and the CT feature map according to the fusion ratio MFR, and splice the added image with the up-sampled image obtained by the up-sampling module of the previous layer to obtain a spliced ​​image containing PET features and CT features at different levels; then up-sample the spliced ​​image through the up-sampling module of this layer, and output the fused image representing the fusion process of the PET image to be processed and the CT image to be processed at different fusion ratios MFR through the up-sampling module of the last layer.

[0037] Compared with the prior art, the above-mentioned method and system for progressive fusion of functional medical images provided by the embodiment of the present invention has the following beneficial effects:

[0038] The present invention improves the generator of generative adversarial networks (GANs) by introducing a controllable adjustment module into a three-dimensional convolutional neural network 3D U-Net. The controllable adjustment module can adjust the fusion ratio MFR between the PET feature map and the CT feature map, so that the generated fusion image represents the fusion process of the to-be-processed PET image and the to-be-processed CT image at different fusion ratios MFR, and can observe the gradual details of the lesion metabolic information and the anatomical structure information, which is suitable for the multimodal fusion requirements of various functional medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The present invention is a flowchart of the overall progressive fusion of PET / CT of a progressive fusion method of functional medical images provided in one embodiment;

[0040] Figure 2 A PET / CT progressive fusion training flow chart of a functional medical image progressive fusion method provided in one embodiment;

[0041] Figure 3 A generator structure diagram of a functional medical image progressive fusion method provided in one embodiment;

[0042] Figure 4A discriminator structure diagram of a progressive fusion method for functional medical images provided in one embodiment;

[0043] Figure 5 A flowchart of PET / CT progressive fusion generation of a functional medical image progressive fusion method provided in an embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] In one embodiment, a method for progressive fusion of functional medical images is provided, the method comprising:

[0046] Acquire the PET images and CT images to be processed.

[0047] A low-frequency feature extraction module for processing PET images and a high-frequency feature extraction module for processing CT images are added to the input end of the three-dimensional convolutional neural network 3D U-Net respectively; the three-dimensional convolutional neural network 3D U-Net contains multiple processing layers composed of corresponding down-sampling modules and up-sampling modules, and a controllable adjustment module is added between every two down-sampling modules and the up-sampling modules corresponding to the two down-sampling modules in the processing layer, and then a splicing operation Cat is added between adjacent up-sampling modules; finally, the convolution operation after each up-sampling module is removed to form an improved 3D U-Net structure, and the improved 3DU-Net structure is used as a generator of generative adversarial networks (GANs).

[0048] The low-frequency feature extraction module of the generator is used to extract features from the PET image to be processed, and the extracted low-frequency features are spliced ​​with the original PET image to obtain a PET feature combination image; the high-frequency feature extraction module of the generator is used to extract features from the CT image to be processed, and the extracted high-frequency features are spliced ​​with the original CT image to obtain a CT feature combination image.

[0049] The generator is trained through PET feature combination images and CT feature combination images, so that the generator can learn to adjust the information weight of PET / CT images and obtain an image fusion model for fusing medical images.

[0050] The PET image and CT image to be processed are input into the image fusion model, and the PET feature combination image is obtained through the low-frequency feature extraction module, and the CT feature combination image is obtained through the high-frequency feature extraction module; the PET feature combination image and the CT feature combination image are downsampled through the downsampling module in each processing layer to obtain a PET feature map and a CT feature map; the fusion ratio MFR between the PET feature map and the CT feature map is adjusted through the controllable adjustment module, the PET feature map and the CT feature map are added according to the fusion ratio MFR, and the added image and the upsampled image obtained by the upsampling module of the previous layer are spliced ​​to obtain a spliced ​​image containing PET features and CT features at different levels; the spliced ​​image is upsampled through the upsampling module of this layer, and the fused image representing the fusion process of the PET image to be processed and the CT image to be processed at different fusion ratios MFR is output through the upsampling module of the last layer.

[0051] The present invention mainly has four key steps: data preprocessing, frequency feature extraction, dual-branch fusion training and progressive fusion generation. The overall flow chart is as follows: Figure 1 shown.

[0052] 1. Data preprocessing

[0053] Before model training, the PET / CT image data is preprocessed to ensure that the model can effectively learn on consistent and standardized image data, thereby generating high-quality fusion images. The steps of data preprocessing include the following aspects:

[0054] (1) Extracting the chest region: In order to focus on key anatomical regions and improve the fusion effect, the present invention extracts the chest region in the PET / CT image data. The extraction of the chest region helps to reduce the interference of irrelevant background information and make the model more focused on the parts of interest such as lesions and tumors.

[0055] (2) Size standardization: PET / CT image data generated by different imaging devices have different resolutions and sizes. In order to ensure the consistency of model input data, the present invention resamples and standardizes all image data to a uniform resolution and size (128×128×128 pixels) to ensure that the model will not be disturbed by size differences during training.

[0056] (3) Intensity normalization: Since the pixel value ranges of PET / CT images vary greatly, in order to further improve the effect of model training, the present invention performs intensity normalization on each image type and scales the pixel values ​​to the range of [0, 1].

[0057] 2. Frequency domain feature extraction

[0058] In order to extract different features of PET / CT images, the present invention introduces a frequency separation method, which decomposes the image into high-frequency and low-frequency components through 3D Fourier transform, and combines each component with the original image to retain the key features of PET / CT:

[0059] Low-frequency feature extraction: The low-frequency component represents the overall structural information of the image. The present invention splices the low-frequency component of the PET image with its original image to retain the metabolic activity characteristics of the PET image. The PET image is Fourier transformed by the Fourier transform module in the low-frequency feature extraction module to obtain the low-frequency features of the PET image. The low-frequency features and the original PET image are spliced ​​by the splicing operation Cat in the low-frequency feature extraction module to obtain a PET feature combined image.

[0060] High-frequency feature extraction: The high-frequency component mainly contains the edge and detail information of the image. The present invention splices the high-frequency component of the CT image with the original image to enhance the anatomical structure information of the CT image. The CT image is Fourier transformed by the Fourier transform module in the high-frequency feature extraction module to obtain the high-frequency features of the CT image. The high-frequency features and the original CT image are spliced ​​by the splicing operation Cat in the high-frequency feature extraction module to obtain a CT feature combined image.

[0061] Combined input: After frequency separation processing, the obtained PET / CT feature combined images are used as dual-branch inputs of the model, enhancing the feature expression capability of each modality.

[0062] 3. Dual-branch fusion training

[0063] In order to learn the gradient characteristics of PET / CT images at different fusion ratios, the present invention designs a dual-branch structure in the GAN generator, and realizes multi-ratio learning and fusion by introducing different MFRs during the training process. The training flow chart is shown in Figure 2 shown.

[0064] Generator design: The generator adopts an improved 3D U-Net structure. Two independent input branches are designed in the encoding stage to receive the spliced ​​PET / CT images respectively. MFR adjustment is designed in the decoding stage to control the fusion ratio MFR of PET / CT.

[0065] MFR design: When decoding each layer of the generator, by adjusting the MFR (such as 0.2, 0.4, 0.6, etc.), the PET and CT features of the corresponding encoding stage are fused using the weighted sum of feature maps, that is, [PET feature map*MFR+CT feature map*(1-MFR)], and then jump-connected with the upsampled feature map to perform a 3D convolution operation to retain the characteristics of each modality and achieve controllable feature ratio adjustment. The generator structure diagram is shown in the figure. Figure 3 shown.

[0066] Discriminator design: The discriminator includes a splicing operation Cat and multiple 4×4 convolutional layers. The splicing operation Cat is used to splice the fused image generated by the generator, the original PET image, and the original CT image to obtain a multimodal fused image. The multimodal fused image is jointly discriminated through multiple convolutional layers to obtain the authenticity of the fused image. The discriminator structure is shown in the figure below. Figure 4 shown.

[0067] 4. Progressive Fusion Generation

[0068] In the test phase, by adjusting the MFR, PET / CT fusion images with different ratios can be generated, thereby generating a progressive fusion process of PET / CT. Figure 5 shown.

[0069] Progressive fusion process: During the testing phase, by adjusting the MFR (such as 0.1, 0.3, 0.5, 0.7, etc.), the generator can generate continuous progressive fusion images between PET / CT images, and obtain an image sequence that gradually transitions from CT features to PET features.

[0070] Dynamic adjustability: Doctors can select the required fusion ratio MFR according to different diagnostic needs and adjust the fusion ratio MFR to view image details at different feature ratios of PET / CT.

[0071] 5. Loss Function

[0072] Different from the traditional generative model that performs single image fusion through pixel-level related losses, this paper proposes an innovative progressive fusion loss (L PF ), the progressive fusion generation of PET / CT was effectively achieved by adjusting the MFR.

[0073] Common losses in previous generative models include but are not limited to L1 similarity loss (L L1 ), L2 similarity loss (L L2 ), the loss construction takes L1 similarity loss as an example. L1 similarity loss is shown in formula (1)

[0074] L L1 (x,y)=E x，y [||0.5×(x+y)-G(x,y)||1]; (1)

[0075] Among them, x represents the PET image and y represents the CT image.

[0076] Progressive fusion loss L PF As shown in formula (2):

[0077]

[0078] Where f represents the expected progressive fused image controlled by the fusion ratio MFR, and MFR represents the fusion ratio.

[0079] Add the discriminator to the fusion image discrimination loss L adv The overall loss of the complete generator is shown in formula (3):

[0080]

[0081] Among them, λ represents the weight of the loss function, G represents the generator model, and D represents the discriminator model.

[0082] The goal of the discriminator is to distinguish whether the input fused image is real or generated by the generator. Through adversarial learning, the discriminator loss guides the system to continuously optimize, outputting a high credibility score for a real fused image and a low credibility score for a generated fused image. The discriminator loss is shown in formula (4):

[0083] L D =E f [logD(f)]+E x,y,MFR [log(1-D(G(x,y,MFR)))]; (4)

[0084] The effect analysis of the present invention is as follows:

[0085] (1) Traditional image fusion methods use fixed-ratio fusion, which limits the doctor's ability to flexibly adjust between different PET / CT information ratios and makes it difficult to generate gradient images with arbitrary ratios between PET / CT.

[0086] By introducing MFR into GANs, the present invention enables the model to learn fusion image features of different proportions during the training phase, thereby generating progressive fusion images of any proportion between PET / CT during the testing phase, and realizing flexible control of PET / CT features. This technology enables doctors to customize the proportion of PET / CT features during image analysis, obtain more diagnostically valuable perspectives, and provide more accurate imaging support for clinical diagnosis.

[0087] (2) Traditional methods make it difficult to maintain the integrity and physiological accuracy of the anatomical structure during the fusion process, which affects the doctor's judgment of the tissues and structures surrounding the lesion.

[0088] The present invention uses frequency separation and feature extraction technology to extract and fuse the low-frequency and high-frequency features of PET / CT images, effectively improving the structural consistency and detail expression ability of the generated images. In addition, feature analysis also improves the recognition of tissue information around the lesion, making the generated fused image not only visually clear, but also consistent with the real physiological structure. This high-precision structure can ensure the reliability and consistency of functional medical image fusion.

[0089] Based on the same inventive concept, the present invention also provides a functional medical image progressive fusion system, which includes:

[0090] The image acquisition module is used to acquire the PET images and CT images to be processed.

[0091] The generator construction module is used to add a low-frequency feature extraction module for processing PET images and a high-frequency feature extraction module for processing CT images to the input end of the three-dimensional convolutional neural network 3D U-Net. The three-dimensional convolutional neural network 3D U-Net contains multiple processing layers composed of corresponding downsampling modules and upsampling modules. A controllable adjustment module is added between every two downsampling modules and the upsampling modules corresponding to the two downsampling modules in the processing layer, and a splicing operation Cat is added between adjacent upsampling modules; finally, the convolution operation after each upsampling module is removed to form an improved 3D U-Net structure, and the improved 3D U-Net structure is used as a generator of the generative adversarial network GANs.

[0092] The feature combination module is used to extract features from the PET image to be processed through the low-frequency feature extraction module of the generator, and to splice the extracted low-frequency features with the original PET image to obtain a PET feature combination image. The feature extraction is performed on the CT image to be processed through the high-frequency feature extraction module of the generator, and to splice the extracted high-frequency features with the original CT image to obtain a CT feature combination image.

[0093] The model training module is used to train the generator through PET feature combination images and CT feature combination images, so that the generator learns to adjust the information weight of PET / CT images and obtains an image fusion model for fusing medical images.

[0094] The image fusion module is used to input the PET image and CT image to be processed into the image fusion model, obtain the PET feature combination image through the low-frequency feature extraction module, and obtain the CT feature combination image through the high-frequency feature extraction module; downsample the PET feature combination image and the CT feature combination image through the downsampling module in each processing layer to obtain the PET feature map and the CT feature map. The fusion ratio MFR between the PET feature map and the CT feature map is adjusted through the controllable adjustment module, the PET feature map and the CT feature map are added according to the fusion ratio MFR, and the added image is spliced ​​with the upsampled image obtained by the upsampling module of the previous layer to obtain a spliced ​​image containing PET features and CT features at different levels. The spliced ​​image is then upsampled through the upsampling module of this layer, and the fused image representing the fusion process of the PET image to be processed and the CT image to be processed at different fusion ratios MFR is output through the upsampling module of the last layer.

[0095] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A progressive fusion method for functional medical images, characterized in that: include: Acquire PET images and CT images to be processed; A low-frequency feature extraction module for processing PET images and a high-frequency feature extraction module for processing CT images are added to the input end of the three-dimensional convolutional neural network 3D U-Net respectively; the three-dimensional convolutional neural network 3D U-Net contains multiple processing layers composed of down-sampling modules and up-sampling modules with corresponding relationships, and a controllable adjustment module is added between every two down-sampling modules and the up-sampling modules corresponding to the two down-sampling modules in the processing layer, and then a splicing operation Cat is added between adjacent up-sampling modules; finally, the convolution operation after each up-sampling module is removed to form an improved 3D U-Net structure, and the improved 3DU-Net structure is used as a generator of the generative adversarial network GANs; The low-frequency feature extraction module of the generator is used to extract features from the PET image to be processed, and the extracted low-frequency features are spliced ​​with the original PET image to obtain a PET feature combination image; the high-frequency feature extraction module of the generator is used to extract features from the CT image to be processed, and the extracted high-frequency features are spliced ​​with the original CT image to obtain a CT feature combination image; The generator is trained by using PET feature combination images and CT feature combination images, so that the generator learns to adjust the information weight of PET / CT images, and obtains an image fusion model for fusing medical images; The PET image and CT image to be processed are input into the image fusion model, and the PET feature combination image is obtained through the low-frequency feature extraction module, and the CT feature combination image is obtained through the high-frequency feature extraction module; the PET feature combination image and the CT feature combination image are downsampled through the downsampling module in each processing layer to obtain a PET feature map and a CT feature map; the fusion ratio MFR between the PET feature map and the CT feature map is adjusted through the controllable adjustment module, the PET feature map and the CT feature map are added according to the fusion ratio MFR, and the added image and the upsampled image obtained by the upsampling module of the previous layer are spliced ​​to obtain a spliced ​​image containing PET features and CT features at different levels; the spliced ​​image is upsampled through the upsampling module of this layer, and the fused image representing the fusion process of the PET image to be processed and the CT image to be processed at different fusion ratios MFR is output through the upsampling module of the last layer.

2. A functional medical image progressive fusion method as claimed in claim 1, characterized in that: The generator is connected to a discriminator of a generative adversarial network GANs including a concatenation operation Cat and a plurality of 4×4 convolutional layers; The fused image generated by the generator, the original PET image and the original CT image are spliced ​​through the splicing operation Cat to obtain a multimodal fused image; the multimodal fused image is jointly judged through multiple convolutional layers to obtain the authenticity of the fused image.

3. The method for progressive fusion of functional medical images according to claim 1, characterized in that: The low-frequency feature extraction module and the high-frequency feature extraction module both include a Fourier transform module and a splicing operation Cat; Performing Fourier transform on the PET image through the Fourier transform module in the low-frequency feature extraction module to obtain the low-frequency features of the PET image, and splicing the low-frequency features and the original PET image through the splicing operation Cat in the low-frequency feature extraction module to obtain a PET feature combined image; The CT image is Fourier transformed through the Fourier transform module in the high-frequency feature extraction module to obtain the high-frequency features of the CT image. The high-frequency features and the original CT image are spliced ​​through the splicing operation Cat in the high-frequency feature extraction module to obtain a CT feature combined image.

4. The method for progressive fusion of functional medical images according to claim 1, characterized in that: Before inputting the PET images and CT images to be processed into the image fusion model, a preprocessing operation of size standardization and intensity normalization is performed; Size normalization means resampling and normalizing the PET image and CT image to be processed to a uniform resolution and size, and intensity normalization means scaling the pixel values ​​of the PET image and CT image to be processed to [0, 1].

5. The method for progressive fusion of functional medical images according to claim 1, characterized in that: The step of obtaining a fused image of the to-be-processed PET image and the to-be-processed CT image specifically includes: The PET feature map and the CT feature map are added according to the fusion ratio MFR to obtain a spliced ​​image of the PET feature map and the CT feature map; the formula of the addition process is: [PET feature map*MFR+CT feature map*(1-MFR)]; The added image and the upsampled image obtained by the upsampling module of the previous layer are stitched together to obtain a stitched image containing PET features and CT features at different levels; The stitched image is then upsampled through the upsampling module of this layer to implement the convolution operation, and a fused image of the PET image to be processed and the CT image to be processed is obtained.

6. The method for progressive fusion of functional medical images according to claim 1, characterized in that: It also includes loss functions, including: The overall loss of the generator is L G The formula is: Among them, the progressive fusion loss L PF The formula is: Discriminator loss L D The formula is: L D =E f [logD(f)]+E x，y，MFR [log(1-D(G(x,y,MFR)))]; Among them, L adv Represents the discriminant loss of the fused image, x represents the PET image, y represents the CT image, f represents the expected progressive fusion image controlled by the controllable adjustment module, λ represents the weight of the loss function, MFR represents the fusion ratio, G represents the generator model, and D represents the discriminator model.

7. A functional medical image progressive fusion system, characterized in that: include: An image acquisition module, used for acquiring PET images and CT images to be processed; A generator construction module is used to add a low-frequency feature extraction module for processing PET images and a high-frequency feature extraction module for processing CT images at the input end of a three-dimensional convolutional neural network 3D U-Net; the three-dimensional convolutional neural network 3DU-Net includes multiple processing layers composed of down-sampling modules and up-sampling modules with corresponding relationships, a controllable adjustment module is added between every two down-sampling modules and the up-sampling modules corresponding to the two down-sampling modules in the processing layer, and a splicing operation Cat is added between adjacent up-sampling modules; finally, the convolution operation after each up-sampling module is removed to form an improved 3D U-Net structure, and the improved 3D U-Net structure is used as a generator of a generative adversarial network GANs; A feature combination module is used to extract features from the PET image to be processed through the low-frequency feature extraction module of the generator, and to splice the extracted low-frequency features with the original PET image to obtain a PET feature combination image; to extract features from the CT image to be processed through the high-frequency feature extraction module of the generator, and to splice the extracted high-frequency features with the original CT image to obtain a CT feature combination image; A model training module is used to train the generator through PET feature combination images and CT feature combination images, so that the generator learns to adjust the information weight of the PET / CT images and obtains an image fusion model for fusing medical images; The image fusion module is used to input the PET image and CT image to be processed into the image fusion model, obtain the PET feature combination image through the low-frequency feature extraction module, and obtain the CT feature combination image through the high-frequency feature extraction module; down-sample the PET feature combination image and the CT feature combination image through the down-sampling module in each processing layer to obtain the PET feature map and the CT feature map; adjust the fusion ratio MFR between the PET feature map and the CT feature map through the controllable adjustment module, add the PET feature map and the CT feature map according to the fusion ratio MFR, and splice the added image with the up-sampled image obtained by the up-sampling module of the previous layer to obtain a spliced ​​image containing PET features and CT features at different levels; then up-sample the spliced ​​image through the up-sampling module of this layer, and output the fused image representing the fusion process of the PET image to be processed and the CT image to be processed at different fusion ratios MFR through the up-sampling module of the last layer.

Images