Image depth super-resolution method based on double converters

Through the deep super-resolution network DTSRNet based on dual converter, the problem of insufficient ADC image resolution and visual quality is solved, which significantly improves the resolution and quality of breast cancer medical images, enhances the accuracy of breast cancer diagnosis, and has the following requirements: Higher robustness.

CN119006279APending Publication Date: 2024-11-22THE CHINESE UNIVERSITY OF HONG KONG
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310561833.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-18
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Traditional ADC images have poor resolution and visual quality, a lot of noise, making it difficult to use for early detection and diagnosis of breast cancer. The effectiveness of existing super-resolution methods in breast cancer medical images needs further verification.

Method used

Using a deep super-resolution network based on a dual converter, DTSRNet converter, converts low-resolution DCE-MRI and ADC images into high-resolution images through U-type networks and residual enhancement upsampling networks, using dual converter modules and images The block merging operation performs downsampling and upsampling, combining the displacement window transformer and the convolution transformer to extract spatial local and global information of the image, fusing residual information to enhance image resolution.

Benefits of technology

Significantly improves the visual resolution and quality of DCE-MRI and ADC images, enhances the texture and structural integrity of the image, improves the prediction and diagnostic accuracy of breast cancer images, and has higher robustness, suitable for other Complex medical image processing tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006279A_ABST
    Figure CN119006279A_ABST
Patent Text Reader

Abstract

An image depth super-resolution method based on a dual converter comprises the following steps: converting a low-resolution DCE-MRI image or an apparent diffusion coefficient ADC image into a high-resolution image by using a pre-trained super-resolution network DTSRNet based on the dual converter; wherein the super-resolution network DTSRNet comprises a U-shaped network SCUNet based on a dual converter and a residual error enhancement upsampling network REUpNet; the SCUNet uses a dual converter module SCTB and an image block merging operation PM to gradually down-sample an input low-resolution image into a bottom-layer feature map, and then uses the SCTB and an up-sampling operation PSUS based on pixel confusion to gradually up-sample the bottom-layer feature map to obtain a primary reconstruction feature map; and the REUpNet performs up-sampling on an input low-resolution image and the primary reconstruction feature map by using a parallel enhanced up-sampling module (EUM), and then fuses the results of the low-resolution image and the primary reconstruction feature map to obtain the high-resolution image. According to the method, the resolution and the visual quality of the DCE-MRI image and the ADC image can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image depth super-resolution method based on a double transformer (Double Transformer-Based Super-Resolution Network, DTSRNet). Background Art

[0002] As the disease with the highest morbidity and mortality rate affecting women's health worldwide in recent decades, early detection and treatment of breast cancer are very important and can significantly improve the survival rate of patients. Traditional X-ray mammography, as the most popular cancer screening and detection tool, has the problem of reduced detection sensitivity in fat and dense breasts. In order to solve this situation, various imaging modes have been widely studied and used. Among them, dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) has become a widely used imaging technology to solve the problem of ambiguous breast lesions due to its significant sensitivity. DCE-MRI can not only express the morphological characteristics of breast lesions, but also reveal the physiological changes and dynamic enhancement properties of contrast agents in tumors. So far, DCE-MRI has been widely used in clinical trials for breast cancer diagnosis, prediction and treatment, and has shown promising practicality in detecting and characterizing early cancers depicted in fat and dense breasts.

[0003] As an alternative and complementary technique to DCE-MRI, diffusion weighted imaging (DWI) has been widely studied as part of multiparametric MRI. The appearance characteristics of DWI images are determined by a parameter, b-value. Higher b-values ​​generate DWI with lower signal-to-noise ratios (SNR), but can provide intensive behavior of molecular motion and water molecule diffusion within tissues. In contrast, DWI with lower b-values ​​can only measure the mild diffusion properties of tissues, but has a higher signal-to-noise ratio and reveals more anatomical information. Specifically, the apparent diffusion coefficient (ADC) derived from DWI can reveal the mobility and impedance of water molecules around microstructures. Increasing evidence shows that ADC images have been widely used as an imaging biomarker for lesion detection and diagnosis, especially for the prediction of tumor histological grade, molecular subtypes, and receptor status.

[0004] Compared with DCE-MRI, ADC images in conventional clinical settings have poor resolution and visual quality and are subject to a lot of noise. This characteristic makes it challenging to apply ADC images to tumor characterization and they are rarely used for radiomics analysis. Therefore, researchers have shown great interest in improving the spatial resolution and visual quality of ADC images. In this case, features derived from ADC images can be used as biomarkers to capture the pathological characteristics and chemotherapy response of breast cancer. So far, in order to promote more reliable tumor prediction and better clinical application of ADC images, researchers have developed a large number of super-resolution (SR) methods to convert low-resolution (LR) images into high-resolution (HR) images.

[0005] Traditional data interpolation techniques such as spline interpolation, bicubic interpolation, and Lanczos resampling are often used to improve image resolution and visual quality. However, these techniques are inevitably affected by block artifacts and blurred edges. Currently, deep learning is used as an alternative to learn an end-to-end mapping function from LR images to their HR counterparts without any manual engineering and image prior knowledge. In addition, recent studies have demonstrated that super-resolution methods based on deep learning can effectively improve image resolution and visual quality in musculoskeletal imaging, fluorescence microscopy, and neuroimaging. Although these SR methods have achieved remarkable performance, their effectiveness on breast cancer tumor medical images, especially on low-resolution ADC images, needs further verification.

[0006] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0007] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide an efficient image depth super-resolution method based on dual transformer to improve the visual resolution of DCE-MRI and ADC images.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] An image depth super-resolution method based on a dual transformer comprises the following steps:

[0010] Use the pre-trained dual transformer-based super-resolution network DTSRNet to convert low-resolution DCE-MRI images or apparent diffusion coefficient ADC images into high-resolution images;

[0011] The super-resolution network DTSRNet includes a dual transformer-based U-type network SCUNet and a residual enhanced upsampling network REUpNet;

[0012] The SCUNet uses a dual transformer module SCTB and an image block merging operation PM to gradually downsample the input low-resolution image into a bottom-level feature map, and then uses SCTB and a pixel confusion-based upsampling operation PSUS to gradually upsample the bottom-level feature map to obtain a primary reconstructed feature map;

[0013] The REUpNet utilizes a parallel enhanced upsampling module EUM to upsample the input low-resolution image and the primary reconstructed feature map respectively, and then fuses the results of the two to obtain the high-resolution image.

[0014] Furthermore, the SCUNet uses a dual transformer module SCTB and an image block merging operation PM to gradually downsample the input low-resolution image to an underlying feature map, specifically including: SCUNet uses three dual transformer modules SCTB and three image block merging operations PM, namely SCTB, PM, SCTB, PM, SCTB and PM, thereby gradually downsampling the input low-resolution image to an underlying feature map.

[0015] Furthermore, the use of SCTB and pixel confusion-based upsampling operation PSUS to gradually upsample the underlying feature map to obtain a primary reconstructed feature map specifically includes: SCUNet performs three pixel confusion-based upsampling operations PSUS and three dual transformer modules SCTB, namely PSUS, SCTB, PSUS, SCTB, PSUS and SCTB, and performs residual connection before processing through each SCTB, thereby gradually upsampling the underlying feature map to obtain a primary reconstructed feature map.

[0016] Furthermore, the dual transformer module SCTB includes two shift window transformers Swin-T and two convolution transformers Conv-T, namely Swin-T, Swin-T, Conv-T and Conv-T, respectively, wherein the Swin-T extracts the long-range dependencies of the context between the image contents to enrich the content of the reconstructed image, and the Conv-T highlights the spatial local context information of the image through convolution operations to obtain clearer image texture.

[0017] Furthermore, the Swin-T includes convolutional layer normalization, a multi-head self-attention module MSA, a residual connection, and a 2-layer multi-layer perceptron MLP, wherein the window-based multi-head self-attention module W-MSA and the shifted window-based multi-head self-attention module SW-MSA are respectively used in two consecutive Swin-T modules, as expressed as follows:

[0018]

[0019]

[0020]

[0021]

[0022] Among them, LN represents the convolution layer normalization operation, MLP represents the 2-layer multi-layer perceptron operation, W-MSA and SW-MSA represent the window-based multi-head self-attention module and the shift window-based multi-head self-attention MSA module operation, and z l They represent the outputs of the MSA module and the MLP module at the lth layer respectively.

[0023] Furthermore, the specific operation of Conv-T is as follows:

[0024] (1) Divide the input into N blocks of equal size Then, the N heads are used for processing respectively. For the nth head, a convolution layer with a convolution kernel size of 1×1 is used to process the query (Query, Q n ), Key (Key, K n ) and value (Value, V n ):

[0025] Q n =Conv1(LN(F n ))

[0026] K n =Conv1(LN(F n ))

[0027] V n =Conv1(LN(F n ))

[0028] Among them, LN represents the convolution layer normalization operation, and Conv1 represents the convolution layer operation with a convolution kernel size of 1×1;

[0029] (2) Calculate the self-attention feature map F SA :

[0030]

[0031] Among them, softmax represents the softmax function operation, Represents matrix multiplication;

[0032] (3) After processing by N heads, N self-attention feature maps are obtained, which are then stacked and concat together. After a convolution layer Conv1 with a convolution kernel size of 1×1 and a ReLU activation function, the output Y is obtained:

[0033] SA=Concat(SA1,…,SA n )

[0034] Y = ReLU(Conv1(SA).

[0035] Furthermore, the REUpNet performs an element-wise addition operation on the results of the parallel processing of the primary reconstructed feature map and the input low-resolution image, fuses the residual information of the input image, and obtains a restored high-resolution image.

[0036] Furthermore, the parallel enhanced upsampling module EUM is used to upsample the input low-resolution image and the primary reconstructed feature map respectively, specifically including one-step upsampling and two-step upsampling, wherein the one-step upsampling operation passes the low-resolution image through a convolution layer, and then directly transforms the input to a high-resolution image of the target resolution through upsampling; the two-step upsampling operation first transforms the primary reconstructed feature map through a convolution operation and an upsampling operation to transform the input to half of the target resolution, and then undergoes another convolution operation and upsampling operation to obtain the final target resolution.

[0037] Furthermore, the pre-training includes training a super-resolution network DTSRNet using pre-processed DCE-MRI images.

[0038] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image depth super-resolution method based on a dual transformer is implemented.

[0039] The present invention has the following beneficial effects:

[0040] The dual transformer-based image deep super-resolution method of the present invention improves the visual resolution of DCE-MRI and ADC images through an efficient dual transformer-based deep super-resolution network DTSRNet. Through the deep learning network designed by the present invention, the resolution and visual quality of the image are enhanced. In a typical application, the present invention can be used to achieve deep super-resolution of breast cancer images, and the obtained high-resolution images are conducive to higher accuracy in subsequent breast cancer prediction and diagnosis tasks. Specifically, the dual transformer model of the present invention can significantly enhance the resolution and visual quality of the image. The dual transformer model can not only fully extract the spatial local information of the image, but also explore the global structural information of the image, so that the restored high-resolution image has more texture and complete structure. In particular, considering that the upsampling process cannot fully restore the detailed information of the image, the present invention utilizes the residual information of the input image and fuses the results from two different upsampling through an enhanced upsampling module to obtain a clearer high-resolution image.

[0041] The preset DTSRNet network can be trained using DCE-MRI images. The resulting DTSRNet model can improve the resolution and visual quality of DCE-MRI images and diffusion-weighted imaging (DWI)-derived apparent diffusion coefficient (ADC) images. It can not only be flexibly used in image super-resolution tasks (such as breast cancer medical images), but can also be used as a preprocessing technique to improve the accuracy of disease diagnosis and prediction tasks based on medical images (such as breast cancer).

[0042] In addition, the method proposed in the present invention can not only enhance the visual resolution of DCE-MRI and ADC images, but also cope with other complex application scenarios with higher robustness. The present invention can be further applied to other tasks such as segmentation, recognition and detection as breast cancer medical image processing software and system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a breast cancer medical image super-resolution model according to an embodiment of the present invention;

[0044] Figure 2 The specific implementation steps of the dual transformer-based deep super-resolution network DTSRNet according to an embodiment of the present invention;

[0045] Figure 3 The overall network structure of the dual transformer-based deep super-resolution network DTSRNet according to an embodiment of the present invention;

[0046] Figure 4 The specific network structure based on the displacement converter Swin-T of the embodiment of the present invention;

[0047] Figure 5 Specific network structure of the convolution transformer Conv-T according to an embodiment of the present invention;

[0048] Figure 6 The super-resolution results of breast cancer DCE-MRI images obtained by the embodiment of the present invention are as follows: (a) original DCE-MRI image, (b) low-resolution DCE-MRI image, and (c) super-resolution DCE-MRI image obtained by the method of the embodiment of the present invention;

[0049] Figure 7 1 is a super-resolution result of a breast cancer ADC image according to an embodiment of the present invention, (a) the original ADC image, (b) the super-resolution ADC rate image obtained by the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0051] See also Figure 1 to Figure 2 Taking the application of enhancing the resolution of breast cancer ADC images as an example, the deep method of image resolution based on dual transformers in the embodiment of the present invention includes the following steps: obtaining a large number of dynamic contrast enhanced MRI images of breast cancer patients, performing breast region segmentation and downsampling on them, and obtaining low-resolution (Low-Resolution, LR) DCE-MRI images. The LR DCE-MRI images are converted into high-resolution (High-Resolution, HR) DCE-MRI images using a preset dual transformer-based super-resolution network DTSRNet.

[0052] In some embodiments, the overall operation flow of the image deep super-resolution method based on the dual transformer includes the following parts: preprocessing of breast cancer images, training of the super-resolution model DTSRNet designed by the present invention based on DCE-MRI images, and enhancing the resolution of breast cancer ADC images using the trained DTSRNet model. The following is a specific description:

[0053] S1. Breast cancer image preprocessing

[0054] Breast cancer image preprocessing first segments the breast region of the input DCE-MRI image and removes the skin and chest wall area around the breast. These areas are not only meaningless for the diagnosis and feature analysis of breast tumors, but also increase the computational complexity in the deep learning network. The present invention segments the breast region of the DCE-MRI image through a preset breast segmentation model (Segment Network, SegNet), and then separates the segmented results from the middle position to obtain breast images on the left and right sides: left breast image and right breast image. In the subsequent deep network training, both the left breast image and the right breast image are used as input, so that the sample training of the training set can be increased to prevent overfitting and obtain more stable training results.

[0055] S2. Training deep super-resolution model DTSRNet based on DCE-MRI images

[0056] According to the preprocessed DCE-MRI images, the preset dual transformer-based deep super-resolution network DTSRNet is trained. Figure 3 , DTSRNet mainly consists of two sub-networks: a dual transformer-based U-shape network (Swin-ConvTransformer-Based U-Shape Network, SCUNet) and a residual enhanced upscale network (ResidualEnhancedUpscaleNetwork, REUpNet).

[0057] like Figure 3 As shown in the figure, the dual transformer-based U-shaped network SCUNet converts the input low-resolution image into a primary reconstruction feature map. Specifically, SCUNet first converts the input into a bottom-level feature (Bottleneck Feature) through three dual transformer modules (Swin-ConvTransformer Block, SCTB) and image block merging (Patch Merging, PM) operations. The specific process is SCTB, PM, SCTB, PM, SCTB, and PM. Subsequently, the bottom-level feature is converted into a primary reconstruction feature map through three pixel confusion-based upsampling operations (Pixel Shuffle Up-Sampling, PSUS) and dual transformer modules (Swin-Conv Transformer Block, SCTB). The specific process is PSUS, SCTB, PSUS, SCTB, PSUS, and SCTB.

[0058] The dual transformer module SCTB consists of two shift window transformers (Swin Transformer Block, Swin-T) and two convolution transformers (Conv Transformer Block, Conv-T). The specific operation process is Swin-T, Swin-T, Conv-T and Conv-T. Swin-T is committed to extracting the long-range dependencies of the context between image contents, so that the reconstructed image has richer content. Conv-T emphasizes the spatial local context information of the image through convolution operations to obtain clearer image texture.

[0059] The residual enhanced upsampling network REUpNet uses the preset enhanced upsampling module (EUM) to process the primary reconstructed feature map and the input low-resolution image in parallel, and then adds the results of the two at the element level to obtain the restored high-resolution image. The residual enhanced upsampling network fuses the residual information of the input image, which can retain more of the content of the original image, so that the restored high-resolution image has richer texture information.

[0060] The enhanced upsampling module EUM expands the spatial resolution of the input image through two different convolution upsampling operations (one-step upsampling and two-step upsampling) to obtain the final high-resolution image. Its specific operations include convolution layers and upsampling at different scales. The one-step upsampling operation passes the input through a convolution layer, and then directly transforms the input to a high-resolution image of the target resolution through upsampling; the two-step upsampling operation first transforms the image to half of the target resolution through a convolution operation and an upsampling operation, and then obtains the final target resolution through another convolution and upsampling operation. Finally, EUM fuses the results of the two upsampling operations into the final high-resolution image. By fusing different upsampling results, the enhanced upsampling module can make up for the deficiency that a single upsampling method cannot fully restore the image content, and enhance the visual quality of the restored image.

[0061] The preprocessed DCE-MRI images are used as training sets to train the dual transformer-based deep super-resolution model DTSRNet of an embodiment of the present invention to obtain a trained DTSRNet model that can convert input images into output images with higher resolution.

[0062] S3. Apply the trained model to the ADC image

[0063] According to the trained DTSRNet model, by inputting a low-resolution ADC image, a higher-resolution super-resolution image HR-ADC can be obtained.

[0064] The method of the embodiment of the present invention can improve the spatial resolution and visual quality of images (including but not limited to breast cancer tumor medical images).

[0065] The method of a specific embodiment of the present invention is further described below with reference to the accompanying drawings.

[0066] See also Figures 1 to 3 , the method of the specific embodiment comprises the following steps:

[0067] S1. Input LR DCE-MRI image I L , based on the preset U-shaped network, a primary reconstruction feature map I′ is obtained L :

[0068] I′ L =UNet(I L )

[0069] Among them, UNet represents the specific operation of the U-type network, which contains three dual transformer modules (Swin-ConvTransformer Block, SCTB) to convert the input image into the bottom feature (Bottleneck Feature, F B ) and three identical dual transformer blocks to restore the underlying features into a primary reconstructed feature map whose resolution is consistent with the input LR image. The preset dual transformer block SCTB consists of a shift window transformer block (Swin TransformerBlock, Swin-T) and a convolution transformer block (Conv Transformer Block, Conv-T).

[0070] Specifically, the input LR image is gradually extracted into the underlying features F through three SCTB modules and the image block fusion module (Patch Merging, PM). B , the specific operations are as follows:

[0071] F down1 =SCTB1(I L )

[0072] F down2 =SCTB2(PM1(F down1 ))

[0073] F down3 =SCTB3(PM2(F down2 ))

[0074] F B =PM3(F down3 )

[0075] Among them, PM i and SCTBi ,i=1,2,3 represents the i-th image block fusion module and SCTB module.

[0076] Specifically, the underlying features F B The underlying features are gradually converted into a primary reconstruction feature map through three SCTB modules, a pixel shuffle up-sample module (PSUS) and a residual connection. The specific operations are as follows:

[0077] F up1 =SCTB1(PSUS1(F B )+F down3 )

[0078] F up2 =SCTB2(PSUS2(F up1 )+F down2 )

[0079] I′ L =SCTB3(PSUS3(F up2 )+F down1 )

[0080] Among them, PSUS i , i=1,2,3 represents the i-th upsampling module operation based on pixel confusion, and + represents the residual connection.

[0081] like Figure 4 As shown, the preset Swin Transformer Block (Swin-T) includes convolutional layer normalization (Layer Normalization, Layer Norm), multi-head self-attention module (Multi-Head SelfAttention), residual connection (Residual Connection) and multi-layer perceptron (Multi-LayerPerception, MLP) containing 2 layers. In the present invention, the window-based multi-head self-attention module (Window-Based Multi-HeadSelf Attention, W-MSA) and the shifted window-based multi-head self-attention module (Shifted Window-BasedMulti-Head Self Attention, SW-MSA) are respectively used in two consecutive Swin-T modules, which are defined as follows:

[0082]

[0083]

[0084]

[0085]

[0086] Among them, LN represents the convolution layer normalization operation, MLP represents the 2-layer multi-layer perceptron operation, W-MSA and SW-MSA represent the window-based multi-head self-attention module and the shift window-based multi-head self-attention module operation. and z l They represent the outputs of the lth layer (S)W-MSA module and the MLP module respectively.

[0087] like Figure 5 As shown, the specific operation of the preset convolution transformer block (Conv Transformer Block, Conv-T) is as follows:

[0088] (1) Input X and divide it into N blocks of the same size Then use N heads to process. For the nth head, firstly, a convolution layer with a convolution kernel size of 1×1 is used to process the query (Query, Q n ), Key (Key, K n ) and value (Value, V n ):

[0089] Q n =Conv1(LN(F n ))

[0090] K n =Conv1(LN(F n ))

[0091] V n =Conv1(LN(F n ))

[0092] Among them, LN represents the convolution layer normalization operation, and Conv1 represents the convolution layer operation with a convolution kernel size of 1×1.

[0093] (2) Calculate the Self-Attention Map F SA :

[0094]

[0095] Among them, softmax represents the softmax function operation, Represents matrix multiplication.

[0096] (3) After processing by N heads, N self-attention feature maps are obtained, which are then stacked (Concatenation, Concat) together, and passed through a convolution layer (Conv1) with a convolution kernel size of 1×1 and a ReLU activation function (ReLU) to obtain the final output Y:

[0097] SA=Concat(SA1,…,SA n )

[0098] Y = ReLU(Conv1(SA)

[0099] S2. According to the preset residual enhanced upsampling network (REUpNet), the primary reconstruction feature map is converted into a high resolution (HR) DCE-MRI image. REUpNet processes the primary reconstruction feature map I′ through two parallel enhanced upsampling modules (EUM) L And the input LR image I L , and then the results of the two are added element-wise to obtain the restored HR image I H :

[0100] I H =EUM(I′ L )+EUM(I L )

[0101] Among them, EUM represents the preset enhanced upsampling module operation.

[0102] S2-1, the preset enhanced upsampling module EUM performs the input LR image I L Perform a resolution reconstruction operation. The specific operations of EUM are as follows:

[0103] I′ H1 =Upsample ×4 (Conv3(I L ))

[0104] I′ H2 =Upsample ×2 (Conv3(Upsample ×2 (Conv3(I L ))))

[0105] I′ H =I′ H1 +I′ H2

[0106] S2-2, the preset enhanced upsampling module EUM reconstructs the primary feature map I′ of the input L Perform resolution reconstruction operation, the specific operations are as follows:

[0107] I″ H1 =Upsample ×4 (Conv3(I′ L ))

[0108] I″ H2 =Upsample ×2 (Conv3(Upsample ×2 (Conv3(I′ L ))))

[0109] I″ H =I″ H1 +I″ H2

[0110] Among them, Conv3 represents the convolution layer operation with a convolution kernel size of 3×3, Upsample ×4 and Upsample ×2 They represent upsampling operations that increase the feature map resolution by 4 times and 2 times, respectively.

[0111] S2-3, the resolution reconstruction results of the input image and the primary reconstruction feature map are fused to obtain the final high-resolution image:

[0112] I H =I′ H +I″ H .

[0113] Figure 6 The super-resolution results of breast cancer DCE-MRI images obtained by the embodiment of the present invention are shown. Figure 6 In the figure, (a) is the original DCE-MRI image, (b) is the low-resolution DCE-MRI image, and (c) is the super-resolution DCE-MRI image obtained by the method of the embodiment of the present invention. Figure 7 The super-resolution result of the breast cancer ADC image according to the embodiment of the present invention is shown. Figure 7 In the figure, (a) the original ADC image, and (b) the super-resolution ADC rate image obtained by the method of the embodiment of the present invention. Figure 6 and Figure 7 It is shown that the embodiment of the present invention can obtain a clearer high-resolution image from a low-resolution image.

[0114] Compared with the prior art, the present invention has the following advantages:

[0115] The image deep super-resolution method based on dual transformer provided by the present invention improves the visual resolution of DCE-MRI and ADC images through an efficient dual transformer based deep super-resolution network DTSRNet. Through the deep learning network designed by the present invention, the resolution and visual quality of the image are enhanced. In its typical application, the present invention can be used to achieve deep super-resolution of breast cancer images, and the obtained high-resolution images are conducive to higher accuracy in subsequent breast cancer prediction and diagnosis tasks. Specifically, the dual transformer model of the present invention can significantly enhance the resolution and visual quality of the image. The dual transformer model can not only fully extract the spatial local information of the image, but also explore the global structural information of the image, so that the restored high-resolution image has more texture and complete structure. In particular, considering that the upsampling process cannot fully restore the detailed information of the image, the present invention utilizes the residual information of the input image and fuses the results from two different upsampling through the enhanced upsampling module to obtain a clearer high-resolution image.

[0116] The preset DTSRNet network can be trained using DCE-MRI images. The resulting DTSRNet model can improve the resolution and visual quality of DCE-MRI images and diffusion-weighted imaging (DWI)-derived apparent diffusion coefficient (ADC) images. It can not only be flexibly used in image super-resolution tasks (such as breast cancer medical images), but can also be used as a preprocessing technique to improve the accuracy of disease diagnosis and prediction tasks based on medical images (such as breast cancer).

[0117] In addition, the method proposed in the present invention can not only enhance the visual resolution of DCE-MRI and ADC images, but also cope with other complex application scenarios with higher robustness. The present invention can be further applied to other tasks such as segmentation, recognition and detection as breast cancer medical image processing software and system.

[0118] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present invention may take the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] The background section of the present invention may contain background information about the problem or environment of the present invention, but does not necessarily describe the prior art. Therefore, the content included in the background section is not an admission by the applicant that the prior art is available.

[0123] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, it can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention. In the description of this specification, the description of the reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In the absence of mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A dual transformer-based image depth super-resolution method, characterized in that: The steps include: Use the pre-trained dual transformer-based super-resolution network DTSRNet to convert low-resolution DCE-MRI images or apparent diffusion coefficient ADC images into high-resolution images; The super-resolution network DTSRNet includes a dual transformer-based U-type network SCUNet and a residual enhanced upsampling network REUpNet; The SCUNet uses a dual transformer module SCTB and an image block merging operation PM to gradually downsample the input low-resolution image into a bottom-level feature map, and then uses SCTB and a pixel confusion-based upsampling operation PSUS to gradually upsample the bottom-level feature map to obtain a primary reconstructed feature map; The REUpNet utilizes a parallel enhanced upsampling module EUM to upsample the input low-resolution image and the primary reconstructed feature map respectively, and then fuses the results of the two to obtain the high-resolution image.

2. The image depth super-resolution method based on dual transformer according to claim 1, characterized in that: The SCUNet uses a dual transformer module SCTB and an image block merging operation PM to gradually downsample the input low-resolution image to an underlying feature map, specifically including: SCUNet uses three dual transformer modules SCTB and three image block merging operations PM, namely SCTB, PM, SCTB, PM, SCTB and PM, thereby gradually downsampling the input low-resolution image to the underlying feature map.

3. The image depth super-resolution method based on dual transformer according to claim 1 or 2, characterized in that: The method uses SCTB and a pixel confusion-based upsampling operation PSUS to gradually upsample the underlying feature map to obtain a primary reconstructed feature map, specifically including: SCUNet performs three pixel confusion-based upsampling operations PSUS and three dual transformer modules SCTB, namely PSUS, SCTB, PSUS, SCTB, PSUS and SCTB, and performs a residual connection before processing through each SCTB, thereby gradually upsampling the underlying feature map to obtain the primary reconstructed feature map.

4. The image depth super-resolution method based on dual transformer according to any one of claims 1 to 3, characterized in that: The dual transformer module SCTB includes two shift window transformers Swin-T and two convolution transformers Conv-T, namely Swin-T, Swin-T, Conv-T and Conv-T, wherein the Swin-T extracts the long-range dependencies of the context between the image contents to enrich the content of the reconstructed image, and the Conv-T highlights the spatial local context information of the image through convolution operations to obtain clearer image texture.

5. The image depth super-resolution method based on dual transformer according to claim 4, characterized in that: The Swin-T includes convolutional layer normalization, multi-head self-attention module MSA, residual connection and 2-layer multi-layer perceptron MLP, wherein the window-based multi-head self-attention module W-MSA and the shifted window-based multi-head self-attention module SW-MSA are respectively used in two consecutive Swin-T modules, as expressed as follows: Among them, LN represents the convolution layer normalization operation, MLP represents the 2-layer multi-layer perceptron operation, W-MSA and SW-MSA represent the window-based multi-head self-attention module and the shift window-based multi-head self-attention MSA module operation, and z l They represent the outputs of the MSA module and the MLP module at the lth layer respectively.

6. The image depth super-resolution method based on dual transformer according to claim 4, characterized in that: The specific operation of Conv-T is as follows: (1) Divide the input into N blocks of equal size Then, the N heads are used for processing respectively. For the nth head, a convolution layer with a convolution kernel size of 1×1 is used to process the query (Query, Q n ), key (Key, K n ) and value (Value, V n ): Q n =Conv1(LN(F n )) K n =Conv1(LN(F n )) V n =Conv1(LN(F n )) Among them, LN represents the convolution layer normalization operation, and Conv1 represents the convolution layer operation with a convolution kernel size of 1×1; (2) Calculate the self-attention feature map F SA : Among them, softmax represents the softmax function operation, Represents matrix multiplication; (3) After processing by N heads, N self-attention feature maps are obtained, which are then stacked and concat together. After passing through a convolution layer Conv1 with a convolution kernel size of 1×1 and a ReLU activation function, the output Y is obtained: SA=Concat(SA1,...,SA n ) Y = ReLU(Conv1(SA).

7. The image depth super-resolution method based on dual transformer according to any one of claims 1 to 6, characterized in that: The REUpNet performs an element-wise addition operation on the results of the parallel processing of the primary reconstructed feature map and the input low-resolution image, fuses the residual information of the input image, and obtains a restored high-resolution image.

8. The image depth super-resolution method based on dual transformer according to any one of claims 1 to 7, characterized in that: The parallel enhanced upsampling module EUM is used to upsample the input low-resolution image and the primary reconstructed feature map respectively, specifically including one-step upsampling and two-step upsampling. The one-step upsampling operation passes the low-resolution image through a convolution layer, and then directly transforms the input to a high-resolution image of the target resolution through upsampling; the two-step upsampling operation first transforms the primary reconstructed feature map through a convolution operation and an upsampling operation to half the target resolution, and then undergoes another convolution operation and upsampling operation to obtain the final target resolution.

9. The image depth super-resolution method based on dual transformer according to any one of claims 1 to 8, characterized in that: The pre-training includes training a super-resolution network DTSRNet using pre-processed DCE-MRI images.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image depth super-resolution method based on dual transformer is implemented.

Citation Information

Cited By

  • Depth map super-resolution reconstruction method based on diffusion model

    CN119887529A