Three-dimensional super-resolution method suitable for CT image sequence

Abstract

The invention provides a three-dimensional super-resolution method suitable for a CT image sequence, and relates to the technical field of medical images. According to the method, image high-frequency information is extracted more accurately through gradual combination of multi-scale deep convolution and an attention mechanism, a medical CT image interlayer super-resolution network which is iteratively refined from coarse to fine is designed, and a cross section slice of a high-resolution middle frame is synthesized by using two continuous slices at spatial positions. The method comprises the following steps of: generating a multi-scale context feature by using a feature pyramid structure, gradually fusing a two-way deformation estimation graph through an IIR (Infinite Impulse Response) module by combining a global two-way distortion function (GBDF) with two-way optical flow information and the multi-scale context feature, complementing detail information for an intermediate frame by combining an STIFOF (Spatial-Temporal Interval-Frame Optimization Function) function to realize motion compensation, and finally obtaining a high-resolution intermediate frame prediction graph.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a three-dimensional super-resolution method suitable for CT image sequences. Background Art

[0002] Computed tomography (CT) is an indispensable medical imaging technology in medical diagnosis. CT three-dimensional imaging technology reconstructs 3D images by stacking sequential two-dimensional CT plane images, which can make the relationship between tissue structures clearer and facilitate the accurate positioning and qualitative diagnosis of lesions. In order to obtain continuous and clear three-dimensional reconstruction effects, the spatial resolution of a single CT image and the inter-layer resolution of adjacent CT images are required to be high, which is bound to increase the CT radiation dose and equipment costs, and increase damage to the human body. In order to take into account the diagnostic requirements and reduce the harm to the patient's body, doctors usually use a relatively low-resolution CT scanning method, which can easily lead to problems such as high noise artifacts and surface discontinuity in the three-dimensional CT imaging model, affecting the accuracy of diagnosis. The three-dimensional CT image of the organ reconstructed based on low spatial resolution and inter-layer resolution has a step-like surface and blurred edges of vascular tissue.

[0003] At present, there has been a lot of work on intra-layer image super-resolution reconstruction, which has improved the quality of single-layer CT images without increasing the acquisition equipment and radiation costs. However, there are relatively few studies on interpolation to improve inter-layer resolution. Existing work mainly focuses on independent intra-layer or inter-layer super-resolution reconstruction, and there are still few reports on combining the two tasks to improve the overall quality of the 3D CT model. In 2024, I 3 -Net attempts to use inter-layer interpolation and intra-layer super-resolution as the upper and lower branches of the convolutional neural network to simultaneously process the CT image sequence, achieving a preliminary fusion of intra-layer super-resolution and inter-layer interpolation. Although this method implements the two tasks in the same network, the performance evaluation index of a single task still only stays on the two-dimensional spatial clarity of the CT image sequence, and does not explore its impact on the three-dimensional reconstruction effect. The inherent logical relationship between the two tasks of inter-layer interpolation and intra-layer super-resolution algorithms and the impact of their sequence on the three-dimensional reconstruction effect are still unclear. Therefore, it is still necessary to further explore the organic fusion method of inter-layer super-resolution and intra-layer interpolation algorithms. To this end, we propose a medical CT image three-dimensional reconstruction network based on intra-layer super-resolution and inter-layer interpolation. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a three-dimensional super-resolution method suitable for CT image sequences, which effectively solves the problem of discontinuity and unclearness of the three-dimensional volume data surface, and reconstructs a high-quality three-dimensional volume data model from a CT sequence with large layer thickness and layer spacing.

[0005] To achieve the above object, the present invention adopts the following technical solution: a three-dimensional super-resolution method applicable to CT image sequences, comprising the following steps:

[0006] S1. Extract continuous CT cross-sectional slice images, select high-resolution CT cross-sectional slice images with low slice thickness and small inter-slice spacing to construct a training set, divide the CT cross-sectional slice images with large slice thickness into a test set and a validation set, construct high-low resolution CT cross-sectional slice image pairs by bicubic interpolation, divide continuous CT cross-sectional slice images into triplets in the inter-slice interpolation network, and use the intermediate frame images as real images to compare with the output images;

[0007] S2, constructing an intra-layer and inter-layer 3D fusion super-resolution network, which is specifically divided into an intra-layer super-resolution network structure and an inter-layer super-resolution network structure;

[0008] S3, training the intra-layer and inter-layer 3D fusion super-resolution network model constructed in S2;

[0009] S4. Input the test set into the intra-layer and inter-layer 3D fusion super-resolution network model trained in S3, and output a dense sequence of high-resolution CT cross-sectional slice images for 3D reconstruction.

[0010] In a preferred embodiment: S1 uses a bicubic interpolation algorithm to perform high-resolution CT cross-sectional slice image processing. Processing to obtain the corresponding low-resolution image

[0011] In a preferred embodiment, the step of constructing an intra-layer and inter-layer 3D fusion super-resolution network in S2 comprises the following steps:

[0012] S2.1, improve the spatial resolution of each cross-sectional slice by intra-slice super-resolution algorithm;

[0013] S2.2, interpolating two adjacent slices through an interlayer interpolation algorithm to generate a slice at an intermediate position with high resolution, thereby obtaining a 2N-1 slice sequence with dense interlayer spacing;

[0014] The S2 is obtained based on the following formula:

[0015]

[0016] Where x and y represent the horizontal and vertical coordinates of the pixel point, and N represents the number of slices in a set of CT image sequences. represents the generated low-resolution cross-sectional image, f SR and f inter They represent the intra-layer super-resolution algorithm and the inter-layer super-resolution algorithm respectively. represents the generated high-resolution cross-sectional image, z represents the number of layers of the high-resolution cross-sectional image, represents the high-resolution cross-sectional image of layer z+0.5, Represents the high-resolution cross-sectional image of layer z+1.

[0017] In a preferred embodiment, constructing the intra-layer super-resolution network structure in S2.1 comprises the following steps:

[0018] S2.1.1. Feature extraction, low-resolution images Extract features through shallow channel reconstruction network;

[0019] S2.1.2, alignment and deep fusion, progressive alignment and deep fusion of multi-scale features are performed through multi-scale attention and serial progressive fusion modules;

[0020] S2.1.3. Construct a super-resolution image by fusing the channel reconstruction information based on ESPCN and the upsampled image based on the original image.

[0021] In a preferred embodiment: the multi-scale attention module in the S2.1.2 alignment and deep fusion aggregates local information through strip convolution, that is, each branch uses two deep strip convolutions to replace the standard deep convolution operation, and then uses the subsequent 1×1 convolution to model the relationship between multi-scale features, and uses the small-core spatial attention mechanism S-ESA to improve the effective utilization of features.

[0022] In a preferred embodiment: the alignment and deep fusion in S2.1.2 specifically designs a multi-scale attention module to fully refine effective information, and progressively aligns local detail features and global semantic features through a serial 1×1 convolution structure to fully aggregate multi-scale feature information.

[0023] In a preferred embodiment: the inter-layer super-resolution network combines context feature extraction, optical flow estimation, and inter-frame synthesis modules into a nested U-shaped structure, and implements repeated iteration of feature information through the inter-frame iterative optimization module IIR of the nested U-shaped structure. A spatial channel attention mechanism module SCA is designed in the inter-frame iterative optimization module IIR, and the spatial receptive field is gradually expanded through 3×3 convolution, and the number of channels is gradually increased to better extract the correlation of inter-frame feature information.

[0024] In a preferred embodiment: the inter-layer super-resolution network designs an inter-frame optimization function based on spatial information, combines the refined bidirectional optical flow information and context features, and generates the final interpolated image through the following formula:

[0025]

[0026] Where w(·) represents the global bidirectional warping function;

[0027] I z ,I z+1 is the input frame;

[0028] Indicates that the direction of the optical flow information of layer 0 is from frame z to frame z+0.5, Indicates that the direction of the optical flow information of layer 0 is from frame z+0.5 to frame z+1.

[0029] and It is a unidirectional intermediate frame image generated by a global bidirectional warping function;

[0030] M0 and M1 are spatial masks with values ​​ranging from 0 to 1 generated by the output feature D0 after Sigmoid function. The spatial masks M0 and M1 are respectively Multiply to obtain a bidirectional intermediate frame image;

[0031] R is the residual feature image extracted by D0 to compensate for texture details;

[0032] I out Represents the output intermediate frame image.

[0033] In a preferred embodiment: the inter-layer super-resolution network also includes a global bidirectional warping function, which combines the forward and backward warping functions and considers the mapping relationship of global source pixels in the bidirectional warping function, wherein the forward warping function calculation model is as follows:

[0034] u=I z+0.5 (p)-(I z (q)+F z→z+0.5 (q))

[0035] b(u)=max(0,1-|u x |)·max(0,1-|u y |)

[0036]

[0037] Where q is the source pixel coordinate;

[0038] I z+0.5 (p) is the estimated value of the backward function;

[0039] u is the difference between the predicted value and the estimated value based on the backward function;

[0040] u x Indicates the offset value of u in the horizontal direction;

[0041] uy Indicates the offset value of u in the ordinate direction;

[0042] b(u) is the weight coefficient;

[0043] is the target pixel of the reconstructed intermediate frame;

[0044] I z (q) is the original frame;

[0045] F z→z+0.5 The optical flow information in the direction from the zth frame to the z+0.5th frame;

[0046] exp represents the Softmax operation on the weight K, and the weight K of each pixel is generated by the intermediate frame based on Average splatting and the original frame I z The Charbonnier Loss between the values ​​of corresponding pixels in (q) is calculated.

[0047] In a preferred embodiment: the specific process of step S3 is:

[0048] Step S3.1: First, define the loss function by using the mean absolute error function as the loss function;

[0049] Step S3.2: train two models, namely the intra-layer super-resolution network and the inter-layer interpolation network; the above algorithm is based on the Windows 10 operating system, using the PyTorch deep learning framework; the hardware configuration is the CPU internal (R) core (TM) i7-6500u @ 2.50 GHz, 8 GB memory, the GPU is NVIDIA GTX 1080, CUDA 11.4 + cudnn 11.3 for GPU acceleration; the Adam optimization algorithm is used, β1 is set to 0.9, β2 is set to 0.999, the momentum parameter is set to 0.9, the batch size is 16, and the iteration is 300 epochs; the learning rate is initially set to 1×10 -4 , and gradually decays to 1×10 according to the cosine function -5 .

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The present invention progressively combines multi-scale deep convolution and attention mechanism to more accurately extract high-frequency information of images, designs an inter-layer super-resolution network for medical CT images that iteratively refines from coarse to fine, synthesizes cross-sectional slices of high-resolution intermediate frames using two consecutive slices in spatial position, generates multi-scale contextual features with a feature pyramid structure, and uses a global bidirectional warping function GBDF to combine bidirectional optical flow information and multi-scale contextual features. The bidirectional deformation estimation map is gradually fused through the IIR module, and the STIFOF function is combined to complete the detail information of the intermediate frame to realize motion compensation, and finally a high-resolution intermediate frame prediction map is obtained.

[0052] 2. The images predicted by the algorithm of the present invention have higher objective index evaluation and better visual perception effect, which verifies the influence of the order of intra-layer super-resolution reconstruction network and inter-layer interpolation reconstruction network on three-dimensional reconstruction. A series of experiments have proved that only by first performing intra-layer super-resolution reconstruction network and then inter-layer interpolation reconstruction network can the inter-layer resolution be improved and the problem of discontinuity and unclearness of the three-dimensional volume data surface can be effectively solved, and a high-quality three-dimensional volume data model can be reconstructed from a CT sequence with large layer thickness and inter-layer spacing. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a work flow chart of a preferred embodiment of the present invention;

[0054] Figure 2 It is an overall flow chart of the intra-layer-inter-layer fusion three-dimensional super-resolution reconstruction method IIF3SR-Net of the preferred embodiment of the present invention;

[0055] Figure 3 It is a diagram of the intra-layer super-resolution network structure of a preferred embodiment of the present invention;

[0056] Figure 4 is a structural diagram of the MSAFE in a preferred embodiment of the present invention;

[0057] Figure 5 It is the Inter-Net network structure diagram;

[0058] Figure 6 This is the IIR module structure diagram;

[0059] Figure 7 The visual results and PNSR images of the reconstructed images upsampled by 2 and 4 times;

[0060] Figure 8 This is a visual comparison of the reconstructed images upsampled by 2 and 4 times;

[0061] Fig. 9 It is a visual comparison diagram of the original CT image and the 3D reconstruction after intra-layer super-resolution. DETAILED DESCRIPTION

[0062] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0063] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0064] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0065] See also Figure 1 - Fig. 9 The present invention provides a three-dimensional super-resolution method applicable to CT image sequences, comprising the following steps:

[0066] S1. Extract continuous CT cross-sectional slice images, select high-resolution CT images with low CT image layer thickness and small inter-slice spacing to construct the training set, and divide the CT images with larger layer thickness into the test set and the validation set. Use the bicubic interpolation method to construct high-low resolution CT image pairs, divide the continuous CT cross-sectional images into triplets in the inter-slice interpolation network, and use the intermediate frames as the real images to compare with the output images.

[0067] S2. In order to improve the planar and inter-layer resolution of CT image sequences, an IIF3SR-Net network is constructed, which is specifically divided into two modules: the Intra-Net intra-layer super-resolution network structure and the Inter-Net inter-layer super-resolution network structure;

[0068] S3, training the network model constructed in step 2;

[0069] S4. Input the test set into the network model trained in step 3, and output a dense sequence of high-resolution CT images for three-dimensional reconstruction.

[0070] In this embodiment, preferably, constructing the IIF3SR-Net network in S2 includes the following steps:

[0071] S2.1. Improve the spatial resolution of each cross-sectional slice through the intra-layer super-resolution algorithm Intra-Net;

[0072] S2.2. Inter-Net is used to interpolate two adjacent slices to generate a slice at an intermediate position with high resolution, thereby obtaining a 2N-1 slice sequence with dense layer spacing.

[0073] In this embodiment, preferably, S2 is obtained according to the following formula:

[0074]

[0075] In the formula, A set of cross-sectional slices are shown.

[0076] f SR and f inter They represent the intra-layer super-resolution algorithm and the inter-layer super-resolution algorithm respectively.

[0077] represents the generated high-resolution cross-sectional image,

[0078] Represents the z-th layer high-resolution image.

[0079] In this embodiment, preferably, constructing the Intra-Net layer super-resolution network structure in S2.1 includes the following steps:

[0080] S2.1.1. Feature extraction, low-resolution images Extract features through shallow channel reconstruction network;

[0081] S2.1.2, alignment and deep fusion, progressive alignment and deep fusion of multi-scale features are performed through multi-scale attention and serial progressive fusion modules;

[0082] S2.1.3. Construct a super-resolution image by fusing the channel reconstruction information based on ESPCN and the upsampled image based on the original image.

[0083] In this embodiment, preferably, the multi-scale attention module in S2.1.2 alignment and deep fusion aggregates local information through strip convolution, that is, each branch uses two deep strip convolutions to replace the standard deep convolution operation, such as a 7×7 two-dimensional convolution is replaced by a pair of 7×1 and 1×7 strip convolutions, and then the relationship between multi-scale features is modeled through the subsequent 1×1 convolution, and the small-core spatial attention mechanism S-ESA is used to improve the effective utilization of features.

[0084] In this embodiment, preferably, the alignment and deep fusion in S2.1.2 are designed, a multi-scale attention module is designed to fully refine the effective information, and a cascaded 1×1 convolution structure is used to progressively align local detail features and global semantic features to fully aggregate multi-scale feature information.

[0085] In this embodiment, preferably, the Inter-Net in S2.2 designs an inter-frame optimization function based on spatial information, combines the refined bidirectional optical flow information and context features, and generates the final interpolated image through the following formula:

[0086]

[0087] Where w represents the global bidirectional distortion function;

[0088] I z ,I z+1 is the input frame;

[0089] Indicates that the direction of the optical flow information of layer 0 is from frame z to frame z+0.5, Indicates that the direction of the optical flow information of layer 0 is from frame z+0.5 to frame z+1.

[0090] and It is a unidirectional intermediate frame image generated by a global bidirectional warping function;

[0091] M0 and M1 are spatial masks with values ​​ranging from 0 to 1 generated by the output feature D0 after Sigmoid function. The spatial masks M0 and M1 are respectively Multiply to obtain a bidirectional intermediate frame image;

[0092] R is the residual feature image extracted by D0 that can compensate for texture details;

[0093] I out Represents the output intermediate frame image

[0094] In this embodiment, preferably, the Inter-Net in S2.2 also includes a global bidirectional warping function GBDF, which combines the forward and backward warping functions and considers the mapping relationship of global source pixels in the bidirectional warping function, wherein the forward warping function calculation model is as follows:

[0095] u=I z+0.5 (p)-(I z (q)+F z→z+0.5 (q))

[0096] b(u)=max(0,1-|u x |)·max(0,1-|u y |)

[0097]

[0098] Where q is the source pixel coordinate;

[0099] Iz+0.5 (p) is the estimated value of the backward function;

[0100] u is the difference between the predicted value and the estimated value based on the backward function;

[0101] u x Indicates the offset value of u in the horizontal direction;

[0102] u y Indicates the offset value of u in the ordinate direction;

[0103] b(u) is the weight coefficient;

[0104] is the target pixel of the reconstructed intermediate frame;

[0105] I z (q) is the original frame;

[0106] F z→z+0.5 The optical flow information from the zth frame to the z+0.5th frame;

[0107] exp represents the Softmax operation on the weight K, and the weight K of each pixel is generated by the intermediate frame based on Average splatting and the original frame I z The Charbonnier Loss between the values ​​of corresponding pixels (q) is calculated.

[0108] The working principle and use process of the present invention:

[0109] S1. Extract the patient's continuous cross-sectional CT slice images, select high-resolution CT images with low CT image thickness and small inter-slice spacing to construct the training set, and divide the CT images with larger slice thickness into the test set and the validation set. Use the bicubic interpolation method to construct high-low resolution CT image pairs, divide the continuous CT images into triplets in the inter-slice interpolation network, and use the intermediate frames as the real images to compare with the output images.

[0110] S2, build the IIF3SR-Net network, including the following steps:

[0111] S2.1. Improve the spatial resolution of each cross-sectional slice through the intra-layer super-resolution algorithm Intra-Net;

[0112] S2.2. Inter-Net is used to interpolate two adjacent slices to generate a slice at an intermediate position with high resolution, thereby obtaining a sequence of 2N-1 slices with dense layer spacing, which is obtained according to the following formula:

[0113]

[0114] In the formula, A set of cross-sectional slices are shown.

[0115] f SR and f inter They represent the intra-layer super-resolution algorithm and the inter-layer super-resolution algorithm respectively.

[0116] represents the generated high-resolution cross-sectional image,

[0117] Represents the z-th layer high-resolution image.

[0118] S2.1. Constructing the intra-Net layer super-resolution network structure, including the following steps:

[0119] S2.1.1. Feature extraction, low-resolution images Extract features through shallow channel reconstruction network;

[0120] S2.1.2, alignment and deep fusion, progressive alignment and deep fusion of multi-scale features are performed through multi-scale attention and serial progressive fusion modules;

[0121] S2.1.3. Construct a super-resolution image by fusing the channel reconstruction information based on ESPCN and the upsampled image based on the original image.

[0122] Multi-scale attention and serial progressive fusion modules are used to fully refine effective information, and the serial 1×1 convolution structure is used to progressively align local detail features and global semantic features to fully aggregate multi-scale feature information.

[0123] In order to better extract the strip tissue in CT images, the multi-scale module in MSAFE aggregates local information through strip convolution, that is, each branch uses two deep strip convolutions to replace the standard deep convolution operation, such as a 7×7 two-dimensional convolution is replaced by a pair of 7×1 and 1×7 strip convolutions, and then the relationship between multi-scale features is modeled by the following 1×1 convolution. Subsequently, the small kernel spatial attention mechanism S-ESA is used to improve the effective use of features, especially to encourage the network to pay full attention to key pathological detail features. Unlike the ESA spatial attention mechanism, S-ESA simplifies redundant convolution groups into residual modules of small kernel convolution to improve the efficiency of extracting local texture detail features. At the same time, S-ESA uses 1×1 convolution layers to reduce the number of channels in the initial stage and restore the number of channels in the feature extraction stage. While reducing the amount of calculation, it can more effectively fuse the effective feature information of CT images and filter artifact noise. In addition, in order to detect spatial information of different scales, Intra-Net still uses the bilinear interpolation method of the large window pooling layer for up and down sampling, which can greatly improve the ability to extract detailed features.

[0124] S2.2. Construct an Inter-Net interlayer super-resolution network structure; Inter-Net combines context feature extraction, optical flow estimation, and inter-frame synthesis modules into a nested U-shaped structure. The IIR module of the nested U-shaped structure realizes repeated iteration of feature information, and a spatial channel attention mechanism SCA module is designed in the IIR. The spatial receptive field is gradually expanded through 3×3 convolutions, and the number of channels is gradually increased to better extract the correlation of inter-frame feature information. Finally, the channel weight coefficient is multiplied by the input feature to enable the network to learn more critical motion estimation relationships. The left downsampling branch uses multiple 3×3 convolutions with a stride of 2, and the right upsampling branch consists of two 4×4 transposed convolutions with a stride of 2 to extract multi-scale features. Each convolution layer is followed by a PRelu activation function to prevent overfitting. Finally, adding a global bidirectional warp function to the residual connection can also promote further joint refinement and propagation of feature information.

[0125] Inter-Net designs an inter-frame optimization function based on spatial information, combines the refined bidirectional optical flow information and context features, and generates the final interpolated image through the following formula:

[0126]

[0127] Where w represents the global bidirectional distortion function;

[0128] I z ,I z+1 is the input frame;

[0129] and It is a unidirectional intermediate frame image generated by a global bidirectional warping function;

[0130] M0 and M1 are spatial masks with values ​​ranging from 0 to 1 generated by the output feature D0 after Sigmoid function. The spatial masks M0 and M1 are respectively Multiply to obtain a bidirectional intermediate frame image;

[0131] R is the residual feature image extracted by D0 that can compensate for texture details.

[0132] Inter-Net also includes a global bidirectional warping function GBDF, which combines the forward and backward warping functions and considers the mapping relationship of global source pixels in the bidirectional warping function. The forward warping function calculation model is as follows:

[0133] u=I z+0.5 (p)-(I z (q)+F z→z+0.5 (q))

[0134] b(u)=max(0,1-|u x |)·max(0,1-|u y |)

[0135]

[0136] Where q is the source pixel coordinate;

[0137] I z+0.5 (p) is the estimated value of the backward function;

[0138] u is the difference between the predicted value and the estimated value based on the backward function;

[0139] u x Indicates the offset value of u in the horizontal direction;

[0140] u y Indicates the offset value of u in the ordinate direction;

[0141] b(u) is the weight coefficient;

[0142] is the target pixel of the reconstructed intermediate frame;

[0143] I z (q) is the original frame;

[0144] F z→z+0.5 It is the optical flow information from the zth frame to the z+0.5th frame;

[0145] exp represents the Softmax operation on the weight K, and the weight K of each pixel is generated by the intermediate frame based on Average splatting and the original frame I z The Charbonnier Loss between the values ​​of the corresponding pixels (q) is calculated, and the specific calculation process is as follows:

[0146]

[0147] As shown in the following formula (1), the input frame is first distorted by Average splatting to pre-generate an intermediate frame image On this basis, using Charbonnier Loss (Its definition is shown in the following formula (2), ε is 10 -3 ) to calculate the weights. Compared with the L1 function, Charbonnier Loss can better distinguish the effects between different pixels, so it is beneficial to optimize the local lesion details. Specifically, the weight K of each pixel is the original image I z (q), and the pre-generated intermediate frame The Charbonnier Loss between , the specific calculation process is shown in formula (3). Here we use the pre-generated intermediate frame Rather than directly adopting I z+1 The calculation is performed on the frame to reduce the gap between two frames, which helps to obtain a more accurate weight coefficient K.

[0148]

[0149] The downsampling branch Pyramid Encoder on the left extracts multi-scale optical flow information and contextual feature information of the image, and combines the optical flow information in the Coarse-to-Fine Frame Synthesis part and Pyramid Features The global bidirectional warping function (GBDF) is used to preliminarily filter artifact noise information and synthesize a rough intermediate frame image. The IIR module in the Coarse-to-Fine Frame Syntheis part aligns and refines the multi-scale bidirectional optical flow information and context features. This coarse-to-fine strategy facilitates the combination of context information and optical flow information at each scale, and more fully aligns global information and local details. Finally, the texture details are supplemented by the post-processing optimization function STIFOF based on spatial information to generate the final high-resolution intermediate frame image. The left half downsampling branch Pyramid Encoder is a feature pyramid structure to extract context information of different scales in each input frame. There are a total of 4 convolution stages: stage0, stage1, stage2 and stage3. Each stage consists of two 3×3 two-dimensional convolutions with strides of 2 and 1 respectively. The first convolution is responsible for downsampling the input features and the second convolution is responsible for extracting rich scale feature information. Each pyramid layer is followed by a PRelu activation function. The number of channels that gradually enrich features as the feature space size decreases is 32, 48, 72 and 96, and each stage corresponds to frame I. z and I z+1 The k-level pyramid features Among them, k∈[0,1,2,3], in the upsampling branch, the inter-frame analysis layer IIR_k+1 combines the bidirectional optical flow field Preliminary synthesis of intermediate frame images and the intermediate feature D output by IIR_k k , reconstructing higher-level intermediate features D with richer details k+1 , as shown in the following formula, the final progressive alignment and fusion is achieved.

[0150]

[0151] The network architecture progressively integrates bidirectional optical flow fields of different scales, pyramid encoder features, global bidirectional distortion compensation, and features reconstructed at the previous scale, which can fully reconstruct missing reference information and achieve more accurate motion estimation. This progressive reconstruction method is conducive to aligning the input feature information from coarse to fine and improving the quality of intermediate feature reconstruction. Therefore, the upsampling branch designed in this paper can make the reconstructed intermediate features and bidirectional optical flow fields gradually and collaboratively refine, generate a fusion mask and compensate for motion detail information, and finally generate a high-resolution intermediate frame. In summary, the inter-frame analysis layer IIR_k is expressed by the above formula, where D k represents the features extracted by the decoder, and [·] represents the Concat operation.

[0152] S3: Train the network model constructed in step 2

[0153] Step S3.1: First, define the loss function. This method uses the mean absolute error function as the loss function.

[0154] Step S3.2: The present invention trains two models, namely Intra-Net and Inter-Net. The above algorithm is based on the Windows 10 operating system and uses the PyTorch deep learning framework. The hardware configuration is a CPU internal (R) core (TM) i7-6500u @ 2.50 ghz, 8g memory, a GPU is NVIDIA GTX 1080, and CUDA 11.4 + cudnn 11.3 is used for GPU acceleration. Using the Adam optimization algorithm, β1 is set to 0.9, β2 is set to 0.999, the momentum parameter is set to 0.9, the batch size is 16, and iterates for 300 epochs. The learning rate is initially set to 1×10 -4 , and gradually decays to 1×10 according to the cosine function -5 .

[0155] S4. Input the test set into the network model trained in step 3, and output a dense sequence of high-resolution CT images for three-dimensional reconstruction.

[0156] In order to better illustrate the performance of IIF3SR-Net, this paper describes the performance comparison experiment of the intra-layer super-resolution reconstruction network Intra-Net, the inter-layer interpolation reconstruction network Inter-Net, and their sequence; the experiment is based on two datasets, 3D-IRCADb and COVID-19. Among them, the 3D-IRCADb dataset contains CT scan image sequences of 10 female and 10 male liver tumor patients. This paper preprocessed it accordingly, extracted the CT cross-sectional slice images of each patient, and selected high-resolution CT images with low layer thickness and small inter-layer spacing. The training set was constructed by 100 sets of LR-HR image pairs, and 737 sets of LR-HR image pairs were finally obtained as the training set, while the remaining 110 sets of high-low resolution CT images with thicker slices were divided into two parts, as the test set and the validation set, respectively. The resolution of each HR image is 512×512 pixels. The COVID-19 dataset is 20 sets of lung CT datasets collected by IEEE8023 for the study of the new coronavirus (COVID-19). The intra-layer resolution of the dataset ranges from 0.6836 to 0.7246 mm, and the inter-layer resolution is 1 mm. After preprocessing, the training set consists of 965 sets of LR-HR CT images, and the test and validation set contains a total of 146 sets of images. In order to meet the training and performance testing of the interpolation network, the continuous tomographic slices of the real thin-layer CT are used as high-resolution CT images during the experiment and divided into triples {I z , I z+0.5 , I z+1}. Among them, I z with I z+1 As the input graph, I out The high-resolution interpolated images generated for the network are obtained by combining with I z+0.5 Real intermediate slices are used for comparative evaluation to verify the model performance. This experiment uses the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), which are widely used in SR quality evaluation of natural images and medical images, as performance indicators to evaluate the performance of CT super-resolution reconstruction and interpolation network models.

[0157] The performance comparison experiments of Intra-Net and Inter-Net networks were conducted based on 3D-IRCADb and COVID-CT datasets respectively; First, this paper compares the performance of 2× and 4× super-resolution reconstruction by using the small model and lagre model of Intra-Net with advanced natural image super-resolution algorithms EDSR, RCAN, RFANet, MFEN, SwinIR, RFLN and existing medical image super-resolution algorithms FAWDN, RDAN, MSAB, MWSR, DRIDSR. The difference between the small and large models lies in the number of deep strip convolution channels. Experiments have shown that more channels help to extract richer multi-scale features. PSNR and SSIM indexes are important performance indicators for reconstruction, as shown in Table 1 and Figure 7 As shown in Figure 2, Intra-Net has higher PSNR and SSIM indicators, which indicates that the quality of SR results is better than other methods and closer to HR images. Compared with all the above methods, the Intra-Net of this paper outperforms the above methods in two CT image datasets and scaling factors of 2× and 4×.

[0158] This paper shows the intuitive visual results of the comparison with other super-resolution algorithms at scaling factors of 2× and 4×. Figure 8 As shown in the figure, because Intra-Net uses multi-scale strip convolution, the images reconstructed by the Intra-Net algorithm are clearer at the edge of the lungs and small lung nodules than other algorithms, the image contrast is more obvious, and it is easier to distinguish different small nodule tissues.

[0159] This article provides a comparison of the effects of 3D reconstruction using the Intra-Net model after super-resolution and 3D reconstruction using the original CT image. Fig. 9 As shown. (a)(c)(d) are the details of the 3D reconstruction of the original CT image, and (b)(e)(f) are the details of the 3D reconstruction after the intra-layer super-resolution reconstruction algorithm. (e) The deep strip convolution of the intra-layer super-resolution reconstruction algorithm extracts more strip-shaped vascular tissue features, and the reconstructed image is clearer than (c) and eliminates many local artifacts and noise. (d) The MSA-PF stage of the intra-layer super-resolution reconstruction algorithm integrates more tissue features and achieves clear visualization of more tiny texture details than (f).

[0160] Table 1 Performance comparison of Intra-Net model and other methods on 3D-IRCADb and COVID-CT datasets

[0161]

[0162]

[0163] This paper selects 7 algorithms that have achieved good results in recent years, including Softmax spaltting, FLAVR, RIFE, CAIN, VFIformer, IFRnet and DQBC. VFIformer is a transformer-based interpolation algorithm, FLAVR, CAIN and DQBC are convolution-based interpolation algorithms, and Softmax spaltting, RIFE and IFRnet are interpolation algorithms based on optical flow information. The quantitative evaluation results of the above algorithms on this dataset are shown in Table 2, where red represents the best results and green represents the suboptimal results.

[0164] Table 2 Comparison of indicators of various algorithms on two test sets

[0165]

[0166] It can be seen from Table 2 that the inter-layer interpolation method based on optical flow information basically exceeds the convolution and transformer based methods in terms of the quality and efficiency of predicting intermediate frames, which proves the practical value of optical flow information in inter-layer super-resolution reconstruction of CT images.

[0167] VFIformer uses the transformer algorithm to estimate complex motion, and FLAVR uses 3D convolution to extract the spatiotemporal features of inter-frame images, but both have the problem of large number of parameters and slow inference speed. The Inter-Net proposed in this paper achieves the best performance in terms of PSNR and SSIM, which shows the visual consistency between the generated image and the real image. This paper measures the complexity of the network by the number of parameters and inference time. The data in Table 2 show that Inter-Net can achieve higher performance with a small number of parameters and is more suitable for modern medical applications.

[0168] S4.3, Ablation experiment, ablation experiments were designed for different modules of Intra-Net and Inter-Net algorithms of IIF3SR-Net. The ablation experiment of Intra-Net algorithm was completed based on 3D-IRCADb dataset with a scaling factor of 2×, and the ablation experiment of Inter-Net algorithm was completed based on COVID-CT dataset; This paper designed ablation experiments for different modules of Intra-Net and Inter-Net algorithms of IIF3SR-Net. The ablation experiment of Intra-Net algorithm was completed based on 3D-IRCADb dataset with a scaling factor of 2×, as shown in Tables 3-5. The ablation experiment of Inter-Net algorithm was completed based on COVID-CT dataset, as shown in Tables 6 and 7.

[0169] Intra-Net verifies the contribution of S-ESA, MSFE and SCR to the network. In each experiment, only one module is deleted and the network performance without this specific module is tested. The results are shown in Table 3. It can be seen that each module is helpful for improving the performance. Deleting the multi-scale module has the greatest impact on the performance, which proves that multi-scale lesion feature information plays an important role in ensuring the pathological invariance of CT images.

[0170] Table 3 Ablation experiments of main modules of Intra-Net

[0171]

[0172] In order to better extract the multi-scale strip lesion features, Intra-Net decomposes a common convolution into a pair of deep strip convolutions. The traditional convolution is a two-dimensional convolution with a kernel of K×K, while the deep strip convolution splits a K×K convolution into deep strip convolutions with kernels of 1×K and K×1. In order to verify the impact of splitting into strip convolutions on the reconstruction results, Intra-Net compares the different results of using traditional convolution and strip convolution in the multi-scale module. As shown in Table 4, splitting the traditional convolution into strip convolutions can reduce the number of parameters, but the cost is the reduction of reconstruction performance. Experiments show that strip convolution can also extract rich multi-scale feature information instead of ordinary convolution, and the number of parameters is less than that of ordinary two-dimensional convolution.

[0173] Table 4 Comparison experiment between 2D ordinary convolution and depth convolution

[0174]

[0175] Network depth is an important factor affecting the reconstruction effect. The deeper the network, the more texture features are extracted and the better the reconstruction effect. However, as the network deepens, the difficulty of model training will also increase, and in severe cases, the gradient may disappear. In order to obtain the best performance, this paper analyzes the impact of the depth of the model on the network. Table 5 shows the relationship between the number of MSAFE layers and PSNR / SSIM. The performance of the deep network is better than that of the shallow network, but the inference time of the model will be increased. When 8 MSAFE modules are stacked continuously, the network performs best, while when only 6 MSA modules are used, the PSNR is not much different, but the time consumption increases. If the MSAFE module is further added, only a small performance improvement will be obtained, but the calculation and training costs will be greatly increased. When it is necessary to balance the two factors of performance and speed, this paper will set the number of layers of the MSAFE module to 6, and when only high-quality CT images are required to be generated, this paper will choose the number of layers of the MSAFE module to be 8.

[0176] Table 5 Relationship between the number of MSAFE layers and PSNR / SSIM

[0177]

[0178] In order to verify the role of the global bidirectional warp function GBDF proposed in this paper, the following ablation experiment is designed for Inter-Net. The effects of different warp functions on the quality of the output intermediate frames are verified respectively. The results are shown in Table 6, and red indicates the best result. Backwardwarp and deformable convolution only take into account the surrounding pixels and ignore the global information. The splatting operation does not distinguish the contribution of different pixels to the target area. The GBDF in this paper solves the above two limiting factors and thus achieves better reconstruction results.

[0179] Table 6 Impact of different distortion functions on network performance

[0180]

[0181] In order to compare the effects of different attention mechanisms on the network's extraction of inter-frame motion mapping relationships, this paper compares the cross-window attention mechanism, channel attention mechanism CA, spatial attention mechanism ESA and the spatial channel attention mechanism SCA proposed in this paper, and the results are shown in Table 7. The cross-window attention mechanism and spatial attention mechanism can make the network pay attention to local details but ignore the correlation between frames, while the lightweight channel attention mechanism enriches channel features and learns the correlation between channels, and more effectively extracts the mapping relationship between frames. The SCA designed in this paper better takes into account the factors ignored by the above algorithms.

[0182] Table 7 Impact of different attention mechanisms on network performance

[0183]

[0184] Aiming at the problem of large surface noise artifacts and step-like information in the three-dimensional reconstruction of medical CT images with large slice thickness and inter-slice spacing, this paper combines plane super-resolution and inter-layer interpolation to propose the IIF3SR-Net three-dimensional super-resolution algorithm. First, this paper proposes a medical image super-resolution network Intra-Net based on multi-scale attention, which progressively combines multi-scale deep convolution and attention mechanism to more accurately extract high-frequency information of the image. Secondly, this paper designs an inter-layer super-resolution network Inter-Net for medical CT images that iteratively refines from coarse to fine. Two continuous slices in spatial position are used to synthesize cross-sectional slices of high-resolution intermediate frames, and multi-scale context features are generated with feature pyramid structure. The global bidirectional warping function GBDF combines bidirectional optical flow information and multi-scale context features, gradually fuses bidirectional deformation estimation maps through IIR modules, and combines STIFOF function to complete the detail information of the intermediate frame to realize motion compensation, and finally obtains a high-resolution intermediate frame prediction map. Compared with other algorithms, the images predicted by this algorithm have higher objective index evaluation and better visual perception effect. Finally, this paper verifies the influence of the order of Intra-Net and Inter-Net algorithms on three-dimensional reconstruction. After a series of experiments, it is proved that only by performing the Intra-Net algorithm first and then the Inter-Net algorithm can the inter-layer resolution be improved and the problem of discontinuity and unclearness of the three-dimensional data surface be effectively solved. A high-quality three-dimensional data model can be reconstructed from a CT sequence with large layer thickness and inter-layer spacing.

[0185] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional super-resolution method suitable for CT image sequences, characterized in that: The steps include: S1. Extract continuous CT cross-sectional slice images, select high-resolution CT cross-sectional slice images with low slice thickness and small inter-slice spacing to construct a training set, divide the CT cross-sectional slice images with large slice thickness into a test set and a validation set, construct high-low resolution CT cross-sectional slice image pairs by bicubic interpolation, divide continuous CT cross-sectional slice images into triplets in the inter-slice interpolation network, and use the intermediate frame images as real images to compare with the output images; S2, constructing an intra-layer and inter-layer 3D fusion super-resolution network, which is specifically divided into an intra-layer super-resolution network structure and an inter-layer super-resolution network structure; S3, training the intra-layer and inter-layer 3D fusion super-resolution network model constructed in S2; S4. Input the test set into the intra-layer and inter-layer 3D fusion super-resolution network model trained in S3, and output a dense sequence of high-resolution CT cross-sectional slice images for 3D reconstruction.

2. A three-dimensional super-resolution method applicable to CT image sequences according to claim 1, characterized in that: In S1, the bicubic interpolation algorithm is used to interpolate high-resolution CT cross-sectional slice images. Processing to obtain the corresponding low-resolution image 3. The three-dimensional super-resolution method applicable to CT image sequences according to claim 1, characterized in that: The construction of the intra-layer and inter-layer 3D fusion super-resolution network in S2 comprises the following steps: S2.1, improve the spatial resolution of each cross-sectional slice by intra-slice super-resolution algorithm; S2.2, interpolating two adjacent slices through an interlayer interpolation algorithm to generate a slice at an intermediate position with high resolution, thereby obtaining a 2N-1 slice sequence with dense interlayer spacing; The S2 is obtained based on the following formula: In the formula, x and y represent the horizontal and vertical coordinates of the pixel point, and N represents the number of slices in a set of CT image sequences; represents the generated low-resolution cross-sectional image, f SR and f inter They represent the intra-layer super-resolution algorithm and the inter-layer super-resolution algorithm respectively. represents the generated high-resolution cross-sectional image, z represents the number of layers of the high-resolution cross-sectional image, represents the high-resolution cross-sectional image of layer Z+0.5, Represents the high-resolution cross-sectional image of layer Z+1.

4. A three-dimensional super-resolution method applicable to CT image sequences according to claim 3, characterized in that: Constructing the intra-layer super-resolution network structure in S2.1 comprises the following steps: S2.1.

1. Feature extraction, low-resolution images Extract features through shallow channel reconstruction network; S2.1.2, alignment and deep fusion, progressive alignment and deep fusion of multi-scale features are performed through multi-scale attention and serial progressive fusion modules; S2.1.

3. Construct a super-resolution image by fusing the channel reconstruction information based on ESPCN and the upsampled image based on the original image.

5. The three-dimensional super-resolution method applicable to CT image sequences according to claim 4, characterized in that: The multi-scale attention module in the S2.1.2 alignment and deep fusion aggregates local information through strip convolution, that is, each branch uses two deep strip convolutions to replace the standard deep convolution operation, and then uses the subsequent 1×1 convolution to model the relationship between multi-scale features, and uses the small-core spatial attention mechanism S-ESA to improve the effective utilization of features.

6. The three-dimensional super-resolution method applicable to CT image sequences according to claim 5, characterized in that: The alignment and deep fusion in S2.1.2 specifically designs a multi-scale attention module to fully refine effective information, and gradually aligns local detail features and global semantic features through a cascaded 1×1 convolution structure to fully aggregate multi-scale feature information.

7. The three-dimensional super-resolution method applicable to CT image sequences according to claim 1, characterized in that: The inter-layer super-resolution network combines context feature extraction, optical flow estimation, and inter-frame synthesis modules into a nested U-shaped structure, and implements repeated iteration of feature information through the inter-frame iterative optimization module IIR of the nested U-shaped structure. A spatial channel attention mechanism module SCA is designed in the inter-frame iterative optimization module IIR, and the spatial receptive field is gradually expanded through 3×3 convolutions, and the number of channels is gradually increased to better extract the correlation of inter-frame feature information.

8. The three-dimensional super-resolution method applicable to CT image sequences according to claim 1, characterized in that: The inter-layer super-resolution network designs an inter-frame optimization function based on spatial information, combines the refined bidirectional optical flow information and context features, and generates the final interpolated image through the following formula: Where w(·) represents the global bidirectional warping function; I z ,I z+1 is the input frame; Indicates that the direction of the optical flow information of layer 0 is from frame z to frame z+0.5, Indicates that the direction of the optical flow information of layer 0 is from frame z+0.5 to frame z+1; and It is a unidirectional intermediate frame image generated by a global bidirectional warping function; M0 and M1 are spatial masks with values ​​ranging from 0 to 1 generated by the output feature D0 after Sigmoid function. The spatial masks M0 and M1 are respectively Multiply to obtain a bidirectional intermediate frame image; R is the residual feature image extracted by D0 to compensate for texture details; I out Represents the output intermediate frame image.

9. A three-dimensional super-resolution method applicable to CT image sequences according to claim 8, characterized in that: The inter-layer super-resolution network also includes a global bidirectional warping function, which combines the forward and backward warping functions and considers the mapping relationship of global source pixels in the bidirectional warping function, wherein the forward warping function calculation model is as follows: u=I z+0.5 (p)-(I z (q)+F z→z+0.5 (q)) b(u)=max(0,1-|u x |)·max(0,1-|u y |) Where q is the source pixel coordinate; I z+0.5 (p) is the estimated value of the backward function; u is the difference between the predicted value and the estimated value based on the backward function; u x Indicates the offset value of u in the horizontal direction; u y Indicates the offset value of u in the ordinate direction; b(u) is the weight coefficient; is the target pixel of the reconstructed intermediate frame; I z (q) is the original frame; F z→z+0.5 The optical flow information in the direction from the zth frame to the z+0.5th frame; exp represents the Softmax operation on the weight K, and the weight K of each pixel is generated by the intermediate frame based on Average splatting and the original frame I z The Charbonnier Loss between the values ​​of corresponding pixels (q) is calculated.

10. The three-dimensional super-resolution method applicable to CT image sequences according to claim 1, characterized in that: The specific process of step S3 is: Step S3.1: First, define the loss function by using the mean absolute error function as the loss function; Step S3.2: train two models, namely the intra-layer super-resolution network and the inter-layer interpolation network; the above algorithm is based on the Windows 10 operating system, using the PyTorch deep learning framework; the hardware configuration is the CPU internal (R) core (TM) i7-6500u @ 2.50 GHz, 8 GB memory, the GPU is NVIDIA GTX 1080, CUDA 11.4 + cudnn 11.3 for GPU acceleration; the Adam optimization algorithm is used, β1 is set to 0.9, β2 is set to 0.999, the momentum parameter is set to 0.9, the batch size is 16, and the iteration is 300 epochs; the learning rate is initially set to 1×10 -4 , and gradually decays to 1×10 according to the cosine function -5 .

Images