A three-dimensional super-resolution method suitable for CT image sequences

Abstract

The application provides a three-dimensional super-resolution method suitable for a CT image sequence, and relates to the technical field of medical images.The application gradually extracts image high-frequency information more accurately by combining multi-scale deep convolution and attention mechanism, designs a medical CT image interlayer super-resolution network which is iteratively refined from coarse to fine, synthesizes a high-resolution cross-section slice of an intermediate frame by using two slices which are continuous in spatial position, generates multi-scale context features by a feature pyramid structure, combines bilateral flow information and multi-scale context features by a global bilateral distortion function (GBDF), gradually fuses bilateral deformation estimation maps by an IIR (Interframe iteration refinement, IIR) module, realizes motion compensation by combining an STIFOF (Spatio-Temporal Inter-Frame Optimization Function, STIFOF) function to complete detail information of the intermediate frame, and finally obtains a high-resolution intermediate frame prediction map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a three-dimensional super-resolution method suitable for CT image sequences. BACKGROUND

[0002] Computed tomography (CT) is an indispensable medical imaging technology in medical diagnosis. The CT three-dimensional imaging technology can make the tissue structure relationship clearer by stacking the sequence of two-dimensional CT planar images to reconstruct a 3D stereoscopic image, which is beneficial to accurate positioning and qualitative diagnosis of lesions. In order to obtain continuous and clear three-dimensional reconstruction effect, it is required that the spatial resolution of a single CT image and the interlayer resolution of adjacent layers of CT images are high, which will inevitably increase the CT radiation dose and equipment cost, and increase the damage to the human body. In order to balance the diagnosis requirements and reduce the harm to the patient's body, doctors usually use a relatively low-resolution CT scanning method, which is easy to cause the three-dimensional CT imaging model to have problems such as high noise artifacts and discontinuous surface, thereby affecting the diagnosis accuracy. The three-dimensional CT image of the organ reconstructed based on the low spatial resolution and interlayer resolution has a step-like surface and a blurred edge at the vascular tissue.

[0003] At present, there are a large number of intra-layer image super-resolution reconstruction works, which improve the quality of a single layer CT image without increasing the acquisition equipment and radiation cost. However, there are relatively few studies on the interlayer resolution enhancement interpolation work. The existing work mainly focuses on independent intra-layer or inter-layer super-resolution reconstruction work, and there are still few reports on the work of combining the two tasks to improve the overall three-dimensional CT model quality. In 2024, I 3 -Net attempts to process the CT image sequence by taking the inter-layer interpolation and intra-layer super-resolution as the upper and lower branches of the convolutional neural network, and realizes the preliminary fusion of the intra-layer super-resolution and the inter-layer interpolation. Although this method realizes the two tasks in the same network, the performance evaluation index of a single task still stays in the two-dimensional spatial clarity of the CT image sequence, and the influence on the three-dimensional reconstruction effect is not discussed. The internal logical relationship between the inter-layer interpolation and the intra-layer super-resolution algorithm and the influence of the sequence on the three-dimensional reconstruction effect are still not clear, so the organic fusion method of the inter-layer super-resolution and the intra-layer interpolation algorithm still needs to be further discussed. Therefore, we propose a medical CT image three-dimensional reconstruction network based on intra-layer super-resolution and inter-layer interpolation. SUMMARY

[0004] Therefore, the present application aims to provide a three-dimensional super-resolution method suitable for CT image sequences, which effectively solves the problems of discontinuous and unclear three-dimensional body data surface, and reconstructs a high-quality three-dimensional body data model from a CT sequence with large layer thickness and interlayer spacing.

[0005] To achieve the above object, the application adopts the following technical scheme: a three-dimensional super-resolution method suitable for CT image sequences, comprising the following steps:

[0006] S1, extract continuous CT cross-sectional slice images, select high-resolution CT cross-sectional slice images with low layer thickness and small interlayer spacing to construct a training set, divide CT cross-sectional slice images with large layer thickness into a test set and a validation set, construct high-low resolution CT cross-sectional slice image pairs through bicubic interpolation, divide the continuous CT cross-sectional slice images into triplets in the interlayer frame interpolation network, and compare the middle frame image as the real image with the output image;

[0007] S2, construct an intra-interlayer three-dimensional fusion super-resolution network, which is specifically divided into an intra-layer super-resolution network structure and an inter-layer super-resolution network structure;

[0008] S3, train the intra-interlayer three-dimensional fusion super-resolution network model constructed in S2;

[0009] S4, input the test set into the intra-interlayer three-dimensional fusion super-resolution network model trained in S3, and output the dense sequence of high-resolution CT cross-sectional slice images for three-dimensional reconstruction.

[0010] In a preferred embodiment: in S1, bicubic interpolation algorithm is used to process high-resolution CT cross-sectional slice images to obtain corresponding low-resolution images

[0011] In a preferred embodiment: the intra-interlayer three-dimensional fusion super-resolution network constructed in S2 comprises the following steps:

[0012] S2.1, improve the spatial resolution of each cross-sectional slice through an intra-layer super-resolution algorithm;

[0013] S2.2, generate a high-resolution intermediate position slice by interlayer frame interpolation on adjacent two slices to obtain a 2N-1 slice sequence with dense interlayer spacing;

[0014] The S2 is based on the following formula:

[0015]

[0016] In the formula, x and y represent the horizontal and vertical coordinates of the pixel point, and N represents the number of slices of a group of CT image sequences. f represents the generated low-resolution cross-sectional image, f SR and f inter respectively represent the intra-layer super-resolution algorithm and the inter-layer super-resolution algorithm, Fig. 2 shows a high-resolution cross-section image generated, z represents the number of layers of the high-resolution cross-section image, Fig. 3 shows a high-resolution cross-section image of the z+0.5 layer, Fig. 4 shows a high-resolution cross-section image of the z+1 layer.

[0017] In a preferred embodiment: the step of constructing the layer-in super-resolution network structure in S2.1 includes the following steps:

[0018] S2.1.1, feature extraction, low-resolution image extract features through a shallow channel reconstruction network;

[0019] S2.1.2, alignment and depth fusion, progressive alignment and depth fusion of multi-scale features through multi-scale attention and cascaded progressive fusion module;

[0020] S2.1.3, constructing a super-resolution image, fusing channel reconstruction information based on ESPCN and up-sampled images based on the original image.

[0021] In a preferred embodiment: the multi-scale attention module in S2.1.2 alignment and depth fusion aggregates local information through strip convolution, i.e. each branch uses two deep strip convolutions instead of standard deep convolution operations, and then uses a 1x1 convolution to model the relationship between multi-scale features, and uses a small kernel space attention mechanism S-ESA to improve the effective use of features.

[0022] In a preferred embodiment: the multi-scale attention module in S2.1.2 alignment and depth fusion is specifically designed to fully refine effective information, and a cascaded 1x1 convolution structure is used to progressively align local detail features and global semantic features to fully aggregate multi-scale feature information.

[0023] In a preferred embodiment: the inter-layer super-resolution network integrates the context feature extraction, optical flow estimation, and inter-frame synthesis modules into a nested U-shaped structure, and uses the inter-frame iterative optimization module IIR of the nested U-shaped structure to realize repeated iteration of feature information, and designs a spatial channel attention mechanism module SCA in the inter-frame iterative optimization module IIR, which gradually expands the spatial receptive field through 3x3 convolution and gradually increases the number of channels to better extract the correlation of inter-frame feature information.

[0024] In a preferred embodiment: the inter-layer super-resolution network designs an inter-frame optimization function based on spatial information, combines the refined bidirectional optical flow information and context features, and uses the following formula to generate the final interpolated image:

[0025]

[0026] where w(·) represents a global bidirectional warping function;

[0027] I z 、I z+1 is an input frame;

[0028] represents the light flow information direction of the 0th layer is from the zth frame to the z+0.5th frame, represents the light flow information direction of the 0th layer is from the z+0.5th frame to the z+1th frame.

[0029] and is a unidirectional intermediate frame image generated through the global bidirectional warping function;

[0030] M0 and M1 are output features D0 after passing through a Sigmoid function to generate a spatial mask with a value range from 0 to 1, and the spatial masks M0 and M1 are respectively multiplied by the unidirectional intermediate frame images to obtain a bidirectional intermediate frame image.

[0031] R is a residual feature image extracted from D0 to compensate for texture details;

[0032] I out represents an output intermediate frame image.

[0033] In a preferred embodiment: the inter-layer super-resolution network further comprises a global bidirectional warping function, which combines a forward warping function and a backward warping function, and considers the mapping relationship of global source pixels in the bidirectional warping function, wherein the forward warping function calculation model is as follows:

[0034] u=I z+0.5 (p)-(I z (q)+F z→z+0.5 (q))

[0035] b(u)=max(0,1-|u x |)·max(0,1-|u y |)

[0036]

[0037] where q is a source pixel coordinate;

[0038] I z+0.5 (p) is a backward function estimation value;

[0039] u is a difference value between a prediction value and the backward function estimation value;

[0040] u x represents a shift value of u in the horizontal coordinate direction;

[0041] uy represents the offset value of u in the ordinate direction;

[0042] b(u) is a weight coefficient;

[0043] is the target pixel of the reconstructed intermediate frame;

[0044] I z (q) is the original frame;

[0045] F z→z+0.5 is the optical flow information in the direction from the z-th frame to the z+0.5-th frame;

[0046] exp represents the Softmax operation on the weight K, and the weight K of each pixel is obtained by the Charbonnier Loss calculation between the values of the corresponding pixels of the intermediate frame generated based on the Average splatting and the original frame I z (q).

[0047] In a preferred embodiment: the specific process of step S3 is:

[0048] Step S3.1: first define the loss function, by using the mean absolute error function as the loss function;

[0049] Step S3.2: train two models, respectively, intra-layer super-resolution network and inter-layer interpolation network; the above algorithm is based on windows10 operating system, using pytorch deep learning framework; the hardware configuration is CPU internal (R) core (TM) i7-6500u@2.50ghz, 8g memory, GPU is NVIDIA GTX 1080, CUDA 11.4+cudnn 11.3 for GPU acceleration; using Adam optimization algorithm, setting β1 to 0.9, β2 to 0.999, momentum parameter to 0.9, batch size to 16, iteration to 300 epochs; the learning rate is initially set to 1×10 -4 , and gradually decays to 1×10 -5 according to the cosine function.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] 1.The application gradually extracts high-frequency information of images more accurately by multi-scale deep convolution and attention mechanism, designs a medical CT image interlayer super-resolution network of iterative refinement from coarse to fine, synthesizes a high-resolution cross-sectional slice of an intermediate frame from two consecutive slices in spatial position, generates multi-scale context features by a feature pyramid structure, combines bidirectional optical flow information and multi-scale context features by a global bidirectional distortion function GBDF, gradually fuses bidirectional deformation estimation maps by an IIR module, combines an STIFOF function to complete detail information of the intermediate frame to realize motion compensation, and finally obtains a high-resolution intermediate frame prediction map.

[0052] 2.The image predicted by the algorithm has high objective index evaluation and good visual perception effect, verifies the influence of the intra-layer super-resolution reconstruction network and the inter-layer frame interpolation reconstruction network on the three-dimensional reconstruction, proves through a series of experiments that only the intra-layer super-resolution reconstruction network is performed and then the inter-layer frame interpolation reconstruction network is used to improve the inter-layer resolution, and the problem of discontinuity and unclearness of the three-dimensional body data surface can be effectively solved, and a high-quality three-dimensional body data model is reconstructed from a CT sequence with large layer thickness and layer spacing. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a workflow diagram of the preferred embodiment of the application;

[0054] Figure 2 is a general flowchart of the intra-layer-inter-layer fusion three-dimensional super-resolution reconstruction method IIF3SR-Net of the preferred embodiment of the application;

[0055] Figure 3 is a structure diagram of the intra-layer super-resolution network of the preferred embodiment of the application;

[0056] Figure 4 is a structure diagram of MSAFE in the preferred embodiment of the application;

[0057] Figure 5 is an Inter-Net network structure diagram;

[0058] Figure 6 is an IIR module structure diagram;

[0059] Figure 7 is a visual result and PNSR diagram of the up-sampling 2 times and 4 times reconstructed images;

[0060] Figure 8 is a visual comparison diagram of the up-sampling 2 times and 4 times reconstructed images;

[0061] Figure 9 is a visual comparison diagram of the original CT image and the three-dimensional reconstruction after the intra-layer super-resolution. DETAILED DESCRIPTION

[0062] The application will be further described below in conjunction with the accompanying drawings and embodiments.

[0063] It should be noted that the following detailed description is illustrative only, and is intended to provide further description in order to provide a fuller enabling teaching of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0064] It is also important to note that the terms "comprises", "comprising", "includes", "including", "contains", "containing" or variations thereof herein, do not specify an exhaustive or complete list of components or features as used by those skilled in the art. Unless otherwise expressly specified, use of the terms "comprises", "comprising", "includes", "including", "contains", "containing" or variations thereof herein, are intended to be inclusive or open-ended and do not exclude additional, unrecited elements or method steps.

[0065] Reference will now be made to Figure 1 Figure 9 The present application provides a three-dimensional super-resolution method suitable for CT image sequences, comprising the following steps:

[0066] S1, extracting continuous CT cross-sectional slice images, selecting high-resolution CT images with low layer thickness and small interlayer spacing to construct a training set, dividing CT images with larger layer thickness into a test set and a validation set, constructing high-low resolution CT image pairs by bicubic interpolation method, dividing continuous CT cross-sectional images into triplets in the interlayer frame interpolation network, and comparing the middle frame as the real image with the output image;

[0067] S2, in order to improve the plane and interlayer resolution of the CT image sequence, an IIF3SR-Net network is constructed, which is specifically divided into two modules, an Intra-Net intra-layer super-resolution network structure and an Inter-Net inter-layer super-resolution network structure;

[0068] S3, training the network model constructed in step 2;

[0069] S4, inputting the test set into the network model trained in step 3, and outputting high-resolution CT images of the dense sequence for three-dimensional reconstruction.

[0070] In this embodiment, preferably, the IIF3SR-Net network constructed in S2 comprises the following steps:

[0071] S2.1, improving the spatial resolution of each cross-sectional slice by the intra-layer super-resolution algorithm Intra-Net;

[0072] ​S2.2, Inter-Net is used to perform inter-layer frame interpolation between two adjacent slices to generate intermediate position slices with high resolution, thereby obtaining a 2N-1 slice sequence with dense inter-layer spacing.

[0073] In this embodiment, preferably, S2 is obtained according to the following formula:

[0074]

[0075] In the formula, A set of cross-sectional slices is shown,

[0076] f SR and f inter respectively represent an intra-layer super-resolution algorithm and an inter-layer super-resolution algorithm,

[0077] represents a generated high-resolution cross-sectional image,

[0078] represents a high-resolution image of the z-th layer.

[0079] In this embodiment, preferably, the construction of the Intra-Net intra-layer super-resolution network structure in S2.1 includes the following steps:

[0080] S2.1.1, feature extraction, low-resolution image extract features through a shallow channel reconstruction network;

[0081] S2.1.2, alignment and depth fusion, progressive alignment and depth fusion of multi-scale features through a multi-scale attention and series type progressive fusion module;

[0082] S2.1.3, construction of a super-resolution image, fusion of channel reconstruction information based on ESPCN and up-sampled images based on the original image.

[0083] In this embodiment, preferably, the multi-scale attention module in S2.1.2 alignment and depth fusion aggregates local information through strip convolution, i.e., each branch uses two deep strip convolutions instead of standard deep convolution operations, such as replacing a 7x7 two-dimensional convolution with a pair of 7x1 and 1x7 strip convolutions, and then modeling the relationship between multi-scale features through a 1x1 convolution, and then through a small kernel space attention mechanism S-ESA to improve the effective use of features.

[0084] In this embodiment, preferably, in S2.1.2 alignment and depth fusion, a multi-scale attention module is designed to fully refine effective information, and a series type 1x1 convolution structure is used to progressively align local detail features and global semantic features to fully aggregate multi-scale feature information.

[0085] In this embodiment, preferably, the Inter-Net in S2.2 designs an inter-frame optimization function based on spatial information, which combines the refined bidirectional optical flow information and context features, and the final interpolated frame image is generated through the following formula:

[0086]

[0087] In the formula, w represents a global bidirectional warping function;

[0088] I z 、I z+1 is an input frame;

[0089] represents the direction of the optical flow information of the 0th layer from the zth frame to the z+0.5th frame, represents the direction of the optical flow information of the 0th layer from the z+0.5th frame to the z+1th frame.

[0090] and is a unidirectional intermediate frame image generated by the global bidirectional warping function;

[0091] M0 and M1 are spatial masks with a value range from 0 to 1 generated by the Sigmoid function from the output feature D0, and the spatial masks M0 and M1 are respectively multiplied by the unidirectional intermediate frame images to obtain bidirectional intermediate frame images.

[0092] R is a residual feature image extracted from D0, which can compensate for texture details;

[0093] I out represents an output intermediate frame image

[0094] In this embodiment, preferably, the Inter-Net in S2.2 further includes a global bidirectional warping function GBDF, which combines a forward warping function and a backward warping function and considers the mapping relationship of global source pixels in the bidirectional warping function, wherein the forward warping function calculation model is as follows:

[0095] u=I z+0.5 (p)-(I z (q)+F z→z+0.5 (q))

[0096] b(u)=max(0,1-|u x |)·max(0,1-|u y |)

[0097]

[0098] In the formula, q is a source pixel coordinate;

[0099] Iz+0.5 (p) is a backward function estimation value;

[0100] u is a difference value between a prediction value and the backward function estimation value;

[0101] u x represents a shift value of u in the horizontal coordinate direction;

[0102] u y represents a shift value of u in the vertical coordinate direction;

[0103] b(u) is a weight coefficient;

[0104] is a target pixel of a reconstructed intermediate frame;

[0105] I z (q) is an original frame;

[0106] F z→z+0.5 is optical flow information in a direction from the z-th frame to the z+0.5-th frame;

[0107] exp represents a Softmax operation on the weight K, and the weight K of each pixel is obtained by Charbonnier Loss calculation between the values of the corresponding pixel points between the intermediate frame generated based on Average splatting and the original frame I z (q).

[0108] Working principle and use flow of the application:

[0109] S1, extract continuous tomographic CT cross-sectional slice images of a patient, select high-resolution CT images with low CT image layer thickness and small layer spacing to construct a training set, divide CT images with larger layer thickness into a test set and a validation set, construct high-low resolution CT image pairs through bicubic interpolation method, divide the continuous CT images into triplets in the inter-layer frame interpolation network, and compare the intermediate frame as a real image with the output image;

[0110] S2, construct an IIF3SR-Net network, including the following steps:

[0111] S2.1, improve the spatial resolution of each cross-sectional slice through an intra-Net inter-layer super-resolution algorithm;

[0112] S2.2, generate an intermediate position slice with high resolution by Inter-Net inter-layer frame interpolation on adjacent two slices to obtain a 2N-1 slice sequence with dense layer spacing, according to the following formula:

[0113]

[0114] In the formula, showing a set of cross-sectional slices,

[0115] f SR and f inter represent intra-layer super-resolution algorithm and inter-layer super-resolution algorithm, respectively,

[0116] represent generated high-resolution cross-sectional images,

[0117] represent high-resolution images of the z-th layer.

[0118] S2.1, constructing an Intra-Net intra-layer super-resolution network structure, comprising the following steps:

[0119] S2.1.1, feature extraction, low-resolution images extracting features through a shallow channel reconstruction network;

[0120] S2.1.2, alignment and depth fusion, performing progressive alignment and depth fusion of multi-scale features through a multi-scale attention and cascaded progressive fusion module;

[0121] S2.1.3, constructing a super-resolution image, fusing channel reconstruction information based on ESPCN and up-sampled images based on the original image.

[0122] The multi-scale attention and cascaded progressive fusion module is used to fully refine effective information and progressively align local detail features and global semantic features through a cascaded 1x1 convolution structure to fully aggregate multi-scale feature information.

[0123] To better extract the strip-shaped organization in the CT image, the multi-scale module in MSAFE aggregates local information through strip-shaped convolution, that is, each branch uses two deep strip-shaped convolutions instead of standard deep convolution operations, such as replacing a 7x7 two-dimensional convolution with a pair of 7x1 and 1x7 strip-shaped convolutions, and then modeling the relationship between multi-scale features through a 1x1 convolution. Subsequently, a small kernel spatial attention mechanism S-ESA is used to improve the effective use of features, especially to encourage the network to fully focus on key pathological detail features. Unlike the ESA spatial attention mechanism, S-ESA simplifies redundant convolution groups into a residual module of small kernel convolution to improve the efficiency of extracting local texture detail features. At the same time, S-ESA uses a 1x1 convolution layer to reduce the number of channels at the beginning and restore the number of channels at the feature extraction stage, which reduces the computational load while more effectively fusing effective feature information of the CT image and filtering artifacts and noise. In addition, to detect spatial information of different scales, Intra-Net still uses a large window pooling layer bilinear interpolation method for up-sampling, which can greatly improve the extraction ability of detail features.

[0124] S2.2, construct the Inter-Net interlayer super-resolution network structure; wherein, the Inter-Net combines the context feature extraction, optical flow estimation, interframe synthesis module into a nested U-shaped structure, and the IIR module of the nested U-shaped structure realizes repeated iteration of feature information, and a spatial channel attention mechanism SCA module is designed in the IIR, which gradually expands the spatial receptive field through 3x3 convolution, and gradually increases the channel number to better extract the interframe feature information correlation. Finally, multiplying the channel weight coefficient with the input feature makes the network learn more critical motion estimation relationship. The left downsampling branch adopts multiple 3x3 convolutions with a stride of 2, and the right upsampling branch is composed of 2 4x4 transposed convolutions with a stride of 2, to extract multi-scale features. There is a PRelu activation function after each convolution layer to prevent overfitting. Finally, adding a global bidirectional warping function in the residual connection can also promote the further joint refinement and propagation of feature information.

[0125] The Inter-Net designs an interframe optimization function based on spatial information, which combines the refined bidirectional optical flow information and context features, and the final interpolated image is obtained by the following formula:

[0126]

[0127] In the formula, w represents a global bidirectional warping function;

[0128] I z 、I z+1 is the input frame;

[0129] and is a unidirectional intermediate frame image generated by the global bidirectional warping function;

[0130] M0 and M1 are spatial masks with a value range of 0 to 1 generated by the output feature D0 after the Sigmoid function, and the spatial masks M0, M1 are multiplied by the unidirectional intermediate frame images to obtain a bidirectional intermediate frame.

[0131] R is a residual feature image extracted from D0 that can compensate for texture details.

[0132] The Inter-Net also includes a global bidirectional warping function GBDF, which combines the forward and backward warping functions and considers the mapping relationship of global source pixels in the bidirectional warping function, wherein the forward warping function calculation model is as follows:

[0133] u = I z+0.5 (p) - (I z (q) + F z→z+0.5 (q))

[0134] b(u) = max(0, 1 - |u x |) · max(0, 1 - |u y |)

[0135]

[0136] where q is a coordinate of a source pixel point;

[0137] I z+0.5 (p) is a backward function estimation value;

[0138] u is a difference between a prediction value and the backward function estimation value;

[0139] u x represents a shift value of u in the horizontal coordinate direction;

[0140] u y represents a shift value of u in the vertical coordinate direction;

[0141] b(u) is a weight coefficient;

[0142] is a target pixel of the reconstructed intermediate frame;

[0143] I z (q) is an original frame;

[0144] F z→z+0.5 is optical flow information from the zth frame to the z+0.5th frame;

[0145] exp represents a Softmax operation on the weight K, and the weight K of each pixel is obtained by Charbonnier Loss calculation between the values of corresponding pixel points of the intermediate frame generated based on Average splatting and the original frame I z (q);

[0146]

[0147] As shown in the following formula (1), first, the input frame is distorted by Average splatting to pre-generate an intermediate frame image On this basis, the weight is calculated using Charbonnier Loss which is defined as shown in the following formula (2), and ε is taken as 10 -3 ). Compared with the L1 function, Charbonnier Loss can better distinguish the effects between different pixel points, and thus is conducive to optimizing the local lesion detail information. Specifically, the weight K of each pixel is obtained by Charbonnier Loss calculation between the values of corresponding pixel points of the original image I z (q) and the pre-generated intermediate frame The Charbonnier Loss between the two frames is calculated as shown in formula (3). Here, a pre-generated intermediate frame is used. Instead of directly using I z+1 Calculations are performed on each frame to reduce the gap between two frames, which helps to obtain more accurate weighting coefficients K.

[0148]

[0149] The left-hand downsampling branch, Pyramid Encoder, extracts multi-scale optical flow information and image contextual features, which are then combined in the Coarse-to Fine Frame Synthesis part. and Pyramid features The Global Bidirectional Distortion Function (GBDF) is used to initially filter artifact noise and synthesize a coarse intermediate frame image. The IIR module in the Coarse-to-Fine Frame Synthes section aligns and refines multi-scale bidirectional optical flow information and contextual features. This coarse-to-fine strategy facilitates the combination of contextual and optical flow information at each scale, resulting in a more comprehensive alignment of global information and local details. Finally, the spatially based post-processing optimization function STIFOF supplements texture details to generate the final high-resolution intermediate frame image. The left-hand downsampling branch, Pyramid Encoder, is a feature pyramid structure to extract contextual information at different scales in each input frame, consisting of four convolutional stages: stage0, stage1, stage2, and stage3. Each stage comprises two 3×3 two-dimensional convolutions with strides of 2 and 1, respectively. The first convolution downsamples the input features, while the second extracts rich scale features. Each pyramid layer is followed by a PRelu activation function. The number of feature channels gradually increases with the shrinking feature space size, becoming 32, 48, 72, and 96, respectively. Each stage corresponds to frame I. z and I z+1 k-level pyramid features Where k∈[0,1,2,3], in the upsampling branch, the inter-frame analysis layer IIR_k+1 is combined with the bidirectional optical flow field. Preliminary synthesized intermediate frame images Intermediate features D output by IIR_k k Reconstructing higher-level intermediate features D with richer details k+1 As shown in the following formula, progressive alignment and blending are ultimately achieved.

[0150]

[0151] The network architecture gradually fuses bidirectional optical flow fields of different scales, pyramid encoder features, global bidirectional warping compensation, and reconstructed features of the previous scale, which can fully reconstruct the missing reference information and achieve more accurate motion estimation. The progressive reconstruction method is beneficial to align the input feature information from coarse to fine, improving the quality of intermediate feature reconstruction. Therefore, the up-sampling branch designed in this paper can gradually refine the reconstructed intermediate features and bidirectional optical flow fields, generate fusion masks, and compensate for motion detail information, finally generating high-resolution intermediate frames. In summary, the inter-frame analysis layer IIR_k is represented by the above formula, where D k represents the features extracted by the decoder, and [·] represents the Concat operation.

[0152] S3, training the network model constructed in step 2

[0153] Step S3.1: First, define the loss function. This method uses the mean absolute error function as the loss function.

[0154] Step S3.2: The present application trains two models, Intra-Net and Inter-Net. The above algorithm is based on windows10 operating system, using pytorch deep learning framework. The hardware configuration is CPU internal (R) core (TM) i7-6500u@2.50ghz, 8g memory, GPU is NVIDIA GTX 1080, CUDA 11.4+cudnn 11.3 for GPU acceleration. Using Adam optimization algorithm, setting β1 to 0.9, β2 to 0.999, momentum parameter to 0.9, batch size to 16, iteration to 300 epochs. The learning rate is initially set to 1×10 -4 , and gradually decays to 1×10 -5 according to the cosine function.

[0155] S4, input the test set into the network model trained in step 3, and output the high-resolution CT image of the dense sequence for three-dimensional reconstruction.

[0156] To better illustrate the performance of IIF3SR-Net, this paper illustrates the performance comparison experiments of the intra-layer super-resolution reconstruction network Intra-Net, the inter-layer interpolation reconstruction network Inter-Net, and their order; The experiment is based on 3D-IRCADb and COVID-19 two data sets, among which, the 3D-IRCADb data set contains CT scan image sequences of 10 female and 10 male liver tumor patients, this paper has carried out corresponding pretreatment on it, extracted the CT cross-sectional slice images of each patient, and selected high-resolution CT images with low layer thickness and small interlayer spacing to construct the training set, finally obtained 737 groups of LR-HR image pairs as the training set, while the remaining 110 groups of high-low resolution CT images with larger layer thickness were divided into two parts, respectively as the test set and the verification set, the resolution of each HR image is 512x512 pixels, the COVID-19 data set is a 20 group of lung CT data set collected by ieee8023 for COVID-19 research, the intra-layer resolution of this data set ranges from 0.6836 to 0.7246 mm, and the inter-layer resolution is 1 mm, after pretreatment, the training set is composed of 965 groups of LR-HR CT images, and the test and verification sets contain a total of 146 images, in order to meet the training and performance test of the interpolation network, the continuous cross-sectional slices of the real thin layer CT are used as high-resolution CT images in the experiment, and are divided into triplets {I z , I z+0.5 , I z+1}. Among them, I z and I z+1 are input images, I out is the high-resolution interpolation image generated by the network, and the performance is verified by comparing with I z+0.5 the real intermediate slice. Peak signal-to-noise ratio PSNR and structural similarity index SSIM widely used for natural image and medical image SR quality evaluation are used as performance indicators to evaluate the performance of CT super-resolution reconstruction and interpolation network model.

[0157] The performance comparison experiments of Intra-Net and Inter-Net networks based on 3D-IRCADb and COVID-CT datasets are carried out; first, the performance comparison of 2x and 4x super-resolution reconstruction of the small model and the large model of Intra-Net and the advanced natural image super-resolution algorithms EDSR, RCAN, RFANet, MFEN, SwinIR, RFLN and the existing medical image super-resolution algorithms FAWDN, RDAN, MSAB, MWSR, DRIDSR is carried out, the difference between the small model and the large model lies in the number of deep strip convolution channels, the experiment proves that more channel numbers are helpful to extract much richer scale features. PSNR and SSIM index are important performance indicators of reconstruction, as shown in Table 1 and Figure 7 Intra-Net has higher PSNR and SSIM index, which indicates that the quality of the SR result is better than other methods and closer to the HR image. Compared with all the above methods, the Intra-Net of the present application is better than the above other methods in the performance of two CT image datasets and the scaling factor of 2x and 4x.

[0158] The present application shows the intuitive visual results compared with other super-resolution algorithms with the scaling factor of 2x and 4x, and the effect is as shown in Figure 8 Since Intra-Net adopts multi-scale strip convolution, the image reconstructed by Intra-Net algorithm is more clear at the edge of the lung and the small nodules in the lung than other algorithms, the image contrast is more obvious, and it is easier to distinguish different small nodule tissues.

[0159] The present application provides the effect comparison chart of three-dimensional reconstruction after super-resolution by Intra-Net model and three-dimensional reconstruction of original CT image, and the effect is as shown in Figure 9 (a) (c) (d) are detail maps of three-dimensional reconstruction of original CT image, (b) (e) (f) are detail maps of three-dimensional reconstruction after layer super-resolution reconstruction algorithm. (e) extracts more strip vascular tissue features by the deep strip convolution of the layer super-resolution reconstruction algorithm, and the reconstructed image is clearer than (c) and eliminates many local false information and noise points. (d) MSA-PF stage of the layer super-resolution reconstruction algorithm fuses more tissue features, and realizes clear visualization effect on more micro-texture details than (f).

[0160] Table 1 Performance comparison of Intra-Net model and other methods on 3D-IRCADb and COVID-CT datasets

[0161]

[0162]

[0163] Seven algorithms with better performance in recent years are selected in this paper, including Softmax spaltting, FLAVR, RIFE, CAIN, VFIformer, IFRnet and DQBC. Among them, VFIformer is a transformer-based interpolation algorithm, FLAVR, CAIN and DQBC are convolution-based interpolation algorithms, and Softmax spaltting, RIFE and IFRnet are optical flow-based interpolation algorithms. The quantitative evaluation results of the above algorithms on this dataset are shown in Table 2, where red represents the optimal result, and green represents the suboptimal result.

[0164] Table 2 Comparison of indicators of various algorithms on two test sets

[0165]

[0166] As can be seen from Table 2, the optical flow-based interlayer interpolation method basically outperforms the convolution and transformer-based methods in predicting the quality and efficiency of the intermediate frame, proving the practical value of optical flow information in CT image interlayer super-resolution reconstruction.

[0167] VFIformer estimates complex motion through a transformer algorithm, and FLAVR uses 3D convolution to extract the spatiotemporal features of interframe images, but both have the problem of large parameter quantity and slow inference speed. The proposed Inter-Net achieves the optimal performance in terms of PSNR and SSIM, indicating the visual consistency of the generated image and the real image. The parameter quantity and inference time are used to measure the complexity of the network, and the data in Table 2 show that Inter-Net can achieve higher performance with small parameter quantity, making it more suitable for modern medical applications.

[0168] S4.3, Ablation experiment, the effect of different modules in Intra-Net and Inter-Net algorithms of IIF3SR-Net An ablation experiment was designed for the effect of different modules in Intra-Net and Inter-Net algorithms of IIF3SR-Net. The ablation experiment of Intra-Net algorithm was completed based on 3D-IRCADb dataset with a scaling factor of 2x, and the ablation experiment of Inter-Net algorithm was completed based on COVID-CT dataset. This paper designed an ablation experiment for the effect of different modules in Intra-Net and Inter-Net algorithms of IIF3SR-Net. The ablation experiment of Intra-Net algorithm was completed based on 3D-IRCADb dataset with a scaling factor of 2x, as shown in Tables 3-5. The ablation experiment of Inter-Net algorithm was completed based on COVID-CT dataset, as shown in Tables 6 and 7.

[0169] Intra-Net verifies the contribution of S-ESA, MSFE and SCR to the network, and each experiment only deletes one module and tests the network performance without the specific module, and the results are shown in Table 3, which shows that each module is helpful to improve the performance, and the deletion of the multi-scale module has the greatest impact on the performance, which proves that the multi-scale lesion feature information plays an important role in ensuring the pathologic invariance of CT images.

[0170] Table 3 Ablation experiment of main modules of Intra-Net

[0171]

[0172] In order to better extract the multi-scale strip lesion features, Intra-Net decomposes a normal convolution into a pair of deep strip convolutions. The traditional convolution is a KxK two-dimensional convolution, while the deep strip convolution splits a KxK convolution into a 1xK and Kx1 deep strip convolution. In order to verify the influence of splitting into strip convolution on the reconstruction result, Intra-Net compares the different results of adopting traditional convolution and strip convolution in the multi-scale module, as shown in Table 4. Splitting the traditional convolution into strip convolution can reduce the parameter amount, but at the cost of reducing the reconstruction performance. Experiments show that strip convolution can also extract rich multi-scale feature information and has less parameter amount than ordinary two-dimensional convolution.

[0173] Table 4 Comparison experiment of two-dimensional ordinary convolution and deep convolution

[0174]

[0175] The depth of the network is an important factor affecting the reconstruction effect. The deeper the network, the more texture features are extracted, and the better the reconstruction effect. However, as the network deepens, the difficulty of model training will also increase, and in severe cases, the gradient may disappear. In order to obtain the optimal performance, this paper analyzes the influence of the depth of the model on the network. Table 5 shows the relationship between the number of MSAFE layers and PSNR / SSIM. The performance of the deep network is better than that of the shallow network, but it increases the inference time of the model. When 8 MSAFE modules are continuously stacked, the performance of the network is the best, and when only 6 MSAFE modules are used, the PSNR is not much different, but the time consumption increases. If the number of MSAFE modules continues to increase, only a small performance improvement will be obtained, but the calculation and training cost will be greatly increased. When balancing the two factors of performance and speed, this paper sets the number of MSAFE modules to 6, and when only high-quality CT images are required, this paper selects the number of MSAFE modules to 8.

[0176] Table 5 Relationship between the number of MSAFE layers and PSNR / SSIM

[0177]

[0178] To verify the role of the global bidirectional distortion function GBDF proposed in this paper, the following ablation experiments are designed for Inter-Net. The influence of different distortion functions on the quality of the output intermediate frame is verified respectively, and the results are shown in Table 6, with red indicating the optimal result. Backwardwarp and deformable convolution only take into account the surrounding pixel points and ignore global information, and the splatting operation does not distinguish the contribution of different pixel points to the target area. The GBDF in this paper solves the above two limiting factors and therefore achieves better reconstruction results.

[0179] Table 6 Influence of different distortion functions on network performance

[0180]

[0181] To compare the promoting effect of different attention mechanisms on the network's extraction of inter-frame motion mapping relationship, this paper compares the cross-window attention mechanism, the channel attention mechanism CA, the spatial attention mechanism ESA and the spatial channel attention mechanism SCA proposed in this paper, and the results are shown in Table 7. Cross-window attention mechanism and spatial attention can make the network pay attention to local details but ignore the relevance between frames, while lightweight channel attention mechanism enriches channel features and learns the relevance between channels, more effectively extracts the mapping relationship between frames, and the SCA designed in this paper better takes into account the factors ignored by the above algorithms.

[0182] Table 7 Influence of different attention mechanisms on network performance

[0183]

[0184] To solve the problem of the noise and the step-like surface of the 3D reconstruction of medical CT images with large slice thickness and slice spacing, this paper proposes a 3D super-resolution algorithm called IIF3SR-Net, which combines plane super-resolution and inter-slice interpolation. First, this paper proposes a medical image super-resolution network called Intra-Net based on multi-scale attention, which gradually extracts more accurate high-frequency information by combining multi-scale deep convolution and attention mechanism. Second, this paper designs an inter-slice super-resolution network called Inter-Net for medical CT images, which synthesizes high-resolution cross-sectional slices from two consecutive slices by using a feature pyramid structure to generate multi-scale context features, and a global bidirectional distortion function (GBDF) to combine bidirectional optical flow information and multi-scale context features. The IIR module gradually fuses the bidirectional deformation estimation map, and the STIFOF function is used to complete the details of the intermediate frame to realize motion compensation, and finally the high-resolution intermediate frame prediction image is obtained. Compared with other algorithms, the predicted image has higher objective evaluation and better visual perception. Finally, this paper verifies the influence of the order of Intra-Net and Inter-Net algorithms on 3D reconstruction, and a series of experiments prove that only by first performing Intra-Net algorithm and then Inter-Net algorithm can the inter-slice resolution be improved to effectively solve the problem of discontinuous and unclear surface of 3D volume data. The CT sequence with large slice thickness and slice spacing is reconstructed into a high-quality 3D volume data model.

[0185] The above description of disclosed embodiments allows those skilled in the art to make or use the application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional super-resolution method suitable for a sequence of CT images, characterized in that, It comprises the following steps: S1, extract continuous CT cross-sectional slice images, select high-resolution CT cross-sectional slice images with low layer thickness and small interlayer spacing to construct a training set, divide CT cross-sectional slice images with large layer thickness into a test set and a validation set, construct high-low resolution CT cross-sectional slice image pairs by bicubic interpolation method, and divide the continuous CT cross-sectional slice images into triplets in the interlayer frame interpolation network, and the middle frame image is compared with the output image as the true image; S2, construct a three-dimensional fusion super-resolution network in and between layers, which is specifically divided into an intra-layer super-resolution network structure and an inter-layer super-resolution network structure; S3, train the intra-layer and inter-layer three-dimensional fusion super-resolution network model constructed in S2; S4, input the test set into the intra-layer and inter-layer three-dimensional fusion super-resolution network model trained in S3, and output the dense sequence of high-resolution CT cross-sectional slice images for three-dimensional reconstruction; In S1, the high-resolution CT cross-section slice image is processed by using a bicubic interpolation algorithm to obtain a corresponding low-resolution image ;​ The construction of the three-dimensional fusion super-resolution network in and between layers in S2 comprises the following steps: S2.1, improve the spatial resolution of each cross-sectional slice by intra-layer super-resolution algorithm; S2.2, generate a high-resolution intermediate position slice by interlayer frame interpolation algorithm on adjacent two slices to obtain a 2N-1 slice sequence with dense interlayer spacing; The S2 is obtained based on the following formula: where x and y represent the horizontal and vertical coordinates of a pixel, and N represents the number of slices of a CT image sequence; denotes the generated low-resolution cross-sectional image, f SR and f inter denote the intra-slice super-resolution algorithm and the inter-slice super-resolution algorithm, respectively, denotes the generated high-resolution cross-sectional image, and z represents the number of layers of the high-resolution cross-sectional image, denotes the high-resolution cross-sectional image of the Z+0.5th layer, denotes the high-resolution cross-sectional image of the Z+1th layer.

2. The three-dimensional super-resolution method suitable for CT image sequences according to claim 1, characterized in that, The construction of the intra-layer super-resolution network structure in S2.1 comprises the following steps: S2.1.1, feature extraction, low resolution image extract features through the shallow channel reconstruction network; S2.1.2, alignment and depth fusion, multi-scale feature progressive alignment and depth fusion are performed through multi-scale attention and cascaded progressive fusion module; S2.1.3, construct a super-resolution image, fuse channel reconstruction information based on ESPCN and up-sampled image based on the original image.

3. The three-dimensional super-resolution method for CT image sequences according to claim 2, characterized in that: The multi-scale attention module in the alignment and depth fusion in S2.1.2 aggregates local information through strip convolution, that is, two deep strip convolutions are used instead of standard deep convolution operation in each branch, and then 1x1 convolution is used to model the relationship between multi-scale features, and S-ESA is used to improve the effective use of features.

4. The three-dimensional super-resolution method for CT image sequences according to claim 3, characterized in that: The multi-scale attention module in the alignment and depth fusion in S2.1.2 is specifically designed to fully refine effective information, and a cascaded 1x1 convolution structure is used to progressively align local detail features and global semantic features to fully aggregate multi-scale feature information.

5. The three-dimensional super-resolution method for CT image sequences according to claim 1, characterized in that, The inter-layer super-resolution network combines the context feature extraction, optical flow estimation and inter-frame synthesis modules into a nested U-shaped structure, and the inter-frame iterative optimization module IIR of the nested U-shaped structure is used to realize repeated iteration of feature information, and the spatial channel attention mechanism module SCA is designed in the inter-frame iterative optimization module IIR, which gradually expands the spatial receptive field and gradually increases the channel number to better extract the correlation of inter-frame feature information.

6. The three-dimensional super-resolution method suitable for CT image sequences according to claim 1, wherein, The inter-layer super-resolution network designs an inter-frame optimization function based on spatial information, combines the refined bidirectional optical flow information and context features, and generates the final interpolated image through the following formula: ， In the formula, w(·) represents a global bidirectional warping function; I z , I z+1 is an input frame; the light flow information direction of the 0th layer is from the zth frame to the z+0.5th frame, the light flow information direction of the 0th layer is from the z+0.5th frame to the z+1th frame; and is a unidirectional intermediate frame image generated by a global bidirectional warping function; M0 and M1 are output features D0 after passing through a Sigmoid function to generate a spatial mask with a value range from 0 to 1, and the spatial masks M0 and M1 are respectively multiplied with the one-way intermediate frame images , to obtain a bidirectional intermediate frame. R is a residual feature image extracted from D0 to compensate for texture details; I out representing an intermediate frame image of the output.

7. The three-dimensional super-resolution method for CT image sequences according to claim 6, characterized in that, The inter-layer super-resolution network further comprises a global bidirectional warping function, which combines a forward warping function and a backward warping function and considers the mapping relationship of global source pixels in the bidirectional warping function, wherein the forward warping function calculation model is as follows: In the formula, q is a source pixel point coordinate; Estimate for backward function; u is a difference value between a prediction value and an estimated value based on a backward function; u x represents the offset value of u in the horizontal direction of the coordinate; u y represents the offset value of u in the ordinate direction; b(u) is a weight coefficient; target pixel of the reconstructed intermediate frame; is the original frame; F z→z+0.5 is the optical flow information for the direction from the z-th frame to the z+0.5-th frame; exp denotes a Softmax operation on the weights K, and the weights K of the individual pixels are obtained from the values of the corresponding pixels between the intermediate frame generated based on Average splatting and the original frame by a Charbonnier Loss calculation between the values of the corresponding pixels.

8. The three-dimensional super-resolution method for CT image sequences according to claim 1, characterized in that: The specific process of the step S3 is as follows: Step S3.1: first, define a loss function by using a mean absolute error function as the loss function; Step S3.2: Train two models, respectively, intra-layer super-resolution network and inter-layer frame interpolation network; the above algorithm is based on windows 10 operating system, using pytorch deep learning framework; hardware configuration is CPU internal (R) core (TM) i7-6500u @ 2.50ghz, 8g memory, GPU is NVIDIA GTX 1080, CUDA 11.4 + cudnn 11.3 for GPU acceleration; using Adam optimization algorithm, setting β1 as 0.9, β2 as 0.999, momentum parameter as 0.9, batch size as 16, iteration 300 epochs; the learning rate is initially set to 1×10 −4 , and gradually decays to 1×10 −5 according to the cosine function.

Images