Ultra-low dose systemic medical image generation method and system based on deformable convolution

Abstract

The invention relates to an ultra-low dose whole body medical image generation method and system based on deformable convolution, and the method comprises the steps: S1, obtaining 3D whole body PET image data, and carrying out the preprocessing of the 3D whole body PET image data, and the preprocessing comprises data enhancement and data division; s2, inputting the preprocessed 3D whole body PET image into a three-dimensional deformable convolutional depth attention network, namely a Deformable 3D U-Net module, and carrying out the processing of the 3D whole body PET image; and S3, an ultra-low dose systemic medical image is generated through a Deformable 3D U-Net module. Compared with the prior art, the pediatric whole body PET imaging method can be used for enhancing pediatric whole body PET imaging without anesthesia, the result accuracy is high, and the model has high robustness.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging engineering technology, and in particular to a method and system for generating ultra-low-dose whole-body medical images based on deformable convolution. Background Art

[0002] [18F]fluorodeoxyglucose positron emission tomography ([18F]FDG PET) imaging has been widely used in clinical practice and research for decades. Recently, the use of PET in pediatric disease has attracted widespread attention and has been shown to be a useful diagnostic tool in the evaluation of pediatric oncological and non-oncological diseases. [18F]FDG PET has been used to diagnose pediatric solid tumors, hematological tumors, post-transplant lymphoproliferative disorder (PTLD), and fever of unexplained origin (FUO). It is routinely used for staging, assessment of treatment response, monitoring of disease progression, and detection of recurrence in a variety of malignancies.

[0003] However, the pediatric PET examination procedure is more complex than adult PET imaging. Immobilization of pediatric patients can be achieved using body wraps or specialized immobilization devices. Parents are often allowed to accompany their children throughout the study to provide emotional support. In pediatric imaging, it is critical to remain still because even small movements may result in artifacts. Anesthesia is often recommended to suppress motion during the examination to ensure image quality. However, anesthesia also has some disadvantages: (1) it may affect the child's neurodevelopment; (2) despite the use of anesthesia, some children may not cooperate or have involuntary movements during the examination; and (3) the potential effect of anesthesia on the biodistribution of the FDG tracer must be considered. Therefore, research is increasingly focused on accelerating PET scanning and accurate diagnosis without anesthesia.

[0004] Various approaches have been used to compensate or correct motion in PET imaging, such as the use of external devices that successfully detect motion. However, these devices often have limitations in clinical application, including limited and complex integration with scanners, additional patient setup time, patient discomfort, and relatively high failure rates in some cases. Another approach is to directly extract motion information from PET data by reconstructing short-duration PET image sequences. It is worth noting that a limitation of current data-driven PET methods is their reliance on CT images for attenuation and scatter correction.

[0005] Ultrafast whole-body PET imaging offers a balance between acquisition time and image reconstruction quality. Whole-body PET / CT is able to acquire PET data for the entire human body simultaneously on a single bed. The scanner has a superior sensitivity of 174 kcps / MBq and has acceptable spatial resolution capabilities in human imaging (≤3.0 mm FWHM near the center of the AFOV). As a result, the sensitivity of PET scanners has been significantly improved and the acquisition time has been significantly reduced. Although there are significant visual differences between whole-body PET images with 30-second and 300-second acquisition times, they do not differ substantially in diagnostic accuracy.

[0006] Deep learning-based frameworks have emerged as powerful tools that provide solutions for various tasks in medical imaging, including disease diagnosis, tumor segmentation, and post-image reconstruction. Several generative models, such as using convolutional neural networks (CNNs) or generative adversarial networks (GANs), have attracted the attention of researchers due to their superior performance in improving PET image quality over traditional methods. However, most deep learning-based methods focus on brain PET images rather than pediatric whole-body PET data. In addition, popular GAN-based frameworks often suffer from instability and model collapse during training, as obtaining Nash equilibrium points is challenging and the training process of GANs is complicated.

[0007] The challenges faced by existing technologies are mainly as follows: 1. Pediatric patients need to remain still through special fixation devices or anesthesia to avoid artifacts caused by slight movements. However, anesthesia may affect children's neurological development, and there are problems with complex operation and unstable effects. 2. Existing motion compensation methods (such as using external devices to detect motion) face complex integration problems, additional setup time, and patient discomfort in clinical applications. Data-driven motion correction methods rely on CT images and have certain limitations. 3. In ultra-fast whole-body PET imaging, high-quality images need to be maintained in an extremely short acquisition time. Although there are visual differences between 30-second and 300-second whole-body PET images, there is no substantial difference in diagnostic accuracy. How to further shorten the acquisition time remains a challenge. 4. Image reconstruction methods based on deep learning (such as GAN) are superior in improving image quality, but the instability and model collapse problems in the training process still need to be solved, especially when processing pediatric whole-body PET data. 5. It is necessary to develop technologies that can achieve high-quality images without anesthesia to reduce the adverse effects of anesthesia on children while ensuring the accuracy and reliability of diagnosis. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an ultra-low-dose whole-body medical image generation method and system based on deformable convolution, which can be used to enhance whole-body PET imaging of children without anesthesia, with high accuracy and high robustness of the model.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A first aspect of the present invention provides a method for generating ultra-low-dose whole-body medical images based on deformable convolution, comprising the following steps:

[0011] S1. Acquire 3D whole-body PET image data, and preprocess the 3D whole-body PET image data, wherein the preprocessing includes data enhancement and data segmentation;

[0012] S2, input the preprocessed 3D whole body PET image into the three-dimensional deformable convolutional deep attention network, namely the Deformable 3D U-Net module for processing;

[0013] S3. Generate ultra-low-dose whole-body medical images through the Deformable 3D U-Net module.

[0014] Furthermore, S1 specifically includes the following steps:

[0015] S1.1. Data enhancement, 3D whole-body PET image data, split the 3D PET image dataset into individual PET image frames, and expand the dataset by rotating, adjusting contrast, and scaling, while normalizing the image size;

[0016] S1.2. Data division: The enhanced image data is divided into a training set and a test set, and both the training set and the test set contain complete 3D whole-body PET images.

[0017] Furthermore, the Deformable 3D U-Net module includes a spatially deformable aggregation module SDAB and a multi-scale self-attention module MSSAM;

[0018] The spatial deformable aggregation module SDAB includes a U-Net network and a 3D deformable convolution layer;

[0019] The spatial deformable aggregation module SDAB works by receiving input features and processing them through 3D convolutional layers, which contain activation functions; applying 3D deformable convolutional layers with activations to extract additional spatial information for motion compensation; adding 3D convolutional layers in parallel to achieve residual projection and enhance the expression capability of SDAB;

[0020] The multi-scale self-attention module MSSAM comprises an input unit, a feature sampling and transformation unit, a multi-scale feature processing unit, a self-attention mechanism unit and a feature fusion unit;

[0021] The input unit receives input data, performs preliminary feature extraction, and generates a feature map;

[0022] The feature sampling and transformation unit samples and transforms the shape of the feature map to generate a multi-scale feature map;

[0023] The multi-scale feature processing unit performs a convolution operation on the multi-scale feature map through a filtering unit to extract multi-scale feature information;

[0024] The self-attention mechanism unit calculates the attention weight in the feature map, generates a self-attention feature map, and ensures the effectiveness and stability of the attention feature through normalization processing;

[0025] The feature fusion unit adds the multi-scale feature map to the self-attention feature map to generate the final output feature.

[0026] Furthermore, S2 specifically includes the following steps:

[0027] S2.1, input the preprocessed 3D whole body PET image to the Deformable 3D U-Net module;

[0028] S2.2, feature extraction and fusion are performed through the spatial deformable aggregation module SDAB and the multi-scale self-attention module MSSAM;

[0029] S2.3. Output the feature map processed by the Deformable 3D U-Net module.

[0030] Furthermore, in S2.2, the processing of the spatial deformable aggregation module SDAB and the multi-scale self-attention module MSSAM includes:

[0031] S2.2.1, the spatial deformable aggregation module SDAB receives the input image in S2.1, extracts spatial information through the U-Net network and the 3D deformable convolution layer, and performs motion compensation;

[0032] S2.2.2, the multi-scale self-attention module MSSAM receives the output of the spatial deformable aggregation module SDAB module, and performs feature fusion through input units, feature sampling and transformation, multi-scale feature processing, and self-attention mechanism;

[0033] S2.2.3. Multi-scale self-attention module MSSAM outputs the fused multi-scale feature map.

[0034] Furthermore, S3 specifically includes the following steps:

[0035] S3.1. Use mean square error as the network loss function to calculate the error between the output of the Deformable 3D U-Net module and the actual image.

[0036] S3.2. Use the standard Adam optimizer to perform gradient descent update on the weight parameters of the Deformable 3D U-Net module according to the error in S3.1.

[0037] A second aspect of the present invention provides an ultra-low-dose whole-body medical image generation system based on deformable convolution, the system comprising:

[0038] 3D whole-body PET image data preprocessing module, used to split, normalize, enhance and partition the original 3D PET image data to generate training sets and test sets;

[0039] A three-dimensional deformable convolutional deep attention network Deformable 3D U-Net module, which includes a spatial deformable aggregation module SDAB and a multi-scale self-attention module MSSAM, is used to process pre-processed 3D whole-body PET images and generate ultra-low-dose whole-body medical images.

[0040] Furthermore, the spatially deformable aggregation module SDAB comprises:

[0041] U-Net network, used to extract input features;

[0042] 3D deformable convolutional layers to extract additional spatial information for motion compensation;

[0043] Parallel 3D convolutional layers are used to implement residual projection and enhance the expressiveness of the spatially deformable aggregation module (SDAB).

[0044] Furthermore, the multi-scale self-attention module MSSAM comprises:

[0045] An input unit, used for receiving input data and performing preliminary feature extraction;

[0046] Feature sampling and transformation unit, used to generate multi-scale feature maps;

[0047] A multi-scale feature processing unit, used to extract multi-scale feature information;

[0048] Self-attention mechanism unit, which is used to calculate the attention weights in the feature map and generate the self-attention feature map;

[0049] The feature fusion unit is used to fuse the multi-scale features with the self-attention features to generate the final output features.

[0050] Furthermore, the 3D whole body PET image data preprocessing module further includes:

[0051] A data augmentation unit that augments the dataset by rotating, adjusting contrast, and scaling;

[0052] The data division unit is used to divide the enhanced image data into a training set and a test set, wherein the training set and the test set both contain complete 3D whole-body PET images.

[0053] Compared with the prior art, the present invention has the following technical advantages:

[0054] 1. Adaptive convolution layer: The deformable convolution layer in the model used in the present invention can adaptively modify the shape of the convolution kernel. Due to the high variability of anatomical structures, this feature is crucial for synthesizing whole-body PET images.

[0055] 2. Enhanced feature capture: Deformable convolutions are better at capturing complex spatial relationships and morphological changes than standard convolutional layers, and therefore extract more accurate and robust features than standard convolutional layers.

[0056] 3. Superior performance on quantitative metrics: Compared with the standard 3D U-Net, the deformable 3D U-Net performs significantly better on quantitative metrics with significant anatomical variation between different patients.

[0057] 4. Realize network training with small data volume: The present invention uses image enhancement to increase the sample size, and on this basis, trains the model and tests the data, thereby avoiding overfitting of network training and improving the robustness of network training.

[0058] 5. Drastically shortened examination time: Pediatric patients can successfully complete whole-body PET examinations without sedation. The synthetic PET images based on 6-second scans generated by this method can meet the needs of clinical diagnosis of physical diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a structural block diagram of a system for generating ultra-low-dose whole-body [18F]FDG PET images by deformable convolution 3D-Uet based on deep learning according to an embodiment of the present invention;

[0060] Figure 2 This is a flowchart of an implementation of a system for generating ultra-low-dose whole-body [18F]FDG PET images by deformable convolution 3D-Uet based on deep learning according to an embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of D3DU-Net construction according to an embodiment of the present invention;

[0062] Figure 4 It is a D3DU-Net network framework of an embodiment of the present invention;

[0063] Figure 5 This is a diagram of the PET image generation result of D3DU-Net according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] Innovatively, the present invention discloses an ultra-low-dose whole-body medical image generation system based on deformable convolution, which pre-processes data through a 3D whole-body PET data pre-processing module, and transmits the processed data to a deep attention network Deformable 3D U-Net module based on deformable convolution, including using a spatial deformable aggregation block (SDAB) and a multi-scale self-attention module (MSSAM). The former adaptively extracts spatial information of a larger receptive field through residual projection in the encoding path, and reduces image noise and reconstructs textures and boundaries in the decoding path; the latter uses multi-scale self-similar priors to capture long-range dependencies and break through scale constraints to generate richer and more realistic details. The system can effectively improve the accuracy and efficiency of ultra-low-dose PET image generation by introducing spatial deformable aggregation, deformable convolution, and summing multi-scale self-attention, and has high robustness.

[0065] The present invention provides an ultra-low-dose whole-body medical image generation system based on deformable convolution to enhance pediatric PET imaging without anesthesia (acquisition time is about 6 seconds, which shortens the scanning time by more than 50 times compared to standard clinical routine).

[0066] The image generation system includes a 3D whole-body PET image data preprocessing module and a three-dimensional deformable convolutional deep attention network Deformable 3D U-Net module;

[0067] The 3D whole-body PET image data preprocessing module includes a data enhancement module and a data division module;

[0068] The data enhancement module: splits the existing 3D PET image data set into individual PET image frames, expands the data set and normalizes the image size;

[0069] The data division module divides the enhanced image data into a training set and a test set; both the training set and the test set contain complete 3D whole-body PET images.

[0070] The deformable convolutional deep attention network Deformable 3D U-Net module includes a spatial deformable aggregation module SDAB and a multi-scale self-attention module MSSAM;

[0071] The spatial deformable aggregation module SDAB: This module contains a U-Net network and a 3D deformable convolution layer; SDAB feeds the input features into a 3D convolution layer with LeakyRelu activation. Then, a 3D deformable convolution layer with LeakyRelu activation is applied to extract additional spatial information for motion compensation. Finally, a parallel 1×1 3D convolution layer is added to achieve residual projection, thereby enhancing the expressive power of SDAB;

[0072] The multi-scale self-attention module MSSAM: This module includes an input unit, feature sampling and transformation, multi-scale feature processing, a self-attention mechanism, and feature fusion. The multi-scale self-attention module MSSAM first receives input data x and y through the input unit, performs preliminary feature extraction, and generates feature maps f(x), g(y), and θ(y). Next, these feature maps are sampled and transformed in shape to generate multi-scale feature maps. These multi-scale feature maps are convolved through the filtering unit to extract multi-scale feature information. Then, the attention weights in the feature map are calculated by the self-attention calculation unit to generate a self-attention feature map, and then the Softmax function is used to normalize the attention weights to ensure the effectiveness and stability of the attention feature. Next, the multi-scale features are fused with the self-attention features, and the multi-scale feature map and the self-attention feature map are added through the feature fusion unit to generate the final output feature z;

[0073] The data enhancement module expands the data set by rotating, adjusting contrast and scaling during image processing.

[0074] Before each frame of 3D whole-body PET image is input into the Deformable 3D U-Net network, a 3D sliding patch strategy is adopted to divide each 3D whole-body PET image into multiple overlapping 3D image patches, achieving resource-efficient reconstruction on a single GPU.

[0075] The spatial deformable aggregation module SDAB includes a total of 9 layers of structures. The first layer includes a convolution layer and a Relu activation function; the second to fourth layers are downsampling structures, each of which includes two convolution layers and two Relu activation functions; the fifth to sixth layers are upsampling structures, including a convolution layer, a deconvolution layer and two Relu activation functions; the seventh layer is a jump transfer structure, which processes the features obtained by downsampling and fuses them with the upsampling results, including two convolution layers, a deconvolution layer and three Relu activation functions. The eighth layer is an offset output structure, including two convolution layers and a Relu activation function; the ninth layer includes a deformable convolution and a Relu activation function, and takes the input image and offset as input to obtain the high-dimensional feature fused feature maps of the image.

[0076] In the spatial deformable aggregation module SDAB, the calculation process of the convolution layer is:

[0077]

[0078] where conv out Output image size for the convolutional layer, conv in is the input image size, padding means padding pixels around the image, kernel_Size is the convolution kernel size, and stride is the stride of the convolution kernel;

[0079] The 3×3×3 convolution kernel G is defined as: G = {(-1,-1,-1), (-1,-1,0), …, (1,1,0), (1,1,1)}, and the feature y(p0) of the convolution layer is:

[0080]

[0081] where p n is the value of the adjacent position in the 3D product sampling grid, w(·) is the weight function, x(·) is the feature of the input image, p0 is the initial position, and N is the size of the sampling grid;

[0082] In the deformable convolution layer, the convolution kernel R is offset {Δp n |n=1,...,N} enhancement, where N=|R|; thus, the feature y(p0) of the deformable convolution is:

[0083]

[0084] Where Δp n Represents the nth convolution sampling grid in the 3×3×3 convolution th The learnable offset corresponding to the value can be generated by parallel 3D convolutional layers of the input feature x. Considering the offset Δpn Typically floating point numbers, with bilinear interpolation producing exact values ​​in fractional places.

[0085] Furthermore, in the spatially deformable aggregation module SDAB, the small and large inputs of MSSAM are defined as x(H×W)a and Where H and W are the height and width of the input feature respectively. Therefore, the expression of MSSAM is:

[0086]

[0087] where i and j represent the indices on the input features x and y, respectively, and outputz.θ(·) represents the feature transformation function, i.e., a simple linear embedding θ(y j )=W θ y j , to generate y j . σ(·) is a scalar function used to normalize all summation locations. φ(·) represents the embedded Gaussian function, which is used to calculate the pairwise association between the input features x and y:

[0088]

[0089] f(x i )=W f x i ,g(y j )=W g y j

[0090] The proposed MSSAM captures more information of long-range relationships and different spatial sizes at multiple scales.

[0091] The Deformable 3D U-Net module uses mean square error (MSE) as the loss function of the network during training. The mean square error (MSE) calculation formula is:

[0092]

[0093] Where n is the number of samples, y i Represents the i-th actual value. represents the i-th predicted value;

[0094] The weight parameter θ of the Deformable 3D U-Net module is updated using the standard Adam optimizer gradient descent, and the formula is:

[0095]

[0096] Where η is the learning rate, θ kis the k-th weight parameter.

[0097] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Component models, material names, connection structures, control methods, algorithms and other features not clearly described in this technical solution are all considered to be common technical features disclosed in the prior art.

[0098] Example 1

[0099] This embodiment provides a method for generating ultra-low-dose whole-body medical images based on deformable convolution. Figure 2 , including the following steps:

[0100] S1. Acquire 3D whole-body PET image data, and preprocess the 3D whole-body PET image data, wherein the preprocessing includes data enhancement and data segmentation;

[0101] In the specific implementation, S1 specifically includes the following steps:

[0102] S1.1. Data enhancement, 3D whole-body PET image data, split the 3D PET image dataset into individual PET image frames, and expand the dataset by rotating, adjusting contrast, and scaling, while normalizing the image size;

[0103] S1.2. Data division: The enhanced image data is divided into a training set and a test set, and both the training set and the test set contain complete 3D whole-body PET images.

[0104] In a specific implementation, the Deformable 3D U-Net module includes a spatially deformable aggregation module SDAB and a multi-scale self-attention module MSSAM;

[0105] The spatial deformable aggregation module SDAB includes a U-Net network and a 3D deformable convolution layer;

[0106] The spatial deformable aggregation module SDAB works by receiving input features and processing them through 3D convolutional layers, which contain activation functions; applying 3D deformable convolutional layers with activations to extract additional spatial information for motion compensation; adding 3D convolutional layers in parallel to achieve residual projection and enhance the expression capability of SDAB;

[0107] The multi-scale self-attention module MSSAM comprises an input unit, a feature sampling and transformation unit, a multi-scale feature processing unit, a self-attention mechanism unit and a feature fusion unit;

[0108] The input unit receives input data, performs preliminary feature extraction, and generates a feature map;

[0109] The feature sampling and transformation unit samples and transforms the shape of the feature map to generate a multi-scale feature map;

[0110] The multi-scale feature processing unit performs a convolution operation on the multi-scale feature map through a filtering unit to extract multi-scale feature information;

[0111] The self-attention mechanism unit calculates the attention weight in the feature map, generates a self-attention feature map, and ensures the effectiveness and stability of the attention feature through normalization processing;

[0112] The feature fusion unit adds the multi-scale feature map to the self-attention feature map to generate the final output feature.

[0113] S2, input the preprocessed 3D whole body PET image into the three-dimensional deformable convolutional deep attention network, namely the Deformable 3D U-Net module for processing;

[0114] In the specific implementation, S2 specifically includes the following steps:

[0115] S2.1, input the preprocessed 3D whole body PET image to the Deformable 3D U-Net module;

[0116] S2.2, feature extraction and fusion are performed through the spatial deformable aggregation module SDAB and the multi-scale self-attention module MSSAM;

[0117] S2.3. Output the feature map processed by the Deformable 3D U-Net module.

[0118] In specific implementation, in S2.2, the processing process of the spatial deformable aggregation module SDAB and the multi-scale self-attention module MSSAM includes:

[0119] S2.2.1, the spatial deformable aggregation module SDAB receives the input image in S2.1, extracts spatial information through the U-Net network and the 3D deformable convolution layer, and performs motion compensation;

[0120] S2.2.2, the multi-scale self-attention module MSSAM receives the output of the spatial deformable aggregation module SDAB module, and performs feature fusion through input units, feature sampling and transformation, multi-scale feature processing, and self-attention mechanism;

[0121] S2.2.3. Multi-scale self-attention module MSSAM outputs the fused multi-scale feature map.

[0122] S3. Generate ultra-low-dose whole-body medical images through the Deformable 3D U-Net module.

[0123] In the specific implementation, S3 specifically includes the following steps:

[0124] S3.1. Use mean square error as the network loss function to calculate the error between the output of the Deformable 3D U-Net module and the actual image.

[0125] S3.2. Use the standard Adam optimizer to perform gradient descent update on the weight parameters of the Deformable 3D U-Net module according to the error in S3.1.

[0126] like Figure 1 As shown, the ultra-low dose systemic [ 18 The F]FDG PET image deformable convolution 3D-Uet generation system includes a 3D whole-body PET image data preprocessing module and a three-dimensional deformable convolution deep attention network Deformable 3DU-Net module.

[0127] The 3D whole-body PET image data preprocessing module includes a data enhancement module and a data division module;

[0128] The data enhancement module: splits the existing 3D PET image data set into individual PET image frames, expands the data set and normalizes the image size;

[0129] The data division module divides the enhanced image data into a training set and a test set; both the training set and the test set contain complete 3D whole-body PET images.

[0130] like Figure 3 and Figure 4 As shown, the three-dimensional deformable convolutional deep attention network Deformable 3D U-Net module (D3DU-Net) includes a spatial deformable aggregation module SDAB and a multi-scale self-attention module MSSAM;

[0131] The spatial deformable aggregation module SDAB includes a total of 9 layers of structures. The first layer includes a convolution layer and a Relu activation function; the second to fourth layers are downsampling structures, each of which includes two convolution layers and two Relu activation functions; the fifth to sixth layers are upsampling structures, including a convolution layer, a deconvolution layer and two Relu activation functions; the seventh layer is a jump transfer structure, which processes the features obtained by downsampling and fuses them with the upsampling results, including two convolution layers, a deconvolution layer and three Relu activation functions. The eighth layer is an offset output structure, including two convolution layers and a Relu activation function; the ninth layer includes a deformable convolution and a Relu activation function, and takes the input image and offset as input to obtain the high-dimensional feature fused feature maps of the image.

[0132] In specific implementation, in the spatial deformable aggregation module SDAB, the calculation process of the convolution layer is:

[0133]

[0134] where conv out Output image size for the convolutional layer, conv in is the input image size, padding means padding pixels around the image, kernel_Size is the convolution kernel size, and stride is the stride of the convolution kernel;

[0135] The 3×3×3 convolution kernel G is defined as: G = {(-1,-1,-1), (-1,-1,0), …, (1,1,0), (1,1,1)}, and the feature y(p0) of the convolution layer is:

[0136]

[0137] where p n is the value of the adjacent position in the 3D product sampling grid, w(·) is the weight function, x(·) is the feature of the input image, p0 is the initial position, and N is the size of the sampling grid;

[0138] In the deformable convolution layer, the convolution kernel R is offset {Δp n |n=1,...,N} enhancement, where N=|R|; thus, the feature y(p0) of the deformable convolution is:

[0139]

[0140] Where Δp n Represents the nth convolution sampling grid in the 3×3×3 convolution th The learnable offset corresponding to the value can be generated by parallel 3D convolutional layers of the input feature x. Considering the offset Δpn Typically floating point numbers, with bilinear interpolation producing exact values ​​in fractional places.

[0141] The small and large inputs of MSSAM are defined as x(H×W) and Where H and W are the height and width of the input feature respectively. Therefore, the expression of MSSAM is:

[0142]

[0143] where i and j represent the indices on the input features x and y, respectively, and θ(·) represents the feature transformation function, i.e., a simple linear embedding θ(y j )=W θ y j , to generate y j . σ(·) is a scalar function used to normalize all summation locations. φ(·) represents the embedded Gaussian function, which is used to calculate the pairwise association between input features x and y:

[0144]

[0145] f(x i )=W f x i ,g(y j )=W g y j

[0146] The proposed MSSAM captures more information of long-range relationships and different spatial sizes at multiple scales.

[0147] The activation functions used in MSSAM are all linear rectifier units except the last layer, and the activation function of the last layer is linear activation. The calculation method of the linear rectifier unit g(z)′ is:

[0148]

[0149] The linear activation function g(z) is calculated as:

[0150] g(z)=z

[0151] The input data in MSSAM is B×(2r+1)×3×H×W, where B is the batch size, 2r+1 is the number of input MRI image frames, 3 is the number of image channels, H is the image height, and W is the image width. In this embodiment, the input MRI image size is 12×3×3×256×256, and the size changes to 12×32×256×256 after the first layer of structure. After three downsamplings from the 2nd to the 4th layers, the data sizes obtained are 12×32×128×128, 12×32×64×64, and 12×32×32×32, respectively. After two upsamplings from the 5th to the 6th layers, the image feature sizes obtained are 12×32×64×64 and 12×32×128×128, respectively. The image feature size obtained after the 7th layer jump transfer structure is 12×32×128×128, which is first merged with the upsampled input and then passed into the upsampling structure. The 8th layer is an offset output structure, and the image feature size obtained is 12×54×256×256. The 9th layer of spatiotemporal deformable convolution structure obtains fused feature maps with an image feature size of 12×64×256×256.

[0152] In specific implementation, the spatial deformable aggregation module SDAB includes a U-Net network, a 3D deformable convolutional attention module, three summed attention modules (Attention Gates module) and an output layer. The deformable convolutional attention module is based on the spatial attention module. The input of the spatial attention module passes through a deformable convolutional layer, and the calculated output of the deformable convolutional layer is added to the output of the spatial attention module to finally obtain the output of the deformable convolutional attention module; the deformable convolutional attention module is added to the first layer skip connection of the U-Net network, and the summed attention module (Attention Gates module) is added to the remaining three layers of skip connection; the high-dimensional feature fused feature maps of the compensated image are input into the U-Net network, the number of input channels is changed to 64, and the image size remains unchanged.

[0153] Then, after 4 downsamplings, each downsampling operation includes a convolution, using a 3×3×3 convolution kernel and transforming the number of channels to double the original number, so that the number of channels of the image changes to 128, 256, 512, and 1024. After each convolution, the nonlinear features are obtained by the Relu activation function. A maximum pooling operation with a pooling kernel of 2×2×2 changes the image size to half of the original size, that is, to 128×128×128, 64×64×64, 32×32×32, and 16×16×16.

[0154] Then, after 4 upsamplings, each upsampling operation includes a convolution, using a 3×3×3 convolution kernel and changing the number of channels to half of the original, so that the number of channels of the image changes to 1024, 512, 256, and 128. After each convolution, the Relu activation function is used to obtain nonlinear features. The image size is doubled by linear interpolation, that is, it becomes 32×32×32, 64×64×64, 128×128×128, and 256×256×256.

[0155] The input also enters the DGPA module to extract global pixel-related features and is concatenated with the output of the DGPA network. The results of the first three downsamplings enter the summed attention network to suppress irrelevant background and are then concatenated with the results of the first three upsamplings.

[0156] In the deep deformable convolution global attention module DGPA, a deformable convolution global attention network is used to extract global pixel-related features. The high-dimensional features I of the compensated MRI image are input into a 3×3 deformable convolution kernel in the deformable convolution attention module. The output O is:

[0157]

[0158] The high-dimensional feature I of the compensated MRI image is input into three 1×1 convolution kernels in the spatial attention module to generate a new feature map Where N is the number of channels of the feature map, M = H × W, that is, the number of pixels of the feature map, and H and W are the height and width of the feature map respectively; perform matrix multiplication on the transposed C and B, and use the softmax formula to obtain the spatial attention map Each element of S is calculated as follows:

[0159]

[0160] In the formula, s ji is the element in row i and column j in S, B i is the i-th row of feature map B, The jth column of the transposed feature map C is used to perform matrix multiplication of the spatial attention map S and the feature map D, and the calculation result is added to the result O of the previous deformable convolution to obtain the final result.

[0161]

[0162] Where α is the weight coefficient, D i is the i-th row of feature map D, O j is the j-th column of output O.

[0163] In the deep deformable convolutional global attention module DGPA, the sum attention module is used to suppress irrelevant background; G is a feature map with a dimension of N×W×H in the downsampling stage, and X is a feature map with a dimension of N×W×H in the upsampling stage. Both are changed to a dimension of F after passing through a 1×1 convolution kernel. int ×W×H, where F int is the dimension parameter in the pre-set U-Net network; the two matrices are added point by point and then passed through a Relu activation layer, the result is passed through a 1×1 convolution kernel and the dimension is changed to 1×W×H, and then the weight coefficient α is obtained through the sigmoid function; the input X is multiplied by the weight coefficient α to obtain the feature map with attention. The output result can be normalized between 0 and 1, and the size of the value represents the size of the correlation between this point and the recognition result. The larger the value, the more likely it is that the object to be recognized is contained here. This suppresses irrelevant areas and improves the accuracy of image recognition.

[0164] During the training process, the mean square error (MSE) is used as the loss function of the network. The mean square error (MSE) calculation formula is:

[0165]

[0166] Where n is the number of samples, y i Represents the i-th actual value. represents the i-th predicted value;

[0167] The weight parameter θ of the Deformable 3D U-Net module is updated using the standard Adam optimizer gradient descent, and the formula is:

[0168]

[0169] Where η is the learning rate, θ k is the k-th weight parameter.

[0170] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0171] In addition, those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the embodiments provided by the present invention, it should be understood that the disclosed system can be implemented in other ways. The division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0172] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional units.

[0174] Application Example 1

[0175] In a specific case where the system of this embodiment is applied, Figure 5 As shown in the figure, the collected 3D whole-body PET image dataset is first divided into a training set and a test set. The spatial deformable aggregation module SDAB is constructed using the U-Net network to obtain the offset of the target image in the 3D whole-body PET image. The offset is fused into the target image through deformable convolution, and then it is input into the multi-scale self-attention module MSSAM to extract the global pixel-related features. Finally, the generated result image is obtained, thereby realizing ultra-fast PET reconstruction. Finally, by comparing the ultra-fast PET images with the corresponding synthetic PET images at different acquisition times, the proposed deformable 3D U-Net shows significant improvements in improving PSNR (by 23.06%, 27.33%, 18.08% and 11.65%, respectively), SSIM (by 2.48%, 1.66%, 1.53% and 1.64%, respectively) and FSIM (by 2.57%, 1.87%, 1.31% and 1.62%, respectively). These improvements are statistically significant in the t-test (p<0.01). Maintain high accuracy and efficiency.

[0176] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. A method for generating ultra-low-dose whole-body medical images based on deformable convolution, characterized in that: The following steps are involved: S1. Acquire 3D whole-body PET image data, and preprocess the 3D whole-body PET image data, wherein the preprocessing includes data enhancement and data segmentation; S2, input the preprocessed 3D whole body PET image into the three-dimensional deformable convolutional deep attention network, namely the Deformable 3D U-Net module for processing; S3. Generate ultra-low-dose whole-body medical images through the Deformable 3D U-Net module.

2. The method for generating ultra-low-dose whole-body medical images based on deformable convolution according to claim 1, characterized in that: S1 specifically includes the following steps: S1.

1. Data enhancement, 3D whole-body PET image data, split the 3D PET image dataset into individual PET image frames, and expand the dataset by rotating, adjusting contrast, and scaling, while normalizing the image size; S1.

2. Data division: The enhanced image data is divided into a training set and a test set, and both the training set and the test set contain complete 3D whole-body PET images.

3. The method for generating ultra-low-dose whole-body medical images based on deformable convolution according to claim 1, characterized in that: The Deformable 3D U-Net module includes a spatially deformable aggregation module SDAB and a multi-scale self-attention module MSSAM; The spatial deformable aggregation module SDAB includes a U-Net network and a 3D deformable convolution layer; The spatial deformable aggregation module SDAB works by receiving input features and processing them through 3D convolutional layers, which contain activation functions; applying 3D deformable convolutional layers with activations to extract additional spatial information for motion compensation; adding 3D convolutional layers in parallel to achieve residual projection and enhance the expression capability of SDAB; The multi-scale self-attention module MSSAM includes an input unit, a feature sampling and transformation unit, a multi-scale feature processing unit, a self-attention mechanism unit, and a feature fusion unit; The input unit receives input data, performs preliminary feature extraction, and generates a feature map; The feature sampling and transformation unit samples and transforms the shape of the feature map to generate a multi-scale feature map; The multi-scale feature processing unit performs a convolution operation on the multi-scale feature map through a filtering unit to extract multi-scale feature information; The self-attention mechanism unit calculates the attention weight in the feature map, generates a self-attention feature map, and ensures the effectiveness and stability of the attention feature through normalization processing; The feature fusion unit adds the multi-scale feature map to the self-attention feature map to generate the final output feature.

4. The method for generating ultra-low-dose whole-body medical images based on deformable convolution according to claim 1, characterized in that: S2 specifically includes the following steps: S2.1, input the preprocessed 3D whole body PET image to the Deformable 3D U-Net module; S2.2, feature extraction and fusion are performed through the spatial deformable aggregation module SDAB and the multi-scale self-attention module MSSAM; S2.

3. Output the feature map processed by the Deformable 3D U-Net module.

5. The method for generating ultra-low-dose whole-body medical images based on deformable convolution according to claim 4, characterized in that: In S2.2, the processing of the spatial deformable aggregation module SDAB and the multi-scale self-attention module MSSAM includes: S2.2.1, the spatial deformable aggregation module SDAB receives the input image in S2.1, extracts spatial information through the U-Net network and the 3D deformable convolution layer, and performs motion compensation; S2.2.2, the multi-scale self-attention module MSSAM receives the output of the spatial deformable aggregation module SDAB module, and performs feature fusion through input units, feature sampling and transformation, multi-scale feature processing, and self-attention mechanism; S2.2.

3. Multi-scale self-attention module MSSAM outputs the fused multi-scale feature map.

6. The method for generating ultra-low-dose whole-body medical images based on deformable convolution according to claim 1, characterized in that: S3 specifically includes the following steps: S3.

1. Use mean square error as the network loss function to calculate the error between the output of the Deformable 3D U-Net module and the actual image. S3.

2. Use the standard Adam optimizer to perform gradient descent update on the weight parameters of the Deformable 3D U-Net module according to the error in S3.

1.

7. An ultra-low-dose whole-body medical image generation system based on deformable convolution, characterized in that: The system includes: 3D whole-body PET image data preprocessing module, used to split, normalize, enhance and partition the original 3D PET image data to generate training sets and test sets; A three-dimensional deformable convolutional deep attention network Deformable 3D U-Net module, which includes a spatial deformable aggregation module SDAB and a multi-scale self-attention module MSSAM, is used to process pre-processed 3D whole-body PET images and generate ultra-low-dose whole-body medical images.

8. The ultra-low-dose whole-body medical image generation system based on deformable convolution according to claim 7, characterized in that: The spatially deformable aggregation module SDAB comprises: U-Net network, used to extract input features; 3D deformable convolutional layers to extract additional spatial information for motion compensation; Parallel 3D convolutional layers are used to implement residual projection and enhance the expressiveness of the spatially deformable aggregation module (SDAB).

9. The ultra-low-dose whole-body medical image generation system based on deformable convolution according to claim 7, characterized in that: The multi-scale self-attention module MSSAM comprises: An input unit, used for receiving input data and performing preliminary feature extraction; Feature sampling and transformation unit, used to generate multi-scale feature maps; A multi-scale feature processing unit, used to extract multi-scale feature information; Self-attention mechanism unit, which is used to calculate the attention weights in the feature map and generate the self-attention feature map; The feature fusion unit is used to fuse the multi-scale features with the self-attention features to generate the final output features.

10. The ultra-low-dose whole-body medical image generation system based on deformable convolution according to claim 7, characterized in that: The 3D whole body PET image data preprocessing module further comprises: A data augmentation unit that augments the dataset by rotating, adjusting contrast, and scaling; The data division unit is used to divide the enhanced image data into a training set and a test set, wherein the training set and the test set both contain complete 3D whole-body PET images.

Images