Deep Learning-Based Methods and Systems for Medical Image Quality Assessment

Abstract

The application provides a medical image quality evaluation method and system based on deep learning, which comprises the following steps: extracting a local feature map of a medical image by using a ConvNeXt model; extracting a global feature map of the medical image by using a Transformer model; obtaining a first prediction score; constructing a teacher model and a student model with the same architecture; iteratively training the student model by inputting the medical image into the student model to obtain a Gaussian vector of a second prediction score; calculating a loss function according to the first prediction score vector and the Gaussian vector of the second prediction score; updating the weights of the student model and the weights of the teacher model; determining whether the number of iterations is greater than or equal to a preset number; if yes, saving the weights of the teacher model; if no, repeating the feature extraction step and the model updating step; loading the weights of the teacher model into a neural network model to obtain a medical image quality evaluation model; and inputting a clinical medical image into the medical image quality evaluation model to obtain a quality evaluation.

Description

Technical Field

[0001] This application relates to the field of image quality assessment technology, specifically to a method and system for medical image quality assessment based on deep learning. Background Technology

[0002] Image quality assessment (IQA) aims to evaluate the quality of an image using automated procedures. This technique can also be used to evaluate the quality of images generated by medical image algorithms, contributing to the development of new medical image algorithms. For example, in computed tomography (CT), excessive radiation doses can have harmful effects on patients; therefore, clinical practice generally uses low-dose CT combined with post-processing of medical image algorithms to obtain images for clinical diagnosis.

[0003] Currently widely used IQA techniques, such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), require gold-standard original reference images to calculate these metrics. However, in clinical practice, reference images corresponding to low-dose medical images are generally unavailable. Furthermore, these algorithms calculate image scores solely by measuring pixel similarity, without considering the impact of image quality on clinical diagnosis. Therefore, their results do not fully align with radiologists' evaluations of image quality, rendering them ineffective in real-world clinical settings.

[0004] Furthermore, many deep learning-based IQA methods have achieved excellent results on natural images, but these methods require large amounts of labeled datasets to train the deep learning models. Although there are many deep learning methods for quality assessment of natural images, the characteristics of natural images differ from those of medical images, making these methods unsuitable for direct application in medical image quality assessment.

[0005] First, the types of contamination in natural and medical images differ. Contamination in natural images includes noise, blurring, and downsampling; contamination in medical images includes low-dose noise, metallic artifacts, and motion artifacts. Furthermore, the content of natural and medical images also differs. Natural images have complex and diverse content, while medical images generally contain large areas of black background. These differences determine that a well-trained IQA deep model for natural images cannot be directly transferred and applied to medical images. Summary of the Invention

[0006] This application provides a method and system for medical image quality assessment based on deep learning, in order to solve the technical problem that existing deep models for natural image quality assessment cannot be directly applied to medical images.

[0007] This application provides a deep learning-based method for medical image quality assessment, including the following steps:

[0008] Obtain medical images and their corresponding label scores;

[0009] Feature extraction steps: The ConvNeXt model is used to extract the local features of the medical image and obtain a local feature map; at the same time, the Transformer model is used to extract the global features of the medical image and obtain a global feature map.

[0010] Based on the deformable convolutional fusion model, and using the local feature map and the global feature map, a Gaussian vector of the first prediction score is obtained;

[0011] Construct two neural network models with the same architecture, and define them as the teacher model and the student model, respectively;

[0012] Iterative training steps: Input the medical image into the student model and obtain the Gaussian vector of the second prediction score;

[0013] The loss function is calculated based on the vector of the first predicted score and the Gaussian vector of the second predicted score;

[0014] Model update steps: Calculate the gradient based on the loss function, and update the weights of the student model and the teacher model;

[0015] Determine whether the number of iterations is greater than or equal to the preset number. If yes, save the weights of the teacher model. If no, iterate through the feature extraction step to the model update step.

[0016] The weights of the saved teacher model are loaded into the neural network model to obtain a medical image quality evaluation model;

[0017] Clinical medical images are acquired and input into the medical image quality evaluation model to obtain the quality evaluation of the clinical medical images.

[0018] Optionally, the steps of acquiring medical images and corresponding label scores may include the following:

[0019] Acquire two or more medical images;

[0020] Each of the medical images is scored at least twice, and the average score is defined as the label score;

[0021] The medical images and their corresponding label scores are selected to form a dataset.

[0022] Optionally, the step of obtaining the first prediction score based on the local feature map and the global feature map includes the following steps:

[0023] The global feature map is subjected to convolutional neural network operations, and the features after convolution are extracted to obtain the displacement.

[0024] offset = CNN(F2)

[0025] Where offset represents the displacement, CNN() represents the convolutional neural network operation, and F2 represents the global feature map;

[0026] The local feature map is aligned with the global feature map using deformable convolution, and fused features are obtained.

[0027] F′1 = DeformConv(F1, offset)

[0028] Where F'1 represents the fused features, DeformConv() represents the deformable convolution operation, F1 represents the local features, and offset represents the displacement.

[0029] The fused features are input into the attention module to obtain the final feature map;

[0030] The final feature map and the global feature map are concatenated along the channel dimension and then input into the fully connected layer to obtain the first prediction score.

[0031] Optionally, before the iterative training step, the following step is also included:

[0032] Initialize teacher model f t (·,θ t ) and student model f s (·,θ s ), where θ t ,θ s These represent the weights of the teacher model and the student model, respectively.

[0033] Optionally, the iterative training step includes the following steps:

[0034] The medical images are preprocessed, including rotation, flipping, and cropping.

[0035] The main body of the preprocessed medical image is cropped, and the background area is discarded;

[0036] The cropped medical image is input into the student model to obtain a Gaussian vector of the second prediction score.

[0037] Optionally, in the preprocessing medical image step,

[0038] The medical image is randomly rotated from 0 to 360°.

[0039] The medical image is randomly flipped.

[0040] Obtain a mini-batch of size B, where each image in the mini-batch is a data-augmented medical image;

[0041] The mini-batch is randomly cut into L×L sizes, where 448≤L≤512.

[0042] Optionally, in the step of calculating the loss function based on the vector of the first predicted score and the Gaussian vector of the second predicted score,

[0043] The relationship between the predicted score and the Gaussian vector of the predicted score is as follows:

[0044]

[0045] Among them, v b,k Let x represent the k-th element in the b-th vector, where m is the length of the Gaussian vector and k is the element index of the Gaussian vector, k = 1, 2, ..., m. b Represents the label score, and x b ∈R 16 Define 4*argmax(f(x)) / m as the predicted score;

[0046] The loss function is:

[0047]

[0048] Where f(y) represents the vector of the first prediction score, and the medical image y∈R B×1×L×L 1 represents the number of channels, B represents the mini-batch size, L×L represents the size of the mini-batch random cropping, v represents the Gaussian vector corresponding to the second prediction score, and l represents the loss function.

[0049] Optionally, the model update step includes the following steps:

[0050] The gradient is calculated based on the loss function, and the weights of the student model are updated using Adam optimization.

[0051] The teacher model is updated using an exponentially weighted average.

[0052] θ s =αθ s +(1-α)θ t .

[0053] Where, θ s The weights θ represent the teacher model weights. s The weights of the student model are represented by α, where α is the weighting coefficient.

[0054] Optionally, the medical image is at least one of CT image, MRI image, PET image, DSA image, and OCT image.

[0055] Accordingly, this application also provides a deep learning-based medical image quality assessment system, which includes a memory and a processor. The memory is used to store executable program code; the processor is connected to the memory and runs a computer program corresponding to the executable program code by reading the executable program code, so as to execute the deep learning-based medical image quality assessment method described above.

[0056] This application provides a deep learning-based method and system for medical image quality assessment. In the Transformer-CNN hybrid architecture image quality assessment neural network based on an attention mechanism, the neural network utilizes a Transformer model to extract global features of CT images, while simultaneously using a ConvNeXt model to extract local features. This allows the attention-based Transformer-CNN hybrid architecture image quality assessment neural network to simulate the expert scoring process, focusing on both local details and the overall appearance of the CT image, thereby improving prediction performance. Furthermore, the global features extracted by the Transformer model are more conducive to identifying key parts of the CT image, manifested as key locations in the CT image, resulting in larger global feature values. This guides local features to also focus more on key areas of the CT image.

[0057] The network architecture of the teacher model and the student model are the same. In each iteration, the parameters of the student model and the teacher model are weighted and averaged. This makes full use of the historical information of the network parameters, which can increase the stability of the neural network performance and improve the generalization ability.

[0058] This application predicts the Gaussian vector corresponding to the quality score, rather than directly predicting the score itself. The standard deviation of the Gaussian vector can be used to measure uncertainty, making the accuracy of predicting the Gaussian vector corresponding to the quality score higher than that of directly predicting the score, thereby improving the accuracy and stability of the predicted score. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1This is a flowchart illustrating the deep learning-based medical image quality assessment method provided in this application.

[0061] Figure 2 This is a flowchart illustrating step S300 in the deep learning-based medical image quality assessment method provided in this application.

[0062] Figure 3 This application presents the quality evaluation results of CT images using a deep learning-based medical image quality evaluation method. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. In addition, it should be understood that the specific embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. In this application, unless otherwise stated, directional terms such as "up," "down," "left," and "right" generally refer to up, down, left, and right in the actual use or working state of the device, specifically the drawing directions in the accompanying drawings.

[0064] This application provides a method and system for medical image quality assessment based on deep learning, which will be described in detail below. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments of this application. Furthermore, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments.

[0065] Please see Figure 1 and Figure 2 This application provides a deep learning-based method for evaluating the quality of medical images, which specifically includes the following steps:

[0066] S100: Obtain medical images and their corresponding label scores;

[0067] The aforementioned medical images are at least one of CT images, MRI images, PET images, DSA images, and OCT images. In this application, CT images are used as an example to illustrate the medical images.

[0068] Step S100 specifically includes the following steps:

[0069] S110. Acquire two or more medical images;

[0070] In this embodiment, a dataset of more than 2,000 low-dose CT images is collected, with each CT image derived from data obtained by low-dose scanning or accelerated scanning.

[0071] S120. For each of the medical images, score it more than twice, and define the average score as the label score;

[0072] Multiple radiology experts were asked to score the quality of each CT image, and the label score (quality score) of each CT image was defined as the average of the expert scores.

[0073] The scoring is based on a rating scale, which includes 5 levels: very poor, poor, average, good, and very good. The corresponding scores for the 5 levels are 1, 2, 3, 4, and 5 points, respectively.

[0074] S130. Select the medical images and their corresponding label scores to form a dataset;

[0075] Define CT image as y i Its corresponding label score is x i Then, based on the paired data (y) i ,x i The dataset formed is

[0076] The entire dataset is divided according to its purpose and function. 80% of the dataset is used as the training set, 10% as the validation set, and 10% as the test set. In this embodiment, the training set has 1600 images, the validation set has 200 images, and the test set has 200 images.

[0077] S200, Feature extraction steps: Use the ConvNeXt model to extract local features of the medical image and obtain a local feature map; simultaneously use the Transformer model to extract global features of the medical image and obtain a global feature map;

[0078] The ConvNeXt model is an adjustment and improvement of the existing classic ResNet50 / 200 network based on some advanced ideas of the Transformer network. It introduces the latest ideas and techniques of the Transformer network into the existing modules of the CNN network, thereby combining the advantages of the two networks and improving the performance of the CNN network.

[0079] The Transformer used in this application is the open-source Vision Transformer. In the Transformer-CNN hybrid architecture image quality assessment neural network based on the attention mechanism, the neural network uses the Transformer model to extract global features of CT images, and at the same time uses the ConvNeXt model to extract local features of CT images.

[0080] The Transformer-CNN hybrid architecture image quality assessment neural network based on the attention mechanism can simultaneously extract global and local features of CT images. This allows the attention mechanism-based Transformer-CNN hybrid architecture image quality assessment neural network to simulate the scoring process of experts, paying attention to both the local details of CT images and the overall appearance of CT images, thereby improving prediction performance.

[0081] The global features extracted by the Transformer model are also more conducive to identifying key parts in CT images. This manifests as key locations in CT images, resulting in larger values ​​for global features, which in turn can guide local features to focus more on key parts of CT images.

[0082] S300. Based on the deformable convolutional fusion model, and using the local feature map and the global feature map, obtain the Gaussian vector of the first prediction score;

[0083] Step S300 specifically includes the following steps:

[0084] S310. Perform convolutional neural network operations on the global feature map, and simultaneously extract the features after convolution to obtain the displacement.

[0085] offset = CNN(F2)

[0086] Where offset represents the displacement, CNN() represents the convolutional neural network operation, and F2 represents the global feature map;

[0087] S320. Align the local feature map with the global feature map using deformable convolution, and obtain the fused features.

[0088] F′1 = DeformConv(F1, offset)

[0089] Where F'1 represents the fused features, DeformConv() represents the deformable convolution operation, F1 represents the local feature map, and offset represents the displacement.

[0090] The Transformer-CNN hybrid architecture based on the attention mechanism in this application uses deformable convolutions to align features. Compared with the simple stacking or feature fusion methods in the prior art, the network prediction results of deformable convolutions are more accurate.

[0091] S330. Input the fused features into the attention module and obtain the final feature map;

[0092] This application adds an attention module to the attention-based Transformer-CNN hybrid architecture. By learning attention, it can better discover pixels that are more important for score evaluation.

[0093] S340. The final feature map and the global feature map are concatenated along the channel dimension and then input into the fully connected layer to obtain the first prediction score;

[0094] The Transformer-CNN hybrid architecture based on the attention mechanism in this application uses a multi-scale fusion strategy, which utilizes CNN features of different scales to focus on different details, thereby further improving the accuracy of extraction and prediction.

[0095] S400. Construct two neural network models with the same architecture and define them as the teacher model and the student model, respectively.

[0096] Initialize teacher model f t (·,θ t ) and student model f s (·,θ s ), where θ t ,θ s These represent the weights of the teacher model and the student model, respectively.

[0097] The network architecture of the teacher model and the student model are the same. In each iteration, the parameters of the student model and the teacher model are weighted and averaged. This makes full use of the historical information of the network parameters, which can increase the stability of the neural network performance and improve the generalization ability.

[0098] The teacher model is used to generate the learning objectives of the student model. The student model learns using the objectives provided by the teacher model. The weights of the teacher model are obtained by weighted averaging of the time memories of the student model. The algorithm is also based on consistency regularization, which assumes that the model's prediction results will not change when a small perturbation noise is added to the input data.

[0099] S500, Iterative training step: Input the medical image into the student model and obtain the Gaussian vector of the second prediction score;

[0100] When experts score CT images, there is a degree of subjectivity, specifically in the lack of strict scoring standards or the possibility that scores for the same CT image may differ at different times. Therefore, the quality score prediction method based on distribution regression in this application—that is, predicting the Gaussian vector corresponding to the quality score rather than directly predicting the score itself—utilizes the standard deviation of the Gaussian vector to measure uncertainty. This results in higher accuracy in predicting the Gaussian vector corresponding to the quality score compared to directly predicting the score, thereby improving the precision and stability of the predicted score.

[0101] Step S500 specifically includes the following steps:

[0102] S510. Preprocess the medical image, including rotation, flipping, and cropping.

[0103] CT images undergo data augmentation including random rotation, random flipping, and random cropping to obtain batch data. The order of rotation, flipping, and cropping can be adjusted; the following preprocessing steps are merely examples:

[0104] In each iteration of the iterative training phase, the CT images y in the dataset are processed. i Rotate randomly from 0 to 360°.

[0105] The medical image is randomly flipped to obtain the data-enhanced result, which is still denoted as y. i ;

[0106] Obtain a mini-batch of size B, where each image in the mini-batch is a data-augmented medical image;

[0107] The mini-batch is randomly cut into L×L sizes, where 448≤L≤512.

[0108] Preprocessing CT images by rotating, flipping, and cropping changes the details, size, and angle of the CT images input into the neural network model, essentially exposing the neural network model to more diverse data.

[0109] S520. Extract the main body of the preprocessed medical image and discard the background area;

[0110] S530. Input the cropped medical image into the student model and obtain the Gaussian vector of the second prediction score;

[0111] S600. Calculate the loss function based on the vector of the first predicted score and the Gaussian vector of the second predicted score;

[0112] Assume the input CT image y∈R of the neural network model B×1×L×L1 represents the number of channels, B represents the mini-batch size, and L×L represents the size of the mini-batch random cropping; in this application, m = 30 and the batch size B = 16. The relationship between the predicted score and the Gaussian vector of the predicted score is as follows:

[0113]

[0114] Among them, v b,k Let x represent the k-th element in the b-th vector, m be the length of the Gaussian vector, k be the element index of the Gaussian vector (k = 1, 2, ..., m), and b be the data number in the mini-batch. b Represents the label score, and x b ∈R 16 Define 4*argmax(f(x)) / m as a floating-point number, which is the predicted score;

[0115] The loss function is:

[0116]

[0117] Where f(y) represents the vector of the first prediction score, and CT image y∈R B×1×L×L 1 represents the number of channels, B represents the mini-batch size, L×L represents the size of the mini-batch random cropping, v represents the Gaussian vector corresponding to the second prediction score, and l represents the loss function.

[0118] S700, Model update steps: Calculate the gradient based on the loss function, and update the weights of the student model and the teacher model;

[0119] In each iteration, the parameters of the student model and the teacher model are weighted and averaged, making full use of the historical information of the network parameters, which can increase the stability of the neural network performance and improve the generalization ability.

[0120] Step S700 specifically includes the following steps:

[0121] S710. Calculate the gradient based on the loss function, and update the weights of the student model through Adam optimization;

[0122] S720. Update the teacher model using an exponentially weighted average.

[0123] θ s =αθ s +(1-α)θ t .

[0124] Where, θ s The weights θ represent the teacher model weights. sThe weights of the student model are represented by α, where α is the weighting coefficient.

[0125] S800. Determine whether the number of iterations is greater than or equal to the preset number. If yes, save the weights of the teacher model. If no, iterate the training steps until the model update step. The preset number can be set to a specific value as needed, but is not specifically limited in this application.

[0126] S900. Load the weights of the saved teacher model into the neural network model to obtain a medical image quality evaluation model.

[0127] S1000: Acquire clinical medical images and input them into the medical image quality evaluation model to obtain the quality evaluation of the clinical medical images.

[0128] Please see Figure 3 Contamination in CT images includes low-dose noise, metal artifacts, and motion artifacts. The less artifacts or noise in a CT image, the higher the quality score of the CT image given by the medical image quality assessment model.

[0129] This application also provides a deep learning-based medical image quality assessment system, which includes a memory and a processor. The memory is used to store executable program code; the processor is connected to the memory and runs a computer program corresponding to the executable program code by reading the executable program code, so as to execute the deep learning-based medical image quality assessment method described above.

[0130] The above provides a detailed description of a deep learning-based medical image quality evaluation method and system. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for medical image quality evaluation based on deep learning, characterized in that, The method comprises the steps of: obtaining a medical image and a corresponding label score; a feature extraction step: extracting local features of the medical image using a ConvNeXt model and obtaining a local feature map; and extracting global features of the medical image using a Transformer model and obtaining a global feature map; based on a deformable convolution fusion model, and using the local feature map and the global feature map, obtaining a Gaussian vector of a first predicted score; two neural network models with the same architecture are constructed and are defined as a teacher model and a student model, respectively; an iterative training step: inputting the medical image into the student model and obtaining a Gaussian vector of a second predicted score; calculating a loss function according to the first predicted score vector and the second predicted score Gaussian vector; an updating model step: calculating the gradient based on the loss function and updating the weights of the student model and the weights of the teacher model; determining whether the number of iterations is greater than or equal to a preset number, if yes, saving the weights of the teacher model, and if no, repeating the feature extraction step to the updating model step; loading the saved weights of the teacher model into the neural network model to obtain a medical image quality evaluation model; collecting a clinical medical image and inputting it into the medical image quality evaluation model to obtain the quality evaluation of the clinical medical image. 2.The deep learning-based medical image quality evaluation method of claim 1, wherein, In the step of obtaining a medical image and a corresponding label score, the following steps are included: collecting more than two medical images; scoring each of the medical images more than twice, and defining the average of the scores as the label score; selecting the medical images and the corresponding label scores to form a data set. 3.The deep learning-based medical image quality evaluation method of claim 1, wherein, In the step of obtaining a first predicted score based on the local feature map and the global feature map, the following steps are included: performing convolutional neural network operation on the global feature map, extracting the features after convolution, and obtaining an offset, offset = CNN(F2) where offset represents the offset, CNN() represents the convolutional neural network operation, and F2 represents the global feature map; aligning the local feature map and the global feature map using deformable convolution to obtain a fusion feature, F'1 = DeformConv(F1, offset) where F'1 represents the fusion feature, DeformConv() represents the deformable convolution operation, F1 represents the local feature, and offset represents the offset; inputting the fusion feature into an attention module to obtain a final feature map; concatenating the final feature map and the global feature map along the channel dimension and inputting them into a fully connected layer to obtain a first predicted score. 4.The deep learning-based medical image quality evaluation method of claim 1, wherein, Before the iterative training step, the following steps are included: Initialize the teacher model f t (·, θ t ) and the student model f s (·, θ s ), where θ t , θ s are the weights of the teacher model and the weights of the student model, respectively. 5.The deep learning-based medical image quality evaluation method of claim 1, wherein, In the iterative training step, the following steps are included: preprocessing the medical image, the preprocessing methods including rotation, flipping and cropping; cutting the main part of the preprocessed medical image and discarding the background area; inputting the cut medical image into the student model to obtain a Gaussian vector of a second predicted score. 6.The deep learning-based medical image quality evaluation method of claim 5, wherein, In the preprocessing medical image step, randomly rotating the medical image by 0-360°; randomly flipping the medical image; obtaining a mini-batch with a batch size of B, wherein each image in the mini-batch is a data-augmented medical image; randomly cropping the mini-batch to a size of LxL, 448≤L≤512. 7.The deep learning-based medical image quality evaluation method of claim 1, wherein, in the step of calculating the loss function according to the vector of the first prediction score and the Gaussian vector of the second prediction score, the relationship between the prediction score and the Gaussian vector of the prediction score is: wherein v b,k represents the kth element in the bth vector, m is the length of the Gaussian vector, k is the element serial number of the Gaussian vector, k = 1, 2, …, m, x b represents the label score, and x b ∈R 16 , and 4*argmax(f(x)) / m is defined as the prediction score. the loss function is: where f(y) represents a vector of first prediction scores, a medical image y e R B×1×L×L , 1 is the number of channels, B represents the batch size of the mini-batch, L x L represents the size of the random cropping of the mini-batch, v represents a Gaussian vector corresponding to the second prediction scores, and l represents a loss function. 8.The deep learning-based medical image quality evaluation method of claim 1, wherein, the step of updating the model comprises the following steps: calculating the gradient based on the loss function and updating the weight of the student model through Adam optimization; updating the teacher model by using exponential weighted average operation, θ s = a θ s + (1 - a) θ t . where θ s denotes the weights of the teacher model, θ s denotes the weights of the student model, and α denotes a weighting factor.

9. The deep learning-based medical image quality evaluation method according to claim 1, wherein the medical image is at least one of a CT image, an MRI image, a PET image, a DSA image, and an OCT image. comprise: 10.A deep learning-based medical image quality evaluation system, characterized by a memory for storing executable program code; and a processor connected to the memory, which runs a computer program corresponding to the executable program code by reading the executable program code, to execute the deep learning-based medical image quality evaluation method according to any one of claims 1-9. ​

Images