Medical image quality assessment method, system and computer readable medium

Abstract

The present invention relates to a medical image quality assessment method, system and computer-readable medium, the medical image quality assessment method comprising: constructing a feature extraction module based on a deep learning model, the feature extraction module is used to generate image features according to an input image; constructing an image segmentation submodel according to the feature extraction module, the image segmentation submodel is used to convert image features into an image segmentation template; constructing an image quality assessment submodel according to the feature extraction module, the image quality assessment submodel is used to generate an image quality score according to image features; jointly training the image segmentation submodel and the image quality assessment submodel to obtain a multi-task network model; and obtaining a quality assessment score of the input image according to the multi-task network model. The present invention can improve the accuracy of image quality assessment.

Description

Technical Field

[0001] The present invention mainly relates to the field of image processing technology, and in particular to a medical image quality evaluation method, system and computer-readable medium. Background Art

[0002] The novel coronavirus infection (COVID-19) has had a huge impact on the health and lives of people around the world. In the diagnosis and treatment of COVID-19, lung CT images play a key role. Many official diagnosis and treatment plans use lung CT imaging as one of the diagnostic criteria. The formulation of COVID-19 treatment plans and efficacy evaluation are also highly dependent on lung CT images. However, excessive X-ray exposure may increase the incidence of cancer, genetic diseases, and leukemia, and the radiation dose will accumulate in the patient's body. Therefore, it is particularly important to control the radiation dose to obtain low-dose CT (LDCT) images.

[0003] In order to reduce the radiation dose to patients during CT scanning, low-dose CT scanning methods can be used, but this method may reduce the quality of CT images. LDCT images are prone to speckle noise, stripe artifacts, and ring artifacts, which will significantly affect the doctor's diagnosis of COVID-19 and may lead to misdiagnosis and treatment. If the image quality problem leads to missed diagnosis, it may delay the patient's treatment and endanger the patient's life; and misdiagnosis, such as diagnosing a disease-free patient as a disease or making an incorrect judgment on the type of lesion, may increase the patient's economic and psychological burden and even delay treatment. Therefore, from the perspective of clinical diagnosis and treatment needs, it is particularly important to evaluate the quality of LDCT images of COVID-19. Through evaluation, LDCT images with unqualified quality can be screened out, and the scanning dose can be reasonably reduced under the premise of ensuring the quality of CT images, thereby ensuring the quality of clinical diagnosis and treatment of COVID-19. In addition, CT image quality assessment (IQA) can also guide the optimization of scanning sequences and parameters, image reconstruction, transmission, compression, acquisition, enhancement, and hardware design to further improve image quality.

[0004] At present, the evaluation of LDCT image quality of COVID-19 mainly relies on the visual inspection of doctors. This method is highly dependent on the level and experience of doctors, and is highly subjective, has poor repeatability, and has a large workload. Therefore, it is necessary to study the objective evaluation method of LDCT quality to avoid the influence of subjective judgment. However, the complexity and diversity of medical images and the lack of reference images have brought great challenges to the research of objective evaluation methods.

[0005] Among IQA methods, there are two main categories: subjective quality evaluation and objective quality evaluation. Subjective quality evaluation methods cannot be promoted on a large scale because of their time-consuming, high cost, many influencing factors, and poor repeatability. Therefore, objective IQA methods that use mathematical models and computer theories to simulate the subjective evaluation of the human visual system (HVS) came into being. Objective quality evaluation methods use computer technology to establish reasonable mathematical models, which can automatically and accurately evaluate the realism, distortion or intelligibility of images, which is conducive to improving the objectivity, repeatability and consistency of IQA.

[0006] In objective IQA, according to the availability of reference image information, it can be divided into full-reference (FR), reduced-reference (RR) and no-reference (NR) image quality assessment methods. For no-reference IQA, it can be divided into models for specific distortion types and general models. Since CT images usually have no reference images in actual clinical applications and often face multiple distortions of non-specific types, it is of theoretical and practical value to study a general no-reference IQA model and perform objective image quality assessment based on the model. However, the image quality evaluation results obtained by the existing general no-reference IQA model are still a certain distance away from the expected goal, and there is a problem of low accuracy in image quality evaluation. Summary of the invention

[0007] The technical problem to be solved by the present application is to provide a medical image quality evaluation method, system and computer-readable medium, which can improve the accuracy of image quality evaluation.

[0008] The technical solution adopted by the present application to solve the above-mentioned technical problems is a medical image quality assessment method, comprising: constructing a feature extraction module based on a deep learning model, the feature extraction module is used to generate image features according to an input image; constructing an image segmentation sub-model according to the feature extraction module, the image segmentation sub-model is used to convert image features into an image segmentation template; constructing an image quality assessment sub-model according to the feature extraction module, the image quality assessment sub-model is used to generate an image quality score according to image features; jointly training the image segmentation sub-model and the image quality assessment sub-model to obtain a multi-task network model; and obtaining a quality assessment score of the input image according to the multi-task network model.

[0009] In one embodiment of the present application, the deep learning model includes a convolutional neural network U-Net, U-Net includes a symmetrically arranged encoder and decoder, and the step of the feature extraction module generating image features according to the input image includes: performing convolution operations and pooling operations on the input image according to the encoder to capture context information; performing convolution operations, upsampling operations and feature fusion operations on the context information according to the decoder to generate image features.

[0010] In one embodiment of the present application, after the feature extraction module generates image features according to the input image, the method further includes: cropping the input image into a plurality of regional image blocks; and learning the mapping relationship between the plurality of regional image blocks and the image features using the following formula:

[0011] Z i =f FE (RI i )

[0012] Among them, f represents the mapping relationship symbol between image features and the i-th region image block, FE represents image feature extraction, and RI i represents the i-th region image block, i is a positive integer, Z i Represents image features.

[0013] In one embodiment of the present application, after the step of constructing the image quality evaluation sub-model according to the feature extraction module, it also includes: using the following loss function L QA To train the image quality assessment sub-model:

[0014]

[0015] Where M represents the number of input images; α represents the sample weight, 0≤α≤1, α is used to balance the importance of positive and negative samples; y i represents the true value of the image quality of the i-th input image, where i is a positive integer; It represents the probability value of the correct prediction of the i-th input image output by the image quality assessment sub-model by taking the sigmoid function; λ represents the focusing parameter, and λ is a constant greater than 0.

[0016] In one embodiment of the present application, the following objective function L is used: TS To jointly train the image segmentation sub-model and the image quality assessment sub-model:

[0017]

[0018] Where β represents the task weight, 0≤β≤1, and β is used to balance the importance of the image segmentation sub-model and the image quality assessment sub-model; γ is a constant, γ∈{0,1}; m irepresents the label value of the i-th input image, m i =1 or m i =0; f TS.LS (Z i ) represents the i-th image segmentation template output by the image segmentation sub-model; RS i Represents the true i-th image segmentation template.

[0019] In one embodiment of the present application, the step of constructing an image segmentation sub-model based on a feature extraction module includes: splicing A convolutional layers at the decoder output end of the feature extraction module, where A is an integer greater than or equal to 1; and learning the mapping relationship between image features and image segmentation templates based on the A convolutional layers.

[0020] In one embodiment of the present application, the step of constructing an image quality assessment sub-model based on a feature extraction module includes: splicing B fully connected layers at the decoder output end of the feature extraction module, where B is an integer greater than or equal to 1; making each fully connected layer fully connected to all activated neurons in the previous layer, thereby fusing the global information of the image features.

[0021] In one embodiment of the present application, before the step of constructing a feature extraction module based on a deep learning model, it also includes: performing data enhancement on the input image according to a rotation and translation method, and / or performing data enhancement on the input image according to an intensity conversion method to simulate intensity inhomogeneity artifacts.

[0022] In one embodiment of the present application, the deep learning model includes a deformable convolution kernel, which adds an offset to the sampling position of the standard convolution kernel. The deformable convolution kernel is used to simulate the multi-scale receptive field and multi-modal receptive field of different objects in the input image.

[0023] In order to solve the above technical problems, the present application also proposes a medical image quality assessment system, comprising: a memory for storing instructions executable by a processor; a processor for executing the instructions to implement the above medical image quality assessment method.

[0024] In order to solve the above technical problems, the present application also proposes a computer-readable medium storing computer program codes, and the computer program codes implement the above medical image quality assessment method when executed by a processor.

[0025] The technical solution of the present application extracts image features of an input image (e.g., a low-dose CT image of the lung) through a feature extraction module, so that the image segmentation submodel and the image quality assessment submodel can share the image features; at the same time, the image segmentation submodel performs the lung region segmentation task and the image quality assessment submodel performs the image quality assessment task, and the two submodels are jointly trained to obtain a multi-task network model. The multi-task network model can transfer the knowledge learned from one task to other tasks by sharing the feature extraction of the effective lung region, so that it can better adapt to different tasks and improve the performance of multiple tasks and the accuracy of image quality assessment.

[0026] Aiming at the characteristics of low-dose CT images that are prone to speckle noise, stripe artifacts and ring artifacts, this application, based on the theory of multi-task learning, deeply explores its perceptual characteristics of image quality and provides a general reference-free IQA model (i.e., a multi-task network model) to make the objective image quality evaluation results more consistent with the subjective evaluation results of humans. This application has important theoretical significance and practical value, and will provide new ideas and new solutions for general reference-free medical IQA, which can reduce the workload of doctors and ensure the quality of clinical diagnosis and treatment of new coronary infection. The solution of this application is also easy to expand to other tissues and organs (such as liver, breast) and other modal medical images (such as ultrasound, MRI), etc., and can also be extended to more application fields, such as optimizing scanning sequences and parameters, image reconstruction, transmission, compression, enhancement, hardware design, etc., and has extensible value. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below with reference to the accompanying drawings, wherein:

[0028] Figure 1 is an exemplary flow chart of a medical image quality assessment method according to an embodiment of the present application;

[0029] Figure 2 is a schematic diagram of a multi-task network model in an embodiment of the present application;

[0030] Figure 3 is a schematic diagram of a deformable convolution framework in one embodiment of the present application;

[0031] Figure 4 is a schematic diagram of the results of a doctor's subjective scoring of different low-dose CT images in one embodiment of the present application;

[0032] Figure 5 is an exemplary flow chart of a low-dose CT image quality evaluation method combining objective evaluation and subjective evaluation in one embodiment of the present application;

[0033] Figure 6It is a system block diagram of a medical image quality assessment system according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below with reference to the accompanying drawings.

[0035] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and therefore the present application is not limited to the specific embodiments disclosed below.

[0036] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0037] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0038] The present application proposes a medical image quality evaluation method, which is equivalent to a general reference-free objective quality evaluation method for medical images, and can be applied in scenarios where objective quality evaluation of complex and diverse low-dose CT images is required. The medical image quality evaluation method of the present application can be executed in a controller of a local computer, and can also be run in a cloud platform. When the medical image quality evaluation method runs on a cloud platform, the data on the local computer interacts with the cloud platform data through a wireless network. Exemplarily, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an interconnected cloud, and multiple clouds, or any combination thereof. The present application does not limit the operating environment of the medical image quality evaluation method.

[0039] Here, the multi-task network model designed in this application is first introduced to facilitate understanding of the technical solution of this application. The medical image quality evaluation method of this application will be introduced later.

[0040] Figure 2 is a schematic diagram of a multi-task network model in an embodiment of the present application, refer to Figure 2As shown, illustratively, when diagnosing a disease, the doctor does not pay attention to the background information of the image. For example, for a lung CT image, only the lung area needs to be paid attention to. Therefore, to evaluate the quality of the CT image, it is necessary to first obtain the effective lung area in the CT image. The present application constructs the multi-task network model 200 based on the characteristics of the LDCT image data of COVID-19. The multi-task network model 200 is a deep learning model based on the convolutional neural network U-Net architecture. The multi-task network model 200 can also be referred to as a LDCT image quality evaluation model based on the multi-task U-Net. The input of the multi-task network model 200 is an input image 210 (for example, a low-dose CT image of the lung). The multi-task network model 200 includes a feature extraction module 220 and a specific task module 230. Hereinafter, Feature Extraction (FeatureExtraction) is referred to as FE, and Task Specific (Task Specific) is referred to as TS.

[0041] Continue to refer Figure 2 As shown, exemplarily, the specific task module 230 includes two tasks, one of which is used to perform lung segmentation (LS) and output an image segmentation template 231, which can highlight the segmented lung contour; the other task is used to perform image quality assessment QA and output an image quality score 232. Performing these two tasks simultaneously allows the multi-task network model 200 to learn the nonlinear mapping from the input image 210 to the effective features of the image quality assessment QA and the lung segmentation LS, and can simultaneously capture relevant information and allow the features learned between different tasks to complement each other, so that the two related tasks promote each other and improve together. In practical applications, the multi-task network model 200 can be trained with training images, and the test images can be input into the trained model to obtain the objective quality evaluation score of LDCT, and the consistency of the objective quality evaluation score with the subjective quality evaluation score can be verified.

[0042] The medical image quality evaluation method of this application will be introduced later.

[0043] Figure 1 is an exemplary flow chart of a medical image quality evaluation method according to an embodiment of the present application, with reference to Figure 1 As shown, the medical image quality evaluation method of this embodiment includes the following steps:

[0044] Step S110: construct a feature extraction module based on a deep learning model, where the feature extraction module is used to generate image features according to an input image.

[0045] Step S120: constructing an image segmentation sub-model according to the feature extraction module, where the image segmentation sub-model is used to convert image features into an image segmentation template.

[0046] Step S130: constructing an image quality assessment sub-model according to the feature extraction module, where the image quality assessment sub-model is used to generate an image quality score according to the image features.

[0047] Step S140: jointly train the image segmentation sub-model and the image quality assessment sub-model to obtain a multi-task network model.

[0048] Step S150: Obtaining a quality evaluation score of the input image according to the multi-task network model.

[0049] The above steps S110 to S150 are described in detail below:

[0050] In step S110, a feature extraction module based on a deep learning model is constructed, and the feature extraction module is used to generate image features according to the input image. Exemplarily, this configuration of the present application can enable network models of other tasks to share the image features.

[0051] In some embodiments, before the step of constructing a feature extraction module based on a deep learning model, it also includes: performing data enhancement on the input image according to a rotation and translation method, and / or performing data enhancement on the input image according to an intensity conversion method to simulate intensity inhomogeneity artifacts. Exemplarily, in practical applications, the input image can be data enhanced according to a rotation and translation method so as to cover random positions and different directions of the lungs, and the image can be intensity-converted according to an intensity conversion method to simulate intensity inhomogeneity artifacts. In other embodiments, the input image can be data enhanced only according to a rotation and translation method, or the input image can be data enhanced only according to an intensity conversion method. The present application does not limit the method of data enhancement. The present application can increase the number of input images by performing data enhancement on the input image, thereby improving the generalization ability of the deep learning model and preventing overfitting.

[0052] In some embodiments, the deep learning model includes a convolutional neural network U-Net, the U-Net includes a symmetrically arranged encoder and decoder, and the step of the feature extraction module generating image features according to the input image includes:

[0053] Step S111: performing convolution and pooling operations on the input image according to the encoder to capture context information;

[0054] Step S112: performing convolution operation, upsampling operation and feature fusion operation on the context information according to the decoder to generate image features.

[0055] For example, reference Figure 2As shown, the bars in the multi-task network model 200 represent feature maps, the numbers above each bar represent the number of channels, and the numbers on the sides of the bar represent the size. For example, the first bar in the feature extraction module 220 has 1 channel and a size of 572×572. Figure 2 The operations indicated by arrows of different colors in the figure are as follows. “conv 3×3, ReLU” indicates a 3×3 convolution operation and uses the activation function ReLU; “copy and crop” indicates a feature connection operation; “max pool2×2” indicates a 2×2 maximum pooling operation; “up-conv 2×2” indicates a 2×2 convolution upsampling operation; “conv1×1” indicates a 1×1 convolution operation; and “flatten” indicates a dimension conversion operation, which converts multi-dimensional data into one-dimensional data.

[0056] Continue to refer Figure 2 As shown, for example, the feature extraction module 220 is constructed based on U-Net, and the internal structure of the feature extraction module 220 is U-shaped. It can be considered that the left half of the U-shape is the encoder, and the right half of the U-shape is the decoder. In step S111, the encoder receives the input image 210 and processes it. The encoder can perform a convolution operation on the input image 210 (for example Figure 2 The 3×3 convolution operation shown in step S221 and the activation function ReLU) and the pooling operation (e.g. Figure 2 In step S112, the decoder may perform a convolution upsampling operation on the context information (e.g., Figure 2 2×2 convolution upsampling operation shown in step S224) and feature fusion operation (e.g. Figure 2 The feature connection operation shown by the arrow in step S223 is copy and crop).

[0057] Exemplarily, this application uses Gaussian distribution to randomly initialize the network parameters of the U-Net model and uses the RMSprop optimizer for training. This setting can better update the parameters of the U-Net model so that the feature extraction module can extract more accurate image features. This application does not limit the method of initializing the model parameters and the optimizer used.

[0058] In some embodiments, the deep learning model includes a deformable convolution kernel, which adds an offset to the sampling position of the standard convolution kernel, and the deformable convolution kernel is used to simulate the multi-scale receptive field and multi-morphological receptive field of different objects in the input image. For example, in the LDCT images of COVID-19 patients, due to the complexity and diversity of the lung morphology of each patient, the traditional convolution kernel structure may be difficult to effectively capture such variable features. For this reason, the present application uses a deformable convolution kernel to simulate the multi-scale and multi-morphological receptive field of the lungs.

[0059] Illustratively, the deformable convolution kernel adds an offset to the standard convolution sampling position so that it can adapt to and focus on lung regions of different shapes and sizes. This approach makes it easier for the convolutional neural network to focus on the region of interest and further capture the basic features of the lungs. In a convolutional neural network, the receptive field refers to the size of the area mapped to the input image by the pixels on the feature map output by each layer, that is, a point on the feature map corresponds to the area on the input image. Using convolution kernels of different sizes means that the network can perceive receptive fields of different sizes (i.e., multi-scale receptive fields), while using deformable convolution means that the network can perceive receptive fields of different shapes (i.e., multi-modal receptive fields).

[0060] Considering that the lungs of each COVID-19 patient have different shapes and sizes, a deformable convolution kernel can be used as Figure 2 Alternatives to some convolution kernels in the above example, using deformable convolution kernels instead Figure 2 The conv 3x3 in this setting can effectively simulate and adapt to the multi-scale and multi-morphological receptive field of the lungs, thereby achieving effective recognition of lung morphology.

[0061] Figure 3 is a schematic diagram of a deformable convolution framework in an embodiment of the present application, with reference to Figure 3 As shown, exemplarily, firstly, an input feature map 310 is obtained and a conventional convolution operation is performed through a convolution kernel 320 to obtain an offset feature map 330; then the offset feature map 330 outputs an offset 340 required by the deformable convolution kernel; then the offset 340 is combined with the conventional convolution kernel to obtain a deformable convolution kernel, and a deformable convolution operation is performed using the deformable convolution kernel, as shown in FIG. Figure 3 As shown by the dotted line in , the output feature map 350 is finally obtained. The size and position of the deformable convolution kernel can be dynamically adjusted according to different image contents to adapt to different shapes of the lungs. In the process of training a multi-task network model based on the deformable convolution kernel, the convolution kernel and offset for generating output features can be learned at the same time. In order to learn the offset, the gradient can be back-propagated through a bilinear interpolation operation.

[0062] In some embodiments, after the feature extraction module generates image features according to the input image, the method further includes:

[0063] Step S113: cropping the input image into a plurality of regional image blocks;

[0064] Step S114: Use the following formula (1) to learn the mapping relationship between multiple regional image blocks and image features:

[0065] Z i =f FE (RI i )(1)

[0066] Among them, f represents the mapping relationship symbol between the image feature and the i-th region image block, FE (Feature Extraction) represents the feature extraction module, FE can also represent image feature extraction, RI i represents the i-th region image block, i is a positive integer, Z i Represents image features.

[0067] For example, in step S113, the input image can be cropped into multiple regional image blocks, and the present application does not limit the method of cropping the input image. Formula (1) can allow the feature extraction module to learn the mapping relationship between the cropped lung regional image blocks and the extracted image features.

[0068] In step S120, an image segmentation sub-model is constructed according to the feature extraction module, and the image segmentation sub-model is used to convert image features into an image segmentation template.

[0069] In some embodiments, the step of constructing an image segmentation sub-model according to the feature extraction module includes:

[0070] Step S121: splicing A convolutional layers at the decoder output end of the feature extraction module, where A is an integer greater than or equal to 1;

[0071] Step S122: learning the mapping relationship between image features and image segmentation templates according to A convolutional layers.

[0072] For example, reference Figure 2 As shown, two convolutional layers (such as Figure 2 The present application does not limit the number of concatenated convolutional layers.

[0073] In step S130, an image quality assessment sub-model is constructed according to the feature extraction module, and the image quality assessment sub-model is used to generate an image quality score according to the image features.

[0074] In some embodiments, the step of constructing an image quality assessment sub-model according to the feature extraction module includes:

[0075] Step S131: splicing B fully connected layers at the decoder output end of the feature extraction module, where B is an integer greater than or equal to 1;

[0076] Step S132: Make each fully connected layer fully connected to all activated neurons in the previous layer, so as to fuse the global information of image features.

[0077] For example, reference Figure 2 As shown, three fully connected layers (such as Figure 2 ), the present application does not limit the number of fully connected layers to be spliced. In some embodiments, a dropout layer may be added after the second fully connected layer, and such a setting may prevent the network from overfitting. The image features extracted by the feature extraction module 220 may be transferred to the image quality assessment task through step S132.

[0078] In some embodiments, after the step of constructing the image quality evaluation sub-model according to the feature extraction module, the following loss function L is used: QA To train the image quality assessment sub-model:

[0079]

[0080] In formula (2), M represents the number of input images; α represents the sample weight, 0≤α≤1, and α is used to balance the importance of positive samples and negative samples; y i represents the true value of the image quality of the i-th input image, where i is a positive integer; represents the probability value of the correct prediction of the i-th input image output by the image quality assessment sub-model by taking the sigmoid function; λ represents the focus parameter, and λ is a constant greater than 0. The above formula (2) is equivalent to the multi-class balanced focal function. Formula (2) can minimize the difference between the true value and the predicted value of the image quality assessment sub-model.

[0081] In step S140, the image segmentation sub-model and the image quality assessment sub-model are jointly trained to obtain a multi-task network model.

[0082] In some embodiments, the following objective function L is used: TS To jointly train the image segmentation sub-model and the image quality assessment sub-model:

[0083]

[0084] In formula (3), β represents the task weight, 0≤β≤1, and β is used to balance the importance of the image segmentation sub-model and the image quality assessment sub-model; γ is a constant, γ∈{0,1}, and by setting γ, the denominator in formula (3) can be avoided to be equal to 0; m i represents the label value of the i-th input image, m i =1 or m i =0; f TS.LS (Z i ) represents the i-th image segmentation template output by the image segmentation sub-model, TS stands for Task Specific, LS stands for Lung Segmentation; RS i Represents the real i-th image segmentation template, RS stands for Real Segmentation. For example, the above formula (3) is equivalent to combining two loss functions to train a multi-task network model, and the difference between the final regression segmentation mask and the corresponding true value is controlled by the mean square error loss.

[0085] Exemplarily, the aforementioned steps S110 to S130 of the present application are equivalent to establishing two single-task neural network models (i.e., an image segmentation sub-model and an image quality assessment sub-model), which can perform lung region segmentation and image quality assessment respectively. The multi-task network model of the present application is equivalent to a multi-task U-Net model. The present application designs a strategy for collaborative training of multiple sub-models. The training strategy can be called a three-step training strategy, including separate training and joint training. Specifically, it includes: (1) training the parameters in the feature extraction module FE and the image segmentation sub-model (lung region segmentation module LS) by minimization; (2) training the weights in the feature extraction module FE and the image quality assessment sub-model QA by optimization; (3) joint training of the feature extraction module FE and the specific task module TS by minimization. By comparing the single-task model with the multi-task model, and comparing the strategies of separate training and collaborative training of multiple sub-networks, the advantages of collaborative training of the multi-task U-Net model can be reflected.

[0086] In step S150, the quality evaluation score of the input image is obtained according to the multi-task network model. Exemplarily, after the input image is sent to the trained multi-task network model, the objective image quality evaluation score of the input image can be obtained, and the consistency between the objective image quality evaluation score and the subjective image quality evaluation score can be verified later. In the process of verifying the consistency between the objective image quality evaluation method and the subjective image quality evaluation method of the present application, the present application divides the LDCT image into a training image and a test image, trains the deep learning model (i.e., the multi-task network model) with the training image, inputs the test image into the trained model, obtains the objective evaluation score of the LDCT and evaluates its consistency with the subjective evaluation.

[0087] Here we introduce the subjective evaluation method standard of this application. For example, CT images are ultimately for the purpose of diagnosis. With this as a guide, it is necessary to study the observation environment, observer selection, test sequence selection, evaluation process and analysis method of subjective evaluation. There is no special standard for the subjective evaluation of medical images, and it is usually based on the subjective evaluation standard of ordinary images. Because there is no reference image for LDCT of COVID-19, this application is based on the Single Stimulus Continuous Quality Evaluation (SSCQ) method. Five experienced chest radiologists are asked to subjectively score the image quality. The actual application of medical images for clinical diagnosis services is fully considered in the subjective scoring process. The score is divided into 0 to 4 points. The higher the score, the better the image quality. The final score is the average of the scores of the five doctors. Figure 4 is a schematic diagram of the results of a doctor's subjective scoring of different low-dose CT images in one embodiment of the present application, Figure 4 LDCT examples of different image qualities are shown in FIG. 4 , for example, the subjective score of image 410 is Bad; the subjective score of image 420 is Poor; the subjective score of image 430 is Fair; the subjective score of image 440 is Good; and the subjective score of image 450 is Excellent.

[0088] After obtaining the doctor's subjective rating of the image quality, the consistency between the medical image quality evaluation method of this application and the doctor's subjective evaluation method is verified. For example, the Spearman Rank Order Correlation Coefficient (ROCC), Pearson Linear Correlation Coefficient (LCC), and mean square error are used to measure the consistency between the objective quality evaluation score of the CT image output by the multi-task network model constructed in this application and the subjective evaluation.

[0089] Here, an example is used to illustrate the process of verifying the consistency between objective evaluation and subjective evaluation in this application. Figure 5 is an exemplary flow chart of a low-dose CT medical image quality evaluation method combining objective evaluation and subjective evaluation in one embodiment of the present application, with reference to Figure 5As shown, in step S510, an LDCT image is acquired; in step S5211, a subjective evaluation method standard is designed; in step S5212, a subjective evaluation of image quality is performed; in step S5221, a quality evaluation model based on multi-task U-Net is constructed; in step S5222, feature extraction is performed; in step S5223, lung region segmentation is performed; in step S5224, image quality evaluation is performed; in step S5225, an objective evaluation score of image quality is obtained; and in step S530, consistency verification of objective evaluation and subjective evaluation is performed.

[0090] The innovative design of this application and the beneficial effects it brings are as follows:

[0091] (1) Construction of a CT image quality assessment model based on multi-task U-Net. In the field of medical image quality assessment (IQA), existing methods often ignore specific areas of concern to doctors, such as the lung area in lung CT images. In order to accurately evaluate the quality of CT images, the first task is to identify and extract the valid lung area in the image. Although some existing IQA methods extract the areas of interest to doctors, they usually use different strategies to independently perform the region segmentation task and the quality assessment task, that is, they regard them as two unrelated tasks. Both tasks require the extraction of features of the region of interest. Existing methods ignore the close connection between the two tasks.

[0092] This application proposes a multi-task U-Net neural network model that can perform lung segmentation and image quality evaluation simultaneously. This design enables the convolutional neural network features learned between the two tasks to complement each other, thereby avoiding the problem of insufficient utilization of effective information due to the separation of two related tasks. This application allows the two related tasks to promote each other, jointly improve the algorithm performance, and improve the generalization ability of the model. At the same time, in order to better train the multi-task model, this application proposes a multi-step training strategy, which effectively solves the problem that the model is not easy to converge. The scheme of this application can provide new ideas for multi-task deep learning methods in the field of IQA. The scheme of this application does not require prior knowledge, can automatically learn image quality features without reference and supervision, and has good versatility. It is expected to open up new ideas and technologies for solving the problems of complex and diverse images and no reference images faced by medical IQA. It has important theoretical significance and application prospects, and can be extended to other tissues and organs (such as liver), other modal medical images (such as MRI), and can also be extended to more application fields, such as optimizing scanning sequences and parameters, image reconstruction, transmission, etc., with extensible value.

[0093] (2) Use deformable convolution to segment lung regions of different shapes. Most medical IQA methods do not take into account that the shape and appearance of the lungs vary from patient to patient. In view of the large differences in the morphology and imaging features of lung images of different COVID-19 patients, this application uses deformable convolution to achieve the receptive field of different lung shapes, which is different from the receptive field of conventional convolution kernels with square shapes. It can better capture the basic characteristics of the image quality of various lung shapes, improve the modeling ability of the network, and play its role in the field of IQA.

[0094] The present application also includes a medical image quality assessment system, including a memory and a processor, wherein the memory is used to store instructions executable by the processor; and the processor is used to execute the instructions to implement the medical image quality assessment method described above.

[0095] Figure 6 is a system block diagram of a medical image quality assessment system according to an embodiment of the present application. Figure 6 As shown, the medical image quality assessment system 600 may include an internal communication bus 601, a processor 602, a read-only memory (ROM) 603, a random access memory (RAM) 604, and a communication port 605. When applied on a personal computer, the medical image quality assessment system 600 may also include a hard disk 606. The internal communication bus 601 can realize data communication between the components of the medical image quality assessment system 600. The processor 602 can make judgments and issue prompts. In some embodiments, the processor 602 can be composed of one or more processors. The communication port 605 can realize data communication between the medical image quality assessment system 600 and the outside. In some embodiments, the medical image quality assessment system 600 can send and receive information and data from the network through the communication port 305. The medical image quality assessment system 600 may also include different forms of program storage units and data storage units, such as a hard disk 606, a read-only memory (ROM) 603 and a random access memory (RAM) 604, which can store various data files used for computer processing and / or communication, and possible program instructions executed by the processor 602. The processor executes these instructions to implement the main part of the method. The result processed by the processor is transmitted to the user device through the communication port and displayed on the user interface.

[0096] The above-mentioned medical image quality assessment method can be implemented as a computer program, stored in the hard disk 606, and loaded into the processor 602 for execution to implement the medical image quality assessment method of the present application.

[0097] The present application also includes a computer-readable medium storing a computer program code, which, when executed by a processor, implements the medical image quality assessment method described above.

[0098] When the medical image quality assessment method is implemented as a computer program, it can also be stored in a computer-readable storage medium as a product. For example, a computer-readable storage medium may include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic strip), an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD)), a smart card, and a flash memory device (e.g., an electrically erasable programmable read-only memory (EPROM), a card, a stick, a key drive). In addition, the various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) that can store, contain and / or carry code and / or instructions and / or data.

[0099] It should be understood that the embodiments described above are only illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or a combination thereof.

[0100] Some aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". The processor may be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. In addition, various aspects of the present application may be expressed as computer products located in one or more computer-readable media, which include computer-readable program codes. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, tapes ...), optical disks (e.g., compact disks CDs, digital versatile disks DVDs ...), smart cards, and flash memory devices (e.g., cards, sticks, key drives ...).

[0101] A computer-readable medium may include a propagated data signal containing computer program code, such as in baseband or as part of a carrier wave. The propagated signal may have a variety of manifestations, including electromagnetic, optical, etc., or a suitable combination. A computer-readable medium may be any computer-readable medium other than a computer-readable storage medium, which may be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit a program for use. The program code on the computer-readable medium may be propagated via any suitable medium, including radio, cable, fiber optic cable, radio frequency signal, or similar medium, or any combination of the above mediums.

[0102] The basic concepts have been described above. Obviously, for those skilled in the art, the above application disclosure is only an example and does not constitute a limitation of the present application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements and amendments to the present application. Such modifications, improvements and amendments are suggested in the present application, so such modifications, improvements and amendments still belong to the spirit and scope of the exemplary embodiments of the present application.

[0103] At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0104] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of the present application are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

Claims

1. A method for evaluating medical image quality, characterized in that: include: Constructing a feature extraction module based on a deep learning model, the feature extraction module is used to generate image features according to an input image; wherein the deep learning model includes a convolutional neural network U-Net, the U-Net includes a symmetrically arranged encoder and decoder, and the step of the feature extraction module generating image features according to the input image includes: performing convolution operations and pooling operations on the input image according to the encoder to capture context information; performing convolution operations, upsampling operations and feature fusion operations on the context information according to the decoder to generate the image features; An image segmentation sub-model is constructed according to the feature extraction module, and the image segmentation sub-model is used to convert the image features into an image segmentation template; the step of constructing the image segmentation sub-model according to the feature extraction module comprises: splicing A convolutional layers at the decoder output end of the feature extraction module, where A is an integer greater than or equal to 1; and learning the mapping relationship between the image features and the image segmentation template according to the A convolutional layers; An image quality assessment submodel is constructed according to the feature extraction module, and the image quality assessment submodel is used to generate an image quality score according to the image features; the step of constructing the image quality assessment submodel according to the feature extraction module comprises: splicing B fully connected layers at the decoder output end of the feature extraction module, where B is an integer greater than or equal to 1; each fully connected layer is fully connected to all activated neurons in the previous layer, so as to fuse the global information of the image features; jointly training the image segmentation sub-model and the image quality assessment sub-model to obtain a multi-task network model; and Obtain a quality evaluation score of the input image according to the multi-task network model.

2. The medical image quality assessment method according to claim 1, characterized in that: After the step of the feature extraction module generating image features according to the input image, the method further includes: Cropping the input image into a plurality of regional image blocks; The mapping relationship between the plurality of regional image blocks and the image features is learned using the following formula: Z i =f FE (RI i ) Wherein, f represents the mapping relationship symbol between the image feature and the i-th region image block, FE represents image feature extraction, and RI i represents the i-th region image block, i is a positive integer, Z i Represents the image features.

3. The medical image quality assessment method according to claim 2, characterized in that: After the step of constructing the image quality evaluation sub-model according to the feature extraction module, the following step is also included: using the following loss function L QA To train the image quality assessment sub-model: Where M represents the number of input images; α represents the sample weight, 0≤α≤1, α is used to balance the importance of positive and negative samples; y i represents the true value of the image quality of the i-th input image, where i is a positive integer; It represents the probability value of the correct prediction of the predicted value of the i-th input image output by the image quality assessment sub-model by taking the sigmoid function; λ represents the focusing parameter, and λ is a constant greater than 0.

4. The medical image quality assessment method according to claim 3, characterized in that: The following objective function L is used TS To jointly train the image segmentation sub-model and the image quality assessment sub-model: Where β represents the task weight, 0≤β≤1, and β is used to balance the importance of the image segmentation sub-model and the image quality assessment sub-model; γ is a constant, γ∈{0,1}; m i represents the label value of the i-th input image, m i =1 or m i =0; f TS.LS (Z i ) represents the i-th image segmentation template output by the image segmentation sub-model; RS i Represents the true i-th image segmentation template.

5. The medical image quality assessment method according to claim 1, characterized in that: Before building the feature extraction module based on the deep learning model, it also includes: Performing data augmentation on the input image according to a rotation and translation method, and / or The input image is data augmented according to an intensity conversion method to simulate intensity inhomogeneity artifacts.

6. The medical image quality assessment method according to claim 1, characterized in that: The deep learning model includes a deformable convolution kernel, which adds an offset to the sampling position of the standard convolution kernel, and the deformable convolution kernel is used to simulate the multi-scale receptive field and multi-modal receptive field of different objects in the input image.

7. A medical image quality assessment system, characterized in that: include: a memory for storing instructions executable by a processor; A processor, configured to execute the instructions to implement the medical image quality assessment method as described in any one of claims 1 to 6.

8. A computer readable medium storing computer program code, characterized in that: When the computer program code is executed by a processor, the computer program code implements the medical image quality assessment method according to any one of claims 1 to 6.

Images