A medical image quality blind evaluation method based on ROI attention

Abstract

The application discloses a medical image quality blind evaluation method based on ROI attention. The deep learning network model in the application comprises a first network unit and a second network unit; the first network unit comprises a ResNet module and an Attention UNet module; the output of the ResNet module is connected with the Attention UNet module; the second network unit comprises a SiPNet module and a RegNet module, and the output of the SiPNet module is connected with the RegNet module. The application adopts a supervised deep learning method, introduces a double attention mechanism on the basis of a U-Net network structure, and mainly concentrates the energy of a feature map in an ROI region, so that the problem that a non-ROI region may greatly reduce an overall image quality score is effectively solved.

Description

Technical Field

[0001] This invention belongs to the fields of medical image processing and computer vision, and relates to a blind evaluation method for medical image quality based on ROI attention. Background Technology

[0002] Given a natural image, a person can easily judge whether the image is clear and what its quality is; similarly, given a medical image, a radiologist can easily judge its quality. However, for a computer, without a reference image, it is difficult to make a quality assessment of an image similar to that of a human or medical expert.

[0003] In image quality assessment, medical images differ significantly from natural images. Medical image quality assessment is diagnosis-oriented; that is, if the region of interest (ROI) is clear, making it easy for radiologists to make a diagnosis during image interpretation, the image quality is considered good; conversely, if the ROI is unclear, the image quality is considered poor. Figure 2 (a), (b), and (c) in the text. Moreover, the quality of a medical image is often uneven, with some areas of good quality and others of poor quality. This completely violates the assumptions of most natural image quality assessment algorithms. These differences mean that two factors need to be considered when designing evaluation methods: 1. The quality of certain key areas in a medical image determines the quality of the entire image; 2. The image quality varies greatly between different areas of a medical image. Summary of the Invention

[0004] This application addresses the shortcomings of existing medical image quality assessment methods by providing a deep learning network model for blind evaluation of medical image quality, a blind evaluation method for medical image quality based on the model, a blind evaluation device for medical image quality, and a computer-readable storage medium.

[0005] The first aspect of this application provides a deep learning network model for blind evaluation of medical image quality, including a first network unit and a second network unit;

[0006] The first network unit includes a ResNet module and an Attention UNet module; the output of the ResNet module is connected to the Attention UNet module.

[0007] The second network unit includes a SiPNet module and a RegNet module, with the output of the SiPNet module connected to the RegNet module;

[0008] The first network unit outputs three paths, each of which applies an attention map to a feature map at each scale through an attention operation; the feature maps at each scale are then input to the second network unit to obtain a medical image quality score.

[0009] Further, the encoder in the Attention UNet module has three layers, each of which is composed of two convolution operations in series.

[0010] The decoder in the Attention UNet module has four layers, each of the first three layers is first connected with the output of the corresponding attention operation A, and then performs convolution operation J1, and the last layer performs convolution operation J3 to output the attention map.

[0011] Further, the two convolution operations in each layer of the encoder are convolution operation J2 and convolution operation J1 in sequence, and the number of convolution kernels is twice the number of input feature map channels of the layer.

[0012] Further, the convolution operation P1 and the convolution kernel J1 in the first three layers of the decoder have the number of convolution kernels being half of the number of input feature map channels of the layer; the feature map output by each layer is output to each SiPNet module via the attention operation M. Figure 1

[0013] Further, the SiPNet module is used to blend the feature map output by the first network unit into a feature vector.

[0014] The RegNet module is used to fit the feature vector output by the SiPNet module into a quality score.

[0015] The second aspect of the application provides a medical image quality blind evaluation method based on ROI attention, which adopts a supervised deep learning method, and the method comprises the following steps:

[0016] Step 1. Construct a deep learning network model for medical image quality blind evaluation;

[0017] Step 2. Define a loss function so that the quality score output by the deep learning network model is close to the expert score;

[0018] Step 3. Give a data set, each medical picture in the data set has a corresponding expert score, and divide the data set into a training set, a validation set and a test set;

[0019] Step 4. Train the deep learning network model until the deep learning network model reaches the best state;

[0020] Step 5. Use the trained deep learning network model to perform inference to obtain the quality score of the image to be evaluated.

[0021] ​The third aspect of the present application provides a ROI attention-based medical image quality blind evaluation device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned ROI attention-based medical image quality blind evaluation method when executing the program.

[0022] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program for executing a ROI attention-based medical image quality blind evaluation method.

[0023] The present application has the following beneficial effects: the present application adopts a supervised deep learning method, introduces a double attention mechanism on the basis of a U-Net network structure, and mainly concentrates the energy of a feature map in an ROI region, thereby effectively solving the problem that a non-ROI region may greatly reduce the overall image quality score. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A deep learning network model structure diagram for medical image quality blind evaluation in the present application;

[0025] Figure 2 Three original images and corresponding attention maps after processing by the present application;

[0026] Figure 3 A ROI attention-based medical image quality blind evaluation device structure diagram. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application. Based on the examples in the present application, all other examples obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0028] The terms used in the present application are only for the purpose of describing specific examples, and are not intended to limit the present disclosure. The singular forms "a", "an" and "the" used in the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein means and includes any or all possible combinations of one or more associated listed items.

[0029] In the present application, various convolution operations and attention operations A and M are involved. For the sake of convenience of description, these operations are classified and marked as follows:

[0030] Convolution operation J1: the convolution kernel size is 3x3, the stride is 1, the padding is 1, and the activation function is ReLU.

[0031] Convolution operation J2: the convolution kernel size is 3x3, the stride is 2, the padding is 1, and the activation function is ReLU.

[0032] Convolution operation J3: the convolution kernel size is 3x3, the stride is 1, the padding is 1, and the activation function is Sigmoid.

[0033] Dilation convolution operation P1: the dilation convolution, the convolution kernel size is 2x2, the stride is 2, the padding is 1, and the activation function is ReLU.

[0034] Attention operation A: expressed by the formula as follows:

[0035]

[0036] y = a x

[0037] where x i represents a feature vector at position i of the feature map H1xW1xC1, g i represents a feature vector at position i of the feature map H2xW2xC2, W x ,W g and Ψ respectively belong to and b1 and b2 are bias terms, and σ1 and σ2 are ReLU and sigmoid activation functions respectively; - is an up-sampling operation, so that the resolution of its output is consistent with ; "·" represents pixel-by-pixel multiplication, and y represents the output of the attention operation A.

[0038] Attention operation M: is an element-wise multiplication, so as to apply the attention map (attention map, Figure 1 H1xW1x1) marked with a blue frame in the upper right corner to each scale feature map. In the supervised learning framework, this operation plays a role in focusing attention on the ROI region. It should be noted that before multiplication, the resolution of the attention map needs to be adjusted by the bilinear interpolation method to match the target feature map.

[0039] One embodiment of the present application provides a deep learning network model for medical image quality blind evaluation, comprising a first network unit and a second network unit;

[0040] The first network unit comprises a ResNet module and an Attention UNet module; an output of the ResNet module is connected to the Attention UNet module;

[0041] The second network unit comprises a SiPNet module and a RegNet module; an output of the SiPNet module is connected to the RegNet module;

[0042] The output of the first network unit has three paths, and each path applies an attention map to each scale feature map through an attention operation; the each scale feature map is input to the second network unit to obtain a medical image quality score.

[0043] As Figure 1 The first network unit is denoted as ROI-guided Dual Attention UNet:

[0044] Input layer: Figure 1 The upper left corner is marked by a red frame; an input is a gray image, and the size is H1xW1x1; the input is used as the input of the following ResNet module.

[0045] ResNet module: see Figure 1 Part (a) in the above, which is composed of two short connections and five identical convolution operations, the convolution kernel size is 3x3, the number of convolution kernels is 32, the stride is 1, the padding is 1, and the activation function is ReLU.

[0046] The ResNet module accepts data input of H1xW1x1 size, and the output is H1xW1xC1, C1=32; the output is used as the input of the following attention UNet module.

[0047] Attention UNet module: the module is divided into an encoding stage and a decoding stage. The encoding stage has three layers, each layer is composed of two convolution operations in series, which are convolution operations J2 and J1 in turn, and the number of convolution kernels is twice the number of channels of the input feature map of the layer. The decoding stage has four layers, and the first three layers are each connected to the output of the corresponding attention operation A after being expanded by the convolution operation P1, and then the convolution operation J1 is executed; the number of convolution kernels of P1 and J1 is half of the number of channels of the input feature map of the layer. The last layer executes the convolution operation J3, and the number of convolution kernels is 1, and an attention map is output.

[0048] In addition, the Attention UNet module also outputs feature maps at each level, which are output to each SiPNet module through the attention operation M. Figure 1 ​

[0049] SiPNet module: the role is to blend the feature maps into a feature vector with the size of 1x1x64. The input feature map of this module first undergoes a convolution operation with a kernel size of 1x1 and a stride of 1, keeping the spatial resolution of the feature map unchanged and the number of channels changing to 64; then, according to the size of the input feature map, different R rounds of convolution operation with a stride of 2 are needed to ensure that the size of the feature map is between 8x8x64 and 15x15x64 before Max Pooling.

[0050] Further, if the size of the input feature map of the SiPNet module is SxSxC, then Finally, the Max pooling operation is performed to obtain a feature vector with the size of 1x1x64. Three feature vectors with the size of 1x1x64 are obtained from SiPNet-1 to SiPNet-3, which are input to the RegNet module to complete the regression fitting of the final quality score.

[0051] RegNet module: accepts outputs from SiPNet-1 to SiPNet-3, and fits three feature vectors with the size of 1x1x64 into a quality score.

[0052] Further, on the one hand, each 64-dimensional feature vector is input to fully connected layers FC1 to FC3, respectively, and outputs Q i representing quality evaluation at different scales; on the other hand, the three 64-dimensional feature vectors are connected together to form a 192-dimensional feature vector, which is then passed through two fully connected layers FC4 and FC5 to obtain the final quality evaluation Q of the entire image.

[0053] Another embodiment of the present application provides a medical image quality blind evaluation method based on ROI attention. A supervised deep learning method is used, which includes the following steps:

[0054] Step 1. Construct a deep learning network model structure, which includes ROI-guided DualAttention UNet, SiPNet, and RegNet modules.

[0055] In a preferred embodiment, the network model is implemented using Pytorch programming.

[0056] Step 2. Define the loss function, which expects Q1, Q2, Q3, and Q output by the fully connected layers FC1 to FC3 of the RegNet module to be as close as possible to the expert score S.

[0057] In a preferred embodiment, the loss function is defined as follows, and the loss function is implemented using Pytorch:

[0058]

[0059] where b i is a hyper-parameter to balance the weights of different levels of quality; the function H is defined as H(x, y) = (x - y) 2 / y 2 is defined in this way is to make Q and Q i as close as possible to S.

[0060] Step 3. Given a dataset, the present application requires that the dataset has more than 1000 medical images, and each image has a quality score from 1 to 10 given by an expert, where 1 is the worst and 10 is the best.

[0061] In a preferred embodiment, ten experts, consisting of five diagnostic radiologists and five radiological technicians with 15 to 25 years of experience, participated in the subjective image quality assessment. Before the quality assessment, a meeting was held to determine the overall standard of the quality standard; in this meeting, the standard was discussed and 20 different chest radiographs were jointly evaluated as a trial practice. Each expert was required to score each image according to their subjective perception of image quality, with 10 representing the best and 1 representing the worst. Finally, the average opinion score (MOS) of the ten experts was used as the final subjective evaluation result.

[0062] The given dataset has a total of 1132 chest radiographs, which are divided into training set, validation set and test set in the ratio of 8:1:1.

[0063] Step 4. Train the model: first, resize each image in the dataset to 1024x1024 resolution, set BatchSize to 4, and when training with GPU, require at least 16GB of video memory; in the loss function (Eq. 1), set

[0064] b i = 0.1, for i = 1, 2, 3

[0065] Use the Adam optimizer and set its parameters β1 = 0.9, β2 = 0.999, ε = 10 -8 and learning rate α = 10 -5 . Set epoch to 100, after so many iterations of training, select the checkpoint that performs best in the validation set as the final medical image quality blind evaluation model of the present application.

[0066] Step 5. Inference: When inferring, only need to adjust the resolution of the image to be evaluated to 1024x1024 by using the bilinear interpolation method, and then input it into the medical image quality blind evaluation model obtained in step 4, and perform the forward propagation process of the model, that is, the quality scores Q1, Q2, Q3 and Q can be obtained, and Q is the final predicted quality score. In addition, while obtaining the quality score, the inference process can also obtain the attention map Figure 1 The H1xW1x1 in the upper right corner is marked with a blue border, and the actual effect can be seen in Figure 2 , Figure 2 (a)-(c) are original images, and the expert scores are 7.6, 8.9 and 7.1 respectively; the quality scores of the three images obtained by the present application are 7.65, 8.81 and 7.23 respectively; (d)-(f) are the corresponding attention maps.

[0067] Another embodiment of the present application also discloses a medical image quality blind evaluation device based on ROI attention, see Figure 3 , comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the medical image quality blind evaluation method based on ROI attention when executing the program.

[0068] Still another embodiment of the present application also discloses a computer readable storage medium, the storage medium stores a computer program, and the computer program is used to execute the above-mentioned medical image quality blind evaluation method based on ROI attention.

[0069] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of each method can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0070] The processor in the present application can include one or more processing cores. The processor invokes data stored in the memory by running or executing instructions, programs, code sets or instruction sets stored in the memory, performs various functions and processes data of the present application. The processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller and a microprocessor. It can be understood that for different devices, the electronic devices used to implement the functions of the above-mentioned processor can also be other, and the embodiments of the present application are not limited specifically.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions.

[0072] The application further gives the corresponding implementation effect; the performance of the image quality evaluation method is usually calculated by the correlation coefficient between the expert score and the model score, and the larger the value is, the closer the model is to the expert score. Table 1 compares the performance of several latest models and the model of the application by using the correlation coefficient. It can be seen that the performance of the model of the application is far ahead.

[0073] In addition, from the application Figure 2 It can be seen from the above that the energy of these attention maps is mainly concentrated in the key ROI region of the lung, which shows that the application indeed focuses the attention of the model on the region of interest (ROI region) of the doctor.

[0074] Table 1 Comparison of the model of the application and several image quality models

[0075] Model Correlation coefficient PLCC AlexNet 0.653 VGGNet 0.602 ResNet 0.664 CaHDC 0.765 The present application 0.922

[0076] The above are preferred embodiments of the application, and are not intended to limit the protection scope of the application, therefore: all equivalent changes made according to the structure, shape, principle of the application should be covered within the protection scope of the application.

Claims

1. A deep learning network model for medical image quality blind assessment, characterized in that: The first network unit and the second network unit are included. The first network unit includes a ResNet module and an Attention UNet module; the output of the ResNet module is connected to the Attention UNet module; The second network unit includes a SiPNet module and a RegNet module, and the output of the SiPNet module is connected to the RegNet module; The output of the first network unit has three paths, and each path applies an attention map to each scale feature map through an attention operation; the scale feature map is input into the second network unit to obtain a medical image quality score; The encoder in the Attention UNet module has three layers, and each layer is formed by two convolution operations in series; The decoder in the Attention UNet module has four layers, and each of the first three layers is connected to the output of the corresponding attention operation A through a convolution operation P1, and then a convolution operation J1 is performed; the last layer performs a convolution operation J3 to output an attention map; The convolution operation P1 and the convolution kernel J1 in the first three layers of the decoder have the same number of convolution kernels, which is half of the channel number of the input feature map of the layer; the feature map output by each layer is output to each SiPNet module through an attention operation M together with the attention map; The SiPNet module is used to blend the feature map output by the first network unit into a feature vector with a size of 1x1x64; the input feature map of the module is first subjected to a convolution operation with a kernel size of 1x1 and a stride of 1 to maintain the spatial resolution of the feature map unchanged and change the channel number to 64; then, according to the different sizes of the input feature map, different R rounds of convolution operation with a stride of 2 are performed to ensure that the feature map size is between 8x8x64 and 15x15x64 before Max Pooling. The RegNet module is used to fit the feature vectors with a size of 1*1*64 output by the three SiPNet modules into a quality score; on one hand, each 64-dimensional feature vector is input into a fully connected layer FC1 to FC3, and a Q i is output, representing the quality evaluation under different scales; on the other hand, the three 64-dimensional feature vectors are connected together to form a 192-dimensional feature vector, and then the final quality evaluation Q of the whole image is obtained through two fully connected layers FC4 and FC5.

2. The deep learning network model of claim 1, wherein: The two convolution operations in each layer of the encoder are convolution operation J2 and convolution operation J1 in sequence; the number of convolution kernels is twice the channel number of the input feature map of the layer.

3. A medical image quality blind evaluation method based on ROI attention, adopting a supervised deep learning method, characterized in that The method comprises the following steps: Step 1. Construct a deep learning network model for blind evaluation of medical image quality; the deep learning network model adopts the deep learning network model of any one of claims 1 to 2; Step 2. Define a loss function so that the quality score output by the deep learning network model approaches the expert score; Step 3. Give a data set, each medical picture in the data set has a corresponding expert score, and divide the data set into a training set, a validation set and a test set; Step 4. Train the deep learning network model until the deep learning network model reaches the best state; Step 5. Use the trained deep learning network model for inference to obtain the quality score of the image to be evaluated.

4. A medical image quality blind evaluation device based on ROI attention, characterized by, The method comprises: The memory, the processor and the computer program stored in the memory and capable of running on the processor, wherein the processor implements the method of blind evaluation of medical image quality based on ROI attention in the above claim 3 when executing the program.

5. A computer readable storage medium, characterized in that, The storage medium stores a computer program for executing the method of blind evaluation of medical image quality based on ROI attention in the above claim 3.

Images