A method and system for evaluating the prognosis of COVID-19 based on multi-time point CT image analysis

Abstract

The application discloses a COVID-19 prognosis evaluation method and system based on multi-time point CT image analysis, for the data corresponding to the patient to be prognostically evaluated, the initial CT data is preprocessed and input into the trained prognosis evaluation model to obtain the probability of whether the patient will have an adverse event; the feature extraction module of the prognosis evaluation model supports processing multiple CT data of a single patient, sharing a network structure, and extracting features between the multiple CT data; a 3D lesion feature Attention module assigns weight information to the features obtained by the feature extraction layer; a feature conversion module combines the longitudinal features of the multi-time CT data extracted by the feature extraction module and the feature weight information obtained by the Attention module, and a classification module automatically evaluates the possibility of an adverse event of the patient; without complex lesion labeling work, the application helps to refine hidden feature mapping, can help clinicians to make targeted treatment schemes in advance to save time, and reduces the probability of occurrence of COVID-19 adverse events.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, and particularly relates to a COVID-19 prognosis evaluation method and system based on multi-time-point CT image analysis. BACKGROUND

[0002] A novel coronavirus can cause acute respiratory disease and even fatal acute respiratory distress syndrome. This coronavirus infection disease is named 2019 coronavirus disease (COVID-19) by the World Health Organization (WHO). By now, the COVID-19 virus has mutated for many generations, and the number of COVID-19 patients in the world is still increasing every day. Nucleic acid detection (RT-PCR) is considered to be the recognized standard for COVID-19 detection, and can make a definite diagnosis of COVID-19 infection. However, nucleic acid detection can only determine whether a patient is infected with COVID-19, and cannot predict how the patient's condition will develop after the disease, such as whether there will be a poor prognosis event (rapid exacerbation of the disease, ICU event, mechanical intubation, death, etc.) in the future. Considering the huge number of COVID-19 virus infections and deaths, it is very important to predict the prognosis of COVID-19 patients early. If it can be predicted whether a patient will have a poor prognosis event in the future, it will help clinicians make targeted clinical treatment decisions in advance, gain effective time for the treatment of the patient, and thus greatly reduce the risk of death caused by a poor prognosis event.

[0003] In clinical practice, doctors often judge whether the patient's condition is worsening according to clinical manifestations. For COVID-19, the rapid deterioration caused by the disease is often within a relatively short period of time, which greatly limits the ability of doctors to make corresponding decisions in advance according to clinical characteristics, so that more effective measures cannot be taken before the patient's condition worsens. In contrast, chest computed tomography (CT) is relatively easy to perform, and will be recorded every few days after admission for COVID-19 patients, which can produce rapid diagnosis. Chest CT of almost all COVID-19 patients has some typical imaging features, such as ground glass opacities (GGO) and consolidation, and these lesions are usually mainly distributed in the lung periphery. Therefore, chest CT can be recommended as the main tool for clinical diagnosis. In particular, the CT images of those patients with good prognosis and those patients with poor prognosis often show different symptoms. In the CT images at different time points in succession, patients with good prognosis often show a tendency to alleviate the above-mentioned CT features, while patients with poor prognosis usually have symptoms that worsen. These differences guarantee the feasibility of using 1-2 chest CT data of patients in the early stage of hospitalization to predict the future prognosis of COVID-19 patients.

[0004] In order to solve the above-mentioned COVID-19 prognosis evaluation problem, the present application proposes a COVID-19 prognosis evaluation method based on multi-time point CT image analysis. In the application process, only 1-2 CT data of the patient in the early stage of hospitalization is needed, and the cumbersome lesion labeling work of the radiologist is not needed. Through automatic data processing and model prediction, the future prognosis of the patient is automatically diagnosed, so as to gain time for the clinician to make targeted treatment decisions and effectively reduce the occurrence of adverse events in the prognosis of the patient. SUMMARY

[0005] The purpose of the present application is to provide a COVID-19 prognosis evaluation method based on multi-time point CT image analysis, to realize early prediction of the future prognosis of the patient, to gain time for the clinician to make targeted treatment decisions, and to effectively reduce the occurrence of adverse events in the prognosis of the patient.

[0006] The purpose of the present application is achieved by the following technical solution: a COVID-19 prognosis evaluation method based on multi-time point CT image analysis, comprising the following steps:

[0007] For the data corresponding to the patient to be prognostically evaluated, the initial 1-2 CT data thereof is selected for preprocessing to obtain 3D CT data;

[0008] The 3D CT data is input into the trained prognosis evaluation model to obtain the probability of whether the patient will have an adverse event;

[0009] The prognosis evaluation model comprises a feature extraction module, a 3D lesion feature Attention module, a feature conversion module and a classification module. The feature extraction module supports processing of single patient multi-time CT data, shares a network structure, and extracts intermediate feature maps between the multi-time CT data. The 3D lesion feature Attention module comprises a 3D channel Attention module and a 3D space Attention module, which are used to assign weight information to the features obtained by the feature extraction layer. The feature conversion module combines the intermediate feature maps extracted by the feature extraction module and the feature weight information obtained by the 3D lesion feature Attention module, enhances the important lesion features extracted by the entire network structure, and weakens the features irrelevant to COVID-19 itself. The final classification module automatically evaluates the possibility of the patient having an adverse event based on the multi-time point CT longitudinal features of the same patient through a classification network that can output prognosis good and prognosis adverse probability values.

[0010] The prognosis evaluation model is obtained through the following steps:

[0011] S1, CT data of COVID-19 patients is preprocessed: including multi-time point CT registration, extraction of lung region, size unification and pixel normalization, generating 3D CT data required by the network input as a data set;

[0012] S2, constructing a longitudinal prognosis prediction network based on multi-time point CT image analysis: the whole network structure supports the input of multi-time point CT data of a single patient, and the longitudinal prognosis prediction network includes a feature extraction module, a 3D lesion feature Attention module, a feature conversion module and a classification module;

[0013] S3, based on the preprocessed training data set, cross-entropy loss as the objective function, using the back propagation algorithm to calculate the gradient of the objective function with respect to the parameters of the longitudinal prognosis prediction network, using the Adam optimization algorithm to learn the optimal parameters of the network, and obtaining the prognosis evaluation model.

[0014] The specific process of the specific preprocessing of the CT data of COVID-19 patients is as follows: first, merging 2D CT slice scanning data into 3D CT data; second, registering the multi-time point CT data of the patient; then, marking a bounding box for each patient's CT data to extract the lung region; third, unifying the size of the CT data for network training; and finally, normalizing the pixel values of the CT data to obtain 3D CT data.

[0015] The feature extraction module includes four layers of 3D convolution structure, as shown in the following table:

[0016] Layer number Layer structure F1 Conv3D(1, 32)@3x3x3 + ReLU + MaxPool3D(2, 2) F2 Conv3D(32, 64)@3x3x3 + ReLU + MaxPool3D(2, 2) F3 Conv3D(64, 128)@3x3x3 + ReLU + MaxPool3D(2, 2) F4 Conv3D(128, 256)@3x3x3 + ReLU + MaxPool3D(2, 2) .

[0017] The 3D lesion feature Attention module includes a channel Attention module and a spatial Attention module, the channel Attention module is used to utilize the relationship between different channels of the feature map, wherein each channel of the feature map is a feature detector, M c The calculation formula is as follows:

[0018]

[0019] Wherein, σ represents the sigmoid function, and share the weights of the MLP with the ReLU activation function, and The spatial Attention module is used to utilize the relationship between the spaces of the feature map, as a supplement to the channel Attention, M s The calculation formula is as follows:

[0020]

[0021] wherein denotes a sigmoid function, f denotes a standard convolutional layer Conv3D(2, 1) with a kernel size of 3x3x3 and a sigmoid activation layer,

[0022] The 3D lesion feature Attention module is a combination of the channel Attention module and the spatial Attention module, denoted as M, and the calculation formula is as follows:

[0023]

[0024] wherein denotes an outer product,

[0025] The combination mode of the intermediate feature map F and the Attention map M is as follows:

[0026] F'=M⊙F

[0027] wherein denotes an element-wise multiplication, denotes the final feature map after the Attention module; according to three spatial dimensions, the feature map F' is divided into 27 blocks, for each block, the average value of all pixels is calculated as the representation of the corresponding block, and a sampled feature map is obtained.

[0028] The input of the classification module is the longitudinal feature vector extracted from multiple CTs of the same patient, and the layer structure of the entire network of the classification module contains 3 fully connected layers, and the last layer outputs the probability of whether the patient will have an adverse event through a SoftMax function.

[0029] In addition, the present application also provides a COVID-19 prognosis evaluation system based on multi-time point CT image analysis, comprising a CT data acquisition and preprocessing module and an evaluation module;

[0030] The CT data acquisition and preprocessing module is used for acquiring the corresponding data of the patient to be prognostically evaluated, selecting the CT data of the initial 1-2 times, pre-processing, and obtaining 3D CT data;

[0031] The evaluation module evaluates the 3D CT data prognosis evaluation model based on the prognosis evaluation model, and obtains the probability of whether the patient will have an adverse event;

[0032] The prognosis evaluation model comprises a feature extraction module, a 3D lesion feature Attention module, a feature conversion module and a classification module; the feature extraction module supports processing multiple CT data of a single patient, shares a network structure and extracts features between the multiple CT data; the 3D lesion feature Attention module comprises a 3D channel Attention module and a 3D space Attention module and is used for assigning weight information to the features obtained by the feature extraction layer; the feature conversion module combines the intermediate feature atlas extracted by the feature extraction module and the feature weight information obtained by the 3D lesion feature Attention module, enhances important lesion features extracted by the entire network structure and weakens features irrelevant to COVID-19 itself, and the final classification module automatically evaluates the possibility of an adverse event of a patient based on longitudinal features of CT of the same patient at multiple time points, through a classification network capable of outputting a good prognosis probability value and a poor prognosis probability value.

[0033] In another aspect, the present application can also provide a computer device comprising a processor and a memory, the memory being used to store a computer executable program, the processor reading the computer executable program from the memory and executing, and the processor executing the computer executable program can realize the COVID-19 prognosis evaluation method based on multi-time point CT image analysis.

[0034] A computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor, and the COVID-19 prognosis evaluation method based on multi-time point CT image analysis can be realized.

[0035] Further, the present application is characterized in that:

[0036] Compared with the prior art, the present application has at least the following beneficial effects:

[0037] The application proposes a COVID-19 prognosis evaluation method based on multi-time point CT image analysis, which only needs 1-2 CT data of a patient in the early stage of hospitalization, without the complex lesion labeling work of professional radiologists, can automatically judge whether the patient will have a poor prognosis event in the future, and can help clinicians to make targeted treatment plan in advance to reduce the probability of COVID-19 poor prognosis event. The method proposed in the application supports longitudinal modeling between multi-time point CT data of the same patient, so that the longitudinal change characteristics of the patient in the early stage can be extracted, and the final diagnostic performance can be effectively improved. The longitudinal prognosis prediction network proposed in the application can refine the hidden feature mapping and focus on important areas during the training process, and can provide more insights in the final decision to ensure stable performance improvement. In actual application, the COVID-19 prognosis evaluation method based on multi-time point CT image analysis proposed in the application shows excellent performance and has good application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 An implementable flowchart of the application.

[0039] Figure 2 An implementable preprocessing process schematic diagram of the application.

[0040] Figure 3 A structure diagram of the longitudinal prognosis prediction network proposed in the application.

[0041] Figure 4 A visualization schematic diagram of the 3D lesion feature Attention module in the method proposed in the application. DETAILED DESCRIPTION

[0042] In order to enable the personnel in the technical field to better understand the technical solutions in the application, the technical solutions in the embodiments of the application will be described clearly and completely below in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by the personnel in the field without creative labor should belong to the protection scope of the application.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs. The terms used in the specification of the application herein are only for the purpose of describing the specific embodiments and are not intended to limit the application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0044] REFERENCE Figure 3The application constructs a longitudinal prognosis prediction network LLP-Net based on multi-time point CT image analysis: the longitudinal prognosis prediction network LLP-Net can support multi-time point CT data of a patient as input, extract longitudinal change features between multiple CTs of the patient in the early stage; the whole LLP-Net includes a feature extraction module, a 3D lesion feature Attention module, a feature conversion module and a classification module, the feature extraction module supports processing multiple CT data of a single patient, shares a network structure, extracts features between multiple CT data, that is, an intermediate feature atlas; the 3D lesion feature Attention module includes a 3D channel Attention module and a 3D space Attention module, which is used to assign weight information to the features obtained by the feature extraction layer, so that the model features are concentrated on feature points with more interpretable meanings; the feature conversion module is used to combine the longitudinal features of the multi-time CT data extracted by the feature extraction module (that is, the intermediate feature atlas) and the feature weight information obtained by the 3D lesion feature Attention module, enhance the important lesion features extracted by the whole network structure, and weaken the features irrelevant to COVID-19 itself, and finally the classification module automatically evaluates the possibility of an adverse event of a patient based on the longitudinal features of the multi-time point CT of the same patient through a classification network that can output a prognosis good and a prognosis poor probability value.

[0045] As shown in Figure 1 The application provides a COVID-19 prognosis evaluation method based on multi-time point CT image analysis, which includes the following steps:

[0046] Specific preprocessing of CT data of COVID-19 patients

[0047] Firstly, merge 2D CT slice scanning data: take single CT data (multi-layer 2D scanning slice data) of the same patient as input, and merge in a dimensional space to obtain 3D CT data, and the specific result is shown in the first row of Figure 2 Secondly, register the multi-time point CD data of the patient: take the first 3D CT data of the patient as a reference, and perform an affine transformation on the other CT data according to the alignment operation of the multiple CT data of each patient, and the specific result is shown in the second row of Figure 2 Then, mark a bounding box for each patient's CT to extract a lung region: according to the bounding box of the corresponding lung region, crop the 3D CT of each patient to extract the lung region for subsequent processing, and the specific result is shown in Figure 2The third row shows; again, unify the size of the lung CT data of all patients: calculate the average size of the cropped CT of all patients, and scale the CT of all patients to the unified average size by sampling. Finally, normalize the pixel values of the CT data: truncate the CT pixel values to the window [-1000, 400], and normalize the CT data pixel intensity to the interval [0, 1], and the specific results are as follows Figure 2 As shown in the fourth row, no tedious lesion labeling work of radiologists is required in the whole process, which can save a lot of time and cost of manual labeling.

[0048] Construction of longitudinal prognosis prediction network based on multi-time point CT image analysis

[0049] The whole network structure can support multi-time point CT data of patients as input, and can extract longitudinal change features between early CTs of patients. For a single patient, the size of the input data is N*L*H*W, where "N" represents the number of CTs input by the patient ("N=1 or 2" means that the 3D CT contains the first CT or the first two CTs of the patient), and L, H and W represent the number of CT scan slices, the height and the width of each scan slice, respectively. The structure of the whole network structure is as shown in Figure 3 The whole network structure is shown in the figure, which includes a feature extraction module, a 3D lesion feature Attention module, a feature conversion module and a classification module.

[0050] The feature extraction module supports processing multiple CT data of a single patient, and the CT data share the network structure, so as to extract the longitudinal features between the multiple CT data. The specific network structure is shown in Figure 3 As shown in the corresponding feature extraction module in the figure, the whole feature extraction module contains four layers of 3D convolution structure, and the network parameters and output size of each layer are shown in Table 1. The size of the finally output feature map is Taking the original input of the size of N*196*262*379 as an example, the actual size of the feature map after the feature extraction layer is N*256*10*14*21.

[0051] Table 1 Network layer structure of feature extraction module (taking the original input of the size of N*196*262*379 as an example)

[0052] Layer number Layer structure Output size F1 Conv3D(1, 32)@3x3x3 + ReLU + MaxPool3D(2, 2) N×32×97×130×188 F2 Conv3D(32, 64)@3x3x3 + ReLU + MaxPool3D(2, 2) N×64×47×64×93 F3 Conv3D(64, 128)@3x3x3 + ReLU + MaxPool3D(2, 2) N×128×22×31×45 F4 Conv3D(128, 256)@3x3x3 + ReLU + MaxPool3D(2, 2) N×256×10×14×21

[0053] The 3D lesion feature Attention module includes a channel Attention module and a spatial Attention module, and the specific network structure is as shown in Figure 3 As shown in the corresponding Attention module in the figure, the Attention map output by the channel Attention module is denoted as M c, the Attention map outputted by the spatial Attention module is denoted as M s The 3D lesion feature Attention module is used to give weight information to the features obtained by the feature extraction layer, so that the model features are concentrated on the feature points with more interpretable meanings.

[0054] The channel Attention module is used to utilize the relationship between different channels of the feature map, wherein each channel of the feature map is a feature detector. Specifically, M c The calculation formula of M

[0055]

[0056] wherein σ represents a sigmoid function, and share the weights of the MLP with the ReLU activation function, and

[0057] The specific network layer structure and the output size of each layer of the channel Attention module are shown in Table 2. Taking the original input with the size of N×196×262×379 as an example, the actual size of M c is N×256×1×1×1.

[0058] Table 2 Network layer structure of the channel Attention module (taking the original input with the size of N×196×262×379 as an example)

[0059] Layer number Layer structure Output size C_avg_1 AdaptiveAvgPool3D(1) N×256×1×1×1 C_avg_2 Conv3D(256, 16)@1x1x1 + ReLU N×16×1×1×1 C_avg_3 Conv3D(16, 256)@1x1x1 N×256×1×1×1 C_max_1 AdaptiveMaxPool3D(1) N×256×1×1×1 C_max_2 Conv3D(256, 16)@1x1x1 + ReLU N×16×1×1×1 C_max_3 Conv3D(16, 256)@1x1x1 N×256×1×1×1 C1 Add(C_avg, C_max) + Sigmoid N×256×1×1×1

[0060] The spatial Attention module is used to utilize the relationship between the spaces of the feature map, which is complementary to the channel Attention, M s The calculation formula of M

[0061]

[0062] wherein σ represents a sigmoid function, f represents a standard convolution layer Conv3D(2, 1) with a kernel size of 3×3×3 and a sigmoid activation layer,

[0063] Table 3 Network layer structure of the spatial Attention unit (taking the original input with the size of N×196×262×379 as an example)

[0064] Layer number Layer structure Output size S_avg_1 Mean(dim=1) N×1×10×14×21 S_max_1 Max(dim=1) N×1×10×14×21 S1 Cat([S_avg, S_max]) N×2×10×14×21 S2 Conv3D(2, 1)@3x3x3 + Sigmoid N×1×10×14×21

[0065] The specific network layer structure and output size of each layer of the spatial Attention unit are shown in Table 3. Taking the original input of N x 196 x 262 x 379 as an example, the actual size of M s is N x 1 x 10 x 14 x 21.

[0066] The final 3D lesion feature Attention module is a combination of channel Attention and spatial Attention, denoted as M, and the calculation formula is as follows:

[0067]

[0068] wherein, represents the outer product, Taking the original input of N x 196 x 262 x 379 as an example, the actual size of M is N x 256 x 10 x 14 x 21.

[0069] The feature conversion module is used to combine the intermediate feature map F extracted by the feature extraction module and the feature weight obtained by the 3D lesion feature Attention module, enhance the important lesion features extracted by the entire network structure, and weaken the features irrelevant to COVID-19 itself. The specific network structure is shown in Figure 3 The combination mode of the intermediate feature map F and the Attention map M is as follows:

[0070] F' = M O F (4)

[0071] wherein, O represents element-wise multiplication, represents the final feature map after the Attention module. Considering the limitation of GPU memory, the final feature map F' as the input of the classifier module is memory-intensive. In order to solve the GPU memory problem, the feature map F' is divided into 27 (3 x 3 x 3) blocks according to the three spatial dimensions. For each block, the average value of all pixels is calculated as the representation of the corresponding block. In this way, the sampled feature map not only can greatly reduce the GPU memory usage during network training, but also retains part of the information of the feature map F' in the spatial dimension. Then, the sampled feature map is flattened into a feature vector.

[0072] Table 4 Network layer structure of feature conversion module (taking the original input of N x 196 x 262 x 379 as an example)

[0073]

[0074] The specific network layer structure of the feature conversion module and the output size of each layer are shown in Table 3. Taking the original input of N x 196 x 262 x 379 as an example, the actual size of the final feature vector is N x 6912.

[0075] The input of the final classification module is the longitudinal feature vector extracted from multiple CTs of the same patient. The layer structure of the entire network contains 3 fully connected layers. The last layer outputs the probability of whether the patient will have an adverse event through the SoftMax function. The specific network structure is shown in Table 4. Figure 3 The probability output by the final network of the corresponding classification module in Table 4 represents the possibility of whether the patient will have an adverse event. The specific network layer structure of the classification module and the output size of each layer are shown in Table 5. Taking the original input of N x 196 x 262 x 379 as an example, the final output is a vector of length 2, and the size of the vector value represents the probability of good prognosis and poor prognosis, respectively.

[0076] Table 5 Network layer structure of the classification module (taking the original input of N x 196 x 262 x 379 as an example)

[0077] Layer number Layer structure Output size CL1 Linear(*, 64) + ReLU 64 CL2 Linear(64, 32) + ReLU 32 CL3 Linear(32, 2) + SoftMax 2

[0078] Training a COVID-19 patient prognosis evaluation model

[0079] Due to the small proportion of adverse events, the data of good prognosis and poor prognosis are not balanced, which makes it impossible to directly divide the data set into training, validation and test subsets in proportion according to the traditional method, which may result in the absence of poor prognosis data in the validation or test set, affecting the rationality of the final model evaluation. In order to solve the imbalance problem, the present application adopts a balancing mechanism to separate the positive and negative samples independently, and randomly divide them into training, validation and test sets in proportion, and then merge the negative and positive samples in these sets to construct the final training, validation and test sets. In the present application, the sampling rates of the training set, the validation set and the test set are set to 80%, 10% and 10%, respectively.

[0080] On the basis of this training, the present application selects cross-entropy loss as the objective function, uses the back propagation algorithm to calculate the gradient of the objective function with respect to the longitudinal prognosis prediction network parameters, uses the Adam algorithm to optimize the longitudinal prognosis prediction network LLP-Net network parameters, and obtains the optimal parameters, so as to obtain the poor prognosis event prediction model.

[0081] Applying the trained model to evaluate the prognosis of patients

[0082] For the patient to be evaluated, 1-2 times of CT data are selected, preprocessed through the preprocessing process recorded in step 1, and input into the prognosis evaluation model trained in step 3, so that the possibility of the patient suffering from an adverse event can be automatically obtained, and auxiliary support for the doctor to formulate a targeted treatment plan in advance is provided, and the possibility of the patient suffering from an adverse event in the future is reduced.

[0083] In another aspect, the application also provides a COVID-19 prognosis evaluation system based on multi-time point CT image analysis, comprising a CT data acquisition and preprocessing module and an evaluation module.

[0084] The CT data acquisition and preprocessing module is used to acquire data corresponding to a patient to be evaluated, select initial 1-2 times of CT data, and preprocess the data to obtain 3D CT data.

[0085] The evaluation module evaluates the 3D CT data based on a prognosis evaluation model to obtain the probability of the patient suffering from an adverse event.

[0086] The prognosis evaluation model comprises a feature extraction module, a 3D lesion feature Attention module, a feature conversion module, and a classification module; the feature extraction module supports processing of single-patient multi-time CT data, shares a network structure, and extracts features between the multi-time CT data; the 3D lesion feature Attention module comprises a 3D channel Attention module and a 3D space Attention module, and is used to assign weight information to the features obtained by the feature extraction layer; the feature conversion module combines the intermediate feature atlas extracted by the feature extraction module and the feature weight information obtained by the 3D lesion feature Attention module, enhances important lesion features extracted by the entire network structure, and weakens features irrelevant to COVID-19 itself; and the final classification module automatically evaluates the possibility of the patient suffering from an adverse event based on the multi-time point CT longitudinal features of the same patient through a classification network capable of outputting prognosis good and prognosis adverse probability values.

[0087] In addition, the application also provides a computer device comprising a processor and a memory, the memory being used to store a computer executable program, the processor reading part or all of the computer executable program from the memory and executing, and the processor executing part or all of the computer executable program to realize the COVID-19 prognosis evaluation method based on multi-time point CT image analysis.

[0088] In another aspect, the application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program being executed by a processor to realize the COVID-19 prognosis evaluation method based on multi-time point CT image analysis.

[0089] The computer device can be a notebook computer, a desktop computer or a workstation.

[0090] The processor can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0091] The memory of the present application can be an internal storage unit of a notebook computer, a desktop computer or a workstation, such as a memory, a hard disk, or can be an external storage unit, such as a mobile hard disk, a flash card.

[0092] The computer-readable storage medium can include computer storage medium and communication medium. The computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. The computer-readable storage medium can include read-only memory (ROM), random access memory (RAM), solid state disk (SSD) or optical disk, etc. Among them, the random access memory can include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0093] The specific implementation of the present application is as follows:

[0094] Embodiment

[0095] The chest CT image data used in the numerical experiment of the present application is provided by the First Affiliated Hospital of Xi'an Jiaotong University after strict data desensitization and is approved by the Ethics Committee of the First Affiliated Hospital of Xi'an Jiaotong University (No. XJTU1AF2020LSK-011, date: March 17, 2020). There are a total of 206 COVID-19 patients in the data set, and the data time is from December 24, 2019 to April 20, 2020, including 192 cases of good prognosis and 14 cases of poor prognosis. In order to ensure the reliability of the experimental results, accuracy and AUC are used as evaluation indexes, and the average value of 10 random experiments is used as verification.

[0096] Table 6: Comparative performance of the LPP-Net method proposed in the present application on CT quantity and network structure setting

[0097] Experimental data Network setting Accuracy AUC Using the first CT Removing the Attention module 0.912 0.906 Using the first CT Adding the Attention module 0.936 0.944 Using the first two CTs Removing the Attention module 0.941 0.907 Using the first two CTs Adding the Attention module 0.947 0.950

[0098] As shown in Table 6, the comparative performance of the LPP-Net method proposed in the application on CT quantity and network structure setting: comparing the first and third rows, the second and fourth rows respectively, it can be concluded that in the case of the same network structure, the prediction performance of the method proposed in the application using the CT data of the patient twice will be significantly better than using the single CT data of the patient, which also reflects the ability of the method designed in the application to extract the change characteristics between multiple time points of CT of the patient; in addition, by comparing the first and second rows, the third and fourth rows respectively, it can be concluded that in the case of the same input data, the use of the 3D lesion feature Attention module designed in the application will greatly improve the performance of the model, as shown in Figure 4 the visualization of the intermediate feature map after the 3D lesion feature Attention module, it can be seen that the Attention feature map is mainly concentrated in the lung, ignoring the influence of other parts such as bones. In other words, the 3D lesion feature Attention module proposed in the application can help refine the hidden feature mapping and focus on the important area of the lung, providing more insight in the final model decision and ensuring stable performance improvement of the model.

[0099] Table 7: Performance comparison of the method proposed in the application and ResNet10, ResNet18 and DeCovNet

[0100]

[0101]

[0102] As shown in Table 7, the comparative performance of the LPP-Net method proposed in the application and the existing method: from the experimental results, it can be seen that whether using the first CT or using the first two CT data, the method proposed in the application has the optimal AUC value and the most stable AUC variance, which also shows that the method proposed in the application has very good performance and better application prospect in COVID-19 prognosis evaluation. By observing the comparison results of using the first CT and using the first two CT under the same network structure in Table 2, it is found that only the method of the application obtains the improvement of prediction performance, while ResNet10, ResNet18 and DeCovNet all obtain different degrees of decline, which also shows that the method proposed in the application can extract effective longitudinal change characteristics from the influence of multiple time points of CT, and then significantly improve the final prediction performance.

[0103] The application provides a COVID-19 prognosis evaluation method and system based on multi-time point CT image analysis, which comprises targeted preprocessing of CT data of COVID-19 patients, construction of a longitudinal prognosis prediction network based on multi-time point CT image analysis, training of a COVID-19 patient prognosis evaluation model, and application of the trained model to evaluate the prognosis of the patient; the COVID-19 prognosis evaluation method based on multi-time point CT image analysis proposed in the application shows excellent performance, input of CT data of the patient in the early stage of hospitalization, without complex lesion labeling work of professional radiologists, helps to refine hidden feature mapping and focus on important areas of the lungs, can automatically judge whether the patient will have a prognosis adverse event in the future; can help clinicians to develop targeted treatment plan in advance to save time and reduce the probability of COVID-19 prognosis adverse events.

[0104] It should be understood that the above description is intended for illustration purposes only and is not intended to be limiting. Many embodiments and many applications other than the examples provided would be apparent to those of ordinary skill in the art from this description. The scope of the present teachings should not be limited to the described examples and should be only determined by the appended claims and their equivalents.

Claims

1. A COVID-19 prognosis evaluation method based on multi-time point CT image analysis, characterized in that, The method comprises the following steps: For the data corresponding to the patient to be prognostically evaluated, the first 1-2 times of CT data are preprocessed to obtain 3D CT data; Input the 3D CT data into the trained prognostic evaluation model to obtain the probability of the patient developing an adverse event; The prognostic evaluation model comprises a feature extraction module, a 3D lesion feature Attention module, a feature conversion module, and a classification module; The feature extraction module supports processing multiple CT data of a single patient, shares a network structure, and extracts intermediate feature maps between the multiple CT data; the 3D lesion feature Attention module comprises a 3D channel Attention module and a 3D spatial Attention module, and is used to assign weight information to the features obtained by the feature extraction layer; The feature conversion module combines the intermediate feature maps extracted by the feature extraction module and the feature weight information obtained by the 3D lesion feature Attention module, enhances the important lesion features extracted by the entire network structure, and weakens the features irrelevant to COVID-19 itself; the final classification module automatically evaluates the possibility of the patient developing an adverse event based on the longitudinal features of the CT of the same patient at multiple time points through a classification network that can output a prognosis good and a prognosis bad probability value; The 3D lesion feature attention module includes a channel attention module and a spatial attention module. The channel attention module is used to utilize the relationship between different channels of a feature map, wherein each channel of the feature map is a feature detector, The calculation formula is as follows: wherein, denotes a sigmoid function, and shares weights with the ReLU activation function, MLP and ; the spatial Attention module is used to utilize the relationship between the spaces of the feature map as a supplement to the channel Attention, The calculation formula is as follows: wherein denotes a sigmoid function, denotes a standard convolutional layer Conv3D(2, 1) with kernel size and a sigmoid activation layer, ; The 3D lesion feature Attention module is a combination of the channel Attention and the spatial Attention module, denoted as M, and the calculation formula is as follows: wherein denotes the exterior product, .

2. The method for COVID-19 prognosis evaluation based on multi-time point CT image analysis according to claim 1, characterized in that, The prognostic evaluation model is obtained through the following steps: S1. Perform targeted preprocessing on the CT data of COVID-19 patients: including multi-time point CT registration, lung region extraction, size unification, and pixel normalization, to generate 3D CT data as a data set required for network input; S2. Construct a longitudinal prognosis prediction network based on multi-time point CT image analysis: the entire network structure supports input of multiple time point CT data of a single patient, and the longitudinal prognosis prediction network comprises a feature extraction module, a 3D lesion feature Attention module, a feature conversion module, and a classification module; S3. Based on the preprocessed training data set, cross-entropy loss is used as the objective function, the back propagation algorithm is used to calculate the gradient of the objective function with respect to the parameters of the longitudinal prognosis prediction network, the Adam optimization algorithm is used to learn the optimal parameters of the network, and the prognostic evaluation model is obtained.

3. The method for COVID-19 prognosis evaluation based on multi-time point CT image analysis according to claim 1, characterized in that, The specific process of the targeted preprocessing of the CT data of COVID-19 patients is as follows: first, merge 2D CT slice scanning data into 3D CT data; second, register the multi-time point CT data of the patient; then, mark a bounding box for the CT data of each patient to extract the lung region; third, unify the size of the CT data for network training; and finally, normalize the pixel values of the CT data to obtain 3D CT data.

4. The method for COVID-19 prognosis evaluation based on multi-time point CT image analysis according to claim 1, characterized in that, The feature extraction module comprises four layers of 3D convolution structures, and the specific structures are shown in the following table: 。 5. The method for COVID-19 prognosis evaluation based on multi-time point CT image analysis according to claim 1, characterized in that, Intermediate feature map and attention map are combined as follows: wherein represents element-wise multiplication, represents the final feature map after the attention module; the feature map is divided into 27 blocks according to three spatial dimensions, for each block, the average value of all pixels is calculated as the representation of the corresponding block, obtaining the sampled feature map. represents element-wise multiplication, represents the final feature map after the attention module; the feature map is divided into 27 blocks according to three spatial dimensions, for each block, the average value of all pixels is calculated as the representation of the corresponding block, obtaining the sampled feature map.

6. The method for COVID-19 prognosis evaluation based on multi-time point CT image analysis according to claim 1, characterized in that, The input of the classification module is the longitudinal feature vector extracted from multiple CTs of the same patient. The entire network of the classification module includes three fully connected layers. The last layer outputs the probability of whether the patient will have an adverse event through the SoftMax function.

7. A COVID-19 prognosis evaluation system based on multi-time point CT image analysis, characterized in that, The method comprises a CT data acquisition and preprocessing module and an evaluation module. The CT data acquisition and preprocessing module is used to acquire the data corresponding to the patient to be evaluated, select the CT data of the initial 1-2 times, and perform preprocessing to obtain 3D CT data. The evaluation module evaluates the 3D CT data based on a prognosis evaluation model to obtain the probability of whether the patient will have an adverse event. The prognosis evaluation model comprises a feature extraction module, a 3D lesion feature Attention module, a feature conversion module, and a classification module. The feature extraction module supports processing multiple CT data of a single patient, shares the network structure, and extracts the features between the multiple CT data. The 3D lesion feature Attention module comprises a 3D channel Attention module and a 3D spatial Attention module, which are used to assign weight information to the features obtained by the feature extraction layer. The feature conversion module combines the intermediate feature atlas extracted by the feature extraction module and the feature weight information obtained by the 3D lesion feature Attention module, enhances the important lesion features extracted by the entire network structure, and weakens the features irrelevant to COVID-19 itself. The final classification module automatically evaluates the possibility of the patient having an adverse event based on the longitudinal features of the CT of the same patient at multiple time points through the classification network that can output the probability of good prognosis and poor prognosis. The 3D lesion feature attention module includes a channel attention module and a spatial attention module. The channel attention module is used to utilize the relationship between different channels of a feature map, wherein each channel of the feature map is a feature detector, The calculation formula is as follows: wherein, denotes a sigmoid function, and shares weights with the ReLU activation function MLP and ; the spatial Attention module is used to utilize the relationship between the spaces of the feature map as a supplement to the channel Attention, The calculation formula is as follows: wherein represents a sigmoid function, represents a standard convolutional layer Conv3D(2, 1) with kernel size and a sigmoid activation layer, ; The 3D lesion feature Attention module is a combination of the channel Attention and the spatial Attention module, denoted as M, and the calculation formula is as follows: wherein denotes the exterior product, .

8. A computer device, comprising: The processor and the memory are included. The memory is used to store a computer executable program. The processor reads the computer executable program from the memory and executes. When the processor executes the computer executable program, the method for COVID-19 prognosis evaluation based on multi-time point CT image analysis in any one of claims 1-6 can be realized.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program. When the computer program is executed by the processor, the method for COVID-19 prognosis evaluation based on multi-time point CT image analysis in any one of claims 1-6 can be realized.

Images