Novel coronavirus infection course stage prediction system based on attention mechanism

Abstract

The invention discloses a novel coronavirus infection course staging prediction system based on an attention mechanism, including a data preprocessing module, a system network module based on an attention mechanism, and a classifier module; the system network module includes a convolution operation unit, a bottleneck unit, an attention mechanism learning unit, a super-resolution sub-pixel upsampling unit, a super-resolution sub-pixel downsampling unit, a summation unit, and a classifier module; a bilinear convolutional neural network (B-CNN) is a high-precision classification algorithm that realizes end-to-end classification, and can well cope with the current situation of insufficient data volume of novel coronavirus infection. The B-CNN model utilizes image second-order statistical information, and models the combined interaction relationship between local features with the characteristics of translation invariance, and realizes weakly supervised classification in the case of only image category labels. At the same time, B-CNN simplifies gradient calculation, making its end-to-end model easier to train.

Description

Technical Field

[0001] The present invention relates to the field of target detection algorithms in image processing, and specifically to a novel coronavirus infection course staging prediction system based on an attention mechanism. Background Art

[0002] The novel coronavirus infection has become a serious public health event of worldwide concern, and high-resolution computerized tomography (HRCT) is the preferred and main imaging examination method. Understanding and mastering the characteristic HRCT manifestations of novel coronavirus infection is of great significance to improving the clinical diagnosis of novel coronavirus infection. Therefore, after many discussions, domestic experts have compiled Chinese guidelines for the imaging staging and manifestations of novel coronavirus infection, imaging manifestations of special populations, and imaging diagnosis based on their own experience and the latest references on novel coronavirus infection, aiming to standardize and guide HRCT examinations in novel coronavirus infection and assist in the clinical diagnosis and treatment of novel coronavirus infection.

[0003] Huawei Cloud has launched an AI-assisted diagnosis service for novel coronavirus infection, which can segment and quantify the ground-glass opacities (GGO) and pulmonary consolidation in patients' lung CT scans to reduce the workload of doctors. However, these technologies mainly assist in the diagnosis of the universal characteristics of patients infected with novel coronavirus, and have not yet achieved accurate diagnosis of the specific characteristics of cases with rare signs of novel coronavirus infection. Accurate diagnosis of the specific characteristics of novel coronavirus infection cases plays an important role in the prevention and control of the epidemic.

[0004] Fine-grained image classification refers to the division of a large category into more detailed subcategories, such as distinguishing bird species, car brands and models, dog breeds, etc. Currently, most fine-grained recognition networks are committed to using deeper convolutional networks or reconstructing the convolution process to achieve the purpose of extracting more image features. However, from the perspective of reducing information loss caused by the pooling process, the network cannot quickly lock on the key target area in the feature image and extract more detailed features of the target area. Summary of the invention

[0005] Based on the above ideas, the present invention proposes a novel coronavirus infection staging course prediction system based on attention mechanism. This system realizes end-to-end weakly supervised classification, which can well cope with the current situation of insufficient novel coronavirus infection data.

[0006] In order to achieve the above purpose, the following technical solutions are adopted:

[0007] A novel coronavirus infection course staging prediction system based on attention mechanism, including a data preprocessing module, a system network module based on attention mechanism, and a classifier module:

[0008] Data preprocessing module: It is used to select CT images containing lesions from HRCT images of infected patients provided by designated hospitals for the treatment of novel coronavirus infection, perform segmentation processing, obtain regions of interest, unify image sizes, and sort out sequence images with different labels, including early stage, progressive stage, severe stage, and dissipated stage; process the labeled sequence images into corresponding one-hot codes according to the levels, and divide them into training sets and validation sets;

[0009] System network module based on attention mechanism: the system network module includes a neural network module integrating attention mechanism and residual module, which is used to input HRCT images into the neural network module for feature extraction. The neural network module adds a channel attention model between the last convolution layer and the bilinear pooling layer, and the features extracted from the two channels are fused in the neural network module using the bilinear pooling method;

[0010] The system network modules include:

[0011] Convolution operation unit: the input image enters the convolution layer for convolution operation;

[0012] Bottleneck unit: The feature map after the convolution operation enters the bottleneck unit, and 1×1conv is used to reduce or expand the feature map dimension;

[0013] Attention mechanism learning unit: adding channel attention weights to learn the attention mechanism;

[0014] Super-resolution sub-pixel upsampling unit: The extracted feature map then enters the super-resolution sub-pixel upsampling unit to obtain a high-resolution large image, which facilitates the next residual module to extract smaller features;

[0015] Super-resolution sub-pixel downsampling unit: The feature maps extracted by the VGG-M and VGG-D networks of B-CNN (Bilinear Convolutional Neural Networks, B-CNN) are merged and then reduced in dimension through super-resolution sub-pixel downsampling;

[0016] Summation unit: sums the bilinear features obtained at all positions as the features of this image;

[0017] Classifier module: The summed features are passed through the classifier to obtain accurate stage prediction.

[0018] The novel coronavirus infection course staging prediction system based on the attention mechanism, the system network module based on the attention mechanism, respectively selects VGG-M and VGG-D, namely VGG-16, as feature extraction networks, on this basis, introduces residual network and attention mechanism, and replaces the last fully connected layer and Softmax layer of the two VGG networks with bilinear pooling layers.

[0019] The novel coronavirus infection course staging prediction system based on the attention mechanism, the convolution operation unit: assuming v = [v1, v2, ..., v c ] is the convolution kernel of Ftr operation, F represents the convolution operation and pooling operation before the last convolution layer, and Ftr represents the convolution operation of the last convolution layer; v c is the parameter of the cth convolution kernel, and the feature map output by the Ftr operation is represented as u=[u1,u2,…,u c ], then the feature map of the cth channel is expressed as follows:

[0020]

[0021] In the above formula, * represents the convolution operation. is a two-dimensional spatial kernel, denoted by v c The convolution kernel of a single channel acts on the corresponding channel of X', where c' represents the number of channel feature maps input to the convolution layer; by c ∈R H ×W Perform global average pooling and global maximum pooling operations to effectively learn the location information and object features of objects. The formal description is shown in the formula:

[0022]

[0023] z c2 =max(u c (i,j))(i∈[1,2,…,H],j∈[1,2,…,W])

[0024] The novel coronavirus infection course staging prediction system based on the attention mechanism, the attention mechanism learning unit: the calculation of the channel attention weight is shown in the formula:

[0025]

[0026] in Refers to the ReLU activation function, s = [s1, s2, …, s c ],W0∈R c / r×c is the weight of the first FC layer, W1∈R c×c / ris the weight of the second FC layer; the obtained channel weight is multiplied element by element with the input feature map to obtain the weighted distribution feature map. The calculation process is shown in the formula:

[0027]

[0028] in F scale (·) is an element-by-element multiplication operation. The weighted feature map has the same size as the original feature map, that is, u c ∈R H×W .

[0029] The novel coronavirus infection course staging prediction system based on the attention mechanism, the super-resolution sub-pixel downsampling unit: replace the last fully connected layer and Softmax layer of the two-way VGG network with a bilinear pooling layer, and perform bilinear combination and pooling on the output results of the two-way feature function to obtain the final bilinear feature. The feature extraction function f(·) of each channel is a feature mapping function, and the mapping relationship is expressed as f:L×I→R c×D , I represents the input image, L represents the position, R c×D Specifically, the input image I and the corresponding position L of the image are mapped into a feature of size c×D through the feature mapping function (feature extractor), and then the matrix outer product operation is used at each position of the feature to aggregate the features into a bilinear feature, that is, f at position l. A and f B The bilinear feature combination formula is:

[0030] Bilinea r(l,I,f A ,f B )=f A (l,I) T f B (l,I)

[0031] At each position of the image, the two networks generate features of size 1×512 respectively, and the features f extracted by the two networks are compared at each position. a and f b Perform outer product operation: X(I) = f a (I)f b (I) We obtain the bilinear feature of this position, with a size of 512×512.

[0032] The novel coronavirus infection course staging prediction system based on the attention mechanism, the summing unit: in order to obtain a bilinear feature vector, a pooling function P is used to aggregate the bilinear features of all positions of the image to obtain a bilinear vector representing the global image Φ(I), and the expression is:

[0033] Φ(I)=∑ l∈L Bilinea r(l,I,f A ,f B )

[0034] If f A and f B The sizes of the extracted feature maps are C×M and C×N respectively, so the size of Φ(I) is M×N. Then Φ(I) is reshaped into a bilinear vector of MN×1;

[0035] The summation pooling method is used to sum the bilinear features obtained at all positions: Its size is also 512×512. Next, the bilinear feature is calculated as follows: Get its signed square root and regularize it: This is used as the feature of the picture and is used for classification.

[0036] The novel coronavirus infection course staging prediction system based on the attention mechanism, the classifier module: the loss function of the entire network is defined as the cross entropy loss m is the number of images, and Y is the known period label, so the gradient of the loss function of A and B is obtained by the chain rule:

[0037]

[0038] The gradient descent algorithm is used to minimize the loss function and the parameters in the deep learning model are adjusted according to the gradient direction. After the model training is completed, the loss function is minimized and the stage Y predicted by the image is output.

[0039] The beneficial effects of the present invention are:

[0040] The method of the present invention constructs and trains a full-course staging model for HRCT images of novel coronavirus infection. The model is improved based on the bilinear convolutional neural network (B-CNN), and the model is trained using HRCT images of infected patients provided by designated hospitals for the treatment of novel coronavirus infection. The trained model is compatible with most of the current medical image acquisition equipment and can be flexibly embedded in various auxiliary diagnosis systems to assist physicians in making fast and accurate stage diagnosis of the disease. The results of the present invention will be applied to multiple fields: hospitals do not need to invest a lot of money in purchasing diagnostic equipment, and existing medical equipment can complete early screening and accurate diagnosis of patients with low cost expenditure; this project will also cultivate a group of important technical backbones for computer-aided diagnosis, who will become the backbone of the project team to support the sustainable development of the project, and their published papers, monographs and other scientific research results will also be applied to clinical treatment.

[0041] The AUC of the staging results of the HRCT image staging model for the entire course of novel coronavirus infection is 91.4%, and the response time for completing a staging task is less than or equal to 2 seconds. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a specific flow chart of the novel coronavirus infection course staging prediction system based on the attention mechanism of the present invention;

[0043] Figure 2 This is a flow chart of the operation of the sub-pixel convolution module of the novel coronavirus infection course staging prediction system based on the attention mechanism of the present invention;

[0044] Figure 3 HRCT characteristic images of each stage of the course of the disease based on the novel coronavirus infection staging prediction system based on the attention mechanism of the present invention; a ultra-early patchy ground-glass in the lower lobe of the lung; b early ground-glass shadows and halo nodular shadows; c progressive stage bilateral ground-glass shadows increase significantly and the range increases; d progressive stage combined with dissipation stage GGO or consolidation lesions increase, subsegmental atelectasis, e dissipation stage patchy shadows evolve into fibrosis, and pulmonary consolidation lesions gradually disappear; f severe diffuse lesions in both lungs, "white lung";

[0045] Figure 4 An image with early labels during the training of the novel coronavirus infection course staging prediction system based on the attention mechanism of the present invention;

[0046] Figure 5 It is an image with a mid-term label when training the novel coronavirus infection staging prediction system based on the attention mechanism of the present invention;

[0047] Figure 6 An image with a late-stage label when training the novel coronavirus infection staging prediction system based on the attention mechanism of the present invention;

[0048] Figure 7 This is a comparison chart of the detection rates of the novel coronavirus infection stage prediction system based on the attention mechanism of the present invention and the other two methods.

[0049] Figure 8 This is a comparison chart of the false detection rates of the novel coronavirus infection staging prediction system based on the attention mechanism of the present invention and the other two methods;

[0050] Fig. 9 This is a comparison chart of the detection time of the novel coronavirus infection staging prediction system based on the attention mechanism of the present invention and the other two methods;

[0051] Fig.10 It is the ROC curve diagram of the novel coronavirus infection course staging prediction system based on the attention mechanism of the present invention and the other two methods. DETAILED DESCRIPTION

[0052] The present invention is described in detail below in conjunction with specific embodiments.

[0053] A novel coronavirus infection stage prediction method based on attention mechanism. The overall process is as follows: Figure 1 As shown in the figure, the image features based on which the stages are divided are as follows Figure 3 As shown, including the following:

[0054] A1 For the HRCT images of infected patients provided by designated hospitals for the treatment of novel coronavirus infection, select the CT images containing lesions, perform segmentation processing, obtain the region of interest, unify the image size, and adjust the input image to 448×448 size. According to the doctor's label, a sequence of images with different labels (early stage, progressive stage, severe stage, and dissipated stage) is obtained;

[0055] A2 processes the labeled sequence images (early stage, progressive stage, severe stage and dissipated stage) into corresponding one-hot codes and divides them into training set and validation set respectively;

[0056] A3 HRCT images are input into the neural network integrating attention mechanism and residual module for feature extraction;

[0057] Step A3 integrates the neural network of the attention mechanism and the residual module as follows Figure 1 As shown, Figure 1 As shown in the figure, the network is improved based on the B-CNN model built by the VGG-16 network. The improvement is mainly reflected in that ResNet directly uses stride=2 convolution for downsampling, and replaces the fully connected layer with the global average pool layer. An important design principle of ResNet is that when the size of the feature map is reduced by half, its number is doubled, maintaining the complexity of the network layer. The features of the image are extracted using the VGG-M and VGG-D dual channels of B-CNN, and then a channel attention model is added between the last convolution layer and the bilinear pooling layer. Secondly, the dual-channel features are fused using the bilinear pooling method, and finally input into the classifier for classification and staging;

[0058] The specific steps of step A3 are as follows:

[0059] A31. Input image enters the convolution layer for convolution operation

[0060] like Figure 1 As shown, Figure 1 In the above example, F represents the convolution operation and pooling operation before the last convolution layer, and Ftr represents the convolution operation of the last convolution layer. Assume v = [v1, v2, …, vc ] is the convolution kernel of Ftr operation, v c is the parameter of the cth convolution kernel. The feature map output by the Ftr operation is represented as u=[u1,u2,…,u c ], then the cth

[0061] The feature map of the channel is expressed as follows:

[0062]

[0063] In the above formula, * represents the convolution operation. is a two-dimensional spatial kernel, denoted by v c The convolution kernel of a single channel acts on the corresponding channel of X', where c' represents the number of channel feature maps input to the convolution layer. c ∈R H ×W Perform global average pooling and global maximum pooling operations to effectively learn the location information and object features of objects. The formal description is shown in the formula:

[0064]

[0065] z c2 =max(u c (i,j))(i∈[1,2,…,H],j∈[1,2,…,W])

[0066] A32. The feature map after the convolution operation enters the "BottleNeck" module

[0067] By using 1×1conv to reduce or expand the feature map dimension, the number of filters of 3×3conv is not affected by the external environment, that is, the input of the previous layer, and naturally its output will not affect the next layer module. It can save computing time and thus reduce the time required for the entire model training.

[0068] A33. Add channel attention weights for attention mechanism learning

[0069] To make full use of the information aggregated in the pooling operation, other operations are needed to capture the dependencies between channels. To achieve this goal, the operation must meet two conditions: it can learn nonlinear relationships and non-mutually exclusive relationships between channels. In order to meet these conditions, the present invention chooses to use a multilayer perceptron with one hidden layer to meet the first condition, and uses the Sigmod activation function to meet the second condition. The calculation of the channel attention weight is shown in the formula:

[0070]

[0071] in Refers to the ReLU activation function, s = [s1, s2, …, s c ],W0∈R c / r×c is the weight of the first FC layer, W1∈R c×c / r is the weight of the second FC layer. The obtained channel weight is multiplied element by element with the input feature map to obtain the weighted distribution feature map. The calculation process is shown in the formula:

[0072]

[0073] in F scale (·) is an element-by-element multiplication operation. The weighted feature map has the same size as the original feature map, that is, u c ∈R H×W .

[0074] A34. The extracted feature map enters the super-resolution sub-pixel upsampling module (PS↑) to obtain a high-resolution large image, which is convenient for the next residual module to extract smaller features.

[0075] Sub-pixel convolution (PS: Sub-pixel convolution) is a clever way to upsample images and feature maps, also known as pixel shuffle. Figure 2 As shown in the figure: the front part is a normal CNN network, and the back part is the sub-pixel convolution operation. First, if you want to enlarge the original image by ×3, you need to generate 3 2 = 9 feature maps of the same size. Nine feature maps of the same size are stitched together into one large image. This is the operation of sub-pixel convolution.

[0076] This is a reverse idea of ​​sampling. If we sample every three points of a ×3 large image, we will get 9 low-resolution images. Therefore, if we can use CNN to obtain 9 low-resolution images that match the distribution, we can form a high-resolution large image.

[0077] A35. The feature maps extracted by B-CNN's VGG-M and VGG-D are merged and then reduced in dimension through the super-resolution sub-pixel downsampling module PS↓

[0078] After extracting the features of the HRCT image using B-CNN's VGG-M and VGG-D, the feature maps extracted by the two networks are merged and then reduced in dimension through the super-resolution sub-pixel downsampling module PS↓. The specific method is as follows: replace the last fully connected layer and Softmax layer of the two VGG networks with bilinear pooling layers, and perform bilinear combination and pooling on the output results of the two feature functions to obtain the final bilinear features. The feature extraction function f(·) of each channel is a feature mapping function, and the mapping relationship is expressed as f:L×I→R c×D , I represents the input image, L represents the position, R c×D Specifically, the input image I and the corresponding position L of the image are mapped into a feature of size c×D through the feature mapping function (feature extractor), and then the matrix outer product operation is used at each position of the feature to aggregate the features into a bilinear feature, that is, f at position l. A and f B The bilinear feature combination formula is:

[0079] Bilinea r(l,I,f A ,f B )=f A (l,I) T f B (l,I)

[0080] At each position of the image, the two networks generate features of size 1×512 respectively, and the features f extracted by the two networks are compared at each position. a and f b Perform outer product operation: X(I) = f a (I)f b (I),,get the bilinear feature of this position, the size is 512×512.

[0081] A36. Sum the bilinear features obtained at all positions as the features of this image

[0082] In order to obtain the bilinear feature vector, the pooling function P is used to aggregate the bilinear features of all positions in the image to obtain the bilinear vector representing the global image Φ(I), which is expressed as:

[0083] Φ(I)=∑ l∈L Bilinea r(l,I,f A ,f B )

[0084] If f A and f BThe sizes of the extracted feature maps are C×M and C×N respectively, so the size of Φ(I) is M×N. Then Φ(I) is reshaped into a bilinear vector of MN×1.

[0085] The summation pooling method is used to sum the bilinear features obtained at all positions: Its size is also 512×512. Next, the bilinear feature is calculated as follows: Get its signed square root and regularize it: This is used as the feature of the picture and is used for classification.

[0086] The summed features of A4 are passed through the classifier to obtain accurate stage prediction.

[0087] During the training process, the two networks can be trained simultaneously, and the entire training process is end-to-end. The loss function of the entire network is defined as the cross entropy loss m is the number of images, and Y is the known period label, so the gradient of the loss function of A and B is obtained by the chain rule:

[0088]

[0089] Use the gradient descent algorithm to minimize the loss function and adjust the parameters in the deep learning model according to the gradient direction. After the model training is completed, the loss function is minimized and the predicted stage Y of the image is output.

[0090] A novel coronavirus infection course staging prediction system based on attention mechanism, including a data preprocessing module, a system network module based on attention mechanism, and a classifier module:

[0091] Data preprocessing module: It is used to select CT images containing lesions from HRCT images of infected patients provided by designated hospitals for the treatment of novel coronavirus infection, perform segmentation processing, obtain regions of interest, unify image sizes, and sort out sequence images with different labels, including early stage, progressive stage, severe stage, and dissipated stage; process the labeled sequence images into corresponding one-hot codes according to the levels, and divide them into training sets and validation sets;

[0092] System network module based on attention mechanism: the system network module includes a neural network module integrating attention mechanism and residual module, which is used to input HRCT images into the neural network module for feature extraction. The neural network module adds a channel attention model between the last convolution layer and the bilinear pooling layer, and the features extracted from the two channels are fused in the neural network module using the bilinear pooling method;

[0093] The system network modules include:

[0094] Convolution operation unit: the input image enters the convolution layer for convolution operation;

[0095] Bottleneck unit: The feature map after the convolution operation enters the bottleneck unit, and 1×1conv is used to reduce or expand the feature map dimension;

[0096] Attention mechanism learning unit: adding channel attention weights to learn the attention mechanism;

[0097] Super-resolution sub-pixel upsampling unit: The extracted feature map then enters the super-resolution sub-pixel upsampling unit to obtain a high-resolution large image, which facilitates the next residual module to extract smaller features;

[0098] Super-resolution sub-pixel downsampling unit: merges the feature maps extracted by the VGG-M and VGG-D networks of B-CNN and performs dimensionality reduction through super-resolution sub-pixel downsampling;

[0099] Summation unit: sums the bilinear features obtained at all positions as the features of this image;

[0100] Classifier module: The summed features are passed through the classifier to obtain accurate stage prediction.

[0101] The novel coronavirus infection course staging prediction system based on the attention mechanism, the system network module based on the attention mechanism, respectively selects VGG-M and VGG-D, namely VGG-16, as feature extraction networks, on this basis, introduces residual network and attention mechanism, and replaces the last fully connected layer and Softmax layer of the two VGG networks with bilinear pooling layers.

[0102] The novel coronavirus infection course staging prediction system based on the attention mechanism, the convolution operation unit: assuming v = [v1, v2, ..., v c ] is the convolution kernel of Ftr operation, F represents the convolution operation and pooling operation before the last convolution layer, and Ftr represents the convolution operation of the last convolution layer; v c is the parameter of the cth convolution kernel, and the feature map output by the Ftr operation is represented as u=[u1,u2,…,u c ], then the feature map of the cth channel is expressed as follows:

[0103]

[0104] In the above formula, * represents the convolution operation. is a two-dimensional spatial kernel, denoted by v c The convolution kernel of a single channel acts on the corresponding channel of X', where c' represents the number of channel feature maps input to the convolution layer; byc ∈R H ×W Perform global average pooling and global maximum pooling operations to effectively learn the location information and object features of objects. The formal description is shown in the formula:

[0105]

[0106] z c2 =max(u c (i,j))(i∈[1,2,…,H],j∈[1,2,…,W])

[0107] The novel coronavirus infection course staging prediction system based on the attention mechanism, the attention mechanism learning unit: the calculation of the channel attention weight is shown in the formula:

[0108]

[0109] in Refers to the ReLU activation function, s = [s1, s2, …, s c ],W0∈R c / r×c is the weight of the first FC layer, W1∈R c×c / r is the weight of the second FC layer; the obtained channel weight is multiplied element by element with the input feature map to obtain the weighted distribution feature map. The calculation process is shown in the formula:

[0110]

[0111] in F scale (·) is an element-by-element multiplication operation. The weighted feature map has the same size as the original feature map, that is, u c ∈R H×W .

[0112] The novel coronavirus infection course staging prediction system based on the attention mechanism, the super-resolution sub-pixel downsampling unit: replace the last fully connected layer and Softmax layer of the two-way VGG network with a bilinear pooling layer, and perform bilinear combination and pooling on the output results of the two-way feature function to obtain the final bilinear feature. The feature extraction function f(·) of each channel is a feature mapping function, and the mapping relationship is expressed as f:L×I→R c×D , I represents the input image, L represents the position, R c×DSpecifically, the input image I and the corresponding position L of the image are mapped into a feature of size c×D through the feature mapping function (feature extractor), and then the matrix outer product operation is used at each position of the feature to aggregate the features into a bilinear feature, that is, f at position l. A and f B The bilinear feature combination formula is:

[0113] Bilinea r(l,I,f A ,f B )=f A (l,I) T f B (l,I)

[0114] At each position of the image, the two networks generate features of size 1×512 respectively, and the features f extracted by the two networks are compared at each position. a and f b Perform outer product operation: X(I) = f a (I)f b (I), get the bilinear feature of this position, the size is 512×512.

[0115] The novel coronavirus infection course staging prediction system based on the attention mechanism, the summing unit: in order to obtain a bilinear feature vector, a pooling function P is used to aggregate the bilinear features of all positions of the image to obtain a bilinear vector representing the global image Φ(I), and the expression is:

[0116] Φ(I)=∑ l∈L Bilinea r(l,I,f A ,f B )

[0117] If f A and f B The sizes of the extracted feature maps are C×M and C×N respectively, so the size of Φ(I) is M×N. Then Φ(I) is reshaped into a bilinear vector of MN×1;

[0118] The summation pooling method is used to sum the bilinear features obtained at all positions: Its size is also 512×512. Next, the bilinear feature is calculated as follows: Get its signed square root and regularize it: This is used as the feature of the picture and is used for classification.

[0119] The novel coronavirus infection course staging prediction system based on the attention mechanism, the classifier module: the loss function of the entire network is defined as the cross entropy loss m is the number of images, and Y is the known period label, so the gradient of the loss function of A and B is obtained by the chain rule:

[0120]

[0121] The gradient descent algorithm is used to minimize the loss function and the parameters in the deep learning model are adjusted according to the gradient direction. After the model training is completed, the loss function is minimized and the stage Y predicted by the image is output.

[0122] Figure 7 This is a comparison chart of the detection rates of the novel coronavirus infection staging prediction system based on the attention mechanism of the present invention and the other two systems. Method 1 uses the traditional B-CNN model for staging prediction without adding the residual module and attention mechanism, and method 2 uses a single VGG-16 model for staging prediction and adds the residual module and attention mechanism; it can be seen that after adding the residual module and attention mechanism, the average detection rate of the novel coronavirus infection staging system is the best, because the addition of the residual module solves the problem of gradient disappearance, and the attention mechanism helps the system find more useful features in decision-making, thereby improving the detection rate.

[0123] Figure 8 This is a comparison chart of the false detection rate of the novel coronavirus infection staging prediction system based on the attention mechanism of the present invention and the other two systems. Method 1 and method 2 describe the same Figure 7 It can be seen that compared with other methods, the system proposed in the present invention can achieve the lowest false detection rate. As the image sequence increases, the false detection rate increases to a certain extent, but the average false detection rate is only 0.06.

[0124] Fig. 9 This is a comparison chart of the detection time of the novel coronavirus infection staging prediction system based on the attention mechanism of the present invention and the other two systems. Method 1 and method 2 describe the same Figure 7 It can be seen that the average decision-making time of the system of the present invention is only 4ms, which is 2.4ms and 7.8ms shorter than the average time of the other two methods, respectively. This shows that adding the attention mechanism and residual network can effectively reduce the running time of the system.

[0125] Fig.10 This is the ROC curve of the novel coronavirus infection stage prediction system based on the attention mechanism of the present invention and the other two systems. Method 1 and method 2 describe the same Figure 7 It can be seen that the AUC value of the present invention reaches 0.914. Compared with the other two methods, the overall accuracy, sensitivity and specificity of the system are higher.

[0126] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A novel coronavirus infection course stage prediction system based on attention mechanism, characterized in that: Including data preprocessing module, system network module based on attention mechanism and classifier module: Data preprocessing module: It is used to select CT images containing lesions from HRCT images of infected patients provided by designated hospitals for the treatment of novel coronavirus infection, perform segmentation processing, obtain regions of interest, unify image sizes, and sort out sequence images with different labels, including early stage, progressive stage, severe stage, and dissipated stage; process the labeled sequence images into corresponding one-hot codes according to the levels, and divide them into training sets and validation sets; System network module based on attention mechanism: The system network module based on attention mechanism includes a neural network module integrating attention mechanism and residual module, which is used to input HRCT images into the neural network module for feature extraction. The neural network module adds a channel attention model between the last convolution layer and the bilinear pooling layer, and uses the bilinear pooling method to fuse the features extracted from the two channels in the neural network module; The system network module based on attention mechanism selects VGG-M and VGG-D as feature extraction networks, respectively, introduces residual network and attention mechanism on this basis, and replaces the last fully connected layer and Softmax layer of the two VGG networks with bilinear pooling layers; The system network module based on the attention mechanism includes: Convolution operation unit: the input image enters the convolution layer for convolution operation; Bottleneck unit: The feature map after the convolution operation enters the bottleneck unit, and 1×1conv is used to reduce or expand the feature map dimension; Attention mechanism learning unit: adding channel attention weight to learn the attention mechanism; the attention mechanism learning unit: the calculation of the channel attention weight is shown in the formula: in Refers to the ReLU activation function, s = [s1, s2, …, s c ],W0∈R c / r×c is the weight of the first FC layer, W1∈R c ×c / r is the weight of the second FC layer; the obtained channel weight is multiplied element by element with the input feature map to obtain the weighted distribution feature map. The calculation process is shown in the formula: in F scale (·) is the element-by-element multiplication operation; the weighted feature map has the same size as the original feature map, that is, u c ∈R H×W ; Super-resolution sub-pixel upsampling unit: The extracted feature map then enters the super-resolution sub-pixel upsampling unit to obtain a high-resolution large image, which facilitates the next residual module to extract smaller features; Super-resolution sub-pixel downsampling unit: The feature maps extracted by the VGG-M and VGG-D networks of B-CNN are merged and then reduced in dimension through super-resolution sub-pixel downsampling; Summation unit: sums the bilinear features obtained at all positions as the features of this image; Classifier module: The summed features are passed through the classifier to obtain accurate stage prediction.

2. According to claim 1, a novel coronavirus infection course staging prediction system based on attention mechanism is characterized in that: The convolution operation unit: Assume v = [v1, v2, ..., v c ] is the convolution kernel of Ftr operation, F represents the convolution operation and pooling operation before the last convolution layer, and Ftr represents the convolution operation of the last convolution layer; v c is the parameter of the cth convolution kernel, and the feature map output by the Ftr operation is represented as u=[u1,u2,…,u c ], then the feature map of the cth channel is expressed as follows: In the above formula, * represents the convolution operation. is a two-dimensional spatial kernel, denoted by v c The convolution kernel of a single channel acts on the corresponding channel of X', where c' represents the number of channel feature maps input to the convolution layer; by c ∈R H×W Perform global average pooling and global maximum pooling operations to effectively learn the location information and object features of objects. The formal description is shown in the formula:

3. According to claim 1, a novel coronavirus infection course staging prediction system based on attention mechanism is characterized in that: The super-resolution sub-pixel downsampling unit described above replaces the last fully connected layer and Softmax layer of the two-way VGG network with a bilinear pooling layer, and performs bilinear combination and pooling on the output results of the two-way feature function to obtain the final bilinear feature; the feature extraction function f(·) of each channel is a feature mapping function, and the mapping relationship is expressed as f:L×I→R c×D , I represents the input image, L represents the position, R c×D Represents a feature of size c×D; specifically, the input image I and the position L corresponding to the image are mapped into a feature of size c×D through a feature mapping function (feature extractor), and then the matrix outer product operation is used at each position of the feature to aggregate the features into a bilinear feature, that is, f at position l A and f B The bilinear feature combination formula is: Bilinea r(l,I,f A ,f B )=f A (l,I) T f B (l,I) At each position of the image, the two networks generate features of size 1×512 respectively, and the features f extracted by the two networks are compared at each position. a and f b Perform outer product operation: X(I) = f a (I)f b (I) We obtain the bilinear feature of this position, with a size of 512×512.

4. According to claim 1, a novel coronavirus infection course staging prediction system based on attention mechanism is characterized in that: The summation unit: In order to obtain a bilinear feature vector, a pooling function P is used to aggregate the bilinear features of all positions of the image to obtain a bilinear vector representing the global image Φ(I), which is expressed as: Φ(I)=∑ l∈L Bilinear r(l,I,f A ,f B ) If f A and f B The sizes of the extracted feature maps are C×M and C×N respectively, so the size of Φ(I) is M×N. Then Φ(I) is reshaped into a bilinear vector of MN×1; The summation pooling method is used to sum the bilinear features obtained at all positions: Its size is also 512×512. Next, the bilinear feature is calculated as follows: Get its signed square root and regularize it: This is used as the feature of the picture and is used for classification.

5. According to claim 1, a novel coronavirus infection course staging prediction system based on attention mechanism is characterized in that: The classifier module described: The loss function of the entire network is defined as the cross entropy loss m is the number of images, and Y is the known period label, so the gradient of the loss function of A and B is obtained by the chain rule: The gradient descent algorithm is used to minimize the loss function and the parameters in the deep learning model are adjusted according to the gradient direction. After the model training is completed, the loss function is minimized and the stage Y predicted by the image is output.

Images