Pneumonia image detection method and device based on hybrid space and inter-channel attention

Abstract

The present invention provides a method and device for detecting pneumonia images based on mixed space and inter-channel attention. The method comprises: step 1: preprocessing the lung X-ray image data; step 2: constructing a first feature network, using the first feature network to extract features from the preprocessed lung X-ray image to obtain a feature map C; step 3: constructing a second feature network, using the second feature network to extract features from the feature map C to obtain a feature map F; step 4: constructing an attention module of mixed space attention and inter-channel attention, using the attention module to process the feature map F to obtain a feature tensor X; step 5: constructing a network classifier, using the network classifier to detect the feature tensor X to obtain a detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image recognition, and in particular to a method and device for detecting pneumonia images based on mixed space and inter-channel attention. Background Art

[0002] Pneumonia is an inflammation that occurs in the terminal airway alveoli and lung interstitium. It can be divided into bacterial pneumonia, viral pneumonia, etc. It has many causes and a high incidence rate. It is one of the most common infectious diseases. Early diagnosis of pneumonia is crucial to its successful treatment. Pneumonia examination methods include X-ray imaging, lung CT, magnetic resonance imaging (MRI), etc. Lung X-ray detection has the advantages of convenient process, low radiation, and low cost. It is the first choice for current clinical testing. However, for doctors, it is a complicated task to check the lesion information in lung medical images through manual reading. The traditional doctor reading method often consumes a lot of time and energy, and the accuracy of the diagnosis mainly depends on the doctor's level and work experience. Due to visual fatigue, environmental interference and other reasons, misdiagnosis and missed diagnosis may occur.

[0003] Since the 21st century, with the development of computer science and technology, mainly image recognition and pattern recognition technology, target detection has emerged. The main task of target detection is to confirm the category and position coordinates of the object in the input image. The types of objects that can be detected are limited by artificially setting the required objects in the image. Since the shape, size and position of objects in each picture are different, improving the accuracy of target detection has always been an issue that needs to be improved. At present, artificial intelligence can achieve expert-level accuracy in auxiliary diagnosis of medical images. The application of artificial intelligence methods in pneumonia diagnosis can effectively improve diagnostic efficiency and quality, and help alleviate the imbalance of medical resources and improve diagnostic efficiency. Lung X-ray image lesion area detection helps doctors make diagnoses by automatically analyzing lung X-ray images and outputting information such as the location and size of the lesion area. However, the lung X-ray image detection task is different from other image detection tasks. Lung X-ray images have the characteristics of high inter-class similarity and low intra-class variability, that is, the image features of different categories have high similarity, and the images of the same category have low differences. Data with such characteristics are prone to model bias and overfitting problems during training, resulting in a decrease in the generalization ability of the network and increasing the difficulty of image recognition. In addition, the effect of lung X-ray pneumonia detection using only traditional networks is not satisfactory, and it is still necessary to improve the classification accuracy by improving the network structure. Summary of the invention

[0004] In order to improve the pneumonia detection effect based on lung X-ray images, the present invention provides a pneumonia image detection method and device based on mixed space and inter-channel attention. The specific scheme is as follows:

[0005] The present invention provides a pneumonia image detection method based on mixed space and inter-channel attention, comprising:

[0006] Step 1: Data preprocessing of lung X-ray images;

[0007] Step 2: construct a first feature network, and use the first feature network to extract features from the preprocessed lung X-ray image to obtain a feature map C;

[0008] Step 3: construct a second feature network, and use the second feature network to extract features from the feature map C to obtain a feature map F;

[0009] Step 4: construct an attention module that mixes spatial attention and inter-channel attention, and use the attention module to process the feature map F to obtain a feature tensor X;

[0010] Step 5: Construct a network classifier, and use the network classifier to detect the feature tensor X to obtain a detection result.

[0011] Furthermore, step 1 specifically includes:

[0012] Step 1.1: Screen out lung X-ray images that do not meet the requirements;

[0013] Step 1.2: Divide the dataset consisting of all lung X-ray images that meet the requirements into training set, validation set and test set;

[0014] Step 1.3: Convert each lung X-ray image in the dataset into an RGB three-channel image;

[0015] Step 1.4: Perform image enhancement on each of the RGB three-channel images;

[0016] Step 1.5: Convert each of the RGB three-channel images after image enhancement into a tensor image;

[0017] Step 1.6: Regularize each channel of each of the RGB three-channel images, and then normalize the tensor image according to the mean vector and standard vector of the three channels;

[0018] Step 1.7: Convert each of the normalized tensor images into a grayscale image.

[0019] Furthermore, step 1.4 specifically includes:

[0020] Step A1: horizontally flipping the RGB three-channel image according to a flipping probability of p=0.5;

[0021] Step A2: adjusting the image properties of the flipped RGB three-channel image, specifically: setting the brightness offset amplitude to 0.5, the contrast offset amplitude to 0.5, the saturation offset amplitude to 0.5, and the hue offset amplitude to 0;

[0022] Step A 3: setting the random cropping area ratio to (0.7, 1.0), randomly cropping the RGB three-channel image after image attribute adjustment to different sizes and aspect ratios, and then scaling the size of the cropped RGB three-channel image to 224×224 pixels;

[0023] Step A4: RandAugmentation is used to automatically augment the RGB three-channel image after scaling each pixel.

[0024] Furthermore, the first feature network uses ResNet101 using Inception convolution as the backbone network, and includes a total of 5 feature extraction layers, namely: a first feature layer, a second feature layer, a third feature layer, a fourth feature layer and a fifth feature layer;

[0025] The feature extraction process of the first feature layer includes: firstly, using 64 convolution kernels with a channel number of 3 to perform a convolution operation on the input preprocessed lung X-ray image, then using a BN layer to perform a BatchNormalization operation, then using a ReLu activation function to process, and finally inputting it into a maximum pooling layer with a channel number of 64 to obtain a feature map C1;

[0026] The feature extraction process of the second feature layer includes two branches, a first branch and a second branch. The two branches repeat the feature extraction operation three times in sequence on the input feature map C1 according to their respective feature extraction processes, and then perform a preset processing operation on the last output of the two branches to obtain the feature map C2; the preset processing operation specifically includes: adding the outputs of the two branches and then processing them using a ReLu activation function;

[0027] The feature extraction process of the third feature layer includes two branches, a third branch and a fourth branch, which perform four feature extraction operations on the input feature map C2 in the third feature layer under different parameter states according to their own feature extraction processes, and then perform the preset processing operation on the output of the last time of the two branches to obtain the feature map C3;

[0028] The feature extraction process of the fourth feature layer includes two branches, a fifth branch and a sixth branch, which perform twenty-three feature extraction operations on the input feature map C3 in the fourth feature layer under different parameter states according to their own feature extraction processes, and then perform the preset processing operation on the output of the last time of the two branches to obtain the feature map C4;

[0029] The feature extraction process of the fifth feature layer includes two branches, the seventh branch and the eighth branch. The two branches perform three feature extraction operations on the input feature map C4 in the fifth feature layer under different parameter states according to their respective feature extraction processes, and then perform the preset processing operation on the last output of the two branches to obtain the feature map C5, and use the feature map C5 as the final feature map C.

[0030] Furthermore, the second feature extraction network adopts an FPN network;

[0031] The feature extraction process of the FPN network includes: using 1×1 convolution to reduce the number of channels of the feature map C5 from 2048 to 256, and then performing an upsampling operation to obtain a feature map of the same size as the feature map C4, recorded as feature map C5_up, and then performing weighted summation of the feature map C5_up and the feature map C4 to obtain the feature map P; using 3×3 convolution to perform feature fusion on the feature map P to obtain the feature map F; and horizontally connecting the feature map F to increase the number of channels to 2048.

[0032] Furthermore, the processing process of the attention module on the feature map F includes:

[0033] Step B1: for each batch of image sets containing m lung X-ray images, each lung X-ray image of size H0×W0 is subjected to the first feature extraction network and the second feature extraction network to obtain a corresponding feature map F, then all feature maps F of each batch of image sets form a feature tensor X;

[0034] Step B2: performing a 1×1 convolution operation on the feature tensor X, and then dividing it by its regularized transpose;

[0035] Step B3: Use the flatten() function to expand the feature tensor X obtained in step B2 from the second dimension, thereby separating the feature tensor of each lung X-ray image into X1, X2, X3, ... X H×W ;

[0036] Step B4: Perform a global average pooling operation on the features at all positions in the feature tensor X to obtain the global class-independent features g:

[0037] Step B5: Calculation By performing a weighted combination of the feature tensors on the scores, the maximum value of all spatial positions of each category is obtained to obtain the class-specific feature tensor a: Where T is a hyperparameter > 0, and Respectively represent X j and X k The transpose of m i represents the classifier parameters of the i-th category;

[0038] Step B6: Get the final f based on the class-specific feature tensor a and the global class-independent feature g i :f i =g+λa i ; and f i The size of is expanded to [m, 2048, 1, 1]; where f i represents the eigenvector of the i-th class;

[0039] Step B7: Put f i The size of is expanded to be the same as the feature tensor X in step B1, and then multiplied by the feature tensor X in step B1 to obtain a new feature tensor X.

[0040] Furthermore, the detection process of the network classifier specifically includes:

[0041] Step 5.1: Apply adaptive average pooling to the feature tensor X;

[0042] Step 5.2: Use the flatten() function to expand the feature tensor X in step 5.1 from the first dimension, and then input the expanded feature tensor X into the fully connected layer for linear transformation to obtain X′, specifically: X′=XA T +b; where A T represents the transpose of A, A represents the parameter matrix of the fully connected layer, and b is the bias row vector;

[0043] Step 5.3: Input the linearly transformed feature vector X′ into the ReLU activation function, specifically: ReLU(X′)=(X′) + =max(0,X′);

[0044] Step 5.4: Input the feature tensor X′ output in step 5.3 into the fully connected layer again for linear transformation, and output a feature tensor X″ with a size of [2048, 2]:

[0045] Step 5.5: Input the linearly transformed feature tensor X″ into the loss function to calculate the loss value; then, the training model is modified through the optimizer, loss function, data evaluation and hyperparameters; wherein, the loss function adopts the binary cross entropy function, specifically: in, C is the number of categories, y represents the true value of the image, Represents the predicted value of the image.

[0046] Furthermore, in step 5.5, the optimizer adopts the SGD optimizer.

[0047] The present invention also provides a pneumonia image detection device based on mixed space and inter-channel attention, comprising:

[0048] A preprocessing module, used for performing data preprocessing on lung X-ray images;

[0049] A first feature network construction module, used to construct a first feature network, and use the first feature network to extract features from the preprocessed lung X-ray image to obtain a feature map C;

[0050] A first feature network construction module is used to construct a second feature network, and use the second feature network to extract features from the feature map C to obtain a feature map F;

[0051] An attention mechanism building module is used to build an attention module that mixes spatial attention and inter-channel attention, and uses the attention module to process the feature map F to obtain a feature tensor X;

[0052] The classifier construction module is used to construct a network classifier, and use the network classifier to detect the feature tensor X to obtain a detection result.

[0053] Beneficial effects of the present invention:

[0054] 1. The present invention incorporates the mixed space and inter-channel attention mechanism into the lung X-ray image classification process, generates class-specific features for the two classes of pneumonia or non-pneumonia, and achieves performance improvement without any additional training burden.

[0055] 2. The present invention applies the attention mechanism to the feature fusion process, which can effectively utilize the information useful for pneumonia detection in different feature extraction layers, suppress irrelevant noise, and thus improve detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flowchart of a pneumonia image detection method based on mixed space and inter-channel attention provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the structure of a first characteristic network provided by an embodiment of the present invention;

[0058] Figure 3A schematic diagram of the structure of a second characteristic network provided by an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of the structure of an attention module for mixed spatial and inter-channel attention provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Example 1

[0062] like Figure 1 As shown, an embodiment of the present invention provides a pneumonia image detection method based on mixed space and inter-channel attention, comprising the following steps:

[0063] S101: performing data preprocessing on the lung X-ray image;

[0064] Specifically, this step includes the following steps:

[0065] S1011: Screen out lung X-ray images that do not meet the requirements;

[0066] S1012: Divide the lung X-ray image dataset into a training set, a validation set, and a test set;

[0067] S1013: converting the lung X-ray image into a three-channel image in RGB order;

[0068] S1014: Perform image enhancement on lung X-ray images;

[0069] As an implementable method, the sub-step specifically includes:

[0070] Step A1: horizontally flip the lung X-ray image according to the probability of p=0.5;

[0071] Step A2: adjusting the image attributes of the lung X-ray image, specifically: setting the brightness offset amplitude to 0.5, the contrast offset amplitude to 0.5, the saturation offset amplitude to 0.5, and the hue offset amplitude to 0;

[0072] Step A 3: randomly crop the lung X-ray images into different sizes and aspect ratios by setting the random cropping area ratio to (0.7, 1.0), and scale the size of the cropped lung X-ray images to 224×224 pixels;

[0073] Step A4: RandAugmentation is used to automatically perform data augmentation on each lung X-ray image.

[0074] S1015: Convert the image format of the lung X-ray image into the Tensor format (i.e., the vector format used in training) and normalize it, i.e., divide each channel by 255;

[0075] S1016: regularizing each channel of the lung X-ray image, and normalizing the tensor image according to the mean vector and standard vector of the three channels;

[0076] Specifically, the mean vector of the given three-channel vector is set to [0,0,0], and the standard vector is set to [1,1,1]. The tensor image is normalized according to the mean vector and standard vector of the three channels, that is:

[0077] S1017: Convert the lung X-ray image into a grayscale image.

[0078] S102: constructing a first feature network, and using the first feature network to extract features from the preprocessed lung X-ray image to obtain a feature map C;

[0079] Specifically, Figure 2 As shown, the first feature network uses ResNet101 using Inception convolution as the backbone network, and includes a total of 5 feature extraction layers, namely: the first feature layer, the second feature layer, the third feature layer, the fourth feature layer and the fifth feature layer;

[0080] The Inception convolution enables the convolution kernel of the original ResNet101 to have independent spaces between different axes (dimensions), channels and layers, that is, it can have different expansion values. For each layer, the expansion values ​​of the two axes of each channel are Determined by the trained supernet, the set d is the set of dilation values ​​of the two axes of all channels in this layer, expressed as follows:

[0081]

[0082] in and represents the expansion value of the x-axis and y-axis in the i-th channel, d max is the maximum expansion value, C out is the number of output channels.

[0083] The supernet has 4 layers, and the training parameters correspond to the second to fifth feature layers of ResNet101. Each layer of the supernet consists of several convolutions covering all possible dilation values. After training the supernet, for each layer, the dilation value is selected according to the principle of minimizing the loss function, and the optimal dilation mode of the layer is determined. The specific method is: for each layer in the supernet, W is the tensor of all original parameters, W i is the tensor of all original parameters of the i-th channel, is the parameter of the dilated convolution kernel of the i-th channel, and is used Represents the stacked i∈{1,2,…,C out}. Since W and Independent of X, optimized by W and The difference and the expected convolution of X are L1, which minimizes L1. X is the input lung X-ray image. Since the difference between each batch of X is not big, the expectation of X can be replaced by the constant α, and the formula is Where 1 is a matrix of all 1s, and * is a convolution operation. When L1 is the smallest, the optimal d for each layer can be determined. Then, the optimal d for each layer is applied as a parameter to ResNet101, and the backbone network of ResNet101 using Inception convolution can be obtained.

[0084] The feature extraction process of the first feature layer includes: firstly, using 64 convolution kernels with a channel number of 3 to perform a convolution operation on the input preprocessed lung X-ray image, then using a BN layer to perform a BatchNormalization operation, then using a ReLu activation function to process, and finally inputting it into a maximum pooling layer with a channel number of 64 to obtain a feature map C1;

[0085] The feature extraction process of the second feature layer includes two branches, a first branch and a second branch. The two branches repeat the feature extraction operation three times in sequence on the input feature map C1 according to their respective feature extraction processes, and then perform a preset processing operation on the last output of the two branches to obtain the feature map C2; the preset processing operation specifically includes: adding the outputs of the two branches and then processing them using a ReLu activation function;

[0086] Among them, the feature extraction process of the first branch includes: using 64 convolution kernels to perform convolution operation on the input feature map, using BN layer to perform Batch Normalization operation, using 64 convolution kernels to perform convolution operation, using BN layer to perform Batch Normalization operation, using 256 convolution kernels to perform convolution operation, and using BN layer to perform BatchNormalization operation; the feature extraction process of the second branch includes: using 256 convolution kernels to perform convolution operation on the input feature map;

[0087] The feature extraction process of the third feature layer includes two branches, a third branch and a fourth branch, which perform four feature extraction operations on the input feature map C2 in the third feature layer under different parameter states according to their own feature extraction processes, and then perform the preset processing operation on the output of the last time of the two branches to obtain the feature map C3;

[0088] Among them, the feature extraction process of the third branch includes: using 128 convolution kernels to perform the first convolution operation on the input feature map, using the BN layer to perform the BatchNormalization operation, using 128 convolution kernels to perform the downsampling operation, using the BN layer to perform the BatchNormalization operation, using 512 convolution kernels to perform the second convolution operation, and using the BN layer to perform the BatchNormalization operation; the feature extraction process of the fourth branch includes: using 512 convolution kernels to perform the downsampling operation on the input feature map;

[0089] The feature extraction process of the fourth feature layer includes two branches, a fifth branch and a sixth branch, which perform twenty-three feature extraction operations on the input feature map C3 in the fourth feature layer under different parameter states according to their own feature extraction processes, and then perform the preset processing operation on the output of the last time of the two branches to obtain the feature map C4;

[0090] Among them, the feature extraction process of the fifth branch includes: using 256 convolution kernels to perform the first convolution operation on the input feature map, using the BN layer to perform the BatchNormalization operation, using 256 convolution kernels to perform the downsampling operation, using the BN layer to perform the BatchNormalization operation, using 1024 convolution kernels to perform the second convolution operation, and using the BN layer to perform the BatchNormalization operation; the feature extraction process of the sixth branch includes: using 1024 convolution kernels to downsample the input feature map;

[0091] The feature extraction process of the fifth feature layer includes two branches, the seventh branch and the eighth branch, which perform three feature extraction operations on the input feature map C4 in the fifth feature layer under different parameter states according to their own feature extraction processes, and then perform the preset processing operation on the output of the last time of the two branches to obtain the feature map C5, and use the feature map C5 as the final feature map C;

[0092] Among them, the feature extraction process of the seventh branch includes: using 512 convolution kernels to perform the first convolution operation on the input feature map, using the BN layer to perform the BatchNormalization operation, using 512 convolution kernels to perform the downsampling operation, using the BN layer to perform the BatchNormalization operation, using 2048 convolution kernels to perform the second convolution operation, and using the BN layer to perform the BatchNormalization operation; the feature extraction process of the eighth branch includes: using 2048 convolution kernels to perform the downsampling operation on the input feature map;

[0093] S103: constructing a second feature network, and using the second feature network to extract features from the feature graph C to obtain a feature graph F;

[0094] Specifically, the second feature network adopts an FPN network; the feature extraction process of the FPN network includes: using a 1×1 convolution to reduce the number of channels of the feature map C5 from 2048 to 256, and then performing an upsampling operation to obtain a feature map of the same size as the feature map C4, recorded as feature map C5_up, and then performing weighted summation of the feature map C5_up and the feature map C4 to obtain the feature map P; using a 3×3 convolution to perform feature fusion on the feature map P to obtain the feature map F; and horizontally connecting the feature map F to increase the number of channels to 2048.

[0095] S104: constructing an attention module (HSC module) that mixes spatial attention and inter-channel attention, and using the attention module to process the feature map F to obtain a feature tensor X;

[0096] Specifically, this step includes the following sub-steps:

[0097] Step B1: for each batch of image sets containing m lung X-ray images, each lung X-ray image of size H0×W0 is subjected to the first feature extraction network and the second feature extraction network to obtain a corresponding feature map F, then all feature maps F of each batch of image sets form a feature tensor X;

[0098] Step B2: performing a 1×1 convolution operation on the feature tensor X, and then dividing it by its regularized transpose;

[0099] Step B3: Use the flatten() function to expand the feature tensor X obtained in step B2 from the second dimension to separate the feature tensor of each lung X-ray image into X1, X2, X3, …, X H×W ;

[0100] Step B4: Perform a global average pooling operation on the features at all positions in the feature tensor X to obtain the global class-independent features g:

[0101] Step B5: Calculation By performing a weighted combination of the feature tensors on the scores, the maximum value of all spatial positions of each category is obtained to obtain the class-specific feature tensor a: Where T is a hyperparameter > 0, and Respectively represent X j and X k The transpose of m i represents the classifier parameters of the i-th category;

[0102] Step B6: Get the final f based on the class-specific feature tensor a and the global class-independent feature g i :f i =g+λa i ; and f i The size of is expanded to [m, 2048, 1, 1]; where f i represents the eigenvector of the i-th class;

[0103] Step B7: Put f i The size of is expanded to be the same as the feature tensor X in step B1, and then multiplied by the feature tensor X in step B1 to obtain a new feature tensor X.

[0104] In this step, the fused feature map F is input into the HSC module to achieve higher accuracy by utilizing the spatial attention of each object category. By obtaining the weights of different channels, the weights of different channels are adjusted to increase the proportion of useful information and reduce the proportion of useless information. By obtaining the weights of all pixels on a feature map, the weights of different pixels are adjusted to increase the proportion of effective features and reduce the impact of background information.

[0105] S105: construct a network classifier, and use the network classifier to detect the feature tensor X to obtain a detection result.

[0106] Specifically, this step includes the following sub-steps:

[0107] S1051: Apply adaptive average pooling to the feature tensor X;

[0108] S1052: Use the flatten() function to expand the feature tensor X in step S1051 from the first dimension, and then input the expanded feature tensor X into the fully connected layer for linear transformation to obtain X′, specifically: X′=XA T +b; where A T represents the transpose of A, A represents the parameter matrix of the fully connected layer, and b is the bias row vector;

[0109] S1053: Input the linearly transformed feature vector X′ into the ReLU activation function, specifically: ReLU(X′)=(X′) + =max(0,X′);

[0110] S1054: The feature tensor X′ outputted from step S1053 is inputted into the fully connected layer again for linear transformation, and a feature tensor X″ with a size of [2048, 2] is outputted:

[0111] S1055: Input the linearly transformed feature tensor X″ into the loss function to calculate the loss value; then, the training model is modified through the optimizer, loss function, data evaluation and hyperparameters; wherein, the loss function adopts the binary cross entropy function, specifically: in, C is the number of categories, y represents the true value of the image, Represents the predicted value of the image.

[0112] As an implementable method, in step S1055, the optimizer adopts SGD optimizer. Specifically:

[0113]

[0114] Among them, α refers to the learning rate, the initial value is 0.01, and each round of iteration is multiplied by 0.1; y (i) -h θ (x (i) ) refers to the loss function, i refers to the number of cycles, j refers to the parameter number, θ j Represents the jth parameter.

[0115] The pneumonia X-ray image detection method based on mixed space and inter-channel attention provided by the present invention performs data preprocessing on the selected lung X-ray images and prepares a data set to enhance image features; the processed lung X-ray images are used to adjust parameters and train a convolutional neural network for feature extraction, the feature extraction network adopts a ResNet101 network, and by constructing a residual block, feature fusion between different feature maps is achieved to solve the problems of gradient vanishing and gradient explosion in the deep network, and convolution operations are performed on the lung X-ray images in the training set in sequence, and then feature fusion is performed, and a mixed space and inter-channel attention mechanism is added, and finally a network classifier is constructed to perform image classification diagnosis, and finally a prediction result is obtained. The present invention utilizes feature fusion and a mixed space and inter-channel attention mechanism to further improve the pneumonia detection accuracy of computer-aided diagnosis technology without increasing the amount of calculation.

[0116] Example 2

[0117] On the basis of the above-mentioned embodiment 1, in the embodiment of the present invention, the size of the lung X-ray image is set to 3×224×224, and the batch is set to 16. The pneumonia image detection method comprises the following steps:

[0118] Step S201: After the first feature network and the second feature network, the size of the feature tensor X composed of all feature maps F of each batch of image sets is [16, 256, 7, 7];

[0119] Step S202: performing a 1×1 convolution operation on the feature tensor X, and then dividing it by its regularized transpose;

[0120] Step S203: Expand the feature tensor X from the second dimension, thereby separating the feature tensor of each lung X-ray image into X1, X2, X3, ... X 49 ;

[0121] Step S204: Perform a global average pooling operation on the features at all positions in the feature tensor X to obtain a global class-independent feature g:

[0122] Step S205: Calculation Take the value of its first dimension; then perform a weighted combination operation of the feature tensor on the score to obtain the maximum value of all spatial positions of each category, and obtain the class-specific feature tensor a:

[0123] Step S206: The final f is obtained by adding the class-specific features and the global class-independent features. i :f i =g+λa i, where λ is 0.1, and the above f i Applied to each score tensor, the final prediction result can be obtained by fusion;

[0124] Step S207: Expand f i The size of is expanded to [16, 2048, 1, 1];

[0125] Step S208: i The size of is expanded to be the same as the feature tensor X in step S201, and then multiplied with the feature tensor X in step S201 to obtain a new feature tensor X;

[0126] Step S209: The new feature tensor X is detected using the network classifier in Example 1 to obtain a detection result, that is, whether the input lung X-ray image is a pneumonia image.

[0127] Example 3

[0128] The embodiment of the present invention also provides a pneumonia image detection device based on mixed space and inter-channel attention, including:

[0129] A preprocessing module, used for performing data preprocessing on lung X-ray images;

[0130] A first feature network construction module, used to construct a first feature network, and use the first feature network to extract features from the preprocessed lung X-ray image to obtain a feature map C;

[0131] A first feature network construction module is used to construct a second feature network, and use the second feature network to extract features from the feature map C to obtain a feature map F;

[0132] An attention mechanism building module is used to build an attention module for mixed spatial and inter-channel attention, and the feature map F is processed by the attention module to obtain a new feature map or feature tensor X;

[0133] The classifier construction module is used to construct a network classifier, and use the network classifier to detect the feature map to obtain a detection result.

[0134] The pneumonia image detection device provided in the embodiment of the present invention is to implement the above-mentioned method embodiment. Its specific functions can be referred to the above-mentioned method embodiment and will not be repeated here.

Claims

1. A pneumonia image detection method based on mixed space and inter-channel attention, characterized in that: include: Step 1: Data preprocessing of lung X-ray images; Step 2: construct a first feature network, and use the first feature network to extract features from the preprocessed lung X-ray image to obtain a feature map C; Step 3: construct a second feature network, and use the second feature network to extract features from the feature map C to obtain a feature map F; Step 4: construct an attention module that mixes spatial attention and inter-channel attention, and use the attention module to process the feature map F to obtain a feature tensor X; The processing process of the attention module on the feature map F includes: Step B1: for each batch of image sets containing m lung X-ray images, each lung X-ray image of size H0×W0 is processed by the first feature network and the second feature network to obtain a corresponding feature map F, then all feature maps F of each batch of image sets form a feature tensor X; Step B2: performing a 1×1 convolution operation on the feature tensor X, and then dividing it by its regularized transpose; Step B3: Use the flatten() function to expand the feature tensor X obtained in step B2 from the second dimension, thereby separating the feature tensor of each lung X-ray image into X1, X2, X3, ... X H×W ; Step B4: Perform a global average pooling operation on the features at all positions in the feature tensor X to obtain the global class-independent features g: Step B5: Calculation By performing a weighted combination of the feature tensors on the scores, the maximum value of all spatial positions of each category is obtained to obtain the class-specific feature tensor a: Where T is a hyperparameter > 0, and Respectively represent X j and X k The transpose of m i represents the classifier parameters of the i-th category; Step B6: Get the final f based on the class-specific feature tensor a and the global class-independent feature g i :f i =g+λa i ; and f i The size of is expanded to [m, 2048, 1, 1]; where f i represents the eigenvector of the i-th class; Step B7: Put f i The size of is expanded to be the same as the feature tensor X in step B1, and then multiplied with the feature tensor X in step B1 to obtain a new feature tensor X; Step 5: construct a network classifier, and use the network classifier to detect the feature tensor X to obtain a detection result; the detection process of the network classifier specifically includes: Step 5.1: Apply adaptive average pooling to the feature tensor X; Step 5.2: Use the flatten() function to expand the feature tensor X in step 5.1 from the first dimension, and then input the expanded feature tensor X into the fully connected layer for linear transformation to obtain X′, specifically: X′=XA T +b; where A T represents the transpose of A, A represents the parameter matrix of the fully connected layer, and b is the bias row vector; Step 5.3: Input the linearly transformed feature vector X′ into the ReLU activation function, specifically: ReLU(X′)=(X′) + =max(0,X′); Step 5.4: Input the feature tensor X′ output in step 5.3 into the fully connected layer again for linear transformation, and output a feature tensor X″ with a size of [2048, 2]; Step 5.5: Input the linearly transformed feature tensor X″ into the loss function to calculate the loss value; then, the training model is modified through the optimizer, loss function, data evaluation and hyperparameters; wherein, the loss function adopts the binary cross entropy function, specifically: in, C is the number of categories, y represents the true value of the image, Represents the predicted value of the image.

2. The pneumonia image detection method based on mixed space and inter-channel attention according to claim 1 is characterized in that: Step 1 specifically includes: Step 1.1: Screen out lung X-ray images that do not meet the requirements; Step 1.2: Divide the dataset consisting of all lung X-ray images that meet the requirements into training set, validation set and test set; Step 1.3: Convert each lung X-ray image in the dataset into an RGB three-channel image; Step 1.4: Perform image enhancement on each of the RGB three-channel images; Step 1.5: Convert each of the RGB three-channel images after image enhancement into a tensor image; Step 1.6: Regularize each channel of each of the RGB three-channel images, and then normalize the tensor image according to the mean vector and standard vector of the three channels; Step 1.7: Convert each of the normalized tensor images into a grayscale image.

3. The pneumonia image detection method based on mixed space and inter-channel attention according to claim 2 is characterized in that: Step 1.4 specifically includes: Step A1: horizontally flipping the RGB three-channel image according to a flipping probability of p=0.5; Step A2: adjusting the image properties of the flipped RGB three-channel image, specifically: setting the brightness offset amplitude to 0.5, the contrast offset amplitude to 0.5, the saturation offset amplitude to 0.5, and the hue offset amplitude to 0; Step A3: setting the random cropping area ratio to (0.7, 1.0), randomly cropping the RGB three-channel image after image attribute adjustment to different sizes and aspect ratios, and then scaling the size of the cropped RGB three-channel image to 224×224 pixels; Step A4: RandAugmentation is used to automatically augment the RGB three-channel image after scaling each pixel.

4. The pneumonia image detection method based on mixed space and inter-channel attention according to claim 1 is characterized in that: The first feature network uses ResNet101 using Inception convolution as the backbone network, and includes 5 feature extraction layers, namely: the first feature layer, the second feature layer, the third feature layer, the fourth feature layer and the fifth feature layer; The feature extraction process of the first feature layer includes: firstly, using 64 convolution kernels with a channel number of 3 to perform a convolution operation on the input preprocessed lung X-ray image, then using a BN layer to perform a BatchNormalization operation, then using a ReLu activation function to process, and finally inputting it into a maximum pooling layer with a channel number of 64 to obtain a feature map C1; The feature extraction process of the second feature layer includes two branches, a first branch and a second branch. The two branches repeat the feature extraction operation three times in sequence on the input feature map C1 according to their respective feature extraction processes, and then perform a preset processing operation on the last output of the two branches to obtain the feature map C2; the preset processing operation specifically includes: adding the outputs of the two branches and then processing them using a ReLu activation function; The feature extraction process of the third feature layer includes two branches, a third branch and a fourth branch, which perform four feature extraction operations on the input feature map C2 in the third feature layer under different parameter states according to their own feature extraction processes, and then perform the preset processing operation on the output of the last time of the two branches to obtain the feature map C3; The feature extraction process of the fourth feature layer includes two branches, a fifth branch and a sixth branch, which perform twenty-three feature extraction operations on the input feature map C3 in the fourth feature layer under different parameter states according to their own feature extraction processes, and then perform the preset processing operation on the output of the last time of the two branches to obtain the feature map C4; The feature extraction process of the fifth feature layer includes two branches, the seventh branch and the eighth branch. The two branches perform three feature extraction operations on the input feature map C4 in the fifth feature layer under different parameter states according to their respective feature extraction processes, and then perform the preset processing operation on the last output of the two branches to obtain the feature map C5, and use the feature map C5 as the final feature map C.

5. The method for detecting pneumonia images based on hybrid space and inter-channel attention according to claim 4, characterized in that: The second feature network adopts an FPN network; The feature extraction process of the FPN network includes: using 1×1 convolution to reduce the number of channels of the feature map C5 from 2048 to 256, and then performing an upsampling operation to obtain a feature map of the same size as the feature map C4, recorded as feature map C5_up, and then performing weighted summation of the feature map C5_up and the feature map C4 to obtain the feature map P; using 3×3 convolution to perform feature fusion on the feature map P to obtain the feature map F; and horizontally connecting the feature map F to increase the number of channels to 2048.

6. The method for detecting pneumonia images based on hybrid space and inter-channel attention according to claim 1, characterized in that: In step 5.5, the optimizer adopts the SGD optimizer.

7. A pneumonia image detection device based on mixed space and inter-channel attention, comprising: A preprocessing module, used for performing data preprocessing on lung X-ray images; A first feature network construction module, used to construct a first feature network, and use the first feature network to extract features from the preprocessed lung X-ray image to obtain a feature map C; A first feature network construction module is used to construct a second feature network, and use the second feature network to extract features from the feature map C to obtain a feature map F; An attention mechanism building module is used to build an attention module that mixes spatial attention and inter-channel attention, and uses the attention module to process the feature map F to obtain a feature tensor X; A classifier construction module, used to construct a network classifier, and use the network classifier to detect the feature tensor X to obtain a detection result; The processing process of the attention module on the feature map F includes: Step B1: for each batch of image sets containing m lung X-ray images, each lung X-ray image of size H0×W0 is processed by the first feature network and the second feature network to obtain a corresponding feature map F, then all feature maps F of each batch of image sets form a feature tensor X; Step B2: performing a 1×1 convolution operation on the feature tensor X, and then dividing it by its regularized transpose; Step B3: Use the flatten() function to expand the feature tensor X obtained in step B2 from the second dimension, thereby separating the feature tensor of each lung X-ray image into X1, X2, X3, ... X H×W ; Step B4: Perform a global average pooling operation on the features at all positions in the feature tensor X to obtain the global class-independent features g: Step B5: Calculation By performing a weighted combination of the feature tensors on the scores, the maximum value of all spatial positions of each category is obtained to obtain the class-specific feature tensor a: Where T is a hyperparameter > 0, and Respectively represent X j and X k The transpose of m i represents the classifier parameters of the i-th category; Step B6: Get the final f based on the class-specific feature tensor a and the global class-independent feature g i :f i =g+λa i ; and f i The size of is expanded to [m, 2048, 1, 1]; where f i represents the eigenvector of the i-th class; Step B7: Put f i The size of is expanded to be the same as the feature tensor X in step B1, and then multiplied with the feature tensor X in step B1 to obtain a new feature tensor X; The detection process of the network classifier specifically includes: Step 5.1: Apply adaptive average pooling to the feature tensor X; Step 5.2: Use the flatten() function to expand the feature tensor X in step 5.1 from the first dimension, and then input the expanded feature tensor X into the fully connected layer for linear transformation to obtain X′, specifically: X′=XA T +b; where A T represents the transpose of A, A represents the parameter matrix of the fully connected layer, and b is the bias row vector; Step 5.3: Input the linearly transformed feature vector X′ into the ReLU activation function, specifically: ReLU(X′)=(X′) + =max(0,X′); Step 5.4: Input the feature tensor X′ output in step 5.3 into the fully connected layer again for linear transformation, and output a feature tensor X″ with a size of [2048, 2]; Step 5.5: Input the linearly transformed feature tensor X″ into the loss function to calculate the loss value; then, the training model is modified through the optimizer, loss function, data evaluation and hyperparameters; wherein, the loss function adopts the binary cross entropy function, specifically: in, C is the number of categories, y represents the true value of the image, Represents the predicted value of the image.

Images