Lung ct image classification system based on domain knowledge and parallel separable convolution swin transformer

Abstract

The application discloses a lung CT image classification system based on domain knowledge and parallel separable convolution Swin Transformer, which comprises a data preprocessing module, an image encoder module, a domain knowledge encoder module, a bidirectional cross attention label alignment module and a classifier module; the ROI region of the HRCT image is subjected to image feature extraction by using the image encoder module, and the extracted image features are guided and constrained by using the text features of COVID-19 cases extracted from the diagnosis report by the domain knowledge encoder module; then the extracted image features and text features are input into the bidirectional cross attention label alignment module for fine-grained alignment; finally, the one-dimensional feature representation of the fused image and text is input into the classifier module composed of self-attention for COVID-19 and other pneumonia category prediction. The application can realize efficient and rapid detection in a short processing time and maintain a high accuracy, and can be used for accurate diagnosis of COVID-19.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical image COVID-19 processing, and uses parallel separable convolution Swin Transformer to extract features of HRCT images, and introduces domain knowledge containing clinical diagnosis information to constrain and guide a precise diagnosis model for distinguishing the differences and connections between COVID-19 and other pneumonia. BACKGROUND

[0002] COVID-19 is a respiratory infectious disease caused by SARS-CoV-2 virus, and its outbreak has caused global public health concern. Because of its high infectivity, it is essential to accurately distinguish COVID-19 from other lung diseases. Currently, reverse transcription polymerase chain reaction (RT-PCR) test can be used to detect whether COVID-19 is infected, but this detection method is time-consuming and may have false negatives, which is not conducive to early diagnosis and treatment of patients. Computed tomography (CT) as a non-invasive imaging technique can detect specific features of COVID-19, such as multiple small patchy shadows or ground glass shadows in both lungs. However, CT images of COVID-19 and other pneumonia may still show some similarities, which poses a barrier to accurate classification. Currently, in the field of medical images, deep convolutional neural networks are usually used to solve medical image classification problems, but there is still no mature technology for accurately distinguishing COVID-19 from other common pneumonia. SUMMARY

[0003] The application discloses a lung CT image classification system based on domain knowledge and parallel separable convolution Swin Transformer, which uses parallel separable convolution Swin Transformer and attention mechanism based on CT images to extract feature differences of COVID-19 and other pneumonia, and introduces domain knowledge containing clinical diagnosis information to guide accurate classification of COVID-19, and has good robustness.

[0004] To achieve the above purpose, the following technical solutions are adopted:

[0005] A lung CT image classification system based on domain knowledge and parallel separable convolution Swin Transformer, comprising a data preprocessing module, an image encoder module, a domain knowledge encoder module, a bidirectional cross-attention label alignment module and a classifier module:

[0006] Data preprocessing module: first, the ROI region of interest is selected from the lung HRCT image, threshold segmentation method is used for segmentation, and the image size is adjusted to 256*256 pixels to enhance the consistency of data and the generalization ability of the model. In order to avoid the overfitting of the model caused by insufficient data, the five-fold cross-validation method is used to divide the training set and the validation set.

[0007] Image encoder module: this module is a multi-stage image feature extraction network based on Swin Transformer, which is composed of parallel separable convolutional Swin Transformer submodules (PSC Swin Transformer) and patch merging submodules, aiming at extracting features from the ROI region of the HRCT image. Then the features extracted at each stage are connected along the channel and input into the spatial attention submodule (SAM) to recalibrate the relative importance of different positions in the feature map. After feature fusion by 1*1 convolution, the output of the last stage is combined through residual connection, and finally the one-dimensional feature representation Z1 of the image is obtained.

[0008] Among them, the parallel separable convolutional Swin Transformer submodule adopts parallel separable convolutional Transformer units based on regular window and moving window division, and the output results of the two parallel branches are connected along the channel, then point-by-point convolution and GeLU activation operation are performed, and finally the obtained one-dimensional sequence is mapped to two-dimensional space. The submodule specifically includes:

[0009] Moving window unit: the input image is respectively input into two branches with offset shift=0 and , which is used to obtain the ability of global and local perception at the same time.

[0010] Module division unit: the image is divided into blocks of the same size and no overlap.

[0011] Position encoding unit: position encoding is added to the input sequence to preserve the relative position information between blocks.

[0012] Separable convolutional Transformer unit: composed of standard multi-head self-attention operation and layer-separable convolution-layer operation. Layer normalization is added before each operation, and jump connection is performed after the module to improve model training and performance.

[0013] Branch fusion unit: the results obtained by the two branches are connected along the channel, and then mapped to two-dimensional space after point-by-point convolution and GeLU activation.

[0014] The Patch Merging submodule merges adjacent image patches to achieve downsampling, enabling better transfer and integration of information between different layers.

[0015] Spatial Attention Submodule: Adjusts the relative importance of spatial location information in the feature map through learnable weights.

[0016] Domain knowledge encoder module: Uses Transformer Block to extract text features from diagnostic information containing domain knowledge.

[0017] Bidirectional cross-attention label alignment module: Input image feature representations and text feature representations from the multimodal feature space into this module to adaptively match and align cross-modal feature representations between medical images and diagnostic reports.

[0018] Classifier module: Input the feature representations of the fused images and text into the classifier to predict the category of COVID-19 and other pneumonias.

[0019] The lung HRCT image classification system, specifically the separable convolutional Transformer unit, consists of standard multi-head self-attention operations and layer-separable convolution-layer operations. Layer normalization is added before each operation, and residual connections are performed after each module to improve model training and performance.

[0020] The formula for the separable convolutional Transformer unit process is expressed as follows:

[0021] F=LCSL(LN(MSA(LN(X))+X))+(MSA(LN(X))+X)

[0022] Where X represents the one-dimensional input sequence, LN represents layer normalization, MSA represents multi-head self-attention operation, LCSL represents layer-separable convolution-layer operation, and F represents the new feature representation obtained after passing through a separable convolution Transformer unit.

[0023] The multi-head self-attention (MSA) operation can be represented by the following formula:

[0024]

[0025]

[0026]

[0027] Where x d-1 This represents the n one-dimensional sequences obtained after moving the window, where n is the number of patches. x represents d-1 The i-th sequence, w q ,w k ,w v These represent the projection matrices of the query, key, and value, respectively. d These are n new one-dimensional sequences whose global information has been extracted after multi-head attention operations. This module uses a multi-head attention mechanism to capture the dependencies between different positions in the sequences. The calculation formula for each attention head is as follows:

[0028]

[0029] Where d represents the dimension of the query, if the number of heads in the multi-head attention is k, then the above operation is repeated k times to obtain the result. By concatenating them, we obtain the final multi-head attention result x. d To keep the computation and number of parameters constant, D needs to be divisible by k. To maintain the integrity of the original image structure and ensure the preservation of spatial information and semantic consistency, after the attention operation, x′ needs to be restored to x through the inverse operation of the second branch module's partitioning movement.

[0030] The Layer-Separable Convolution-Layer (LSCL) operation consists of depthwise separable convolutions and linear layers. Compared to conventional convolution operations, depthwise separable convolutions reduce the number of parameters and computational cost of the model to a certain extent, while improving the computational efficiency of the model while maintaining good feature extraction performance. The operation process is as follows:

[0031] First, the input passes through a linear layer and an activation function to obtain... After projecting it into a two-dimensional space, depthwise separable convolution and nonlinear function operations are performed, and finally it is flattened into a one-dimensional space and passed through a linear layer to obtain the output.

[0032] Depthwise separable convolution consists of depthwise convolution (DW Conv) and pointwise convolution (PW Conv). Unlike traditional convolution, which uses a single kernel to process all channels simultaneously, depthwise convolution applies a 3×3 kernel to each channel and then stacks the results along the channel direction to obtain an output feature map with the same number of channels as the input. Pointwise convolution performs a 1×1 convolution operation on the output of the depthwise convolution. The kernel size is 1×1×C1×C2, where C1 represents the number of input channels and C2 represents the number of output channels, effectively performing a linear combination of the input feature maps along the depth dimension.

[0033] The formula for layer-separable convolution is as follows:

[0034] O1 = Reshape(σ(FC(X)),(H,W))

[0035] O2=σ(Conv(DW Conv(Conv(O1))))

[0036] O = FC(Reshape(O2,(H×W,1)))

[0037] Where X represents the input sequence, FC represents a fully connected layer (FC), and σ(.) represents the PReLU activation function. ReLU Based on this, a learnable parameter is introduced, allowing the slope of the negative part to be adaptively adjusted. (H,W) represents the height and width of the two-dimensional image obtained after reshaping the one-dimensional sequence, Conv represents a 1×1 convolution operation, DW Conv represents a depthwise convolution operation, and (H×W,1) represents the size of the two dimensions obtained after flattening the two-dimensional image. O1 represents the result obtained after reshaping through a non-linear layer, O2 represents the result obtained after O1 undergoing a depthwise separable convolution operation, and the one-dimensional sequence O represents the output result of LCSL.

[0038] The lung HRCT image classification system described above, specifically the Patch Merging submodule, requires downsampling using Patch Merging in each stage except the first stage to capture broader contextual information and achieve ordered information transfer between feature maps of different resolutions. Specifically, Patch Merging divides adjacent pixels into patches, concatenates pixels at the same position within each patch to obtain C feature maps, then connects these C feature maps along the channel direction, and uses a normalization layer to stabilize the input distribution of each layer. Finally, a fully connected layer performs a linear transformation along the channel direction.

[0039] This invention sets the parameter p in Patch Merging to 2, that is, to stitch new patches with a positional interval of 2 in the row and column directions. This achieves the effect of halving the height and width of the feature map and doubling the number of channels. This helps to reduce the size of the feature map while retaining key information, thereby speeding up the model's calculation speed and improving the model's ability to perceive large-scale targets and semantic associations.

[0040] The aforementioned lung HRCT image classification system, specifically the spatial attention submodule, concatenates the outputs F1, F2, F3, and F4 from each stage to obtain a feature map. Formulate it as

[0041] F = [f 1,1 , ..., f i,j f H,W ]

[0042] in Let C represent the feature at spatial location (i,j), and let C represent the number of channels in the feature map, where i∈{1,...,H},j∈{1,...,W}.

[0043] By using weights The convolution kernel compresses the feature map along the channels. The specific operation is q = ω * F, where F represents the feature map and * represents a 1×1 convolution operation. This represents the projection matrix, where each element q i,j Let represent the linear combination of all channels located at spatial position (i,j), where i∈{1,...,H},j∈{1,...,W}.

[0044] q is activated by the Sigmoid activation function σ(.). i,j Scaling to [0,1] is used to calibrate the relative importance of spatial locations, and the process is expressed by the following formula:

[0045] X SA =[σ(q) 1,1 )f1,1 ,...,σ(q i,j )f i,j ,...,σ(q H,W )f H,W ]

[0046] Where X SA This represents the recalibrated feature map.

[0047] The lung HRCT image classification system, specifically the domain knowledge encoder module, first extracts diagnostic information including CRP values, segmental atelectasis, multiple pulmonary cysts, hemoptysis volume, and lymphocyte count, and encodes this information into the input format required by the domain knowledge encoder. The domain knowledge encoder is essentially a Transformer block, comprising a linear embedding layer, multi-head self-attention (MSA), and a multilayer perceptron (MLP). The module's formula is expressed as follows:

[0048] Z2=MLP(LN(MSA(LN(X t ))+X t ))+(MSA(LN(X t ))+X t )

[0049] Where X t LN represents the new feature representation obtained by mapping the input data to a low-dimensional space, and Z2 represents the one-dimensional text feature representation obtained by the domain knowledge encoder module.

[0050] The lung HRCT image classification system described above, specifically the bidirectional cross-attention marker alignment module, first projects the one-dimensional image feature representation Z1 and the one-dimensional text feature representation Z2 onto a multimodal feature space, respectively, to obtain new image representation Z′1 and text representation Z′2. The two feature representations are then input into the bidirectional cross-attention marker alignment module for fine-grained alignment, resulting in new feature representations Z3 and Z4. This process is expressed by the following formula:

[0051]

[0052]

[0053]

[0054] Where Z′1 represents the d-dimensional image feature representation, and Z′2 represents the d-dimensional text feature representation. These represent the projection matrices of the query, key, and value, respectively, and the Attention matrix.i This represents the attention mechanism.

[0055] Then, {Z3,Z4} is convolved with a 1×1 layer to obtain the feature representation Z5 that fuses the information from both the image and text modalities. The formula for this process is as follows:

[0056] Z5 = Conv(concat(Z3,Z4))

[0057] Here, concat(.) means concatenation along the channel, and Conv means 1×1 convolution operation.

[0058] The lung HRCT image classification system, specifically the classifier module, introduces a special classification label [CLS] before the feature representation Z5, performs a self-attention operation on the new feature representation, and obtains the output corresponding to the label [CLS], which is the classification result of COVID-19 and other pneumonias.

[0059] During training, the cross-entropy loss function is used to calculate the difference between the predicted class and the true class, and the COVID-19 accurate diagnostic model is trained by minimizing the loss function.

[0060] The beneficial effects of this invention are:

[0061] This invention employs a lung HRCT image classification system based on domain knowledge and a parallel separable convolutional Swin Transformer to classify COVID-19 and other pneumonias. The parallel separable convolutional Swin Transformer extracts image features from the Regions of Interest (ROIs) of HRCT images, and utilizes specific features of COVID-19 cases extracted from diagnostic reports for guidance and constraints, thereby significantly improving diagnostic accuracy and providing doctors with a powerful auxiliary decision-making tool. This invention achieves efficient and rapid detection in a short processing time while maintaining high accuracy. Even in complex detection environments, the model exhibits excellent robustness and can therefore be applied to the diagnosis of COVID-19. Attached Figure Description

[0062] Figure 1 This describes the overall network architecture of a lung HRCT image classification system based on domain knowledge and parallel separable convolutional Swing Transformer.

[0063] Figure 2 This describes the overall network architecture for an image encoder.

[0064] Figure 3 This describes the network architecture of the parallel separable convolutional Swing Transformer (PSC Swing Transformer) submodule in an image encoder.

[0065] Figure 4 (a) is the network architecture of the separable convolutional Transformer unit (SeqConv Transformer).

[0066] Figure 4 (b) is a layer-separable convolutional-layer (LCSL) network architecture.

[0067] Figure 5 This describes the specific operational process of Spatial Attention (SAM).

[0068] Figure 6 The network architecture for the domain knowledge encoder.

[0069] Figure 7 This is a comparison of the test set accuracy of the present invention with that of two other methods. Detailed Implementation

[0070] The present invention will be described in detail below with reference to specific embodiments.

[0071] A lung HRCT image classification system based on domain knowledge and parallel separable convolutional Swin Transformer, the overall network architecture is as follows: Figure 1 As shown, it includes:

[0072] A1. Data Preprocessing Module: Extracts the region of interest from HRCT images using a threshold segmentation method and standardizes the image size to 256×256 pixels;

[0073] To enable convolutional neural networks to focus more intently on the target region of interest and learn more general features, image preprocessing and cropping are typically required. However, most cropping methods result in damage and deformation of the extracted target region, hindering lesion identification. This invention employs threshold segmentation to extract the Region of Interest (ROI) from lung HRCT images. While preserving the shape and contour information of the target region, it accurately separates the target region. After obtaining the ROI, the image size is standardized to 256×256 pixels to improve data consistency and model generalization ability. To avoid overfitting due to insufficient data, this invention uses five-fold cross-validation to divide the training and validation sets, improving the model's robustness.

[0074] A2. Input the ROI region image into the image encoder module to extract the global features of the region and obtain the image feature representation Z1;

[0075] The deep neural network structure of the image encoder involved in step A2 is as follows: Figure 2As shown, this network is an improvement on the SwinTransformer. The main improvement lies in redesigning the SwinTransformer backbone module: the Parallel Separable Convolutional SwinTransformer (PSC SwinTransformer). The features extracted at each stage are concatenated along the feature channels, and then the spatial attention submodule using the spatial attention mechanism (SAM) is used to recalibrate the relative importance of the spatial information of the given feature map. After changing the number of channels through 1×1 convolution, it is residually concatenated with the output of the last stage to finally obtain the feature representation of the target region of the HRCT image.

[0076] The key design feature of the parallel separable convolutional Swing Transformer submodule is the use of a Layer-Separable Convolution-Layer (LSCL) to replace the MLP in the traditional Transformer encoder. This solves the problem of spatial information loss caused by dimensionality reduction. Unlike the two cascaded Swing Transformer Blocks, which consist of multi-head self-attention W-MSA and SW-MSA based on regular and moving windows respectively, this invention uses a parallel separable convolutional Transformer (SepConv Transformer) based on regular and moving window partitioning. The parallel output results are then concatenated along the channels for further processing.

[0077] The specific steps of step A2 are as follows:

[0078] A21. The input image enters the PSC Swin Transformer module of stage one to perform a depthwise separable convolution operation to obtain the output F1.

[0079] like Figure 3 As shown, the input image Entering offset shift=0 and respectively The input image x has two branches, where (H, W) represents the size of the input image. After entering the branch where shift=0, the input image x is first divided into non-overlapping patches of the same size. For example, each patch consists of 4×4 pixels, resulting in... To match the sequence input of a traditional Transformer module, x needs to be... p Flatten, get in Referring to the Vision Transformer, positional encoding is added to each sequence to preserve the relative positional information between blocks. Another branch sets... The input image x will be moved down and to the right simultaneously. We obtain x′ from 1 pixel. Then, similar to the Swin Transformer, we fix the overlapping part of x and x′, cut the remaining part of x, and move it to the corresponding position of x′ to ensure that x and x′ have the same size. The following steps are the same as the first branch, performing module division to obtain... Flatten x p get Position encoding was then added.

[0080] We input the sequences obtained from the two branches into the separable convolutional Transformer unit, as follows: Figure 4 As shown in (a), this module consists of standard multi-head self-attention (MSA) and layer-separable convolution-layer (LSCL). Layer normalization (LN) is added before each module, and skip connections are performed after the modules to improve model training and performance, and enhance the model's generalization ability.

[0081] The formula for the separable convolutional Transformer unit process is expressed as follows:

[0082] F=LCSL(LN(MSA(LN(X))+X))+(MSA(LN(X))+X)

[0083] Where X represents the one-dimensional input sequence, LN represents layer normalization, MSA represents multi-head self-attention operation, LCSL represents layer-separable convolution-layer operation, and F represents the new feature representation obtained after passing through a separable convolution Transformer unit.

[0084] Multi-Head Self-Attention (MSA):

[0085] The self-attention of multiple bulls can be expressed by the following formula:

[0086]

[0087]

[0088]

[0089] Where x d-1 This represents the n one-dimensional sequences obtained after moving the window, where n is the number of patches. x represents d-1 The i-th sequence, w q ,w k ,w v These represent the projection matrices of the query, key, and value, respectively. d These are n new one-dimensional sequences whose global information has been extracted after multi-head attention operations. This module uses a multi-head attention mechanism to capture the dependencies between different positions in the sequences. The calculation formula for each attention head is as follows:

[0090]

[0091] Where d represents the dimension of the query, if the number of heads in the multi-head attention is k, then the above operation is repeated k times to obtain the result. By concatenating them, we obtain the final multi-head attention result x. d To keep the computation and number of parameters constant, D needs to be divisible by k. To ensure the consistency of the relative positional relationships and overall semantic information of the images, after the attention operation, x′ needs to be restored to x through the inverse operation of the second branch module's movement.

[0092] Layer-Separable Convolution-Layer (LSCL):

[0093] While MSA possesses strong learning capabilities and can effectively extract features and capture dependencies, the generation of attention weights itself lacks direct interpretability. Therefore, convolutional operations are needed to compensate for the learning capacity and enhance the interpretability of the process. Furthermore, since the input to MSA is a one-dimensional sequence, Transformers typically learn it using an MLP. Although positional encoding is introduced to avoid the loss of spatial information caused by dimensionality reduction from 2D to 1D, a better solution remains to operate directly on the image in 2D space. Therefore, this invention uses LSCL instead of the MLP in the Transformer.

[0094] LSCL consists of depthwise separable convolution and linear layers. Compared to conventional convolution operations, depthwise separable convolution can maintain good feature extraction performance while reducing the number of parameters, computational cost and other expenses.

[0095] like Figure 4 As shown in (b), we first obtain the following using a linear layer and activation function: After mapping it to a two-dimensional space, depthwise separable convolution and activation functions are used for learning. Then, it is flattened back to a one-dimensional space and passed through linear layers to obtain the output. Depthwise separable convolution consists of depthwise convolution (DW Conv) and pointwise convolution (PW Conv). The depthwise convolution is implemented using 3×3 convolutions, where each kernel operates on only one channel, and each channel is convolved by only one kernel. That is, the number of input channels, kernels, and output channels remain consistent. However, this operation, which operates on each channel individually, ignores the feature information between channels. Therefore, pointwise convolution is needed to supplement the feature information at the same spatial location in different channels. Pointwise convolution is implemented using 1×1 convolutions, with a kernel size of 1×1×C1×C2, where C1 represents the number of input channels and C2 represents the number of output channels. This means that the input feature map is weighted and combined along the depth direction.

[0096] The formula for a layer-separable convolution layer is as follows:

[0097] O1 = Reshape(σ(FC(X)),(H,W))

[0098] O2=σ(Conv(DW Conv(Conv(O1))))

[0099] O = FC(Reshape(O2,(H×W,1)))

[0100] Where X represents the input sequence, FC represents a fully connected layer (FC), and σ(.) represents the PReLU activation function. ReLU Based on this, a learnable parameter is introduced, allowing the slope of the negative part to be adaptively adjusted. (H,W) represents the height and width of the two-dimensional image obtained after reshaping the one-dimensional sequence, Conv represents a 1×1 convolution operation, DW Conv represents a depthwise convolution operation, and (H×W,1) represents the size of the two dimensions obtained after flattening the two-dimensional image. O1 represents the result obtained after reshaping through a non-linear layer, O2 represents the result obtained after O1 undergoing a depthwise separable convolution operation, and the one-dimensional sequence O represents the output result of LCSL.

[0101] After passing through a separable convolutional Transformer, the two parallel branches (shift=0 and...) are... The output of the circuit is connected along the channels to complement the missing block edge information. Finally, it is projected onto a two-dimensional space and the output of stage one is obtained by pointwise convolution.

[0102] A22. The output F1 of stage one is processed by Patch Merging and then enters the PSC Swin Transformer module to obtain the output F2 of stage two. This process is repeated twice to obtain the output F3 of stage three and the output F4 of stage four.

[0103] In order to enable information exchange and integration among multi-level feature representations, each stage except stage one needs to be downsampled through Patch Merging to reduce the resolution in order to form a hierarchical structure.

[0104] Patch merging specifically includes the following operations:

[0105] 1. Input feature map: Assume the size of the input feature map is H x W x C, where H and W represent the height and width of the feature map, respectively, and C represents the number of channels of the feature map.

[0106] 2. Patch division: Divide the feature map into non-overlapping patches of size p×p×C, where p represents the height and width of the patch, usually taken as 2.

[0107] 3. Patch stitching: Patch the pixels at the same position in each patch together to form p×p×C patches of size 1. The feature map.

[0108] 4. Feature Map Fusion: The concatenated feature maps are normalized using layers to mitigate internal covariate bias, and then passed through fully connected layers to change the number of channels. Assuming p = 2, after Patch Merging, a feature map with halved height and width and doubled channels is obtained, with a size of [size missing].

[0109] A23. Connect the outputs F1, F2, F3, and F4 of each stage along the channels, enter the Spartial Attention Module (SAM) to recalibrate the relative importance of each position, and then perform residual connection with F4 after 1×1 convolution to obtain the feature map F5 of the HRCT image. Finally, flatten F5 to obtain the one-dimensional image feature representation Z1 of the target region.

[0110] Spatial attention SAM, such as Figure 5 As shown, we concatenate F1, F2, F3, and F4 to obtain the feature map. Formulate it as

[0111] F = [f 1,1 , ..., f i,j f H,W ]

[0112] in Let C represent the feature at spatial location (i,j), and let C represent the number of channels in the feature map, where i∈{1,...,H},j∈{1,...,W}.

[0113] By using weights The convolution kernel compresses the feature map along the channels. The specific operation is q = ω * F, where F represents the feature map and * represents a 1×1 convolution operation. This represents the projection matrix, where each element q i,j Let represent the linear combination of all channels located at spatial position (i,j), where i∈{1,...,H},j∈{1,...,W}.

[0114] q is activated by the Sigmoid activation function σ(.). i,j Scaling to [0,1] is used to calibrate the relative importance of spatial locations, and the process is expressed by the following formula:

[0115] X SA =[σ(q) 1,1 )f 1,1 ,...,σ(q i,j )f i,j ,...,σ(q H,W )f H,W ]

[0116] Where X SA This represents the recalibrated feature map. X SA After reducing the channel dimension through 1×1 convolution, the feature map F5 is obtained by adding it element-wise to F4. This process is represented as follows:

[0117] F5 = Conv(X) SA )+F4

[0118] Here, Conv represents a 1×1 convolution operation, and finally F5 is flattened to obtain the one-dimensional feature representation Z1 of the ROI region.

[0119] A3. Input clinical information related to COVID-19 diagnosis into the domain knowledge encoder, extract patient-specific features, and obtain a one-dimensional feature representation of domain knowledge Z2.

[0120] The specific steps are as follows:

[0121] A31. Extract and analyze the specific features of COVID-19, and perform preprocessing operations to meet the input requirements of the domain knowledge encoder.

[0122] Given the potential similarities in CT images between COVID-19 and other pneumonias, more accurate COVID-19 classification requires guidance based on quantified diagnostic information. Therefore, after in-depth discussions and exchanges with COVID-19 experts, the following specific characteristics of patients were extracted:

[0123] 1. CRP value

[0124] C-reactive protein (CRP) is an inflammatory marker whose concentration is typically significantly elevated in COVID-19 patients and higher than in other types of pneumonia. Measuring CRP levels in the blood allows for the assessment of the inflammatory response in patients, which is of great significance for the diagnosis and monitoring of COVID-19.

[0125] 2. Segmental atelectasis

[0126] Segmental atelectasis refers to the partial or complete obstruction of air passages in one or more segments of the lung, preventing normal airflow in the corresponding lung segment. This feature may occur in COVID-19 patients, but is relatively rare in other types of pneumonia. By observing and analyzing lung CT images, we can identify and quantify the degree and extent of segmental atelectasis, thus providing important diagnostic clues for the classification of COVID-19 and other pneumonias.

[0127] 3. Multiple pulmonary cysts

[0128] Multiple lung sacs may be observed in specific cases of COVID-19 patients. Multiple lung sacs refer to the formation of multiple cavities or sac-like structures within the lungs, which is also one of the characteristics of COVID-19.

[0129] 4. Amount of hemoptysis

[0130] Hemoptysis is relatively rare in COVID-19 patients, but it can occur in certain specific cases. Assessing the amount of hemoptysis can provide additional information about lung lesions and help differentiate COVID-19 from other lung diseases.

[0131] 5. Lymphocyte count

[0132] Lymphocytes are an essential part of the body's immune system, and their decline may be related to viral invasion and the immune system's response. In COVID-19 patients, lymphocyte counts typically show a downward trend, while in contrast, lymphocyte counts in other pneumonia patients may be slightly elevated or remain relatively stable.

[0133] Guided by the aforementioned specific feature constraints, the model will have more accurate discrimination and diagnostic capabilities. After that, preprocessing operations such as building a vocabulary, digitization, and generating input tensors are required.

[0134] A32. Input the specific features into the domain knowledge encoder to obtain a one-dimensional feature representation Z2 that integrates diagnostic information.

[0135] Domain knowledge encoders, such as Figure 6 As shown, it includes a linear embedding layer, which maps the input data to a low-dimensional space through linear transformation to obtain a new feature representation X. t The rest of the code is the same as the transformer block, consisting of a multi-head self-attention (MSA) and a multilayer perceptron (MLP). Each module is normalized by a layer normalization (LN) before the output Z2 is obtained by applying residual connections.

[0136] The formula for this process is as follows:

[0137] Z2=MLP(LN(MSA(LN(X t ))+X t ))+(MSA(LN(X t ))+X t )

[0138] A4. Use an image encoder and a domain knowledge encoder to obtain one-dimensional representations of global features in the image and diagnostic information, respectively. Train the global image representation learning function F. i Projecting the image feature representation onto the multimodal feature space yields Z′1, Z1′∈R. d Similarly, the global text representation learning function F is trained. t Projecting the domain knowledge feature representation onto the multimodal feature space yields Z′2, Z2′∈R. d This provides a unified low-dimensional feature representation for the subsequent full integration of image and text information.

[0139] A5. Input the image representation Z′1 and text representation Z′2 of the multimodal feature space into the bidirectional cross-attention label alignment module for fine-grained alignment to obtain new feature representations Z3 and Z4. Connect Z3 and Z4 along the channels and perform a 1×1 convolution to obtain the feature representation Z5 that fuses the image and text.

[0140] We use an efficient bidirectional cross-attention label alignment module to adaptively match and align cross-modal feature representations between medical images and diagnostic reports, the process of which is expressed as follows:

[0141]

[0142]

[0143]

[0144] Where Z′1 represents the d-dimensional image feature representation, and Z′2 represents the d-dimensional text feature representation. These represent the projection matrices of the query, key, and value, respectively, and the Attention matrix. i This represents the attention mechanism.

[0145] Then, {Z3,Z4} is convolved with a 1×1 layer to obtain the feature representation Z5 that fuses the information from both the image and text modalities. The formula for this process is as follows:

[0146] Z5 = Conv(concat(Z3,Z4))

[0147] Here, concat() means concatenation along the channels, and Conv means 1×1 convolution operation.

[0148] A6. Introduce a special classification label [CLS] before the feature representation Z5, perform a self-attention operation on the new feature representation, and the output corresponding to the label [CLS] is the binary classification result of COVID-19 and other pneumonias.

[0149] A7. The entire network uses cross-entropy loss to minimize the distance between the predicted class and the true class.

[0150] Cross-entropy loss (CEL) is one of the commonly used loss functions in classification tasks. It is used to measure the difference between the predicted class and the true class. The formula for cross-entropy loss in a standard classification problem is as follows:

[0151]

[0152] Here, p represents the predicted class probability of the sample, and y represents whether the sample is correctly classified. When y is 1, the closer p is to 1, the smaller the loss; conversely, when y is 0, the closer p is to 0, the smaller the loss, which aligns with the direction of optimization.

[0153] To simplify the expression, we define the piecewise function F as follows:

[0154]

[0155] The standard cross-entropy loss is then restated as:

[0156] CEL(p,y)=-log(F y (p))

[0157] To address the issue of imbalanced data classes, a coefficient α∈[0,1] is added to the standard cross-entropy loss function to adjust the weights of different classes, thus obtaining the Balanced Cross-Entropy Loss Function (BCEL):

[0158] BCEL(p,y)=-F y (α)log(F y (p))

[0159] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

[0160] Figure 7 This figure shows a comparison of the test accuracy of the lung HRCT image classification system based on domain knowledge and parallel separable convolutional Swin Transformer of this invention with two other methods. Method 1 combines a deep dense aggregation structure with DenseNet-201 for class prediction, while Method 2 integrates SE and multi-scale convolution for classification. As can be seen from the figure, the accuracy of this invention is the highest compared to the other two methods, ultimately reaching 99.89%.

[0161] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A lung CT image classification system based on domain knowledge and parallel separable convolutional Swin Transformer, characterized in that, It includes a data preprocessing module, an image encoder module, a domain knowledge encoder module, a bidirectional cross-attention label alignment module, and a classifier module; Data preprocessing module: The threshold segmentation method is used to extract the region of interest (ROI) from the lung HRCT images, and the image size is uniformly set to 256*256 pixels. The five-fold cross-validation method is used to divide the training set and the validation set. Image encoder module: Used to extract features from the ROI region in HRCT images, then concatenates the features extracted at each stage along the channels, inputs them to the spatial attention submodule to recalibrate the relative importance of the spatial information of the feature maps, and finally processes them. After feature fusion via convolution, the output of the final stage is residually concatenated to obtain a one-dimensional feature representation of the image. The image encoder module includes a parallel separable convolutional Swing Transformer submodule, a PatchMerging submodule, and a spatial attention submodule. The parallel separable convolutional Swin Transformer submodule uses parallel separable convolutional Transformer units based on regular and moving window partitioning. It connects the outputs of the two parallel branches along the channels and then performs pointwise convolution and GeLU activation operations. Finally, it maps the resulting one-dimensional sequence to two-dimensional space. The Patch Merging submodule merges adjacent image patches to achieve downsampling, so as to transfer and integrate information between different levels. Spatial Attention Submodule: Employs a spatial attention mechanism (SAM) to adjust the relative importance of information at different locations in the feature map through adaptive weights; Domain knowledge encoder module: Uses transformer blocks to extract text features from diagnostic information containing domain knowledge; Bidirectional cross-attention label alignment module: The image feature representation extracted by the image encoder module and the text feature representation extracted by the domain knowledge encoder module are input into the bidirectional cross-attention label alignment module, which adaptively matches and aligns them with the cross-modal feature representations between medical images and diagnostic reports; Classifier module: Input the feature representations of the fused images and text into the classifier to predict the category of COVID-19 and other types of pneumonia.

2. The lung CT image classification system according to claim 1, characterized in that, The parallel separable convolution SwinTransformer submodule specifically includes: a moving window unit: the input image is shifted by an offset... and The network has two branches to improve the perception and processing of block edge information; a module partitioning unit: divides the image into blocks of the same size that do not overlap; a position encoding unit: adds position encoding to the input sequence to preserve the relative position information between blocks; a separable convolutional Transformer unit: consists of standard multi-head self-attention operations and layer-separable convolution-layer operations; layer normalization is performed before each operation, and skip connections are performed after the modules to enhance the training and performance of the network; a branch fusion unit: connects the results obtained from the two branches along the channels, and maps them to two-dimensional space after pointwise convolution and GeLU activation.

3. The lung CT image classification system according to claim 2, characterized in that, The separable convolutional Transformer unit consists of standard multi-head self-attention operations and layer-separable convolution-layer operations; a layer normalization operation is added before each operation, and residual connections are performed after the module to improve the training and performance of the model; the process formula of the separable convolutional Transformer unit is expressed as follows: ;in This represents a one-dimensional input sequence. Presentation layer normalization, This indicates a bullish self-attention strategy. Representation layer - separable convolution - layer operations, This represents the new feature representation obtained after passing through a separable convolutional Transformer unit; where multi-head self-attention can be expressed by the following formula: ; ; ;in This represents the n one-dimensional sequences obtained after moving the window, where n is the number of patches. express The i-th sequence, These represent the projection matrices of the query, key, and value, respectively. These are n new one-dimensional sequences that have been processed by multi-head attention to extract global information. Different feature representations are learned using a multi-head attention mechanism. The formula for calculating each attention head is as follows: Where d represents the query dimension, if the number of heads in multi-head attention is k, then the above operation is repeated k times to obtain the result. By connecting them together, we obtain the final multi-head attention result. To keep the computation and number of parameters constant, D needs to be divisible by k. The Layer-Separable Convolution-Layer (LSCL) operation consists of depthwise separable convolutions and linear layers. The process is as follows: First, the input sequence undergoes a linear transformation followed by a weighted summation, and then a nonlinear transformation is introduced to obtain... After projecting it into a two-dimensional space, depthwise separable convolution and non-linear transformation operations are performed. Finally, it is flattened into a one-dimensional space and passed through a linear layer to obtain the output. The depthwise separable convolution consists of depthwise convolution (DW Conv) and pointwise convolution (PW Conv). The depthwise convolution applies a separate layer to each channel. A convolutional kernel processes only one channel of the input feature map, and then the results of each channel are stacked along the channel direction to obtain the output of a depthwise convolution. Pointwise convolution is performed on the output of the depthwise convolution. Convolution operation, the size of the convolution kernel is 1 C1 represents the number of input channels, and C2 represents the number of output channels, which means a linear combination of the input feature maps in the depth dimension; the layer-separable convolution-layer operation formula is expressed as follows: ; ; Where X represents the input sequence and FC represents the fully connected layer (FC). This represents the PReLU activation function. PReLU introduces a learnable parameter on top of ReLU, allowing the slope of the negative part to be adaptively adjusted. This represents the height and width of the two-dimensional image obtained by reshaping a one-dimensional sequence. express Convolution operation, This represents a depthwise convolution operation. This represents the size of the two dimensions obtained after flattening a two-dimensional image; This represents the result obtained after reshaping using a nonlinear layer. express The result obtained after depthwise separable convolution operation is represented by a one-dimensional sequence O, which indicates the output of LCSL.

4. The lung CT image classification system according to claim 1, characterized in that, The Patch Merging submodule is designed for information exchange and integration between multi-level feature representations. Except for the first stage, each stage requires downsampling via Patch Merging. This involves dividing adjacent pixels into a patch, concatenating pixels at the same position in each patch to obtain C feature maps, connecting these C feature maps along the channel direction, using a normalization layer to stabilize the input distribution of each layer, and finally performing a linear transformation along the channel direction using a fully connected layer. If new patches are formed by concatenating pixels at position intervals of 2 in the row and column directions, a feature map with halved height and width and doubled channels is obtained.

5. The lung CT image classification system according to claim 1, characterized in that, The spatial attention submodule described above will process the outputs of each stage. Cascading them together yields feature maps Formulate it as follows: ;in Indicates spatial location The feature at the location, where C represents the number of channels in the feature map. ; by using weights The convolution kernel compresses the feature map along the channel, specifically by: ,in Representing feature maps, express Convolution operation, This represents the projection matrix, where each element... Indicates spatial location A linear combination of all channels, ; Activated by the Sigmoid function Will Scaling to [0,1] is used to calibrate the relative importance of spatial locations, and the process is expressed by the following formula: ;in This represents the recalibrated feature map.

6. The lung CT image classification system according to claim 1, characterized in that, The domain knowledge encoder module first extracts diagnostic information including CRP value, segmental atelectasis, multiple pulmonary cysts, hemoptysis volume, and lymphocyte count, and encodes it into the input format required by the domain knowledge encoder. The domain knowledge encoder is essentially a Transformer block, comprising a linear embedding layer, multi-head self-attention (MSA), and a multilayer perceptron (MLP). The module's formula is as follows: ;in LN stands for Layer Normalization, which represents the new feature representation obtained by mapping input data to a low-dimensional space. This represents the one-dimensional text feature representation obtained by the domain knowledge encoder module.

7. The lung CT image classification system according to claim 1, characterized in that, The aforementioned bidirectional cross-attention label alignment module first represents the one-dimensional features of the image. One-dimensional feature representation of text The images are projected onto the multimodal feature space respectively to obtain new image representations. and text representation Next, the two feature representations are input into the bidirectional cross-attention label alignment module for fine-grained alignment, resulting in new feature representations. The formula for this process is as follows: ; ; ;in This represents the d-dimensional image feature representation. Representing d-dimensional text features , These represent the projection matrices of the query, key, and value, respectively. Indicate the attention mechanism; go through Convolution yields feature representations that fuse information from both image and text modalities. The formula for this process is as follows: ;in, Indicates connection along the channel. express Convolution operation.

8. The lung CT image classification system according to claim 7, characterized in that, The classifier module, in feature representation A special classification label [CLS] is introduced, and the new feature representation is subjected to self-attention operation. The output corresponding to the label [CLS] is the classification result of COVID-19 and other pneumonias. During training, the cross-entropy loss function is used to calculate the difference between the predicted class and the true class. The model for accurate diagnosis of COVID-19 is trained by minimizing the loss function.

Images