A multi-stage classification method based on CIP pneumonia multi-modal data

Abstract

The present application relates to the field of medical image and natural language technology, and discloses a multi-stage classification method based on CIP pneumonia multi-modal data, wherein the CT image of a patient with initial diagnosis is subjected to a first-stage classification model trained offline to obtain a first-stage classification prediction result and probability; the CT image, electronic medical record text and physiological examination report of a patient with re-diagnosis are obtained at least once, and a second-stage classification model trained offline is used to obtain CT image features, electronic medical record text features and physiological examination report features, as well as a second-stage classification prediction result and probability; the CT image features, electronic medical record text features and physiological examination report features obtained each time are constructed into a prognosis recovery sequence, and the trend of the prognosis recovery sequence is analyzed by using a Sen+Mann-Kendall test method. The present application has high prediction accuracy by using a three-stage classification prediction method, and can provide a reference for doctors.

Description

Technical Field

[0001] This invention belongs to the field of medical image and natural language processing, specifically relating to a multi-stage classification method based on CIP pneumonia multimodal data. Background Technology

[0002] Checkpoint-inhibitor pneumonitis (CIP) is a fatal lung injury caused by treatment with immune checkpoint inhibitors (ICIs), with diverse clinical, imaging, and pathological manifestations. The pathogenesis of CIP is currently unclear, with an overall incidence of 3%–5%, while in the real world, the incidence can reach as high as 19%, and the mortality rate can be as high as 12.8%–22.7%. Discussions with experts from collaborating hospitals have revealed that the diagnosis and treatment of CIP involves three stages: initial diagnosis, clinical diagnosis, and prognostic treatment. In the initial diagnosis stage, CT imaging is a crucial basis for doctors to determine whether a patient may have CIP. However, in the clinical diagnosis stage, due to the nonspecificity of CIP imaging and clinical manifestations, a comprehensive analysis combining clinical symptoms, imaging findings, examinations, and medications is necessary. This complex assessment relies heavily on the doctor's clinical experience and professional competence, resulting in significant subjectivity and high rates of misdiagnosis and missed diagnosis. In addition to imaging features, electronic medical records and physiological test reports contain a wealth of information closely related to the diagnosis, such as the patient's lung infection, tumor progression (carcinomatous lymphangioma, pulmonary tumor embolism), history of ICI use, physical signs and blood oxygen saturation, and pulmonary function tests. During the prognostic and treatment phase of CIP, because the medication used to treat CIP pneumonia patients is closely related to their clinical manifestations, doctors will assess the patient's prognostic and recovery outcomes to understand the disease progression and adjust treatment and medication plans accordingly.

[0003] Currently, methods using deep learning for classifying CT image data or medical record text data are widely applied. For example, CN117668760A, "A Multimodal Immunosuppressant-Associated Pneumonia Classification Method Based on Deep Learning," inputs CT images and electronic medical records into a multimodal CIP pneumonia classification network for classification prediction, obtaining classification results and probabilities. Another example is CN118335341A, "A Similar Medical Record Recommendation Method Based on Multi-Feature Fusion of CIP Pneumonia Medical Record Text," which uses three fields—chief complaint, present illness, and past medical history—from CIP pneumonia electronic medical record text for analysis, employing an approximate nearest neighbor query method based on a proximity graph to retrieve similar medical records in the medical record text feature database. However, current methods using target classification networks for classification prediction are mostly applied to single CT image data or medical record text data, and there is still no method for classifying and predicting CT image data and medical record text data for different stages of CIP pneumonia separately. Therefore, existing technologies need improvement. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multi-stage classification method based on CIP pneumonia multimodal data, which is used to perform multi-stage classification and prediction of multimodal data combining CIP pneumonia CT image data and electronic medical record text data.

[0005] To address the aforementioned technical problems, this invention provides a multi-stage classification method based on CIP pneumonia multimodal data, comprising the following process:

[0006] S1. Obtain CT image data of patients visiting the hospital for the first time, and after scaling, input it into the offline-trained first-stage classification model to obtain the first-stage classification prediction results and probabilities.

[0007] S2. Acquire CT image data, electronic medical record text, and physiological examination report data of patients with at least one follow-up visit. After preprocessing, use these data as input to the offline-trained second-stage classification model to obtain CT image features V. Image Electronic medical record text features V record and physiological test report characteristics V report Then, the classification prediction results and probabilities of the second stage are obtained through the multimodal feature fusion and classification module;

[0008] S3, convert the CT image features V obtained each time after step S2. Image Electronic medical record text features V record and physiological test report characteristics V reportAfter normalization, the Euclidean distance between each vector and the standard eigenvector is calculated and averaged to construct a prognostic recovery sequence. The Sen+Mann-Kendall test is used to analyze the trend of the prognostic recovery sequence and obtain the estimated effect.

[0009] As an improvement to the multi-stage classification method based on CIP pneumonia multimodal data of this invention:

[0010] The first-stage classification model includes: sequentially connected convolutional and pooling layers, four consecutive DPN_CBAM dual-path layers, pooling layers, and a classifier;

[0011] The DPN_CBAM dual-path layer includes a Transition module, three Concat operations, and a Bottleneck-CBAM module. The input features are used as inputs to the Transition module and the first Concat operation, respectively. The first Concat operation concatenates the input features with the features output by the Transition module, and the output is used as the input to the Bottleneck-CBAM module and the second Concat operation. The second Concat operation concatenates the output of the Bottleneck-CBAM module with the output of the first Concat operation. The output of the Bottleneck-CBAM module and the output of the Transition module are added together, and then used together with the output of the second Concat operation as the input to the third Concat operation.

[0012] As a further improvement to the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0013] The Bottleneck-CBAM module includes the Bottleneck module and the CBAM module. The Bottleneck module includes three convolutional layers, and the CBAM module includes a channel attention part and a spatial attention part.

[0014] As a further improvement to the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0015] The CT image feature extraction module is based on the first-stage classification model, and replaces the final classifier of the first-stage classification model with a multilayer perceptron.

[0016] As a further improvement to the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0017] The electronic medical record feature extraction module is the RoBERTa-wwm-ext model;

[0018] The feature extraction module for the physiological test report is the DeepFM model.

[0019] As a further improvement to the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0020] The multimodal feature fusion and classification module includes a cross-attention-based multimodal feature fusion network, MLP, and softmax function, which integrates the CT image features V Image Electronic medical record text features V record and physiological test report characteristics V report The features are input pairwise into the Cross Attention module to calculate the cross-modal features, and then the electronic medical record text features V are input. record CT image features V Image and physiological test report characteristics V report The concat operation is performed on the cross features to obtain the fused feature V. fusion Then it goes through the MLP and softmax functions.

[0021] As a further improvement to the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0022] The process of obtaining the standard feature vector is as follows:

[0023] Multimodal data, including CT image data, electronic medical record text, and physiological examination report data, were collected from a control group without CIP pneumonia. These data were then preprocessed and passed through CT image feature extraction module, electronic medical record feature extraction module, and physiological examination report feature extraction module to obtain standard feature vectors for the CT image data, electronic medical record text, and physiological examination report data.

[0024] As a further improvement to the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0025] The preprocessing operation specifically includes:

[0026] CT image data is scaled down.

[0027] After filtering stop words in the chief complaint, past medical history, and present medical history fields of electronic medical records, the length of excessively long sequences is limited to 512 by random pruning, and the part with insufficient length is filled with [SEP].

[0028] For discrete data in physiological test reports, one-hot encoding is used to extend the values ​​of discrete features to Euclidean space; for continuous data in physiological test reports, normalization methods are used.

[0029] As a further improvement to the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0030] The offline training process for the first-stage classification model and the second-stage classification model is as follows:

[0031] A multimodal diagnosis and treatment dataset for CIP pneumonia was collected. Each sample included CT images of the initial admission and subsequent treatments, electronic medical record text, and physiological test report data. After data anonymization, the diagnostic results were labeled and then divided into training set, validation set, and test set in an 8:1:1 ratio. The training set, validation set, and test set were then input into the first-stage model and the second-stage model, respectively, for training. After setting the training parameters, the Adam optimizer was used to optimize the network, and a cosine learning rate decay function was used to control the learning rate decay. The total training epochs were 50, and the test set was used for validation.

[0032] The beneficial effects of this invention are mainly reflected in:

[0033] 1. This invention uses a three-stage classification prediction method to construct a classification method for multimodal data of CIP pneumonia. The datasets used in the algorithm are important data bases for doctors to judge CIP pneumonia in the actual diagnosis process. Therefore, it can greatly assist doctors in actual diagnosis and treatment in terms of data rationality and algorithm process close to reality.

[0034] 2. The first-stage classification model and the second-stage CT image feature extraction module of the present invention improve the network's ability to express key features by introducing a convolutional attention module (CBAM).

[0035] 3. In the second-stage classification model proposed in this invention, a prediction algorithm based on multimodal data fusion is employed. Patient CT images, electronic medical records, and physiological test reports are used as model inputs. Corresponding feature extraction modules are built based on the characteristics of each modality of data. Then, a cross-attention method is used to fuse multimodal features. Finally, the features are input into a classifier to classify and obtain the probability of CIP pneumonia. High prediction accuracy is achieved on multimodal datasets collected from collaborating hospitals. This model can provide doctors with a reference and is of significant value in reducing the missed diagnosis and misdiagnosis rates of CIP pneumonia. Attached Figure Description

[0036] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic diagram of the structure of the first-stage classification model of the present invention;

[0038] Figure 2This is a schematic diagram of the basic unit structure of a DPN network;

[0039] Figure 3 This is a schematic diagram of the Convolutional Attention Module (CBAM) structure.

[0040] Figure 4 This is the CT image feature extraction module of the present invention;

[0041] Figure 5 This is the electronic medical record feature extraction module of the present invention;

[0042] Figure 6 This is the feature extraction module for the physiological examination report of the present invention;

[0043] Figure 7 This is the multimodal feature fusion and classification module of the present invention. Detailed Implementation

[0044] The present invention will be further described below with reference to specific embodiments, but the scope of protection of the present invention is not limited thereto:

[0045] Example 1: A multi-stage classification method based on CIP pneumonia multimodal data. This method constructs classification and evaluation models for three different stages to classify and predict CT images, medical record text data, and physiological test reports at different diagnostic stages of CIP pneumonia. The specific process is as follows:

[0046] S1. Construct a first-stage classification model based on CT images.

[0047] S101, CT Image Dataset Acquisition and Processing

[0048] CT images of patients' lungs were acquired from collaborating hospitals. These images included those labeled with lung diseases (such as CIP pneumonia, lung cancer, and pulmonary nodules) and those labeled with normal lungs. All images were resized to 224×224 pixels for the network structure. Four data augmentation methods—random horizontal flipping, random cropping, Gaussian noise, and data padding—were used to process the CT image data. The dataset was then divided into training, validation, and test sets in an 8:1:1 ratio.

[0049] S102. Construction of the backbone network (DPN_CBAM dual-path layer) of the first-stage classification model

[0050] The first-stage classification model uses a dual-path DPN network as the backbone network and introduces a Convolutional Block Attention Module (CBAM) to enhance the network's ability to express key features, thereby improving the overall performance of the network.

[0051] A DPN network is composed of multiple modular basic units stacked together, each of which employs a bottleneck layer design, such as... Figure 2 As shown, residual connection paths and dense connection paths are combined by connecting 1×1 convolutional layers, 3×3 convolutional layers, and 1×1 convolutional layers. The two branches share the first 1×1 convolutional layer. The 3×3 convolutional layer uses grouped convolution, grouping the input channels and performing convolution operations on each group with its own convolutional kernel. The output of the second 1×1 convolutional layer enters both the residual connection path branch and the dense connection path branch, and the output features of the two paths are merged through concat before entering the next basic unit. In addition to useful pathological information, the information extracted by each convolutional layer of the DPN network also contains some useless background information interference. In actual clinical practice, doctors observe and analyze detailed features such as nodules or masses, consolidation or interstitial changes, and traction bronchiectasis. Therefore, a Convolutional Attention Module (CBAM) is introduced to enhance the network's ability to express key features.

[0052] The Channel Attention Module (CBAM) is responsible for adjusting the importance of features across different channels, while the Spatial Attention Module (SAM) adjusts the importance of different locations within the feature map. CBAM adaptively adjusts the weights of the feature map in both channel and spatial dimensions, thereby strengthening the identification of key features. A detailed structure diagram of CBAM is shown below. Figure 3 As shown:

[0053] The CBAM module can be seamlessly inserted into any CNN architecture. Assume feature map F ∈ R. C×H×W As input, it first passes through a channel attention map M c ∈R C×1×1 We obtain the channel attention map F′, and then process it through the spatial attention map M. s ∈R 1×H×W The final optimized feature map F″ is obtained, and the calculation process is shown in the following two equations. Represents the element-wise multiplication operation of a matrix:

[0054]

[0055] Specifically, for the input feature map, the channel attention module first generates two different spatial context descriptors through max pooling and average pooling in the channel dimension, respectively. and Then and The input is fed into a shared fully connected layer for feature fusion and compression, and bitwise addition is used to fuse the output channel attention map M. c (F)∈R C×H×W The calculation process is shown in the following formula, where σ is the Sigmoid activation function, and this feature map encodes what needs to be highlighted.

[0056]

[0057] The spatial attention module follows the channel attention module, focusing more on the location of key information. First, it performs max pooling and average pooling on the channel weight matrix F′ in the spatial dimension, generating two different descriptors. and The two are then concatenated to generate an efficient channel descriptor. A convolutional layer is then used to generate a spatial attention map M from the concatenated channel descriptor. s (F′)∈R C×H×W The calculation process is shown in the following formula, where σ represents the Sigmoid activation function, f 7×7 This represents a convolution operation with a kernel size of 7×7, and the feature map encodes which regions are highlighted or suppressed.

[0058]

[0059] Each DPN_CBAM dual-path layer includes a Transition module, three Concat operations, and a Bottleneck-CBAM module, such as Figure 1As shown, the features input to the DPN_CBAM dual-path layer serve as inputs to the Transition module and the first Concat operation, respectively. The Transition module is used for the transition between two adjacent DPN_CBAM dual-path layers, helping to reduce the size of the feature maps and thus the number of model parameters, promoting smooth information transfer. The first Concat operation concatenates the input features of the main path and the output features of the Transition module along the channel dimension, and the output serves as the input to the Bottleneck-CBAM module and the second Concat operation. The second Concat operation concatenates and integrates the output of the Bottleneck-CBAM module and the output of the first Concat operation with channel- and spatially weighted features. The resulting features contain both channel- and spatial weighted information. The outputs of the Bottleneck-CBAM module and the Transition module are added together and used as input to the third Concat operation, along with the output of the second Concat operation. This concatenates the final outputs of the current layer's residual path and dense path again, ensuring that the features of both paths can continue to be passed to the next layer, achieving multi-layer feature reuse.

[0060] Furthermore, to better extract key features from CT images of CIP pneumonia patients, this invention integrates the CBAM module into the Bottleneck layer of each basic unit on the basis of the DPN backbone network, forming a new Bottleneck-CBAM module, the specific structure of which is as follows: Figure 1 As shown in the upper part.

[0061] The Bottleneck-CBAM module includes the Bottleneck module and the CBAM module. In the Bottleneck module, dimensionality reduction is performed first through a 1×1 convolutional layer, then the features are convolved through a 3×3 convolutional layer, and finally the dimensionality is increased through a 1×1 convolutional layer. Therefore, the Bottleneck module reduces the amount of computation and parameters while ensuring the efficiency of feature transfer, making deep networks more efficient in processing high-dimensional features and avoiding the explosive growth of computation. The CBAM module consists of two parts: channel attention and spatial attention. First, the features pass through the channel attention part. Global average pooling (AvgPool) and global max pooling (MaxPool) are performed on the input feature map, generating two distinct channel description vectors. These vectors retain global channel information. These pooling vectors are then passed through a shared multilayer perceptron (MLP), outputting two new weight vectors representing the weighting coefficients in the channel dimension. These weights are then normalized to values ​​between 0 and 1 using a sigmoid activation function. These weights are then multiplied element-wise with the input feature map to obtain the spatially weighted feature map. Next, the spatial attention part performs attention calculations in the spatial dimension after the channel attention. First, global average pooling and max pooling are performed on the channel-weighted feature map, resulting in two two-dimensional spatial description maps. These spatial description maps are then passed through a 7x7 convolutional layer to generate a new spatial weight map. The spatial weights are then normalized to values ​​between 0 and 1 using a sigmoid function. These weights are then multiplied element-wise with the input feature map to obtain the spatially weighted feature map. The final output features are the result of weighted processing in both channel and spatial dimensions, thereby improving the overall network's ability to express key information.

[0062] S103. Overall Network Construction of the First-Stage Classification Model

[0063] The first-stage classification model (i.e., the DPNCT-Net network) is based on the DPN network, but all four stages of the DPN network are replaced with the DPN_CBAM dual-path layer of this invention. The overall architecture is as follows: Figure 1As shown, the network includes an initial convolutional and pooling layer, four DPN_CBAM dual-path layers in the middle, and a final pooling layer and classifier. First, the CT image dataset is preprocessed to obtain an input image matrix of size 224×224×3. This matrix then passes through an initial 7×7 convolutional layer and a 3×3 max-pooling layer for feature extraction and dimensionality reduction, outputting a feature matrix of size 56×56×64. This feature matrix is ​​then input into the DPN_CBAM dual-path layer. The DPNCT-Net network uses four DPN_CBAM dual-path layers in its design; the number of blocks in each layer determines the overall network depth. This invention references DPN92 in its parameter design. The final convolutional layer outputs a feature matrix of size 7×7×2560. After passing through a 7×7 adaptive average pooling layer, a feature vector of size BatchSize×1×1×2560 is obtained. Finally, an FC+Softmax classifier is used to output the classification result and probability. Based on the classification results, doctors need to further combine electronic medical records or physiological test reports to make a clinical diagnosis and determine whether the patient has CIP pneumonia. Marking the patient as not having the disease does not rule out the possibility of having CIP pneumonia. This invention can only provide the probability of a patient's initial diagnosis of CIP pneumonia to assist doctors in screening. Specific judgment and further clinical diagnosis still need to be combined with the doctor's actual clinical experience.

[0064] S2. Construct a second-stage classification model based on multimodal data fusion.

[0065] Based on the classification results of the first stage, doctors manually determine which cases require further classification and prediction, and then use the second-stage classification model for further classification and prediction. The preprocessed CT image data, electronic medical record text, and physiological examination report data are then processed by the CT image feature extraction module, the electronic medical record feature extraction module, and the physiological examination report feature extraction module, respectively, to obtain the feature representations V of the CT images, electronic medical record text, and physiological examination reports. Image V record and V report Then, the classification prediction results and probabilities for the second stage are obtained through the multimodal feature fusion and classification module.

[0066] S201. Data Acquisition and Preprocessing of Electronic Medical Records and Physiological Examination Reports

[0067] The electronic medical records and physiological examination reports of the present invention have the same data source as in S101.

[0068] Electronic medical records mainly include the patient's basic information, chief complaint, past medical history, present medical history, personal history, and preliminary diagnosis fields. After filtering the chief complaint, past medical history, and present medical history fields for stop words, excessively long sequences are randomly pruned to a length of 512. The length of the preprocessed text sequences is fixed at 512, and any insufficient length is padded with [SEP]. Excessively long text is truncated. Physiological test report data can be divided into two main categories: continuous numerical data, such as routine blood tests, routine urine tests, and routine myocardial enzyme tests; and discrete data, such as physical examinations and pulmonary ventilation and diffusion function tests. This invention employs different preprocessing methods for these two types of data: for discrete data, one-hot encoding is used to extend the values ​​of discrete features to Euclidean space; while for continuous data, considering the significant differences between the reference values ​​and units of continuous data indicators, a normalization method is used to balance the model's focus on continuous data indicators.

[0069] S202, CT Image Feature Extraction Module

[0070] The CT image data used in the second stage is identical to that used in the first stage of S1 in terms of data size and format. The only difference is that the multimodal dataset in the second stage only involves labeling whether or not the patient has CIP pneumonia. Therefore, a CT image feature extraction module is built on the DPNCT-Net network proposed in S1, replacing the final classifier of the DPNCT-Net network with a multilayer perceptron. The specific structure is as follows: Figure 4 As shown, unlike the first-stage DPNCT-Net network, the image feature matrix passes through an adaptive average pooling layer and then through a multilayer perceptron (MLP). This process removes redundant information from the high-dimensional feature vectors while simultaneously reducing the dimensionality of the features to obtain the image modal features V. Image .

[0071] S203, Electronic Medical Record Feature Extraction Module

[0072] To address the significant variation in field lengths across medical record text datasets, this invention selects the RoBERTa-wwm-ext Chinese pre-trained model, which excels in medical text classification, to construct the electronic medical record feature extraction module. The BERT model employs a bidirectional Transformer structure to extract contextual information and demonstrates superior performance across multiple text processing tasks. RoBERTa represents several improvements over BERT, achieving even better performance, primarily in the following aspects:

[0073] (1) Dynamic mask. Compared with the static masking strategy of the BERT model, the RoBERTa model copies the pre-trained data 10 times and randomly selects 15% of the tokens in each copy for masking, so that the model can obtain more complete semantic information.

[0074] (2) Remove the NSP task. RoBERTa removes the NSP task, which limits the model's length, making it difficult for the model to learn long-distance dependency information effectively. During training, multiple consecutive sentences are input at a time until the maximum length of 512 is reached.

[0075] (3) Larger corpus size and batch size. Compared to the 13G of training data and a batch size of 256 for the BERT model, RoBERTa expands the training data to 160G and sets the batch size to 8k, giving the model stronger learning and generalization capabilities.

[0076] The overall process of feature extraction from electronic medical records is as follows: Figure 5 As shown, in this invention, the preprocessed sequences of chief complaint, present illness, and past medical history fields from step S201 are input into the RoBERTa-wwm-ext model module to obtain feature V. zs V xbs and V jws This model employs numerous multi-head attention mechanisms to extract contextual semantic information, resulting in a sequence representation vector after passing through 12 transformer layers.

[0077] S = {v0, v1, ..., v} L-1}=RoBERTa(X)

[0078] In the formula L is the sequence length of the model (512), and d is the hidden layer dimension of the model (768).

[0079] Then, vector concatenation is performed to obtain the complete modal features V of the medical record text. record .

[0080] S204, Physiological Test Report Feature Extraction Module

[0081] Physiological test report data can be divided into two categories: one is continuous numerical data, such as blood routine and urine routine; the other is discrete data, such as physical examination and pulmonary ventilation and diffusion function test. After one-hot encoding and normalization preprocessing in step S201, a large number of sparse features, dense features and cross information between features are obtained.

[0082] DeepFM is a model that integrates Deep Neural Network (DNN) and Factor Machine (FM), capable of processing sparse and dense features separately and simultaneously learning the interaction information between high-order and low-order features. Therefore, this invention uses a DeepFM-based model to construct a feature extraction module for physiological test reports.

[0083] DeepFM's main structure comprises two parts: FM and DNN, which respectively implement low-order feature vector interactions and perform high-dimensional representation of the interaction features. The outputs of FM and DNN are combined (y FM and y DNN The input is fed into the Sigmoid function to obtain the model's prediction result:

[0084]

[0085] The FM (Fast Moving Factor) part primarily performs feature interactions on low-order feature vectors, learning the relationships and weights between features through latent factor decomposition. Its core is the second-order feature interaction of feature vectors. The DNN (Derivative Neural Network) part mainly learns high-order feature vector interactions to extract deeper semantic information from physiological test reports. The specific calculation processes for the FM and DNN parts are as follows:

[0086]

[0087] y DNN =σ(W l+1 a l +b l+1 )

[0088] Where n is the number of variables, w0 is the bias term, and x i w represents the i-th component of the eigenvector x. i w ij The coefficients represent the corresponding features; l represents the number of hidden layers, σ is the ReLU activation function, and W... l+1 b l+1 These represent weights and biases, respectively. l It is the output of the l-th hidden layer.

[0089] The overall structure of the DeepFM-based physiological test report feature extraction module is as follows: Figure 6As shown, the input layer includes preprocessed one-hot encoded features and normalized continuous features, representing the encoded vector of a certain dimension category. The embedding layer extracts the latent features to be learned, and after calculating the weights of the second-order interaction terms, it maps them to a low-dimensional feature vector. In the FM layer, the left side calculates the first-order term, while the inner product operation on the right side is used to calculate the second-order feature interaction of FM. Simultaneously, in the hidden layers, the expressive power of the features is continuously enhanced through fully connected layers. Finally, the output module integrates the output of the FM layer and the output of the hidden layers to obtain a more complete feature representation V. report .

[0090] S205, Multimodal Feature Fusion and Classification Module

[0091] In obtaining the feature representation of CT images, electronic medical record texts, and physiological test reports, V Image V record and V report Subsequently, this invention employs a feature fusion method based on multimodal cross-attention to mine the correlation information between various modalities, thereby improving the expressive power of the post-modal fused features. The cross-attention mechanism establishes the correlation between different modal inputs through the form of vector inner products, and its calculation method is as follows:

[0092]

[0093] Here, x and y represent the features of different modes.

[0094] The structure of a multi-modal cross-attention-based feature fusion network (MMCA-Net) is as follows: Figure 7 As shown. During multimodal feature fusion, the electronic medical record text features V... record CT image features V Image and physiological test report characteristics V report The features are input pairwise into the Cross Attention module to calculate the cross-modal features, and then the original electronic medical record text features V are processed. record CT image features V Image and physiological test report characteristics V report The concat operation is performed on all intersecting features to obtain the fused feature representation V. fusion The encoding classification layer includes an MLP and a softmax function, which fuses the feature vector V. fusion The input is fed into a learnable encoder to further extract deeper information, mapping the abstract feature vector back to the label space for feature extraction. Finally, the softmax function is connected to normalize it to a probability distribution, and the results and probabilities of the second-stage classification prediction are output.

[0095] The patient's treatment process generates multiple CT image data, electronic medical record texts, and physiological examination report data at different times. Therefore, multiple second-stage classification prediction results and probabilities can be generated. The features generated in each classification prediction process are represented as V. Image V record and V report The data will be stored for use in the subsequent third-stage analysis.

[0096] S3. Third-stage evaluation algorithm based on Sen+Mann-Kendall test

[0097] To address the potential for incomplete multimodal data, such as missing modal data from CT images or examination reports, this invention, after discussions with collaborating physicians, will not use the fused feature vector V obtained in step S205 during the third-stage evaluation. fusion Instead of performing analysis, the feature vector V obtained in each classification prediction process in the second stage is used. record V Imag and V report The normalized features (obtained in steps S202-204) are then calculated, and the Euclidean distance between each normalized modal feature vector and the standard feature vector is taken and averaged to obtain the patient's current prognosis and rehabilitation sequence.

[0098] The process of obtaining the standard feature vector is as follows:

[0099] Multimodal data from a control group without CIP pneumonia, provided by professional physicians, were collected, including CT image data, electronic medical record text, and physiological examination report data. Then, according to steps S201-S204, the CT image data, electronic medical record text, and physiological examination report data were preprocessed and features were extracted, which were then used as standard feature vectors for the CT image data, electronic medical record text, and physiological examination report data, respectively.

[0100] Finally, the Sen+Mann-Kendall test was used to analyze the trend of the prognostic rehabilitation sequence, and the evaluation value effect was assigned to assess the effectiveness of rehabilitation treatment. The specific values ​​of the evaluation effect are {2: significant improvement, 1: improvement, 0: no significant change, -1: deterioration, -2: significant deterioration}.

[0101] The Sen method is a slope estimation method that uses the sign of the trend degree β to reflect the trend of the sequence. When β < 0, it indicates a downward trend, while β > 0 indicates an upward trend. The calculation formula is as follows:

[0102]

[0103] The median function represents the median, x j xi Let j represent the j-th data point and i-th data point in the time series.

[0104] The Mann-Kendall method measures the significance of a trend by the absolute value of a statistic. Its calculation process is as follows: For a sequence D = {d1, d2, ..., d...} n The significance test statistic is selected according to the sequence length n: when n < 10, the test statistic S is calculated as follows:

[0105]

[0106] Where sgn(x) is the sign function, n is the length of the sequence, and d j d i Let S represent the j-th and i-th data points in the sequence. When n≥10, the statistic S approximately follows a normal distribution. The test statistic Z is used.

[0107]

[0108]

[0109] Where Var(S) represents the variance of the S-statistic, and Z represents the standardized Mann-Kendall statistic.

[0110] In this invention, the number of tests for the same patient in the dataset is generally less than 10; therefore, trend analysis is performed using trend degree β and significance test statistic S. At a given significance level α, when β > 0 and |S| ≥ S... α / 2 The sequence shows a significant upward trend when β>0 and |S|<S α / 2 The sequence shows a non-significant upward trend; similarly, when β < 0 and |S| ≥ S α / 2 The sequence shows a significant decreasing trend when β < 0 and |S| < S α / 2 The sequence showed a non-significant downward trend.

[0111] The Sen method is a robust nonparametric statistical technique for trend detection, effectively handling errors and outliers, but it cannot determine the significance of a trend. The Mann-Kendall method, on the other hand, does not require assumptions about the data distribution and measures the significance of a trend through the absolute value of the statistic. Combining the Sen and Mann-Kendall methods provides a more comprehensive assessment of the trend and significance of sequential data.

[0112] S4, Training Model

[0113] S401, Training Dataset

[0114] The CIP pneumonia multimodal diagnosis and treatment dataset used in this invention comes from the laboratory's partner hospitals and includes CT images, electronic medical record texts, and physiological test reports. After data anonymization, the diagnostic results were labeled by experts from the partner hospitals. A total of 261 patients were labeled as having CIP pneumonia, and 292 patients were labeled as not having CIP pneumonia. Most of the non-CIP patients had other lung diseases (mostly lung cancer, pulmonary nodules, etc.). Statistically, the dataset contains 13,372 CT images and 1,383 electronic medical records. Each patient includes multiple electronic medical records from initial admission and subsequent treatments, as well as several physiological test reports.

[0115] S402, Training Configuration

[0116] Training was performed on a CentOS operating system server, with the development environment based on the PyTorch deep learning framework. The specific hardware and software configurations are shown in Table 1.

[0117] Table 1

[0118] name Environment configuration operating system CentOS 7.3.1611 processor Intel(R)Xeon(R)CPU 12*E5-2609v3@1.9GHz graphics card Tesla P4 8GB Memory 128GB Development Environment Python 3.8 + PyTorch 1.13

[0119] S403, Training Process

[0120] First, the dataset is split, with 80% of the patient samples selected as the training set, 10% as the validation set, and 10% as the test set. The training set is then input into the networks of steps S1 and S2 for training. During training, the Adam optimizer is used to optimize the network, with the first-order momentum parameter set to 0.9, the second-order momentum parameter set to 0.999, the batch size set to 32, the initial learning rate set to 0.001, and a cosine learning rate decay function used to control the learning rate decay. The weight decay is set to 0.0001. The total training duration is 50 epochs, and the test set is used for validation.

[0121] S5, Online Use

[0122] When a patient first visits, CT image data is acquired, scaled to 224×224 pixels, and input into the first-stage classification model (DPNCT-Net network) to obtain the classification prediction results and probability of CIP pneumonia. Doctors decide whether to perform further second-stage classification prediction for the patient based on the classification prediction results and probability and clinical experience.

[0123] In the second stage, CT image data, electronic medical record text, and physiological examination report data of follow-up patients are collected and preprocessed separately. Then, the preprocessed CT image data, electronic medical record text, and physiological examination report data are processed by the CT image feature extraction module, the electronic medical record feature extraction module, and the physiological examination report feature extraction module, respectively, to obtain CT image features V. Image Electronic medical record text features V record and physiological test report characteristics V report Then, the classification prediction results and probabilities of the second stage are obtained through the multimodal feature fusion and classification module, which are used to assist doctors in diagnosis and treatment and for analysis and evaluation of the third stage algorithm.

[0124] For patients who have visited the clinic at least twice, if an assessment and scoring are required, then the modality feature vectors (CT image features V) obtained from each second-stage classification model should be calculated. Image Electronic medical record text features V record and physiological test report characteristics V report The Euclidean distance between the target vector and the standard eigenvector is calculated, and multiple Euclidean distances are used to construct a prognostic rehabilitation sequence. The Sen+Mann-Kendall test is then used to analyze the trend of the prognostic rehabilitation sequence and obtain the estimated effect, thereby evaluating the effectiveness of rehabilitation treatment.

[0125] experiment:

[0126] 1: DPNCT-Net structure ablation experiment.

[0127] To address the complex features present in CT images, this invention employs a DPN dual-path structure based on densely connected paths and residual connected paths to construct an image feature extraction module, and introduces a convolutional attention module for optimization. The effectiveness of each module design is verified through ablation experiments, and the specific experimental results are shown in Table 2.

[0128] Table 2

[0129] Model Accuracy (%) AUC(%) F1 score (%) ResNet-101 80.66 82.96 79.89 DenseNet-161 78.42 80.17 77.38 DPN-92 81.72 84.01 80.39 This invention (DPNCT-Net) 83.80 85.54 82.44

[0130] Based on the experimental results in the table, the DPN model (see Chen Y, Li J, Xiao H, et al. Dual path networks[J]. Advances in neural information processing systems, 2017, 30.) outperforms ResNet (see He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:770-778.) and DenseNet (see Huang G, Liu Z, VanDer Maaten L, et al. Densely connected convolutional networks[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:4700-4708.) in all metrics, demonstrating the advantages of the dual-path structure, which effectively combines the advantages of ResNet in reusing refined features with the advantages of DenseNet in exploring new feature details. After introducing the CBAM module, the accuracy of the DPNCT ​​network was further improved by 2.08% compared to the DPN network, which verifies that the CBAM module can enhance useful information from both channel and spatial dimensions and improve the network's ability to express key features.

[0131] 2: Comparative experiment of multimodal feature fusion methods.

[0132] To verify the effectiveness of the cross-attention-based multimodal feature fusion method used in this invention, this experiment introduces different multimodal feature fusion methods for comparative analysis. The feature direct concatenation fusion method CFN, tensor fusion network TFN, and low-rank multimodal fusion LWF were selected to replace the multimodal feature fusion method of this algorithm. Specific experimental results are shown in Table 3.

[0133] Table 3

[0134] Multimodal feature fusion method Accuracy (%) AUC(%) F1 score (%) CFN 82.51 85.49 80.58 TFM 85.36 87.32 84.20 LMF 85.91 87.92 84.63 The method of this invention (MMCA) 87.12 89.81 85.86

[0135] According to the experimental results in the table, the CFN method, which is based on direct feature concatenation, performs the worst. This is because direct feature concatenation ignores the modality-specific statistical properties and intermodal correlation information. The tensor-based feature fusion methods TFN and LMF (see Liu Z, Shen Y, Lakshminarasimhan VB, et al. Efficient low-rank multimodal fusion with modality-specific factors[J]. arXiv preprint arXiv:1806.00064,2018.) also perform poorly, indicating that on the dataset of this invention, the tensor outer product method is insufficient for fully mining the complex correlations between multimodal data. The multimodal feature fusion method MMCA based on cross-attention proposed in this invention performs best on all indicators, demonstrating that the cross-attention mechanism can maximize the retention of key information extracted from each modality and fully mine the intermodal correlation information to obtain a complete fused feature vector. The experimental results prove the effectiveness of the multimodal feature fusion method based on cross-attention proposed in this chapter.

[0136] Finally, it should be noted that the above examples are merely some specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A multi-stage classification method based on CIP pneumonia multimodal data, characterized in that: The process includes: S1. Obtain CT image data of patients visiting the hospital for the first time, and after scaling, input it into the offline-trained first-stage classification model to obtain the first-stage classification prediction results and probabilities. The first-stage classification model includes: sequentially connected convolutional and pooling layers, four consecutive DPN_CBAM dual-path layers, pooling layers, and a classifier; The DPN_CBAM dual-path layer includes a Transition module, three Concat operations, and a Bottleneck-CBAM module. The input features are used as inputs to the Transition module and the first Concat operation, respectively. The first Concat operation concatenates the input features with the features output by the Transition module, and the output is used as the input to the Bottleneck-CBAM module and the second Concat operation. The second Concat operation concatenates the output of the Bottleneck-CBAM module with the output of the first Concat operation. The output of the Bottleneck-CBAM module and the output of the Transition module are added together, and then used together with the output of the second Concat operation as the input to the third Concat operation. The Bottleneck-CBAM module includes a Bottleneck module and a CBAM module. The Bottleneck module includes three convolutional layers, and the CBAM module includes a channel attention part and a spatial attention part in sequence. S2. Acquire CT image data, electronic medical record text, and physiological examination report data of patients with at least one follow-up visit. After preprocessing, use these data as input to the offline-trained second-stage classification model to obtain CT image features. Features of electronic medical record text Characteristics of physiological test reports Then, the classification prediction results and probabilities of the second stage are obtained through the multimodal feature fusion and classification module; S3. Calculate the CT image features obtained each time after step S2. Features of electronic medical record text Characteristics of physiological test reports After normalization, the Euclidean distance between each vector and the standard eigenvector is calculated and averaged to construct a prognostic recovery sequence. The Sen + Mann-Kendall test is used to analyze the trend of the prognostic recovery sequence and obtain the estimated effect.

2. The multi-stage classification method based on CIP pneumonia multimodal data according to claim 1, characterized in that: The CT image feature extraction module is based on the first-stage classification model, replacing the final classifier of the first-stage classification model with a multilayer perceptron.

3. The multi-stage classification method based on CIP pneumonia multimodal data according to claim 2, characterized in that: The electronic medical record feature extraction module is the RoBERTa-wwm-ext model; The feature extraction module for the physiological test report is the DeepFM model.

4. The multi-stage classification method based on CIP pneumonia multimodal data according to claim 3, characterized in that: The multimodal feature fusion and classification module includes a cross-attention-based multimodal feature fusion network, MLP, and softmax function to fusion the CT image features. Features of electronic medical record text Characteristics of physiological test reports The features are input pairwise into the Cross Attention module to calculate the cross features between modalities, and then the electronic medical record text features are input. CT image features Characteristics of physiological test reports and cross features The operation yields the fused features Then it goes through the MLP and softmax functions.

5. The multi-stage classification method based on CIP pneumonia multimodal data according to claim 4, characterized in that: The process of obtaining the standard feature vector is as follows: Multimodal data, including CT image data, electronic medical record text, and physiological examination report data, were collected from a control group without CIP pneumonia. These data were then preprocessed and passed through CT image feature extraction module, electronic medical record feature extraction module, and physiological examination report feature extraction module to obtain standard feature vectors for the CT image data, electronic medical record text, and physiological examination report data.

6. The multi-stage classification method based on CIP pneumonia multimodal data according to claim 5, characterized in that: The preprocessing operation specifically includes: CT image data is scaled down. After filtering stop words in the chief complaint, past medical history, and present medical history fields of electronic medical records, the length of excessively long sequences is limited to 512 by random pruning, and the part with insufficient length is filled with [SEP]. For discrete data in physiological test reports, one-hot encoding is used to extend the values ​​of discrete features to Euclidean space; for continuous data in physiological test reports, normalization methods are used.

7. A multi-stage classification method based on CIP pneumonia multimodal data according to claim 6, characterized in that: The offline training process for the first-stage classification model and the second-stage classification model is as follows: A multimodal diagnosis and treatment dataset for CIP pneumonia was collected. Each sample included CT images of the initial admission and subsequent treatments, electronic medical record text, and physiological test report data. After data anonymization, the diagnostic results were labeled and then divided into training set, validation set, and test set in an 8:1:1 ratio. The training set, validation set, and test set were then input into the first-stage model and the second-stage model, respectively, for training. After setting the training parameters, the Adam optimizer was used to optimize the network, and a cosine learning rate decay function was used to control the learning rate decay. The total training epochs were 50, and the test set was used for validation.

Images