Multi-stage classification method based on CIP pneumonia multi-modal data

Abstract

The invention relates to the technical field of medical images and natural languages, and discloses a multi-stage classification method based on CIP pneumonia multi-modal data, and the method comprises the steps: obtaining a first-stage classification prediction result and probability through a first-stage classification model which is obtained through offline training of a CT image of a first diagnosis patient; obtaining at least one CT image, an electronic medical record text and a physiological examination report of a re-visit patient, and obtaining CT image features, electronic medical record text features, physiological examination report features and a second-stage classification prediction result and probability through a second-stage classification model which is trained offline; and constructing the CT image features, the electronic medical record text features and the physiological detection report features obtained each time into a prognosis rehabilitation sequence, and analyzing the trend of the prognosis rehabilitation sequence by using a Sen + Mann-Kendall test method. The three-stage classification prediction method is high in prediction accuracy and can provide reference for doctors.

Description

Technical Field

[0001] The present invention belongs to the field of medical image and natural language processing, and specifically relates to a multi-stage classification method based on CIP pneumonia multimodal data. Background Art

[0002] Checkpoint-Inhibitor Pneumonitis (CIP) is a fatal lung injury caused by immune checkpoint inhibitors (ICIs) treatment, with different clinical, imaging and pathological manifestations. At present, the pathogenesis of CIP in clinical practice is still unclear, with a total incidence of 3% to 5%. In the real world, the incidence of CIP can even reach 19%, and the mortality rate can be as high as 12.8% to 22.7%. After discussions with experts from cooperative hospitals, it was learned that the diagnosis and treatment process of CIP is divided into three stages, namely the initial diagnosis stage, the clinical diagnosis stage, and the prognosis and treatment stage. In the initial diagnosis stage of CIP pneumonia, CT images are an important basis for doctors to judge whether patients may have CIP pneumonia. However, in the clinical diagnosis stage of CIP, due to the non-specificity of CIP images and clinical manifestations, it is necessary to combine clinical symptoms, imaging manifestations, examinations and medication for comprehensive analysis. The complex evaluation work relies more on the clinical experience and professional ability of doctors, which is subjective and has high misdiagnosis and missed diagnosis rates. In addition to imaging features, electronic medical records and physiological test reports contain a large amount of information closely related to the diagnosis results, such as patients' lung infections, tumor progression (malignant lymphangiopathy, pulmonary tumor embolism), ICIs medication history, physical signs and blood oxygen saturation, pulmonary function tests, etc. In the CIP prognosis and treatment stage, since the treatment and medication of CIP pneumonia patients are closely related to their clinical manifestations, doctors will evaluate the prognosis and rehabilitation treatment effects of patients to understand the trend of the disease, and adjust the treatment and medication plan according to the evaluation results.

[0003] At present, the method of classifying CT image data or medical record text data using deep learning methods is widely used. For example, the publication number is CN117668760A "A multimodal immunosuppressant-related pneumonia classification method based on deep learning", which inputs CT images and electronic medical records into the multimodal CIP pneumonia classification network for classification prediction to obtain classification results and probabilities. For example, the publication number is CN118335341A "Similar medical record recommendation method based on multi-feature fusion of CIP pneumonia medical record text", which uses the three fields of chief complaint, current medical history, and past history in the CIP pneumonia electronic medical record text for analysis, and uses the approximate nearest neighbor query method based on the neighbor graph to retrieve similar medical records to obtain the vector to be retrieved in the medical record text feature library. However, the method of classification prediction using the target classification network currently used is mostly used for single CT image data or medical record text data, and there is no method for classifying and predicting CT image data and medical record text data at each stage of CIP pneumonia. Therefore, it is necessary to improve the existing technology. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multi-stage classification method based on CIP pneumonia multimodal data, which is used to perform multi-stage classification prediction on multimodal data combining CT image data of CIP pneumonia and electronic medical record text data.

[0005] In order to solve the above technical problems, the present invention provides a multi-stage classification method based on CIP pneumonia multimodal data, which includes the following process:

[0006] S1, obtaining CT image data of patients who visit the hospital for the first time, and inputting them into the first-stage classification model trained offline after scaling, to obtain the first-stage classification prediction results and probabilities;

[0007] S2. Obtain CT image data, electronic medical record text, and physiological examination report data of patients who have had at least one follow-up visit, and use them as input to the second-stage classification model trained offline after preprocessing operations to obtain CT image features V Image , Electronic medical record text features V record and physiological detection report characteristics V report , and then obtain the classification prediction results and probabilities of the second stage through multimodal feature fusion and classification modules;

[0008] S3, each time the CT image feature V obtained in step S2 is Image , Electronic medical record text features V record and physiological detection report characteristics V reportAfter normalization, the Euclidean distances between the standard eigenvectors were calculated and averaged to construct a prognosis-rehabilitation sequence. The trend of the prognosis-rehabilitation sequence was analyzed using the Sen+Mann-Kendall test method to obtain the estimated effect.

[0009] As an improvement of the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0010] The first stage classification model includes: a sequentially connected convolutional layer and pooling layer, four consecutive DPN_CBAM dual-path layers, a pooling layer and a classifier;

[0011] The DPN_CBAM dual-path layer includes a Transition module, three Concat operations and a Bottleneck-CBAM module. The input features are respectively used as inputs of the Transition module and the first Concat operation. The first Concat operation concatenates the input features with the features output by the Transition module, and the output is used as the input of the Bottleneck-CBAM module and the second Concat operation; the second Concat operation concatenates the output of the Bottleneck-CBAM module with the output of the first Concat operation; the output of the Bottleneck-CBAM module and the output of the Transition module are added, and the output is used together with the output of the second Concat operation as the input of the third Concat operation.

[0012] As a further improvement of the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0013] The Bottleneck-CBAM module includes a Bottleneck module and a CBAM module, the Bottleneck module includes three convolutional layers, and the CBAM module includes a channel attention part and a spatial attention part in sequence.

[0014] As a further improvement of the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0015] The CT image feature extraction module is based on the first-stage classification model, and replaces the last classifier of the first-stage classification model with a multi-layer perceptron.

[0016] As a further improvement of the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0017] The electronic medical record feature extraction module is the RoBERTa-wwm-ext model;

[0018] The physiological detection report feature extraction module is a DeepFM model.

[0019] As a further improvement of the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0020] The multimodal feature fusion and classification module includes a multimodal feature fusion network based on cross attention, MLP and softmax function, which transforms the CT image feature V Image , Electronic medical record text features V record and physiological detection report characteristics V report The two are input into the Cross Attention module to calculate the cross features between the modalities, and then the electronic medical record text features V record , CT image features V Image and physiological detection report characteristics V report And the cross features are concat operated to obtain the fused feature V fusion , and then pass through MLP and softmax function.

[0021] As a further improvement of the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0022] The process of obtaining the standard feature vector is as follows:

[0023] Multimodal data of the control group without CIP pneumonia were collected, including CT image data, electronic medical record text and physiological examination report data, and then preprocessed and passed through the CT image feature extraction module, electronic medical record feature extraction module and physiological test report feature extraction module respectively to obtain standard feature vectors of CT image data, electronic medical record text and physiological examination report data.

[0024] As a further improvement of the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0025] The pre-processing operation is specifically as follows:

[0026] For CT image data, scaling is used;

[0027] After filtering the stop words for the chief complaint, past medical history, and present medical history fields in the electronic medical record, the length of the overlong sequence was limited to 512 by random trimming, and the insufficient length was filled with [SEP];

[0028] For discrete data in physiological test report data, One-hot encoding is used to expand the values ​​of discrete features to Euclidean space; for continuous data in physiological test report data, normalization method is used.

[0029] As a further improvement of the multi-stage classification method based on CIP pneumonia multimodal data of the present invention:

[0030] The offline training process of the first-stage classification model and the second-stage classification model is as follows:

[0031] A multimodal diagnosis and treatment dataset of CIP pneumonia was collected. Each sample contained CT images of the initial hospitalization and subsequent diagnosis and treatment, electronic medical record text, and physiological test report data. The diagnosis results were annotated after data desensitization, and then divided into training set, verification machine, and test set at a ratio of 8:1:1; the training set, verification machine, and test set were respectively input into the first-stage model and the second-stage model for training. After setting the training parameters, the Adam optimizer was used to optimize the network, and the cosine learning rate decay function was used to control the learning rate decay. The total number of training rounds was 50 epochs, and the test set was used for verification.

[0032] The beneficial effects of the present invention are mainly reflected in:

[0033] 1. The present invention adopts a three-stage classification prediction method to construct a classification method for CIP pneumonia multimodal data. The data sets used by the algorithm are all important data bases for doctors to judge CIP pneumonia in the actual diagnosis process. Therefore, in terms of data rationality and the algorithm process being in line with reality, it can well assist doctors in actual diagnosis and treatment.

[0034] 2. The first-stage classification model and the second-stage CT image feature extraction module of the present invention improve the network's ability to express key features by introducing a convolutional attention module (CBAM).

[0035] 3. In the second-stage classification model proposed by the present invention, a prediction algorithm based on multimodal data fusion is adopted, and the patient's CT images, electronic medical records and physiological test report data are used as model inputs. According to the characteristics of each modal data, a corresponding feature extraction module is built, and then the multimodal features are fused by the cross-attention method, and finally the features are input into the classifier to classify and obtain the probability of CIP pneumonia. The prediction accuracy is high on the multimodal data set collected by the cooperative hospital. It can provide a reference for doctors and is of great value in reducing the missed diagnosis rate and misdiagnosis rate of CIP pneumonia diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0037] Figure 1 It is a structural schematic diagram of the first stage classification model of the present invention;

[0038] Figure 2It is a structural diagram of the basic unit of the DPN network;

[0039] Figure 3 Schematic diagram of the convolutional attention module CBAM structure;

[0040] Figure 4 It is a CT image feature extraction module of the present invention;

[0041] Figure 5 The electronic medical record feature extraction module of the present invention;

[0042] Figure 6 The physiological examination report feature extraction module of the present invention;

[0043] Figure 7 It is the multimodal feature fusion and classification module of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto:

[0045] Example 1: A multi-stage classification method based on CIP pneumonia multimodal data, by constructing three different stages of classification evaluation models, to classify and predict CT images, medical record text data and physiological test reports at different diagnostic stages of CIP pneumonia. The specific process is as follows:

[0046] S1. Construct the first stage classification model based on CT images

[0047] S101. CT image data set acquisition and processing

[0048] Collect CT images of the patient's lungs. The CT images were taken from cooperative hospitals, including CT images of patients with lung diseases (including CIP pneumonia, lung cancer, lung nodules and other lung diseases) and CT images of normal lungs. All images were resized to 224×224 pixels required by the network structure. Then, four data enhancement methods, including random horizontal flipping, random cropping, Gaussian noise and data filling, were used to process the CT image data. The data set was divided into training set, validation set and test set in a ratio of 8:1:1.

[0049] S102, the backbone network (DPN_CBAM dual-path layer) of the first stage classification model is built

[0050] The first-stage classification model uses a dual-path DPN network as the backbone network, and introduces a convolutional block attention module (CBAM) to improve the network's ability to express key features, thereby improving the overall performance of the network.

[0051] The DPN network is composed of multiple modular basic units stacked together, each of which adopts a bottleneck layer design, such as Figure 2 As shown in the figure, the residual connection path and the dense connection path are combined together by connecting the 1×1 convolution layer, the 3×3 convolution layer and the 1×1 convolution layer. Two of the branches share the first 1×1 convolution layer. The 3×3 convolution layer uses a grouped convolution method to group the input channels, and each group performs a convolution operation with its own convolution kernel. The output of the second 1×1 convolution layer enters the residual connection path branch and the dense connection path branch respectively. The output features of the two paths are merged by Concat and enter the next basic unit. In addition to useful pathological information, the information extracted by each convolution layer of the DPN network also contains some useless background information interference. In the actual diagnosis and treatment process of doctors, doctors will observe and analyze detailed features such as nodules or mass shadows, consolidation or interstitial changes, and tractional bronchiectasis. Therefore, the convolutional attention module (CBAM) is introduced to improve the network's ability to express key features.

[0052] The Channel Attention Module is responsible for adjusting the importance of features between different channels, while the Spatial Attention Module is used to adjust the importance of different positions in the feature map. CBAM can adaptively adjust the weight assignment of feature maps in both channel and spatial dimensions, thereby strengthening the identification of key features. The specific structure diagram of CBAM is shown in the figure below. Figure 3 Shown:.

[0053] The CBAM module can be seamlessly inserted into any CNN structure. Assuming the feature map F∈R C×H×W As input, it first passes through a channel attention map M c ∈R C×1×1 Get the channel attention map F′, and then pass it through the spatial attention map M s ∈R 1×H×W The final optimized feature map F″ is obtained. The calculation process is shown in the following two formulas: Represents matrix element-wise multiplication:

[0054]

[0055] Specifically, for the input feature map, the channel attention module first generates two different spatial context descriptors through maximum pooling and average pooling in the channel dimension respectively. and Then and Send it to a shared fully connected layer for feature fusion and compression, and use bitwise addition to fuse the output channel attention map M c (F)∈R C×H×W The calculation process is shown in the following formula, where σ is the Sigmoid activation function, and the feature map encodes what needs to be highlighted.

[0056]

[0057] The spatial attention module comes after the channel attention module and focuses more on the location of key information. First, the channel weight matrix F′ is processed by maximum pooling and average pooling in the spatial dimension to generate two different descriptors: and Then the two are concatenated to generate an efficient channel descriptor. For the concatenated channel descriptor, a convolutional layer is used to generate a spatial attention map M s (F′)∈R C×H×W The calculation process is shown in the following formula, where σ represents the Sigmoid activation function, f 7×7 Represents a convolution operation with a convolution kernel size of 7×7. The feature map encodes which area is highlighted or suppressed.

[0058]

[0059] Each DPN_CBAM dual-path layer includes a Transition module, three Concat operations and a Bottleneck-CBAM module, such as Figure 1As shown in the figure, the input features of the DPN_CBAM dual-path layer are used as the input of the Transition module and the first Concat operation respectively. The Transition module is used for the transition of two adjacent DPN_CBAM dual-path layers, which helps to reduce the size of the feature map to reduce the number of model parameters and promote the smooth transmission of information. The first Concat operation concatenates the input features of the main path and the features output by the Transition module in the channel dimension, and the output is used as the input of the Bottleneck-CBAM module and the second Concat operation. The second Concat operation concatenates the output of the Bottleneck-CBAM module and the output of the first Concat operation through channel and spatial weighted features, so that the processed features contain both weighted information in the channel dimension and weighted information in the spatial dimension. After the output of the Bottleneck-CBAM module and the output of the Transition module are added, they are used together with the output of the second Concat operation as the input of the third Concat operation, and the final outputs of the residual path and dense path of the current layer are concatenated again to ensure that the features of the two paths can continue to be passed to the next layer, realizing multi-layer feature reuse.

[0060] In addition, in order to better extract the key features of CT images of CIP pneumonia patients, the present invention integrates the CBAM module into the Bottleneck layer of each basic unit on the basis of the DPN backbone network to form a new Bottleneck-CBAM module. The specific structure is as follows: Figure 1 Shown in the upper part.

[0061] The Bottleneck-CBAM module includes the Bottleneck module and the CBAM module. In the Bottleneck module, the dimension is first reduced through a 1×1 convolution layer, and then the features are convolved through a 3×3 convolution layer, and then the dimension is increased through a 1×1 convolution layer. Therefore, the Bottleneck module reduces the amount of calculation and parameters while ensuring the efficiency of feature transfer, making the deep network more efficient when processing high-dimensional features and avoiding the explosive growth of calculations. The CBAM module includes the channel attention part and the spatial attention part in sequence. First, the features are passed through the channel attention part, and the input feature map is subjected to global average pooling (AvgPool) and global maximum pooling (MaxPool) respectively to generate two different channel description vectors. These two description vectors retain the global channel information. These two pooling vectors pass through a shared multi-layer perceptron (MLP) to output two new weight vectors, which represent the weight coefficients in the channel dimension respectively. Then, through the Sigmoid activation function, these weights are normalized to values ​​between 0 and 1, and these weights are multiplied element by element with the input feature map to obtain the feature map after spatial weighting. Secondly, after the spatial attention part, the attention calculation in the spatial dimension is performed after the channel attention. First, the channel weighted feature map is subjected to global average pooling and maximum pooling respectively to obtain two two-dimensional spatial description maps. These two spatial description maps are passed through a 7x7 convolution layer to generate a new spatial weight map. Then, through the Sigmoid function, the spatial weights are normalized to values ​​between 0 and 1, and these weights are multiplied element by element with the input feature map to obtain the feature map after spatial weighting. The final output features are the result of weighted processing in both channel and spatial dimensions, thereby improving the overall network's ability to express key information.

[0062] S103. Overall network construction of the first stage classification model

[0063] The first stage classification model (i.e., DPNCT-Net network) is based on the DPN network, and the four stages of the DPN network are replaced by the DPN_CBAM dual-path layer of the present invention. The overall architecture is as follows: Figure 1As shown, it includes the initial convolution layer and pooling layer, the middle 4 DPN_CBAM dual-path layers, and the final pooling layer and classifier. First, the CT image data set is preprocessed to obtain an input image matrix of size 224×224×3, which is sequentially subjected to the initial 7×7 convolution and 3×3 maximum pooling layers for feature extraction and dimensionality reduction, and the output feature matrix of size 56×56×64. The feature matrix is ​​then input into the DPN_CBAM dual-path layer. The DPNCT-Net network uses 4 DPN_CBAM dual-path layers in design. The number of blocks in each layer determines the depth of the entire network. The patent of this invention refers to DPN92 in the specific parameter design. The last convolution layer outputs a feature matrix of size 7×7×2560. After passing through the 7×7 adaptive average pooling layer, a feature vector of size BatchSize×1×1×2560 is obtained. Finally, the FC+Softmax classifier is used to output the classification results and probabilities. Based on the classification results, the doctor needs to further combine the electronic medical record text or physiological test report to make a clinical diagnosis to determine whether the patient suffers from CIP pneumonia; being marked as not suffering from the disease does not rule out the possibility of suffering from CIP pneumonia. The present invention can only give the probability that the patient may suffer from CIP pneumonia at the initial diagnosis to assist the doctor in screening. The specific judgment and further clinical diagnosis still need to be combined with the doctor's actual diagnosis and treatment experience.

[0064] S2. Construct the second stage classification model based on multimodal data fusion

[0065] Based on the classification results of the first stage, the doctor manually determines that further classification and prediction are needed, and then further classification and prediction are performed through the second stage classification model. The pre-processed CT image data, electronic medical record text and physiological examination report data are respectively passed through the CT image feature extraction module, the electronic medical record feature extraction module and the physiological test report feature extraction module to obtain the feature representation V of the CT image, the electronic medical record text and the physiological test report. Image 、V record and V report Afterwards, the second stage classification prediction results and probabilities are obtained through multimodal feature fusion and classification modules.

[0066] S201. Data collection and preprocessing of electronic medical records and physiological examination reports

[0067] The data sources of the electronic medical records and physiological examination reports of the present invention are consistent with those in S101.

[0068] The electronic medical record mainly includes the patient's basic information, chief complaint, past history, current medical history, personal history and preliminary diagnosis fields. The chief complaint, past history and current medical history fields are selected for stop word filtering, and the length of the super long sequence is limited to 512 by random clipping. The length of the text sequence after data preprocessing is fixed to 512, and the insufficient length is filled with [SEP], and the text with too long length is truncated. Physiological test report data can be divided into two categories, one is numerical continuous data, such as blood routine, urine routine, myocardial enzyme spectrum routine examination, etc.; the other is discrete data, such as physical examination, pulmonary ventilation diffusion function examination, etc. In view of the characteristics of these two types of data, the present invention adopts different preprocessing methods respectively: for discrete data, One-hot encoding is used to expand the value of discrete features to Euclidean space; for continuous data, considering that there is a large difference between the reference value and unit of continuous data indicators, a normalization method is used to balance the model's attention to continuous data indicators.

[0069] S202, CT image feature extraction module

[0070] The CT image data used in the second stage are the same as those used in the first stage of S1 in terms of data size and data format, except that the multimodal data set in the second stage only involves the annotation of whether or not patients have CIP pneumonia. Therefore, a CT image feature extraction module is built on the basis of the DPNCT-Net network proposed in S1, and the final classifier of the DPNCT-Net network is replaced with a multi-layer perceptron. The specific structure is as follows: Figure 4 As shown in Figure 2, the difference from the first-stage DPNCT-Net network is that the image feature matrix passes through the multi-layer perceptron (MLP) after passing through the adaptive average pooling layer, which removes the redundant information of the high-dimensional feature vector and realizes feature dimensionality reduction to obtain the image modality feature V Image .

[0071] S203, electronic medical record feature extraction module

[0072] In view of the large difference in the length of each field in the medical record text dataset, this paper selects the Chinese pre-trained model RoBERTa-wwm-ext, which has performed well in medical text classification, to build an electronic medical record feature extraction module. The Bert model uses a bidirectional Transforme structure to extract context information and performs well in multiple text processing tasks. RoBERTa has many improvements based on Bert and has achieved better performance, mainly reflected in:

[0073] (1) Dynamic mask. Compared with the static mask strategy adopted by the Bert model, the RoBERTa model copies the pre-trained data 10 times and randomly selects 15% of the tokens for masking in each copy, so that the model can obtain more complete semantic information.

[0074] (2) Remove the NSP task. RoBERTa removes the NSP task, which has certain restrictions on the length of the model, making it difficult for the model to learn long-distance dependency information well. During training, multiple consecutive sentences are input each time until the maximum length reaches 512.

[0075] (3) Larger corpus size and batch size. Compared with the 13GB training data and 256 batch sizes of the Bert model, RoBERTa expands the training data to 160GB and sets the batch size to 8k, which enables the model to have stronger learning and generalization capabilities.

[0076] The overall process of electronic medical record feature extraction is as follows Figure 5 The present invention inputs the main complaint, current medical history and past medical history field sequences preprocessed in step S201 into the RoBERTa-wwm-ext model module to obtain the feature V zs 、V xbs and V jws The model uses a large number of multi-head attention mechanisms to extract contextual semantic information, and obtains the representation vector of the sequence after 12 transformer layers:

[0077] S={v0,v1,…,v L-1 =RoBERTa(X)

[0078] In the formula L is the sequence length of the model, 512, and d is the hidden layer dimension of the model, 768.

[0079] Then, vector concatenation is performed to obtain the complete medical record text modality feature V record .

[0080] S204, physiological detection report feature extraction module

[0081] Physiological test report data can be divided into two categories: one is numerical continuous data, such as blood routine, urine routine, etc.; the other is discrete data, such as physical examination, pulmonary ventilation and diffusion function test, etc. After one-hot encoding and normalization preprocessing in step S201, a large number of sparse features and dense features as well as cross-information between features are obtained.

[0082] DeepFM is a model that combines a deep neural network (DNN) and a factor machine (FM), which can process sparse features and dense features separately and learn the interactive information of high-order and low-order features at the same time. Therefore, the present invention adopts a physiological detection report feature extraction module based on the DeepFM model.

[0083] The main structure of DeepFM consists of two parts: FM and DNN, which respectively realize the interaction of low-order feature vectors and the high-dimensional representation of the interaction features. FM and DNN ) is input into the Sigmoid function to obtain the prediction result of the model:

[0084]

[0085] The FM part mainly performs feature interaction on low-order feature vectors, and learns the connection and weight between features by decomposing potential factors. Its core is to perform second-order feature interaction on feature vectors. The DNN part mainly learns the interaction of high-order feature vectors to extract deeper semantic information in physiological detection reports. The specific calculation process of the FM and DNN parts is as follows:

[0086]

[0087] y DNN =σ(W l+1 a l +b l+1 )

[0088] Where n is the number of variables, w0 is the bias term, and x i represents the i-th component of the eigenvector x, w i 、w ij represents the coefficient of its corresponding feature; l represents the number of hidden layers, σ is the Relu activation function, W l+1 , b l+1 Represent weight and bias respectively, a l is the output of the lth hidden layer.

[0089] The overall structure of the physiological detection report feature extraction module based on DeepFM is as follows Figure 6As shown. The input layer includes the one-hot encoded features after data preprocessing and the normalized continuous features, which represent the encoding vector of a certain dimensional category. The embedding layer extracts the potential features to be learned, and calculates the second-order cross-term weights and maps them to low-dimensional feature vectors. In the FM layer, the left part calculates the first-order terms, and the inner product operation on the right is used to calculate the second-order feature interaction of FM; at the same time, in the hidden layer, the fully connected layer continuously enhances the feature expression ability. Finally, the output module integrates the output of the FM layer and the output of the hidden layer to obtain a more complete feature representation V report .

[0090] S205, multimodal feature fusion and classification module

[0091] In obtaining the feature representation V of CT images, electronic medical records and physiological test reports Image 、V record and V report Finally, the present invention adopts a feature fusion method based on multimodal cross attention to mine the correlation information between each modality, thereby improving the expressive power of the post-modal fusion feature. The cross attention mechanism is the correlation between different modal inputs in the form of vector inner product, which is calculated as follows:

[0092]

[0093] Where x and y represent the characteristics of different modes.

[0094] The structure of the multi-modal feature fusion network based on cross attention (Multi-Modal Cross Attention Net, MMCA-Net) is as follows Figure 7 When multimodal features are fused, the electronic medical record text feature V record , CT image features V Image and physiological detection report characteristics V report The two are input into the Cross Attention module to calculate the cross features between the modalities, and then the original electronic medical record text features V record , CT image features V Image and physiological detection report characteristics V report And all cross features are concat operated to obtain the fused feature representation V fusion The encoding classification layer includes MLP and softmax function, which converts the fused feature vector V fusion The input is sent to a learnable encoder to further extract deep information, and the abstract feature vector is mapped back to the tag space for feature extraction. Finally, the Softmax function is connected to normalize it to the probability distribution, and the results and probabilities of the second-stage classification prediction are output.

[0095] The patient's treatment process will generate multiple CT image data, electronic medical record text and physiological examination report data at different times, so multiple second-stage classification prediction results and probabilities can be generated. The features generated in each classification prediction process are represented by V Image 、V record and V report Store for subsequent third-stage analysis.

[0096] S3. The third stage evaluation algorithm based on Sen+Mann-Kendall test

[0097] In the process of prognosis and treatment, there may be incomplete multimodal data, missing CT image or test report modality data. After discussion with the cooperating doctors, the present invention will not use the fused feature vector V obtained in step S205 in the third stage evaluation. fusion Instead, the feature vector V obtained in each classification prediction process in the second stage is record 、V Imag and V report The normalized features (i.e., those obtained in steps S202-204) are then normalized, and then the Euclidean distances between the normalized feature vectors of each modality and the standard feature vector are calculated and averaged to obtain the current prognosis and rehabilitation sequence of the patient.

[0098] The process of obtaining the standard feature vector is as follows:

[0099] Multimodal data of a control group without CIP pneumonia provided by professional physicians, including CT image data, electronic medical record text, and physiological examination report data, are collected. Then, the CT image data, electronic medical record text, and physiological examination report data are preprocessed and features are extracted according to steps S201-S204, respectively, so as to serve as standard feature vectors of the CT image data, electronic medical record text, and physiological examination report data, respectively.

[0100] Finally, the Sen+Mann-Kendall test method was used to analyze the trend of the prognosis rehabilitation sequence, and the evaluation value effect was assigned to evaluate the effect of rehabilitation diagnosis and treatment. The specific value of the valuation effect is {2: significantly improved, 1: improved, 0: no significant change, -1: worsened, -2: significantly worsened}.

[0101] The Sen method is a slope estimation method that uses the positive or negative trend degree β to reflect the trend of the sequence. When β<0, it indicates a downward trend, and when β>0, it indicates an upward trend. The calculation formula is as follows:

[0102]

[0103] Among them, the median function means taking the median, x j 、xi represents the j-th data point and the i-th data point in the time series.

[0104] The Mann-Kendall method can measure the significance of the trend by the absolute value of the statistic. The calculation process is as follows: For the sequence D = {d1, d2, ..., d n}, select different significant test statistics according to the sequence length n: when n<10, the test statistic S is calculated as follows:

[0105]

[0106] Where sgn(x) is the sign function, n is the length of the sequence, and d j ,d i Represents the jth data point and the ith data point in the sequence. When n ≥ 10, the statistic S approximately follows a normal distribution, and the test statistic Z is used:

[0107]

[0108]

[0109] Where Var(S) represents the variance of the S statistic and Z represents the standardized Mann-Kendall statistic.

[0110] The number of tests for the same patient in the data set of the present invention is generally less than 10, so the trend degree β and the significant test statistic S are used for trend analysis. Under a given significance level α, when β>0 and |S|≥S α / 2 The sequence shows a significant upward trend when β>0 and |S|<S α / 2 The sequence shows an insignificant upward trend; similarly, when β < 0 and |S| ≥ S α / 2 The sequence shows a significant downward trend. When β<0 and |S|<S α / 2 The sequence showed a non-significant downward trend.

[0111] The Sen method is a robust nonparametric statistical technique for trend detection. It can handle errors and outliers well, but it cannot determine the significance of the trend. The Mann-Kendall method does not require assumptions about the distribution of the data, and can measure the significance of the trend by the absolute value of the statistic. Combining the Sen method and the Mann-Kendall method can more comprehensively measure the trend and significance of sequence data.

[0112] S4. Training model

[0113] S401. Training Dataset

[0114] The CIP pneumonia multimodal diagnosis and treatment data set used in the present invention comes from the laboratory's cooperative hospital, including CT images, electronic medical record texts, and physiological test report data. After data desensitization, the diagnosis results are annotated by experts from the cooperative hospital. Among them, there are 261 cases of patient data marked as CIP pneumonia, 292 cases marked as not suffering from CIP pneumonia, and most non-CIP patients are infected with other lung diseases (mostly lung cancer, lung nodules, etc.). According to statistics, the data set contains a total of 13,372 CT images and 1,383 electronic medical records, and each patient contains multiple electronic medical records and several physiological test reports for initial admission and subsequent diagnosis and treatment.

[0115] S402, training configuration

[0116] Training is performed in a Centos operating system server. The development environment is based on the Pytorch deep learning framework. The specific software and hardware configurations are shown in Table 1:

[0117] Table 1

[0118] name Environment Configuration operating system CentOS7.3.1611 processor Intel(R)Xeon(R)CPU 12*E5-2609v3@1.9GHz Graphics Tesla P4 8GB Memory 128GB Development Environment Python 3.8 + PyTorch 1.13

[0119] S403, Training Process

[0120] First, the data set is split and 80% of the patient samples are selected as the training set, 10% of the patient samples are selected as the validation set, and 10% of the patient samples are selected as the test set. The training sets are respectively input into the networks of step S1 and step S2 for training respectively. The model of the present invention uses the Adam optimizer to optimize the network during training, the first-order momentum parameter is set to 0.9, the second-order momentum parameter is set to 0.999, the batch size is set to 32, the initial learning rate is set to 0.001, and the cosine learning rate decay function is used to control the learning rate decay, the weight decay is set to 0.0001, the total training round is 50 epochs, and the test set is used for verification.

[0121] S5. Online use

[0122] When the patient first visits the doctor, the CT image data is obtained and scaled to 224×224 pixels and input into the first-stage classification model (DPNCT-Net network) to obtain the classification prediction results and probabilities of CIP pneumonia. The doctor decides whether the patient needs further second-stage classification prediction based on the classification prediction results and probabilities as well as clinical experience.

[0123] In the second stage, the CT image data, electronic medical record text and physiological examination report data of the returning patients are collected and preprocessed respectively. Then, the preprocessed CT image data, electronic medical record text and physiological examination report data are respectively passed through the CT image feature extraction module, the electronic medical record feature extraction module and the physiological test report feature extraction module to obtain the CT image feature V Image , Electronic medical record text features V record and physiological detection report characteristics V report Finally, the second-stage classification prediction results and probabilities are obtained through multimodal feature fusion and classification modules to assist doctors in diagnosis and treatment and for analysis and use in the third-stage evaluation algorithm.

[0124] For patients who have been to the hospital for at least two visits, if evaluation and scoring are required, the feature vectors of each modality (CT image feature V Image , Electronic medical record text features V record and physiological detection report characteristics V report ) and the standard eigenvector, and then construct the prognosis rehabilitation sequence with multiple Euclidean distances. The trend of the prognosis rehabilitation sequence is analyzed by the Sen+Mann-Kendall test method to obtain the estimated effect, thereby evaluating the effect of rehabilitation diagnosis and treatment.

[0125] experiment:

[0126] 1: DPNCT-Net structure ablation experiment.

[0127] In view of the complex features existing in CT images, the present invention adopts a DPN dual-path structure based on a dense connection path and a residual connection path to build an image feature extraction module, and introduces a convolutional attention module for optimization. The effectiveness of each module design is verified by designing an ablation experiment. The specific experimental results are shown in Table 2:

[0128] Table 2

[0129] Model Accuracy (%) AUC (%) F1-score(%) ResNet-101 80.66 82.96 79.89 DenseNet-161 78.42 80.17 77.38 DPN-92 81.72 84.01 80.39 The present invention (DPNCT-Net) 83.80 85.54 82.44

[0130] According to the experimental results in the table, the DPN (see Chen Y, Li J, Xiao H, et al. Dual path networks [J]. Advances in neural information processing systems, 2017, 30.) model outperforms ResNet (see He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.) and DenseNet (see Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 4700-4708.) in all indicators, proving the advantages of using a dual-path structure, which can effectively combine the advantages of ResNet's reuse of refined features with DenseNet's exploration of new feature details. After the introduction of the CBAM module, the accuracy of the DPNCT ​​network was further improved by 2.08% compared with the DPN network, verifying that the CBAM module can enhance useful information from both channel and spatial dimensions and improve the network's ability to express key features.

[0131] 2: Comparative experiment of multimodal feature fusion methods.

[0132] In order to verify the effectiveness of the multimodal feature fusion method based on cross attention adopted by the present invention, this experiment introduces different multimodal feature fusion methods for comparative analysis, and selects the feature direct splicing fusion method CFN, tensor fusion network TFN, and low-rank multimodal fusion LWF to replace the multimodal feature fusion of this algorithm respectively. The specific experimental results are shown in Table 3:

[0133] Table 3

[0134] Multimodal feature fusion method Accuracy (%) AUC (%) F1-score(%) CFN 82.51 85.49 80.58 TFM 85.36 87.32 84.20 LMF 85.91 87.92 84.63 The method of the present invention (MMCA) 87.12 89.81 85.86

[0135] According to the experimental results in the table, the method CFN based on direct feature concatenation performs the worst, because direct feature concatenation ignores the modality-specific statistical properties and the associated feature information between modalities. The tensor-based feature fusion methods TFN and LMF (see Liu Z, Shen Y, Lakshminarasimhan VB, et al. Efficient low-rank multimodal fusion with modality-specific factors [J]. arXiv preprint arXiv: 1806.00064, 2018.) also perform poorly, indicating that the complex correlation between multimodal data is not fully mined by tensor outer product on the dataset of the present invention. The multimodal feature fusion method MMCA based on cross attention proposed in the present invention performs best in all indicators, indicating that the cross attention mechanism can maximize the retention of key information extracted from each modality and fully mine the associated information between modalities to obtain a complete fusion feature vector. The experimental results prove the effectiveness of the multimodal feature fusion method based on cross attention proposed in this chapter.

[0136] Finally, it should be noted that the above examples are only some specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there are many variations. All variations that can be directly derived or associated with the content disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A multi-stage classification method based on CIP pneumonia multimodal data, characterized by: The process includes: S1, obtaining CT image data of patients who visit the hospital for the first time, and inputting them into the first-stage classification model trained offline after scaling, to obtain the first-stage classification prediction results and probabilities; S2. Obtain CT image data, electronic medical record text, and physiological examination report data of patients who have had at least one follow-up visit, and use them as input to the second-stage classification model trained offline after preprocessing operations to obtain CT image features V Image , Electronic medical record text features V record and physiological detection report characteristics V report , and then obtain the classification prediction results and probabilities of the second stage through multimodal feature fusion and classification modules; S3, each time the CT image feature V obtained in step S2 is Image , Electronic medical record text features V record and physiological detection report characteristics V report After normalization, the Euclidean distances between the standard eigenvectors were calculated and averaged to construct a prognosis-rehabilitation sequence. The trend of the prognosis-rehabilitation sequence was analyzed using the Sen+Mann-Kendall test method to obtain the estimated effect.

2. A multi-stage classification method based on CIP pneumonia multimodal data according to claim 1, characterized in that: The first stage classification model includes: a sequentially connected convolutional layer and pooling layer, four consecutive DPN_CBAM dual-path layers, a pooling layer and a classifier; The DPN_CBAM dual-path layer includes a Transition module, three Concat operations and a Bottleneck-CBAM module. The input features are respectively used as inputs of the Transition module and the first Concat operation. The first Concat operation concatenates the input features with the features output by the Transition module, and the output is used as the input of the Bottleneck-CBAM module and the second Concat operation; the second Concat operation concatenates the output of the Bottleneck-CBAM module with the output of the first Concat operation; the output of the Bottleneck-CBAM module and the output of the Transition module are added, and the output is used together with the output of the second Concat operation as the input of the third Concat operation.

3. A multi-stage classification method based on CIP pneumonia multimodal data according to claim 2, characterized in that: The Bottleneck-CBAM module includes a Bottleneck module and a CBAM module, the Bottleneck module includes three convolutional layers, and the CBAM module includes a channel attention part and a spatial attention part in sequence.

4. A multi-stage classification method based on CIP pneumonia multimodal data according to claim 3, characterized in that: The CT image feature extraction module is based on the first-stage classification model, and replaces the last classifier of the first-stage classification model with a multi-layer perceptron.

5. A multi-stage classification method based on CIP pneumonia multimodal data according to claim 4, characterized in that: The electronic medical record feature extraction module is the RoBERTa-wwm-ext model; The physiological detection report feature extraction module is a DeepFM model.

6. A multi-stage classification method based on CIP pneumonia multimodal data according to claim 5, characterized in that: The multimodal feature fusion and classification module includes a multimodal feature fusion network based on cross attention, MLP and softmax function, which transforms the CT image feature V Image , Electronic medical record text features V record and physiological detection report characteristics V report The two are input into the Cross Attention module to calculate the cross features between the modalities, and then the electronic medical record text features V record , CT image features V Image and physiological detection report characteristics V report And the cross features are concat operated to obtain the fused feature V fusion , and then pass through MLP and softmax function.

7. A multi-stage classification method based on CIP pneumonia multimodal data according to claim 6, characterized in that: The process of obtaining the standard feature vector is as follows: Multimodal data of the control group without CIP pneumonia were collected, including CT image data, electronic medical record text and physiological examination report data, and then preprocessed and passed through the CT image feature extraction module, electronic medical record feature extraction module and physiological test report feature extraction module respectively to obtain standard feature vectors of CT image data, electronic medical record text and physiological examination report data.

8. A multi-stage classification method based on CIP pneumonia multimodal data according to claim 7, characterized in that: The pre-processing operation is specifically as follows: For CT image data, scaling is used; After filtering the stop words for the chief complaint, past medical history, and present medical history fields in the electronic medical record, the length of the overlong sequence was limited to 512 by random trimming, and the insufficient length was filled with [SEP]; For discrete data in physiological test report data, One-hot encoding is used to expand the values ​​of discrete features to Euclidean space; for continuous data in physiological test report data, normalization method is used.

9. A multi-stage classification method based on CIP pneumonia multimodal data according to claim 8, characterized in that: The offline training process of the first-stage classification model and the second-stage classification model is as follows: A multimodal diagnosis and treatment dataset of CIP pneumonia was collected. Each sample contained CT images of the initial hospitalization and subsequent diagnosis and treatment, electronic medical record text, and physiological test report data. The diagnosis results were annotated after data desensitization, and then divided into training set, verification machine, and test set at a ratio of 8:1:1; the training set, verification machine, and test set were respectively input into the first-stage model and the second-stage model for training. After setting the training parameters, the Adam optimizer was used to optimize the network, and the cosine learning rate decay function was used to control the learning rate decay. The total number of training rounds was 50 epochs, and the test set was used for verification.

Images