Honeycomb lung disease condition prediction method based on multi-modal time sequence data

Abstract

The invention provides a cellular lung disease condition prediction method based on multi-modal time sequence data, and belongs to the technical field of cellular lung disease condition prediction. The technical problem to be solved is to provide a cellular lung disease condition prediction method based on multi-modal time sequence data. According to the technical scheme, the method comprises the steps that CT description texts and CT image data of a patient at different time points are collected, then the collected data are cleaned and labeled, and a comprehensive time sequence data set is constructed in combination with the data and used for training and testing a model; a time sequence model is built, the time sequence model comprises an input layer, a wavelet transform layer, a Transform layer, a fusion layer and an output layer, a built time sequence data set is input into the input layer, time sequence data is converted into embedded representation, and the input layer processes multi-modal data and prepares to be input into a subsequent layer; the method is used for predicting the condition of the honeycomb lung disease.

Description

Technical Field

[0001] The present invention provides a method for predicting a condition of a honeycomb lung disease based on multimodal time series data, and belongs to the technical field of predicting a condition of a honeycomb lung disease. Background Art

[0002] Honeycomb lung is the main histopathological feature of idiopathic pulmonary fibrosis (IPF), which is a honeycomb change formed by various advanced lung diseases. Most patients have no obvious early symptoms, and missed diagnosis and misdiagnosis are common. Most patients are misdiagnosed as chronic obstructive pulmonary disease (COPD), asthma or congestive heart failure when they first visit the doctor, which delays the precious early diagnosis and treatment time. In addition, during the development of IPF, the disease progression of different patients varies greatly.

[0003] Honeycomb lung lesions are often distributed in the middle and lower lobes of the lungs, forming vesicles with a diameter of about 1 cm, arranged like honeycombs or honeycombs. Due to their stable structure and fixed area of ​​interest, CT images are prone to information redundancy. The existing disease prediction methods are insufficient in processing complex time series data and multi-scale feature extraction, which will lead to large errors in the prediction results and unsatisfactory prediction effects. Summary of the invention

[0004] In order to overcome the deficiencies in the prior art, the present invention aims to solve the technical problem of providing a method for predicting the condition of honeycomb lung disease based on multimodal time series data.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for predicting the condition of honeycomb lung disease based on multimodal time series data, comprising the following condition prediction steps:

[0006] Step S1: Collect the patient’s CT description text and CT image data at different time points through the hospital’s electronic medical record system and radiology department reports, and then clean and annotate the collected data;

[0007] Step S2: By combining CT image data and text data at different time points, a comprehensive time series dataset is constructed for training and testing the model;

[0008] Step S3: Building a time series model, including:

[0009] The input layer receives the time series dataset.

[0010] Wavelet transform layer, used to perform wavelet transform on input data and extract multi-scale frequency domain features;

[0011] Transformer layer, used to process time series data and capture long-term dependencies;

[0012] The fusion layer is used to fuse the multi-scale frequency domain features extracted by the wavelet transform layer with the long-term dependencies;

[0013] The output layer is used to output the prediction results and predict the development status of the disease;

[0014] Step S4: Input the constructed time series dataset into the input layer, convert the time series data into an embedded representation, and process the multimodal data by the input layer to prepare for input into subsequent layers;

[0015] Step S5: Input the time series data into the wavelet transform branch, perform wavelet transform on the input data, extract multi-scale frequency domain features, and the generated wavelet features are represented as F wavelet ;

[0016] Step S6: Input the time series data into the Transformer branch, directly process the time series data, capture the long-term dependencies in the data, and generate the Transformer feature representation as F trans ;

[0017] Step S7: After going through the two feature extraction paths, the data is input into the fusion layer of the two feature vectors and fused using a gating mechanism;

[0018] Step S8: Input the fused feature vector into the fully connected layer to generate the final feature classification vector, output the prediction result, and predict the future development status of the disease;

[0019] Step S9: define the loss function used in the training process as a cross entropy loss function;

[0020] Step S10: back-propagate the loss value of the loss function back to the model, calculate the gradient and update the parameters of the model to minimize the loss value;

[0021] Step S11: Update model parameters using Adam optimizer;

[0022] Step S12: For the feature classification vector generated in step S8, the loss is calculated through the classification head, and the inverse gradient calculation is performed to adjust the parameters of the model;

[0023] Step S13: Finally define and output the category label:

[0024] Label 0 means that the disease is predicted to get better in the future;

[0025] Label 1 means that the symptoms will not change;

[0026] Label 2 means the condition will continue to worsen.

[0027] The specific method for cleaning and labeling the data in step S1 is:

[0028] Clean the image data, select the 2D slices that can reflect the lesions in the acquired data, unify the image size to 256*256 pixels, and then reconstruct the existing 2D slice data into 3D NRRD data through reconstruction technology after cleaning the image data;

[0029] Clean the text data to remove text noise and standardize medical terms and units;

[0030] Manually annotate CT image data, extract lesion areas and generate binary mask images, annotate text data and extract disease-related information.

[0031] The specific method for constructing the time series data set in step S2 is:

[0032] Step S21: Construct a time series data set, including four columns of data:

[0033] The first column stores the patient number, which uniquely identifies each patient;

[0034] The second column stores the time point, recording the specific time point of data collection;

[0035] The third column stores the CT image data, which stores the digital representation of the CT image at each time point;

[0036] The fourth column stores text data, which stores the digital representation of the CT description data corresponding to each time point;

[0037] Each piece of data in the constructed dataset is composed of the above content, providing input for the designed model;

[0038] Step S22: Dataset preparation:

[0039] The data input to the model includes digital representations of CT image data and text data, wherein the CT image data is subjected to feature extraction through a convolutional neural network to generate a feature vector, and the text data is subjected to word vector conversion through an embedding layer and further subjected to a Transformer network to generate a feature vector, and the above-generated feature vectors are combined into a comprehensive input data;

[0040] Define label data: Each time series data corresponds to a label data, which indicates the predicted future state of the disease. The label data is divided into three categories: the disease state improves, remains the same, and continues to worsen.

[0041] The specific method of processing multimodal data by the input layer in step S4 is:

[0042] Step S41: inputting the sorted multimodal data, including CT image data I, text description data T, and time series data S;

[0043] Step S42: Process the CT image data I, text description data T, and time series data S. The processing expression is:

[0044] F I =CNN(I);

[0045] F T =NLP(T);

[0046] F S =CNN(S);

[0047] Then the processed multimodal data are fused to obtain a comprehensive feature vector representation, which is expressed as:

[0048] F fusion =concat(F I ,F T ,F S ).

[0049] The specific method of performing wavelet transform on the input data in step S5 is:

[0050] Step S51: Use a low-pass filter h[n] and a high-pass filter g[n] to filter and downsample the fused vector to obtain four sub-bands of different resolutions: low frequency-low frequency LL, low frequency-high frequency LH, high frequency-low frequency HL, and high frequency-high frequency HH. The sub-band expression is:

[0051]

[0052] Step S52: After obtaining four sub-bands, a feed-forward neural network is used for processing to enhance the features of each sub-band. The processing expression is:

[0053] F ffn1 =FFN([LL+LH+HL+HH]);

[0054] Step S53: After passing through the feedforward neural network, the extracted features are converted into the spatial domain by inverse wavelet transform, which is represented by F ICWT1 , providing input for subsequent processing;

[0055] Step S54: The feature F converted to the spatial domain in step S54 ICWT1 Perform wavelet transform again and decompose it into four different sub-bands LL, LH, HL, HH;

[0056] Step S55: sum the three sub-bands except the low-frequency-low-frequency sub-band, then concatenate them with the low-frequency-low-frequency sub-band to obtain the feature representation in the frequency domain, and then enhance the feature through the feedforward neural network to obtain the frequency domain feature of the data. The calculation formula is:

[0057] F ffn2 =FFN([LL, LH+HL+HH]);

[0058] Step S56: Perform inverse wavelet transform on the extracted features and convert them back to the spatial domain for further convolution operation and feature extraction. The calculation formula is:

[0059] F wavelet =ICWT(F ffn2 ).

[0060] The specific method for processing time series data by the Transformer branch in step S6 is:

[0061] Step S61: Data F represented by the comprehensive feature vector fusion Specifically, it is a d×n feature matrix, where d is the feature dimension of a patient's data at a time point, and n is the sequence length. It is represented as a data format suitable for inputting Transformer [x 1 , x 2 , …, x n ], where each x i is a eigenvector, denoted by F seq ;

[0062] Step S62: introduce time series data information through position coding, and use sine and cosine functions to generate a position coding vector PE, which is expressed as:

[0063]

[0064] In the formula, pos is the position of the element in the sequence, i is the index of the feature dimension, and d model is the dimension of the model;

[0065] Add the calculated position encoding vector PE to the feature vector F processed by the input layer seq In this way, the data input into the Transformer model contains both feature information and the position information of each feature in the time series. The expression is:

[0066] X seq =F seq +PE;

[0067] Step S63: Process data in the multi-head self-attention layer:

[0068] The input feature sequence is divided into multiple self-attention heads, and attention calculation is performed independently on each attention head. seq The query matrix Q, key matrix K and value matrix V are generated by linear transformation, and the expression is:

[0069]

[0070] Among them, W Q , W K and W V are all trainable weight matrices;

[0071] Then calculate the attention weight, the calculation formula is:

[0072]

[0073] Among them, d k is the dimension of the bond matrix;

[0074] Finally, the outputs of all attention heads are concatenated and linearly transformed to obtain the final feature representation, which is expressed as:

[0075] HultiHead(Q,K,V)=concat(head 1 , head 2 , ..., head h )W O ;

[0076] Among them, headi represents the attention output of the i-th head, and WO is the weight matrix;

[0077] Step S64: Processing data in a feed-forward neural network layer:

[0078] The feature vector at each position is further processed by a feed-forward neural network, which consists of two linear layers activated by a RELU function in the middle;

[0079] Step S65: The output data is processed using layer normalization and residual connection, and the extracted features are fused through a gated fusion mechanism and the features extracted by the wavelet transform branch to predict the future development of the disease.

[0080] The specific method of using the gating mechanism to perform fusion in step S7 is:

[0081] is the wavelet transform feature F wavelet and Transformer feature F transSet the gating unit separately to control the flow of each feature. The expression is:

[0082] G wavelet =σ(W G-wavelet F wavelet +b G-wavelet );

[0083] G trans =σ(W G-trans F trans +b G-trans );

[0084] Then, the features of the gated units are weighted and summed to generate the gated fusion features. The calculation formula is:

[0085] F gated-fusion =G wavelet ⊙F wavelet +G trans ⊙F trans .

[0086] The specific method of using the cross entropy loss function for training in step S9 is:

[0087] Step S91: define the cross entropy loss function, the expression is:

[0088]

[0089] Among them, N represents the number of samples, C represents the number of categories, and y i,j represents the true label of the jth category of the i-th sample, Represents the predicted value of the jth category of the i-th sample;

[0090] Step S92: input the processed data into the model for forward propagation to obtain the prediction result of the model;

[0091] Step S93: input the prediction result of the model and the real label of the data into the cross entropy loss function to calculate the loss value of the model;

[0092] Step S94: back-propagating the loss value of the model back to the model to guide the model to update parameters, thereby reducing the loss value of the model;

[0093] Step S95: Repeat steps S92 to S94 until the model converges or reaches a predetermined number of training rounds.

[0094] The specific method of using the Adam optimizer to update the model parameters in step S11 is:

[0095] Step S111: Set the initial parameters of the model to θ; set the initial learning rate α to 0.001; set the decay rate β of the first-order momentum estimate1 is 0.9 and the decay rate of the second-order momentum estimate is β 2 is 0.999; set the minimum constant ε to 10 to prevent division by zero -8 ; Initialize the first-order momentum estimate m 0 = 0 and the second-order momentum estimate v 0 =0;

[0096] Step S112: For each training sample, calculate the gradient of the model under the current parameters:

[0097] At time step t, calculate the loss function f(θ t ) The gradient of the parameter θ Reflects the current parameter θ t The update direction and update amplitude;

[0098] Step S113: define the first-order momentum estimate m t is the current gradient g t The exponentially weighted moving average is calculated as:

[0099] m t =β 1 ·m t-1 +(1-β 1 )·g t ;

[0100] Among them, β 1 is the momentum decay rate, used to smooth instantaneous changes in the gradient;

[0101] Step S114: Define the second-order momentum estimate v t is the current gradient g t The exponentially weighted moving average of the square of is calculated as:

[0102] v t =β 2 ·v t-1 +(1-β 2 )·g t 2 ;

[0103] Among them, β 2 is the decay rate of the second-order momentum, which is used to smooth the instantaneous change of the square of the gradient;

[0104] Step S115: Using the bias-corrected first-order momentum estimate and second-order momentum estimate, update the learning rate to reduce the bias caused by the initialization momentum estimate:

[0105] For m t and v t To perform bias correction, the calculation formula is:

[0106]

[0107] For m t and v t After the deviation correction, it is used to update the parameter θ, and the calculation formula is:

[0108]

[0109] Among them, α is the learning rate, which is used to control the step size of each update; after deviation correction, Makes the model update smoother; Adjust the update amplitude of the model to prevent the step size from being too large or too small;

[0110] Step S116: Repeat steps S112 to S115 until the loss function of the model converges or reaches a predetermined number of training rounds.

[0111] The beneficial effects of the present invention compared with the prior art are as follows: the present invention proposes a method for predicting honeycomb lung disease based on multimodal time series data, which mainly predicts the development direction of lung disease based on CT images and text descriptions. The method integrates wavelet transform technology and Transformer model for time series analysis to achieve classification and progression prediction of honeycomb lung lesions; in order to enhance the frequency domain feature extraction capability of the system, the present invention adopts wavelet transform method to process data, and can perform multi-resolution analysis of CT images in the frequency domain to capture the global structure and local detail changes of the image, and by combining wavelet transform with Transformer model, the entire prediction model can not only realize the classification of honeycomb lung lesions, but also more accurately predict the progression of the disease, further improving the diagnostic accuracy and the efficiency of clinical decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] The present invention will be further described below in conjunction with the accompanying drawings:

[0113] Figure 1 A flow chart of the steps of predicting lung disease condition according to the present invention;

[0114] Figure 2 This is a flow chart of the steps of processing data using the time series model and wavelet transform method in the present invention. DETAILED DESCRIPTION

[0115] In order to reduce the burden on doctors and improve the accuracy of diagnosis, the present invention uses computer-aided technology to classify honeycomb features in CT images, combines time series analysis and wavelet transform technology, and further predicts the progression of lung diseases with high accuracy, which can provide comprehensive decision support for clinicians. The prediction method mainly includes:

[0116] First, the CT image is decomposed into multiple scales using wavelet transform. Wavelet transform can decompose the image into components of different scales, capture the local details and overall structural information in the image, and obtain rich and accurate image features. These features are of great significance for the identification and progression prediction of the disease.

[0117] The extracted wavelet features are then fused with the patient’s clinical data to form a time series dataset containing multimodal information, which includes not only the multi-scale features of the image but also the patient’s historical condition data, treatment plan, and other clinical indicators;

[0118] Next, we build a prediction model based on Transformer. Transformer network can handle long-term dependencies and is particularly suitable for modeling time series data.

[0119] Finally, by designing a reasonable network structure and optimizing hyperparameters, Transformer can effectively learn patterns and trends in time series data and predict future disease progression.

[0120] Furthermore, the honeycomb lung disease condition prediction method based on multimodal time series data proposed in the present invention specifically includes the following condition prediction steps:

[0121] Step S1: Collect the CT description text and CT image data of each patient at different time points through the hospital's Electronic Medical Record System (EMR) and radiology department reports, and then clean and annotate the collected data to ensure data quality and consistency.

[0122] First, the image data is cleaned, and the 2D slices that can reflect the lesions in the collected data are selected, and the image size is unified to 256*256 pixels. After the image data is cleaned, the existing 2D slice data is reconstructed into 3D NRRD data through reconstruction technology. For text data, it is necessary to remove noise (such as patient personal information, examination time, etc.), standardize medical terms and units. Then, the CT image data is manually annotated, the lesion area is extracted and a binary mask is generated. The text data is annotated to extract disease-related information (such as lesion location, lesion nature, and lesion degree).

[0123] Step S2: Constructing a time series data set. The method proposed in the present invention constructs a comprehensive time series data set for training and testing the model by combining CT image data and text data at different time points. The data set includes four parts: patient number, time point, CT image data and text data. The data finally input into the model includes the digital representation of CT image data and text data, and the label data is used to predict the future state of the disease (get better, remain the same or continue to worsen).

[0124] Step S21: Construct a time series data set, including four columns of data:

[0125] The first column stores the patient number, which uniquely identifies each patient; the second column stores the time point, which records the specific time point of data collection; the third column stores the CT image data, which stores the digital representation of the CT image at each time point; the fourth column stores the text data, which stores the digital representation of the CT description data corresponding to each time point. Each piece of data in the constructed data set is composed of the above content, providing input for the designed model.

[0126] Step S22: Dataset preparation:

[0127] Data input to the model: including digital representations of CT image data and text data. CT image data is feature extracted through a convolutional neural network (CNN) to generate feature vectors; text data is converted into word vectors through an embedding layer and further generated into feature representations through a Transformer network. These feature vectors are combined into a comprehensive input data.

[0128] Label data: Each time series data has a corresponding label data, which indicates the predicted future state of the disease. These labels are divided into three categories: the disease state gets better, stays the same, or continues to get worse. Label data is used to supervise the model training process, helping the model learn how to predict the future state of the disease based on the input CT images and text data.

[0129] The time series dataset constructed through the above steps provides comprehensive data support for subsequent model training and testing, ensuring that the model can accurately predict the progression of lung diseases based on time series CT images and text data.

[0130] Step S3: Build a time series model. It includes an input layer that receives four columns of time series data sets; a wavelet transform layer that performs wavelet transform on the input data to extract multi-scale frequency domain features; a Transformer layer that processes time series data to capture long-term dependencies; a fusion layer that fuses the multi-scale frequency domain features extracted by the wavelet transform layer with the long-term dependencies; and an output layer that outputs prediction results to predict the development status of the disease.

[0131] Step S4: The constructed dataset enters the input layer of the model. The input layer accepts four columns of time series data, including patient number, time point, CT image data, and text data; these data are converted into embedded representations and prepared to be input into subsequent layers.

[0132] Step S5: Enter the first branch, the wavelet transform branch. Perform wavelet transform on the input data to extract multi-scale frequency domain features. The generated wavelet features are represented as F wavelet .

[0133] Step S6: Enter another branch, the Transformer branch. Directly process time series data, capture long-term dependencies in the data, and generate Transformer features represented as F trans .

[0134] Step S7: After going through the two feature extraction paths, the two feature vectors enter the fusion layer, where the gating mechanism is used for fusion. First, the wavelet transform feature F wavelet and Transformer feature F trans Gating units are set up separately to control the flow of each feature.

[0135] G wavelet =σ(W G-wavelet F wavelet +b G-wavelet ); (1)

[0136] G trans =σ(W G-trans F trans +b G-trans ); (2)

[0137] Next, the features of the gated units are weighted and summed to generate the gated fusion features.

[0138] F gated-fusion =G wavelet ⊙F wavelet +G trans ⊙F trans ; (3)

[0139] Step S8: After fusing the extracted feature vectors through the gated fusion mechanism, they are input into the fully connected layer to generate the final feature representation. The final prediction result is output to predict the future development status of the disease (get better, stay the same, or continue to worsen).

[0140] Step S9: define the loss function used in the training process as a cross entropy loss function.

[0141] Step S10: Back propagation: Back propagate the loss value back to the model, calculate the gradient and update the model parameters to minimize the loss value.

[0142] Step S11: Define optimizer: Use Adam optimizer to update model parameters.

[0143] Step S12: For the feature classification vector generated in step S8, the loss is calculated through the classification head, and the inverse gradient calculation is performed to adjust the parameters of the model.

[0144] Step S13: The output category label 0 represents that the disease is predicted to get better in the future, 1 represents that the symptoms will not change, and 2 represents that the symptoms will continue to worsen.

[0145] Furthermore, the input layer processing multimodal data process adopted in step S4 of the present invention has the following specific steps and processes:

[0146] Step S41: inputting the sorted multimodal data, including CT image data I, text description data T and time series data S.

[0147] Step S42: First, the CT image data I, the text description data T and the time series data S are processed:

[0148] F I =CNN(I); (4)

[0149] F T =NLP(T); (5)

[0150] F S =CNN(S); (6)

[0151] Then, the processed multimodal data is fused to obtain a comprehensive feature vector representation:

[0152] F fusion =concat(F I ,F T ,F S ); (7)

[0153] Furthermore, the specific step details and process of step S5 are as follows:

[0154] In this layer, the feature vector F fused in the input layer is fusion Perform wavelet transform (Continuous Wavelet Transform, CWT) to obtain sub-band features of different scales and resolutions.

[0155] Step S51: Use a low-pass filter h[n] and a high-pass filter g[n] to filter and downsample the fused vector to obtain four sub-bands of different resolutions: LL (low frequency-low frequency), LH (low frequency-high frequency), HL (high frequency-low frequency) and HH (high frequency-high frequency).

[0156]

[0157] Step S52: After obtaining four sub-bands, the features of each sub-band are further enhanced by processing using a feed-forward neural network (FNN).

[0158] F ffn1 =FFN([LL+LH+HL+HH]); (12)

[0159] Step S53: After passing through the feedforward neural network, the extracted features are converted into the spatial domain by inverse continuous wavelet transform (ICWT), which is represented by F ICWT1 , providing input for subsequent processing.

[0160] Step S54: The feature F converted to the spatial domain in the previous step ICWT1 Wavelet transform is performed again and decomposed into four different sub-bands LL, LH, HL and HH.

[0161] Step S55: First, sum the three sub-bands except the low-frequency-low-frequency sub-band, and then concatenate them with the low-frequency-low-frequency sub-band to obtain the feature representation in the frequency domain. Then enhance the feature through the feedforward neural network:

[0162] F ffn2 =FFN([LL,LH+HL+HH]); (13)

[0163] Step S56: In this way, the frequency domain features of the data are obtained. After that, the extracted features are converted back to the spatial domain by inverse wavelet transform for further convolution operation and feature extraction:

[0164] F wavelet =ICWT(F ffn2 ); (14)

[0165] Furthermore, the specific step details and process of the adopted Transformer branch, i.e. step S6, are as follows:

[0166] In this branch, the fused comprehensive vector F has been obtained through the previous steps fusion, which is a high-dimensional feature vector that combines text features, image features, and time series features. In order to use the Transformer to process this feature vector, it is necessary to convert this feature vector into sequence data. After such processing, the temporal relationship in the input data can be more effectively captured by using the Transformer network.

[0167] Step S61: Data F processed by the input layer fusion is a d×n feature matrix, where d is the feature dimension of a patient's data at a time point, and n is the sequence length. They are represented in a data format suitable for input to Transformer [x 1 , x 2 , …, x n ], where each x i is a eigenvector, denoted by F seq .

[0168] Step S62: Since the Transformer model itself does not have a built-in mechanism for processing time series data information, time series data information is introduced through positional encoding (PE). Positional encoding (PE) can position the features of the input model data in the time series, which can help the Transformer model capture time series data information more accurately. Positional encoding is generated by using sine and cosine functions to ensure that each feature has a different positional encoding and there is a smooth change between these encodings. The formula is as follows:

[0169]

[0170] Among them, pos is the position of the element in the sequence, i is the index of the feature dimension, and d model is the dimension of the model. Sin and cosine functions are used because they have different frequencies and can distinguish features at different positions in the sequence.

[0171] Add the calculated position encoding vector PE to the feature vector F processed by the input layer seq In this way, the data input into the Transformer model contains not only feature information, but also the position information of each feature in the time series:

[0172] X seq =F seq +PE; (17)

[0173] After this step, the Transformer model can better capture time series dependencies and ensure that the model can process time series relationships through location information, thereby improving the accuracy of model predictions.

[0174] Step S63: Multi-Head Self-Attention Layer: Divide the input feature sequence into multiple self-attention heads, and each head performs attention calculation independently to learn different feature representations. In the Multi-Head Self-Attention mechanism, for each self-attention head, the model calculates an attention matrix, which reflects the relationship between each element in the sequence and other elements.

[0175] Specifically, for each attention head, the input feature vector X seq Generate the query matrix Q, key matrix K and value matrix V through linear transformation:

[0176]

[0177] Among them, W Q , W K and W V are all trainable weight matrices. Next, calculate the attention weight using the following formula:

[0178]

[0179] Among them, d k is the dimension of the key matrix. The obtained attention weights can represent the importance of each element in the input sequence features to other elements.

[0180] For each attention head, the model will calculate in parallel, and each head will learn different feature representations independently. Finally, the outputs of all attention heads are concatenated together and linearly transformed to obtain the final feature representation:

[0181] HultiHead(Q,K,V)=concat(head 1 , head 2 , ..., head h )W O ; (20)

[0182] Among them, headi represents the attention output of the i-th head, and WO is the weight matrix.

[0183] Step S64: Feedforward neural network layer: The feature vector at each position is further processed by a feedforward neural network. This network layer consists of two linear layers, which are activated by a RELU function in the middle.

[0184] Step S65: layer normalization and residual connection: After going through the multi-head self-attention mechanism and feedforward neural network, layer normalization and residual connection are used to stabilize the model training and improve the performance of the model.

[0185] By using the above process, the Transformer network can effectively process the temporal information in time series data.

[0186] Finally, the extracted features are fused through the gated fusion mechanism and the features extracted by the wavelet transform branch for subsequent prediction of the future development of the disease.

[0187] Furthermore, the cross entropy loss function used, that is, the specific steps and process of step S9 are as follows:

[0188] Step S91: define a cross entropy loss function. The present invention is a classification task of time series analysis, so the cross entropy loss function commonly used for classification tasks is used, which is defined as follows:

[0189]

[0190] Among them, N represents the number of samples, C represents the number of categories, and y i,j represents the true label of the jth category of the i-th sample, Represents the predicted value of the jth category of the i-th sample.

[0191] Step S92: Forward propagation: Input the processed data into the model for forward propagation to obtain the model's prediction results.

[0192] Step S93: Calculate the loss function: input the model's prediction results and the true labels of the data into the cross entropy loss function to calculate the loss value of the model.

[0193] Step S94: Back propagation: back propagate the loss value of the model back to the model to guide the model to update parameters, thereby reducing the loss value of the model.

[0194] Step S95: Repeat training: Repeat the above process until the model converges or reaches a predetermined number of training rounds.

[0195] Furthermore, the specific steps and process of step S11 are as follows:

[0196] Step S111: Initialize parameters: set the initial parameters of the model to θ; set the initial learning rate α to 0.001; set the decay rate β of the first-order momentum estimate 1 is 0.9 and the decay rate of the second-order momentum estimate is β 2 is 0.999; set the minimum constant ε to 10 to prevent division by zero -8 ; Initialize the first-order momentum estimate m0 = 0 and the second-order momentum estimate v 0 =0.

[0197] Step S112: Calculate the gradient: For each training sample, calculate the gradient of the model under the current parameters. At time step t, calculate the loss function f(θ t ) The gradient of the parameter θ g t , reflecting the current parameter θ t The update direction and update amplitude.

[0198] Step S113: Calculate the first-order momentum estimate: Calculate the current gradient g t The exponentially weighted moving average of is called the first-order momentum estimate m t :

[0199] m t =β 1 ·m t-1 +(1-β 1 )·g t ; (twenty two)

[0200] Among them, β 1 is the momentum decay rate, used to smooth the instantaneous change of the gradient; momentum m t Accumulating past gradient information makes the update direction more stable.

[0201] Step S114: Calculate the second-order momentum estimate: Calculate the current gradient g t The exponentially weighted moving average of the square of is called the second-order momentum estimate v t :

[0202] v t =β 2 ·v t-1 +(1-β 2 )·g t 2 ; (twenty three)

[0203] Among them, β 2 is the decay rate of the second-order momentum, which is used to smooth the instantaneous change of the square of the gradient; the second-order momentum estimate v t The variance information of the gradient can be captured to adjust the learning rate.

[0204] Step S115: Update the learning rate: Use the bias-corrected first-order momentum estimate and second-order momentum estimate to update the learning rate to reduce the bias caused by the momentum estimate during initialization.

[0205] First m t and v t To perform bias correction:

[0206]

[0207] Then m t and v t After bias correction, it is used to update the parameter θ:

[0208]

[0209] Among them, α is the learning rate, which is used to control the step size of each update; after deviation correction, Makes the model update smoother; Adjust the model update amplitude to prevent the step size from being too large or too small.

[0210] Step S116: Repeat training: Repeat the above steps until the loss function of the model converges or reaches a predetermined number of training rounds.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting the condition of honeycomb lung disease based on multimodal time series data, characterized by: The following steps are included in the disease prediction process: Step S1: Collect the patient’s CT description text and CT image data at different time points through the hospital’s electronic medical record system and radiology department reports, and then clean and annotate the collected data; Step S2: By combining CT image data and text data at different time points, a comprehensive time series dataset is constructed for training and testing the model; Step S3: Building a time series model, including: The input layer receives the time series dataset; Wavelet transform layer, used to perform wavelet transform on input data and extract multi-scale frequency domain features; Transformer layer, used to process time series data and capture long-term dependencies; The fusion layer is used to fuse the multi-scale frequency domain features extracted by the wavelet transform layer with the long-term dependencies; The output layer is used to output the prediction results and predict the development status of the disease; Step S4: Input the constructed time series dataset into the input layer, convert the time series data into an embedded representation, and process the multimodal data by the input layer to prepare for input into subsequent layers; Step S5: Input the time series data into the wavelet transform branch, perform wavelet transform on the input data, extract multi-scale frequency domain features, and the generated wavelet features are represented as F wavelet ; Step S6: Input the time series data into the Transformer branch, directly process the time series data, capture the long-term dependencies in the data, and generate the Transformer feature representation as F trans ; Step S7: After going through the two feature extraction paths, the data is input into the fusion layer of the two feature vectors and fused using a gating mechanism; Step S8: Input the fused feature vector into the fully connected layer to generate the final feature classification vector, output the prediction result, and predict the future development status of the disease; Step S9: define the loss function used in the training process as a cross entropy loss function; Step S10: back-propagate the loss value of the loss function back to the model, calculate the gradient and update the parameters of the model to minimize the loss value; Step S11: Update model parameters using Adam optimizer; Step S12: For the feature classification vector generated in step S8, the loss is calculated through the classification head, and the inverse gradient calculation is performed to adjust the parameters of the model; Step S13: Finally define and output the category label: Label 0 means that the disease is predicted to get better in the future; Label 1 means that the symptoms will not change; Label 2 means the condition will continue to worsen.

2. The method for predicting honeycomb lung disease based on multimodal time series data according to claim 1, characterized in that: The specific method for cleaning and labeling the data in step S1 is: Clean the image data, select the 2D slices that can reflect the lesions in the acquired data, unify the image size to 256*256 pixels, and then reconstruct the existing 2D slice data into 3D NRRD data through reconstruction technology after cleaning the image data; Clean the text data to remove text noise and standardize medical terms and units; Manually annotate CT image data, extract lesion areas and generate binary mask images, annotate text data and extract disease-related information.

3. The method for predicting honeycomb lung disease based on multimodal time series data according to claim 2, characterized in that: The specific method for constructing the time series data set in step S2 is: Step S21: Construct a time series data set, including four columns of data: The first column stores the patient number, which uniquely identifies each patient; The second column stores the time point, recording the specific time point of data collection; The third column stores the CT image data, which stores the digital representation of the CT image at each time point; The fourth column stores text data, which stores the digital representation of the CT description data corresponding to each time point; Each piece of data in the constructed dataset is composed of the above content, providing input for the designed model; Step S22: Dataset preparation: The data input to the model includes digital representations of CT image data and text data, wherein the CT image data is subjected to feature extraction through a convolutional neural network to generate a feature vector, and the text data is subjected to word vector conversion through an embedding layer and further subjected to a Transformer network to generate a feature vector, and the above-generated feature vectors are combined into a comprehensive input data; Define label data: Each time series data corresponds to a label data, which indicates the predicted future state of the disease. The label data is divided into three categories: the disease state improves, remains the same, and continues to worsen.

4. The method for predicting the condition of honeycomb lung disease based on multimodal time series data according to claim 3, characterized in that: The specific method of processing multimodal data by the input layer in step S4 is: Step S41: inputting the sorted multimodal data, including CT image data I, text description data T, and time series data S; Step S42: Process the CT image data I, text description data T, and time series data S. The processing expression is: I =CNN(I); F T =NLP(T); F S =CNN(S); Then the processed multimodal data are fused to obtain a comprehensive feature vector representation, which is expressed as: F fusion =concat(F I ,F T ,F S )。 5. The method for predicting the condition of honeycomb lung disease based on multimodal time series data according to claim 4, characterized in that: The specific method of performing wavelet transform on the input data in step S5 is: Step S51: Use a low-pass filter h[n] and a high-pass filter g[n] to filter and downsample the fused vector to obtain four sub-bands of different resolutions: low frequency-low frequency LL, low frequency-high frequency LH, high frequency-low frequency HL, and high frequency-high frequency HH. The sub-band expression is: Step S52: After obtaining four sub-bands, a feed-forward neural network is used for processing to enhance the features of each sub-band. The processing expression is: F ffn1 =FFN([LL+LH+HL+HH]); Step S53: After passing through the feedforward neural network, the extracted features are converted into the spatial domain by inverse wavelet transform, which is represented by F ICWT1 , providing input for subsequent processing; Step S54: The feature F converted to the spatial domain in step S54 ICWT1 Perform wavelet transform again and decompose it into four different sub-bands LL, LH, HL, HH; Step S55: sum the three sub-bands except the low-frequency-low-frequency sub-band, then concatenate them with the low-frequency-low-frequency sub-band to obtain the feature representation in the frequency domain, and then enhance the feature through a feedforward neural network to obtain the frequency domain feature of the data. The calculation formula is: F ffn2 =FFN([LL,LH+HL+HH]); Step S56: Perform inverse wavelet transform on the extracted features and convert them back to the spatial domain for further convolution operation and feature extraction. The calculation formula is: F wavelet =ICWT(F ffn2 )。 6. The method for predicting honeycomb lung disease based on multimodal time series data according to claim 5, characterized in that: The specific method for processing time series data by the Transformer branch in step S6 is: Step S61: Data F represented by the comprehensive feature vector fusion Specifically, it is a d×n feature matrix, where d is the feature dimension of a patient’s data at a time point, and n is the sequence length. It is represented as a data format suitable for inputting Transformer [x1, x2, …, x n ], where each x i is a eigenvector, denoted by F seq ; Step S62: introduce time series data information through position coding, and use sine and cosine functions to generate a position coding vector PE, which is expressed as: In the formula, pos is the position of the element in the sequence, i is the index of the feature dimension, and d model is the dimension of the model; Add the calculated position encoding vector PE to the feature vector F processed by the input layer seq In this way, the data input into the Transformer model contains both feature information and the position information of each feature in the time series. The expression is: X seq =F seq +PE; Step S63: Process data in the multi-head self-attention layer: The input feature sequence is divided into multiple self-attention heads, and attention calculation is performed independently on each attention head. seq The query matrix Q, key matrix K and value matrix V are generated by linear transformation, and the expression is: Among them, W Q , W K and W V are all trainable weight matrices; Then calculate the attention weight, the calculation formula is: Among them, d k is the dimension of the bond matrix; Finally, the outputs of all attention heads are concatenated and linearly transformed to obtain the final feature representation, which is expressed as: HultiHead(Q,K,V)=concat(head1,head2,...,head h )W O ; Among them, headi represents the attention output of the i-th head, and WO is the weight matrix; Step S64: Processing data in a feed-forward neural network layer: The feature vector at each position is further processed by a feed-forward neural network, which consists of two linear layers activated by a RELU function in the middle; Step S65: The output data is processed using layer normalization and residual connection, and the extracted features are fused through a gated fusion mechanism and the features extracted by the wavelet transform branch to predict the future development of the disease.

7. The method for predicting honeycomb lung disease based on multimodal time series data according to claim 6, characterized in that: The specific method of using the gating mechanism to perform fusion in step S7 is: is the wavelet transform feature F wavelet and Transformer feature F trans Set the gating unit separately to control the flow of each feature. The expression is: G wavelet =σ(W G-wavelet F wavelet +b G-wavelet ); G trans =σ(W G-trans F trans +b G-trans ); Then, the features of the gated units are weighted and summed to generate the gated fusion features. The calculation formula is: F gated-fusion =G wavelet ⊙F wavelet +G trans ⊙F trans 。 8. The method for predicting honeycomb lung disease based on multimodal time series data according to claim 7, characterized in that: The specific method of using the cross entropy loss function for training in step S9 is: Step S91: define the cross entropy loss function, the expression is: Among them, N represents the number of samples, C represents the number of categories, and y i,j represents the true label of the jth category of the i-th sample, Represents the predicted value of the jth category of the i-th sample; Step S92: input the processed data into the model for forward propagation to obtain the prediction result of the model; Step S93: input the prediction result of the model and the real label of the data into the cross entropy loss function to calculate the loss value of the model; Step S94: back-propagating the loss value of the model back to the model to guide the model to update parameters, thereby reducing the loss value of the model; Step S95: Repeat steps S92 to S94 until the model converges or reaches a predetermined number of training rounds.

9. The method for predicting honeycomb lung disease based on multimodal time series data according to claim 8, characterized in that: The specific method of using the Adam optimizer to update the model parameters in step S11 is: Step S111: setting the initial parameters of the model to θ; setting the initial learning rate α to 0.001; setting the decay rate β1 of the first-order momentum estimation to 0.9 and the decay rate β2 of the second-order momentum estimation to 0.999; Set the minimum constant ε to 10 to prevent division by zero. -8 ; Initialize the first-order momentum estimate m0=0 and the second-order momentum estimate v0=0; Step S112: For each training sample, calculate the gradient of the model under the current parameters: At time step t, calculate the gradient of the loss function f(θt) with respect to the parameter θ g t , reflecting the current parameter θ t The update direction and update amplitude; Step S113: define the first-order momentum estimate m t is the current gradient g t The exponentially weighted moving average is calculated as: m t =β1·m t-1 +(1-β1)·g t ; Among them, β1 is the momentum decay rate, which is used to smooth the instantaneous changes of the gradient; Step S114: Define the second-order momentum estimate v t is the current gradient g t The exponentially weighted moving average of the square of is calculated as: v t =β2·v t-1 +(1-β2)·g t 2 ; Where β2 is the decay rate of the second-order momentum, which is used to smooth the instantaneous change of the squared gradient; Step S115: Using the bias-corrected first-order momentum estimate and second-order momentum estimate, update the learning rate to reduce the bias caused by the initialization momentum estimate: For m t and v t To perform bias correction, the calculation formula is: For m t and v t After the deviation correction, it is used to update the parameter θ, and the calculation formula is: Among them, α is the learning rate, which is used to control the step size of each update; after deviation correction, Makes the model update smoother; Adjust the update amplitude of the model to prevent the step size from being too large or too small; Step S116: Repeat steps S112 to S115 until the loss function of the model converges or reaches a predetermined number of training rounds.

Images