A multi-modal asynchronous time series prediction method and device based on an attention mechanism
By constructing a multimodal asynchronous time series prediction method based on an attention mechanism, the modeling challenge of asynchronous time series data in clinical scenarios is solved, enabling more accurate disease prediction and early diagnosis, simplifying the model structure and improving the utilization efficiency of sparse data.
Patent Information
- Application Number
- CN202310400988.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-14
- Publication Date
- 2026-08-25
- Estimated Expiration
- 2043-04-14
AI Technical Summary
Existing technologies struggle to effectively utilize multimodal asynchronous time-series data in clinical settings for accurate prediction, resulting in poor early disease diagnosis and prevention outcomes.
A multimodal asynchronous temporal prediction method based on attention mechanism is adopted. By constructing a nonlinear temporal attention matrix and a simple attention mechanism, combined with static features and temporal features, the prediction results are generated using max pooling operation, and the model is iteratively trained using cross-entropy loss function.
It improves the accuracy of asynchronous time series prediction, enables early screening and prevention of diseases, simplifies the model structure, and enhances the efficiency of utilizing sparse data.
Smart Images
Figure CN116487030B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence-assisted diagnosis and treatment technology, specifically to a multimodal asynchronous temporal prediction method and apparatus based on an attention mechanism. Background Technology
[0002] In recent years, deep learning has made significant progress in processing time-series data, such as in natural language processing and speech recognition. However, the processing of time-series data in clinical settings remains in the exploratory stage. In clinical practice, the need for multiple and irregular examinations to monitor disease progression generates a continuous stream of multimodal, heterogeneous, and temporally misaligned time-series data. This data is often closely related to the patient's condition; beyond the clinical test results themselves, the timing and time intervals of the tests can reflect the patient's condition to some extent. Therefore, modeling and deep mining of multimodal asynchronous time-series data in clinical settings can effectively achieve early clinical diagnosis and prevention, and has significant practical implications.
[0003] In clinical practice, modeling multimodal asynchronous time-series data is a formidable challenge, which is determined by the characteristics of real-world clinical time-series data: First, it is always time-series, and due to inconsistent measurement time intervals for the same indicators and inconsistent measurement frequencies for different indicators, it is always asynchronous in time series; in addition, it is always multimodal, including but not limited to imaging data, clinical laboratory indicators, laboratory values, etc.
[0004] With the rapid development of deep learning, there are increasingly more methods for modeling asynchronous time-series data analysis. One of the most classic methods is the recurrent neural network (RNN). Traditional neural networks only focus on the processing at the current time step, without considering what information from the previous time step can be used in the next. However, RNNs have a self-pointing loop to pass information from the current time step to the next, achieving a memory function. Long Short-Term Memory (LSTM) networks have also solved the gradient vanishing and gradient exploding problems that may occur in traditional RNNs, achieving "long-term" memory. However, RNNs are essentially discrete approximations of a process and assume that the time intervals of the time-series data are synchronous. When the data is sampled irregularly, this discretization method usually fails.
[0005] Neural differential equations (NDEs) are a common choice for continuous modeling of processes, describing the continuous changes of a process. Specifically, neural ordinary differential equations (NDEs) describe the changes of processes determined by a single variable over time. If the differential equation is given in an appropriate form, it can be solved analytically. NDEs bring time series data modeling from the discrete world to the continuous world. However, they have a fundamental problem: the solution of an NDE is determined by its initial conditions, lacking a mechanism to adjust the trajectory based on subsequent observations. In reality, clinical data contains various modalities, such as imaging data, clinical laboratory indicators, and laboratory values, and is almost always sampled irregularly. Data from different time points has varying degrees of influence on the prediction results. Therefore, the characteristics of NDEs limit their clinical predictive performance. Research has proposed the concept of neural controlled differential equations, which solves the problem that the solution of NDEs is determined by initial values, allowing for further utilization of subsequently generated time series data while ensuring the continuous change of the hidden state. However, it does not take into account the weighting of different time points in the prediction. In reality, clinical data that is further away from the prediction time usually has a smaller impact on the clinical prediction results.
[0006] Therefore, there is a need for a multimodal asynchronous time series prediction method that can effectively balance clinical information at different times before the predicted time point. This method can unify clinical data from different modalities, make better use of asynchronous clinical time series data, and combine it with static demographic information of patients to obtain more accurate clinical prediction results, thereby achieving early diagnosis and prevention and reducing the incidence rate. Summary of the Invention
[0007] To address the shortcomings of existing technologies, the present invention aims to provide a multimodal asynchronous time series prediction method and apparatus based on an attention mechanism. This method is designed to solve time series prediction tasks in clinical scenarios, filling the gap in asynchronous time series modeling in clinical settings. It helps to achieve early screening and prevention of diseases. The invention proposes a simple nonlinear attention mechanism that can effectively utilize the time series information and the information of the data itself in asynchronous time series data to generate time series attention and self-attention simply and efficiently, thereby improving the accuracy of asynchronous time series prediction while ensuring the simplicity and lightweight nature of the model.
[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0009] Acquire multimodal data from medical practice and preprocess the acquired multimodal data to achieve format uniformity and temporal alignment;
[0010] Construct a nonlinear temporal attention matrix and an asynchronous temporal prediction module based on a simple attention mechanism;
[0011] Based on the asynchronous temporal prediction module, the temporal attention matrix and the static and temporal features of multimodal data are input to obtain the output static and temporal features. The output static and temporal features are then concatenated, and the prediction result is obtained based on the max pooling operation.
[0012] Based on the cross-entropy loss between the obtained prediction results and the multimodal data labels, the temporal attention matrix and the asynchronous temporal data prediction module are iteratively trained.
[0013] Multimodal asynchronous time series prediction is achieved based on the trained asynchronous time series data prediction module.
[0014] Based on the above technical solutions, the multimodal data includes radiological, laboratory, and clinical test data.
[0015] Based on the above technical solution, the specific steps for constructing the nonlinear temporal attention matrix include:
[0016] A nonlinear temporal attention matrix generation module is constructed, and the time point sequence is transposed and subtracted to obtain an initialized linear temporal attention matrix;
[0017] The initialized linear temporal attention matrix is input into the nonlinear temporal attention generation module to obtain the nonlinear temporal attention matrix;
[0018] The specific implementation process for obtaining the nonlinear temporal attention matrix is as follows:
[0019] t = [t1, t2, ..., t T ]
[0020] temporal_attn=MLP(repeat_t-t′)
[0021] Where t is a 1*T time interval sequence, t T Let T be the T-th time interval, repeat_t be the result of repeating t T times, t′ be the transpose of t, temporal_attn be the temporal attention matrix, and MLP be the multilayer perceptron.
[0022] Based on the above technical solution, in addition to obtaining the nonlinear temporal attention matrix, it also includes:
[0023] The nonlinear temporal attention generation module is trained based on the backpropagation of the loss function.
[0024] Based on the above technical solution, after constructing the nonlinear temporal attention matrix, it also includes: adding observation intensity to the sparse data in the aligned multimodal data;
[0025] The method for adding observation intensity to sparse data in aligned multimodal data is as follows:
[0026] Based on the actual records at each position in the sparse time series data, a mask for the time series data is obtained, and an accumulation operation is performed in the time dimension to obtain the number of records of each feature before the current time series data, which is used as the observation intensity.
[0027] Based on the above technical solution, the asynchronous time series data prediction module includes multiple nonlinear simple asynchronous time series attention layers, a fully connected layer, and a pooling layer;
[0028] The nonlinear simple asynchronous timing attention layer is specifically as follows:
[0029] SimTA_plus(X,t)=softmax(A)σ(f(X))
[0030] Where SimTA_plus is a nonlinear simple asynchronous temporal attention layer, X is the input variable, t is the time interval sequence, softmax is the activation function, σ is the activation function, f is the fully connected layer, and A is the attention matrix.
[0031] Based on the above technical solution, the asynchronous temporal prediction module takes the temporal attention matrix and the static and temporal features of the multimodal data as input to obtain the output static and temporal features. Specific steps include:
[0032] The static and temporal features of the multimodal data are used as input to the current nonlinear simple asynchronous temporal attention layer;
[0033] The input static features are passed through a fully connected layer to obtain the output static features of the current nonlinear simple asynchronous temporal attention layer.
[0034] The static features and temporal features of the input are concatenated to generate a self-attention matrix. The generated self-attention matrix is then added to the temporal attention matrix to obtain the total attention matrix.
[0035] The input temporal features are passed through a fully connected layer to obtain processed temporal features. The processed temporal features are then multiplied by the total attention matrix to obtain the output temporal features of the current nonlinear simple asynchronous temporal attention layer.
[0036] The static and temporal features of the current nonlinear simple asynchronous temporal attention layer are used as inputs to the next nonlinear simple asynchronous temporal attention layer, and this process is repeated until the static and temporal features of the last nonlinear simple asynchronous temporal attention layer are output.
[0037] Based on the above technical solution, the step of concatenating the output static features and temporal features, and obtaining the prediction result based on max pooling, includes the following specific steps:
[0038] The static features and temporal features output from the last nonlinear simple asynchronous temporal attention layer are concatenated to obtain the concatenated result;
[0039] The concatenated result is input into the classifier to obtain the predicted probability at different time points, and max pooling is performed on the concatenated result in the time dimension to obtain the prediction result.
[0040] Based on the above technical solution, the iterative training of the temporal attention matrix and the asynchronous temporal data prediction module according to the cross-entropy loss between the obtained prediction results and the multimodal data labels includes the following steps:
[0041] The cross-entropy loss between the prediction results and the multimodal data labels is calculated based on the loss function, and the temporal attention matrix and the asynchronous temporal data prediction module are iteratively trained.
[0042] The loss function is:
[0043]
[0044] Where L represents the degree of loss, N represents the amount of multimodal data, and y i Let p be the label of the i-th multimodal data. i The probability of predicting the i-th multimodal data as positive.
[0045] The present invention also provides a multimodal asynchronous timing prediction device based on an attention mechanism, comprising:
[0046] The acquisition module is used to acquire multimodal data in medical practice and preprocess the acquired multimodal data to achieve format uniformity and time alignment.
[0047] The building blocks are used to construct nonlinear temporal attention matrices and asynchronous temporal prediction modules based on simple attention mechanisms.
[0048] The execution module is used to construct an asynchronous temporal prediction module based on the building module. It takes the temporal attention matrix and the static and temporal features of the multimodal data as inputs, obtains the output static and temporal features, concatenates the output static and temporal features, obtains the prediction result based on the max pooling operation, iteratively trains the temporal attention matrix and the asynchronous temporal data prediction module based on the cross-entropy loss between the obtained prediction result and the label, and realizes multimodal asynchronous temporal prediction based on the trained asynchronous temporal data prediction module.
[0049] Compared with the prior art, the advantages of the present invention are as follows:
[0050] (1) This invention proposes a multimodal asynchronous time series data prediction method and device based on attention mechanism to solve the time series prediction task in clinical scenarios, fill the gap in asynchronous time series modeling in clinical scenarios, and help to achieve early screening and prevention of diseases.
[0051] (2) This invention proposes a simple nonlinear attention mechanism that can effectively utilize the temporal information and data information in asynchronous time series data to generate temporal attention and self-attention simply and efficiently, thereby improving the accuracy of asynchronous time series prediction and ensuring the simplicity and lightweight of the model.
[0052] (3) This invention adds an observation intensity index to sparse record datasets, fully explores the information carried by the data itself, effectively utilizes the occurrence frequency information of different time series features, and improves the accuracy of asynchronous time series prediction. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1 This is a flowchart illustrating a multimodal asynchronous temporal prediction method based on an attention mechanism in an embodiment of the present invention.
[0055] Figure 2 This is a visualization diagram of the temporal attention matrix during the model training phase in an embodiment of the present invention;
[0056] Figure 3 This is a schematic diagram illustrating the visualization effect of the patient's temporal attention matrix in an embodiment of the present invention. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.
[0058] See Figure 1 As shown, this embodiment of the invention provides a multimodal asynchronous temporal prediction method based on an attention mechanism, including the following steps:
[0059] S1: Acquire multimodal data from medical practice and preprocess the acquired multimodal data to achieve format uniformity and temporal alignment;
[0060] S2: Construct a nonlinear temporal attention matrix and an asynchronous temporal prediction module based on a simple attention mechanism;
[0061] S3: Based on the asynchronous temporal prediction module, the temporal attention matrix and the static and temporal features of the multimodal data are input to obtain the output static and temporal features. The output static and temporal features are then concatenated, and the prediction result is obtained based on the max pooling operation.
[0062] S4: Based on the cross-entropy loss between the obtained prediction results and the multimodal data labels, iteratively train the temporal attention matrix and the asynchronous temporal data prediction module;
[0063] S5: Implement multimodal asynchronous time series prediction based on the asynchronous time series data prediction module after training.
[0064] This involves acquiring multimodal data from medical practice and preprocessing it. The first step in preprocessing is to standardize the data format. In this example, the multimodal data includes CT images, laboratory values, and clinical indicators. Laboratory values and clinical indicators can be easily converted into feature name and feature value formats. For CT images, they are first resampled to a 1*1*1 volume, and then the corresponding affine omics features are used to replace the CT images themselves, thus standardizing the data format. The extraction of CT image omics features is a well-known technique in this field and will not be elaborated upon here. Next, different time-series features are aligned on a time series. This is organized into a CSV table, where the first row is the feature name, the first column is the time series, and each position in the table represents the value of the feature at the corresponding time. If there is no record, it is filled with NAN. Additionally, static information such as gender, age, and height is collected for each patient. Finally, the above data is normalized. Normalization is a well-known technique in this field and will not be elaborated upon here.
[0065] Then, a nonlinear temporal attention matrix is constructed. Generally, the closer the detection result is to the time point to be predicted, the greater its influence on the prediction result. The detection results are those obtained from the multimodal data. First, the corresponding time point sequences in the detection results are transposed and subtracted to obtain an initial temporal attention matrix. Then, this matrix is passed through a two-layer multilayer perceptron (MLP) to obtain a nonlinear temporal attention matrix. Finally, an asynchronous temporal prediction module is constructed based on a simple attention mechanism.
[0066] The asynchronous temporal prediction module is constructed by inputting the temporal attention matrix, static features and temporal features of multimodal data into the module. The output static features and temporal features are then concatenated and the final prediction result is obtained based on the max pooling operation.
[0067] Based on the cross-entropy loss between the obtained prediction results and the sample labels, the temporal attention matrix and the asynchronous temporal data prediction module are iteratively trained. The samples are obtained from the multimodal data. Then, multimodal asynchronous temporal prediction is realized based on the trained asynchronous temporal data prediction module.
[0068] In this invention, the multimodal data includes radiological, laboratory, and clinical test data.
[0069] The multimodal data acquired in medical practice includes radiological, laboratory, and clinical testing data.
[0070] In this invention, the specific steps for constructing the nonlinear temporal attention matrix include:
[0071] A nonlinear temporal attention matrix generation module is constructed, and the time point sequence is transposed and subtracted to obtain an initialized linear temporal attention matrix;
[0072] The initialized linear temporal attention matrix is input into the nonlinear temporal attention generation module to obtain the nonlinear temporal attention matrix;
[0073] The specific implementation process for obtaining the nonlinear temporal attention matrix is as follows:
[0074] t = [t1, t2, ..., t T ]
[0075] temporal_attn=MLP(repeat_t-t′)
[0076] Where t is a 1*T time interval sequence, t T Let T be the T-th time interval, repeat_t be the result of repeating t T times, t′ be the transpose of t, temporal_attn be the temporal attention matrix, and MLP be the multilayer perceptron.
[0077] First, a nonlinear temporal attention matrix generation module is constructed. For an example with a time series of 1*T, it is repeated T times to obtain repeat_t. Then, the time series is transposed and the difference is taken to obtain an initialized linear temporal attention matrix. The initialized linear temporal attention matrix is then input into the constructed nonlinear temporal attention matrix generation module to obtain a nonlinear temporal attention matrix. The specific implementation process of obtaining the nonlinear temporal attention matrix is as follows:
[0078] t = [t1, t2, ..., t T ]
[0079] temporal_attn=MLP(repeat_t-t′)
[0080] In this invention, while obtaining the nonlinear temporal attention matrix, it also includes:
[0081] The nonlinear temporal attention generation module is trained based on the backpropagation of the loss function.
[0082] That is, while obtaining the nonlinear temporal attention matrix, the nonlinear temporal attention generation module is also trained based on the backpropagation of the loss function.
[0083] In this invention, after constructing the nonlinear temporal attention matrix, the method further includes: adding observation intensity to the sparse data in the aligned multimodal data;
[0084] The method for adding observation intensity to sparse data in aligned multimodal data is as follows:
[0085] Based on the actual records at each position in the sparse time series data, a mask for the time series data is obtained, and an accumulation operation is performed in the time dimension to obtain the number of records of each feature before the current time series data, which is used as the observation intensity.
[0086] For the example with a time-series data size of T*C, firstly, a mask for the time-series data is obtained based on whether there is an actual record at each position. Positions with actual records in the time-series data are marked as 1, and positions without actual records are marked as 0. Here, the mask is a masking function. Then, an accumulation operation is performed along the time dimension to obtain the number of records for each feature before the corresponding time point in the time-series data, which is used as the observation intensity. During training, the observation intensity is input as a new channel to the asynchronous time-series data prediction module.
[0087] In this invention, the asynchronous time-series data prediction module includes multiple nonlinear simple asynchronous time-series attention layers, a fully connected layer, and a pooling layer;
[0088] The nonlinear simple asynchronous timing attention layer is specifically as follows:
[0089] SimTA_plus(X,t)=softmax(A)σ(f(X))
[0090] Where SimTA_plus is a nonlinear simple asynchronous temporal attention layer, X is the input variable, t is the time interval sequence, softmax is the activation function, σ is the activation function, f is the fully connected layer, and A is the attention matrix.
[0091] That is, the asynchronous time-series data prediction module in this embodiment of the invention includes multiple nonlinear simple asynchronous time-series attention layers, a fully connected layer and a max-pooling layer.
[0092] The nonlinear simple asynchronous timing attention layer is specifically as follows:
[0093] SimTA_plus(X,t)=softmax(A)σ(f(X))
[0094] Where A is the attention matrix, and the attention here includes two parts: one is the temporal attention generated by the temporal attention matrix obtained in the above steps, and the other is the self-attention of the data itself; the temporal attention matrix is shared by all nonlinear simple asynchronous temporal attention layers.
[0095] In this invention, the asynchronous temporal prediction module takes the temporal attention matrix and the static and temporal features of the multimodal data as input to obtain the output static and temporal features. The specific steps include:
[0096] The static and temporal features of the multimodal data are used as input to the current nonlinear simple asynchronous temporal attention layer;
[0097] The input static features are passed through a fully connected layer to obtain the output static features of the current nonlinear simple asynchronous temporal attention layer.
[0098] The static features and temporal features of the input are concatenated to generate a self-attention matrix. The generated self-attention matrix is then added to the temporal attention matrix to obtain the total attention matrix.
[0099] The input temporal features are passed through a fully connected layer to obtain processed temporal features. The processed temporal features are then multiplied by the total attention matrix to obtain the output temporal features of the current nonlinear simple asynchronous temporal attention layer.
[0100] The static and temporal features of the current nonlinear simple asynchronous temporal attention layer are used as inputs to the next nonlinear simple asynchronous temporal attention layer, and this process is repeated until the static and temporal features of the last nonlinear simple asynchronous temporal attention layer are output.
[0101] See also Figure 2As shown, the static and temporal features of the acquired multimodal data are used as input to the current nonlinear simple asynchronous temporal attention layer. After input, the static features are passed through a fully connected layer to obtain the output static features of the current nonlinear simple asynchronous temporal attention layer. The input static and temporal features are concatenated to generate a self-attention matrix. The generated self-attention matrix is added to the temporal attention matrix obtained in the above steps to obtain the total attention matrix. The input temporal features are passed through another fully connected layer to obtain the processed temporal features. The processed temporal features are multiplied by the total attention matrix to obtain the output temporal features of the current nonlinear simple asynchronous temporal attention layer. The output static and output temporal features of the current nonlinear simple asynchronous temporal attention layer are used as input to the next nonlinear simple asynchronous temporal attention layer, and this process is repeated until the last nonlinear simple asynchronous temporal attention layer outputs static and temporal features, obtaining the output static and temporal features of the last nonlinear simple asynchronous temporal attention layer. Figure 2 In this context, layer n represents the nth layer of a nonlinear simple asynchronous timing attention layer.
[0102] In this invention, the steps of concatenating the output static features and temporal features, and obtaining the prediction result based on max pooling, specifically include:
[0103] The static features and temporal features output from the last nonlinear simple asynchronous temporal attention layer are concatenated to obtain the concatenated result;
[0104] The concatenated result is input into the classifier to obtain the predicted probability at different time points, and max pooling is performed on the concatenated result in the time dimension to obtain the prediction result.
[0105] The static features and temporal features output from the last nonlinear simple asynchronous temporal attention layer are concatenated to obtain the concatenated result. The concatenated result is then input into a classifier, which is a fully connected layer, to obtain the prediction probabilities at different time points. Max pooling is then performed on the time dimension to obtain the final prediction result.
[0106] In this invention, the iterative training of the temporal attention matrix and the asynchronous temporal data prediction module based on the cross-entropy loss between the obtained prediction results and the multimodal data labels includes the following steps:
[0107] The cross-entropy loss between the prediction results and the multimodal data labels is calculated based on the loss function, and the temporal attention matrix and the asynchronous temporal data prediction module are iteratively trained.
[0108] The loss function is:
[0109]
[0110] Where L represents the degree of loss, N represents the amount of multimodal data, and y i Let p be the label of the i-th multimodal data. i The probability of predicting the i-th multimodal data as positive.
[0111] After the asynchronous time series prediction module obtains the prediction result, it needs to iteratively train the time series attention matrix and the asynchronous time series data prediction module based on the cross-entropy loss between the prediction result and the sample labels. The sample is the obtained time series data; wherein, the loss function is:
[0112]
[0113] This invention also provides a multimodal asynchronous time series prediction device based on an attention mechanism, comprising:
[0114] The acquisition module is used to acquire multimodal data in medical practice and preprocess the acquired multimodal data to achieve format uniformity and time alignment.
[0115] The building blocks are used to construct nonlinear temporal attention matrices and asynchronous temporal prediction modules based on simple attention mechanisms.
[0116] The execution module is used to construct an asynchronous temporal prediction module based on the building module. It takes the temporal attention matrix and the static and temporal features of the multimodal data as input, obtains the output static and temporal features, concatenates the output static and temporal features, obtains the prediction result based on the max pooling operation, iteratively trains the temporal attention matrix and the asynchronous temporal data prediction module based on the cross-entropy loss between the obtained prediction result and the label, and realizes multimodal asynchronous temporal prediction based on the asynchronous temporal data prediction module trained by the training module.
[0117] This invention also provides a multimodal asynchronous temporal prediction device based on an attention mechanism, comprising an acquisition module, a construction module, and an execution module. The acquisition module acquires multimodal data from medical practice and preprocesses the acquired multimodal data to achieve format unification and temporal alignment. The multimodal data from medical practice includes radiological, laboratory, and clinical test data. The construction module constructs a nonlinear temporal attention matrix and an asynchronous temporal prediction module based on a simple attention mechanism. The execution module, based on the asynchronous temporal prediction module constructed by the construction module, inputs the static and temporal features of the multimodal data to obtain the static and temporal features output by the asynchronous temporal prediction module. The output static and temporal features are then concatenated, and a prediction result is obtained based on a max-pooling operation. The temporal attention matrix and the asynchronous temporal prediction module are iteratively trained based on the cross-entropy loss between the obtained prediction result and the labels of the multimodal data. Multimodal asynchronous temporal prediction is then implemented based on the trained asynchronous temporal prediction module.
[0118] The following comparative experiments will verify the beneficial effects of the present invention.
[0119] The data used in this trial included follow-up data from 99 patients, totaling 793 CT images, 1335 laboratory values, 99 clinical data points, and 320 response assessments. The model's task was to predict the progression of response to anti-PD-1 immunotherapy in non-small cell lung cancer (NSCLC) using static and all sequence data 90 days prior to response assessment, specifically in a three-category classification: CR / PR (complete / partial response), SD (stable disease), or PD (progressive disease). All data were further divided into dynamic and static data. Comparative methods included LSTM, NeuralCDE, GRU-ODE, and ODE-RNN; the method described in this invention is based on a specific implementation.
[0120] The evaluation metric for the prediction task is AUC, the area under the ROC curve. Its value is between 0 and 1, with the closer to 1 indicating a better model performance.
[0121] Table 1. AUC of the model for predicting immunotherapy response
[0122] LSTM 0.707 0.713 ODE-RNN 0.737 0.732 GRU-ODE 0.749 0.764 NeuralCDE 0.751 0.798 SimTA_plus 0.833 0.831
[0123] As shown in Table 1, our method consistently outperforms all comparable methods. Furthermore, we compared the validation AUC and test AUC. While NeuralCDE also achieved a high test AUC, there is a significant difference between its validation and test AUC. This may be because the neural differential equations are too complex for this small dataset, resulting in poor robustness. Secondly, our method improves prediction performance while maintaining a simple and lightweight model.
[0124] Furthermore, Figure 3 The visualization of the temporal attention matrices of several representative patients during the model training phase of this invention is shown. Each temporal attention matrix has a shape of T×T, where T is the length of the patient's time series, and the value at position (i,j) represents the attention intensity between time points t_i and t_j. Figure 3 In each box, the actual color on the left is blue, and the actual color on the right is red. Red indicates stronger attention, and blue indicates the opposite. A common feature of these attention matrices is that attention gradually strengthens over time, which aligns with our intuitive understanding. Furthermore, the attention matrix intensity of the two patients in the first row changes uniformly over time, but the attention intensity of the two patients in the second row suddenly increases at the last point in time. This matches the original data for the corresponding patients; most of the data for patients p004163 and p007957 is missing, with more complete records available in the last few hours or even the last hour. This also verifies that the proposed method can simultaneously capture both linear and nonlinear temporal attention.
[0125] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
[0126] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
Claims
1. A multimodal asynchronous temporal series prediction method based on an attention mechanism, characterized in that, Includes the following steps: Acquire multimodal data from medical practice and preprocess the acquired multimodal data to achieve format uniformity and temporal alignment; Construct a nonlinear temporal attention matrix and an asynchronous temporal prediction module based on a simple attention mechanism; Based on the asynchronous temporal prediction module, the temporal attention matrix and the static and temporal features of multimodal data are input to obtain the output static and temporal features. The output static and temporal features are then concatenated, and the prediction result is obtained based on the max pooling operation. Based on the cross-entropy loss between the obtained prediction results and the multimodal data labels, the temporal attention matrix and the asynchronous temporal data prediction module are iteratively trained. Multimodal asynchronous time series prediction is achieved based on the trained asynchronous time series data prediction module; The multimodal data includes CT images, laboratory values, clinical data, and response assessments; the multimodal asynchronous temporal prediction includes predicting the progression of response to anti-PD-1 immunotherapy in non-small cell lung cancer. The specific steps for constructing the nonlinear temporal attention matrix include: A nonlinear temporal attention matrix generation module is constructed, and the time point sequence is transposed and subtracted to obtain an initialized linear temporal attention matrix; The initialized linear temporal attention matrix is input into the nonlinear temporal attention generation module to obtain the nonlinear temporal attention matrix; The specific implementation process for obtaining the nonlinear temporal attention matrix is as follows: in, 1* The time interval sequence, For the first A time interval, for repeat The result of this time. for transpose, For the temporal attention matrix, It is a multilayer perceptron.
2. The multimodal asynchronous time series prediction method based on an attention mechanism as described in claim 1, characterized in that, In addition to obtaining the nonlinear temporal attention matrix, it also includes: The nonlinear temporal attention generation module is trained based on the backpropagation of the loss function.
3. The multimodal asynchronous time series prediction method based on attention mechanism as described in claim 1, characterized in that: After constructing the nonlinear temporal attention matrix, the following steps are also included: adding observation intensity for sparse data in the aligned multimodal data; The step of adding observation intensity to the sparse data in the aligned multimodal data is as follows: Based on the actual records at each position in the sparse time series data, a mask for the time series data is obtained, and an accumulation operation is performed in the time dimension to obtain the number of records of each feature before the current time series data, which is used as the observation intensity.
4. The multimodal asynchronous time series prediction method based on attention mechanism as described in claim 1, characterized in that: The asynchronous time series data prediction module includes multiple nonlinear simple asynchronous time series attention layers, a fully connected layer, and a pooling layer. The nonlinear simple asynchronous timing attention layer is specifically as follows: in, It is a nonlinear simple asynchronous timing attention layer. For input variables, It is a time interval sequence. For normalized exponential functions, For activation function, It is a fully connected layer. This is the attention matrix.
5. The multimodal asynchronous temporal prediction method based on an attention mechanism as described in claim 4, characterized in that, The asynchronous temporal prediction module takes the temporal attention matrix and the static and temporal features of the multimodal data as input, and outputs the static and temporal features. Specific steps include: The static and temporal features of the multimodal data are used as input to the current nonlinear simple asynchronous temporal attention layer; The input static features are passed through a fully connected layer to obtain the output static features of the current nonlinear simple asynchronous temporal attention layer. The static features and temporal features of the input are concatenated to generate a self-attention matrix. The generated self-attention matrix is then added to the temporal attention matrix to obtain the total attention matrix. The input temporal features are passed through a fully connected layer to obtain processed temporal features. The processed temporal features are then multiplied by the total attention matrix to obtain the output temporal features of the current nonlinear simple asynchronous temporal attention layer. The static and temporal features of the current nonlinear simple asynchronous temporal attention layer are used as inputs to the next nonlinear simple asynchronous temporal attention layer, and this process is repeated until the static and temporal features of the last nonlinear simple asynchronous temporal attention layer are output.
6. The multimodal asynchronous time series prediction method based on an attention mechanism as described in claim 5, characterized in that, The steps for concatenating the output static features and temporal features, and obtaining the prediction result based on max pooling, include: The static features and temporal features output from the last nonlinear simple asynchronous temporal attention layer are concatenated to obtain the concatenated result; The concatenated result is input into the classifier to obtain the predicted probability at different time points, and the concatenated result is subjected to max pooling operation in the time dimension to obtain the prediction result.
7. The multimodal asynchronous temporal prediction method based on an attention mechanism as described in claim 1, characterized in that, The step of iteratively training the temporal attention matrix and the asynchronous temporal data prediction module based on the cross-entropy loss between the obtained prediction results and the multimodal data labels includes the following steps: The cross-entropy loss between the prediction results and the multimodal data labels is calculated based on the loss function, and the temporal attention matrix and the asynchronous temporal data prediction module are iteratively trained. The loss function is: in, As to the extent of the loss, For the amount of multimodal data, For the first Labels for multimodal data. For the first The probability of predicting a multimodal data point as positive.
8. A multimodal asynchronous time series prediction device based on an attention mechanism, employing the multimodal asynchronous time series prediction method based on an attention mechanism as described in any one of claims 1-7, characterized in that, include: The acquisition module is used to acquire multimodal data in medical practice and preprocess the acquired multimodal data to achieve format uniformity and time alignment. The building blocks are used to construct nonlinear temporal attention matrices and asynchronous temporal prediction modules based on simple attention mechanisms. The execution module is used to construct an asynchronous temporal prediction module based on the building module. It takes the temporal attention matrix and the static and temporal features of the multimodal data as inputs, obtains the output static and temporal features, concatenates the output static and temporal features, obtains the prediction result based on the max pooling operation, iteratively trains the temporal attention matrix and the asynchronous temporal data prediction module based on the cross-entropy loss between the obtained prediction result and the label, and realizes multimodal asynchronous temporal prediction based on the trained asynchronous temporal data prediction module.