Clinical test result prediction model driven by multi-modal data

Abstract

The invention provides a clinical test result prediction model driven by multi-modal data, which relates to the technical field of digital data construction and comprises a data acquisition and preprocessing module, a feature extraction and fusion module, a self-adaptive prediction model, a feedback and updating module and a visualization module. By fusing multi-modal information such as images, genomes, laboratory data and text records, the scheme can more comprehensively obtain multi-dimensional health data of the patient. The feature dynamic weight distribution module dynamically adjusts the weights of different modal data based on global and local attention mechanisms, and ensures the prominent contribution of key features in different prediction scenes, thereby significantly improving the prediction precision and stability of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of digital data construction, and specifically to a clinical trial result prediction model driven by multimodal data. Background Art

[0002] According to a large language model system for clinical trial data analysis and interpretation disclosed in Chinese patent publication number "CN118116536A", the system includes a user interface, a large language model, a natural language processing module, a data storage and management module, a data preprocessing and feature extraction module, a data cleaning and conversion module, a result interpretation and visualization module, a data analysis and modeling module, and a reporting and inference module; the large language model is used to deeply understand and generate natural language text and process complex text data in clinical trials; the data analysis and modeling module is used to analyze and model data using machine learning or statistical methods, extract information, discover patterns, and predict results; the reporting and inference module is used to automatically generate high-quality clinical trial reports and summaries, improve the readability and comprehensibility of research results, and reduce the workload of doctors and researchers.

[0003] According to the multimodal clinical risk prediction method and system based on sampling time interval disclosed in Chinese patent publication number "CN117457209A", the relative offset and absolute offset are extracted from the patient's original electronic health record time series data and initialized and embedded; the patient's original ECG time series data is processed using a neural network method and a self-attention mechanism to generate an ECG data representation vector; the original electronic health record representation vector and the processed ECG data representation vector are feature fused to obtain the patient's fused feature representation vector; based on the offset-guided GRU model, time series features are jointly extracted from the relative offset embedding representation, the absolute offset embedding representation and the fused feature representation vector to obtain the patient's time series feature representation; the patient's time series feature representation vector is input into a classifier to evaluate the patient's relevant clinical risks.

[0004] The above patent documents and prior art have the following technical problems when used:

[0005] Problem 1: Most existing clinical trials do not have a clear prediction system. Even if some clinical technologies have predictions, they usually rely on a single modality or static feature fusion method, fail to fully utilize the complementary advantages of multimodal data, and easily lead to data redundancy or loss of key features;

[0006] Problem 2: Existing models lack dynamic adjustment when data distribution changes over time and lack the ability to provide real-time feedback on new data, which causes model performance to decline as test data changes and lacks adaptability. Summary of the invention

[0007] Technical issues solved

[0008] In view of the shortcomings of the prior art, the present invention provides a multimodal data-driven clinical trial results prediction model to solve the following problems:

[0009] 1. Most existing clinical trials do not have a clear prediction system. Even if some clinical technologies have predictions, they usually rely on a single modality or static feature fusion method, fail to fully utilize the complementary advantages of multimodal data, and easily lead to data redundancy or loss of key features;

[0010] 2. The existing model lacks dynamic adjustment when data distribution changes over time, and lacks the ability to provide real-time feedback on new data, which causes the model performance to decline as the test data changes and lacks adaptability.

[0011] Technical Solution

[0012] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multimodal data-driven clinical trial result prediction model, including a data acquisition and preprocessing module, a feature extraction and fusion module, an adaptive prediction model, a feedback and update module, and a visualization module;

[0013] The data acquisition and preprocessing module is used to collect clinical trial data from multiple different sources, including imaging data, genomic data, laboratory test data, clinical text records and patient demographic information, and preprocess each data type separately, wherein the imaging data is processed by denoising and normalization, the genomic data is processed by feature encoding and dimensionality reduction, and the text records are processed by natural language processing technology to extract key information, and the different modal data are aligned according to timestamps to generate continuous time series data;

[0014] The feature extraction and fusion module is used to extract features from the preprocessed data after being processed by the data acquisition and preprocessing module, extract image features using a convolutional neural network, extract text features based on a natural language processing model, extract key features from genomic data through a graph neural network or a dimensionality reduction algorithm, and dynamically adjust the weights of each modality feature through a multimodal fusion network based on an attention mechanism to generate a high-dimensional feature representation to capture the potential correlation of multimodal data;

[0015] The adaptive prediction model includes an adaptive learning module, a deep prediction network and a loss function based on multi-objective optimization. The adaptive learning module dynamically adjusts the network structure and parameters according to the characteristics of clinical trial data and sample distribution. The deep prediction network adopts a bidirectional long short-term memory network or a Transformer-based time series model to improve the prediction accuracy of data at different time points. The loss function combines prediction accuracy, data noise and outlier factors for multi-objective optimization to ensure the balance of the model in terms of accuracy, robustness and noise resistance.

[0016] The feedback and update module is used to receive feedback data from clinical trials in real time, incorporate new feedback data into model training using incremental learning or online learning mechanisms to improve the prediction effect of the model, and fine-tune model parameters and network levels in different trial scenarios through transfer learning technology to achieve dynamic update and adaptive optimization of the model;

[0017] The visualization module includes an interpretable model and a visualization dashboard. The interpretable model uses the SHAP value / LIME algorithm to generate explanatory information of the prediction results and display the contribution of different modal data to the prediction results. The visualization dashboard is used to display the predicted risk value, effectiveness probability, sample feature change trend and other information in real time to assist clinical researchers and doctors to quickly understand the prediction results.

[0018] The data acquisition and preprocessing module further includes an image denoising submodule and a genomic data denoising submodule. The image denoising submodule uses an adaptive Gaussian filtering algorithm based on a multi-scale convolutional neural network (MSCNN) to denoise the image data to remove image noise while retaining the edge features of key lesion areas; the genomic data denoising submodule uses a clustering algorithm combined with a principal component analysis (PCA) method to separate and compress the noise in the genomic data to maintain the biological relevance of the data and reduce the interference of irrelevant variables.

[0019] Preferably, the image denoising submodule further includes a generative adversarial network (GAN) structure based on content-aware denoising, wherein the generative adversarial network structure includes a generator and a discriminator, wherein the generator is used to enhance key feature areas of the image data, and the discriminator is used to screen images with the best denoising effect, and generate high-resolution and noise-free images through multiple rounds of iterative optimization, thereby improving the contribution of image data to model training.

[0020] Preferably, the feature extraction and fusion module includes a feature dynamic weight allocation module based on a multi-layer attention mechanism, which calculates the global attention and local attention weights of each modal feature through a dual attention layer, wherein the global attention layer allocates modal weights based on data relevance, and the local attention layer performs weighting according to the feature sparsity of the sample data and the importance of rare features, so as to improve the discrimination of key features and dynamically balance the contribution of different modalities to the prediction results.

[0021] Preferably, the feature dynamic weight allocation module further includes a time window-based weight update submodule, which is used to monitor in real time the impact of each modal feature on the prediction accuracy in different time windows, and dynamically adjust the weighted parameters of the modal features based on the reinforcement learning algorithm, so that the adaptability of the model under new data conditions is enhanced, ensuring that the prediction accuracy is continuously optimized as the experiment progresses.

[0022] Preferably, the deep prediction network of the adaptive prediction model adopts a hybrid neural network structure of a bidirectional long short-term memory network (Bi-LSTM) and a Transformer architecture to achieve joint modeling of short-term dependencies and long-term dependencies. The Bi-LSTM is used to capture the short-term correlation characteristics of time series data, and the Transformer structure is used to extract long-term dependencies to achieve multi-dimensional modeling of data at different time scales, thereby improving the responsiveness of the prediction model to data trends and the accuracy of predictions.

[0023] Preferably, the deep prediction network further includes an adaptive feedback control layer, which has a built-in adaptive learning algorithm to monitor the prediction deviation of the model in real time, adjust the hierarchy and parameter configuration of Bi-LSTM and Transformer according to the deviation, so that the prediction model is automatically optimized in a new data environment, avoiding the decline in prediction performance of the model due to changes in data distribution during the progress of clinical trials.

[0024] Preferably, the feedback and update module further includes a real-time data evaluation submodule, which includes an anomaly detection and quality control algorithm. The real-time data evaluation submodule performs multi-dimensional quality evaluation on the received feedback data. The anomaly detection algorithm uses a multivariate analysis technique based on isolation forest to identify and remove outliers in the data, and evaluates data quality based on data integrity, timestamp consistency and data noise level through a quality control algorithm to ensure that only high-quality data is used for model updating, thereby reducing the impact of data noise on prediction results.

[0025] Preferably, the feedback and update module also includes a dynamic model fine-tuning module, which automatically selects historical data similar to the new feedback data for matching, and fine-tunes the weight parameters of the model based on this data, so that the model can adapt to the continuous changes in data in clinical trials, so as to ensure the accuracy and consistency of the prediction model in different trial stages, thereby realizing adaptive dynamic optimization of the model.

[0026] Preferably, the prediction process of the multimodal data-driven clinical trial outcome prediction model is as follows:

[0027] Sp1: Collect clinical trial data from different data sources, including imaging data (such as MRI, CT, etc.), genomic data, laboratory test data, clinical text records and patient demographic information.

[0028] Sp2: Perform preliminary processing on the image data, including format conversion, size adjustment, and input into the image denoising submodule for denoising.

[0029] Sp2.1: The image denoising submodule applies an adaptive Gaussian filtering algorithm based on a multi-scale convolutional neural network (MSCNN) to remove background noise from the image while enhancing edge features.

[0030] Sp2.2: Use the generative adversarial network (GAN) with content-aware denoising to further optimize the denoising effect and ensure the preservation of details in the lesion area.

[0031] Sp3: Processing genomic data, applying principal component analysis (PCA) combined with clustering algorithms to separate noise and maintain biological relevance.

[0032] Sp4: Apply natural language processing (NLP) models to text records to extract key medical terms and information and generate structured data.

[0033] Sp5: Align data of different modalities based on the timestamp information of the data to ensure the consistency and timing of the data in preparation for subsequent analysis.

[0034] Sp6: Input the preprocessed data into the feature extraction module to extract features from image data, text data, and genome data respectively.

[0035] Sp6.1: Apply convolutional neural network (CNN) to extract features from image data and generate multi-dimensional image feature vectors.

[0036] Sp6.2: Embed and encode text data based on the NLP model, extract key words and contextual relationships, and generate text features.

[0037] Sp6.3: Use graph neural networks (GNNs) to analyze genomic data, or extract gene features through dimensionality reduction algorithms to form genomic feature vectors.

[0038] Sp7: Input feature vectors of different modalities into the feature dynamic weight allocation module of the multi-layer attention mechanism for feature fusion.

[0039] Sp7.1: The global attention layer analyzes the modal correlation between data and assigns global weights.

[0040] Sp7.2: The local attention layer performs weighted adjustments based on feature sparsity and feature importance to improve the significance of key features.

[0041] Sp7.3: During the feature fusion process, the contribution of different modal features within a specific time window is monitored in real time through a reinforcement learning algorithm, and the fusion weights are dynamically adjusted to generate a comprehensive feature vector.

[0042] Sp8: The fused comprehensive feature vector is input into the adaptive prediction model, including the adaptive learning module, the deep prediction network and the multi-objective optimization loss function.

[0043] Sp8.1: The adaptive learning module adjusts the network structure and parameters according to the distribution characteristics of clinical trial data to ensure that the model structure adapts to changes in data.

[0044] Sp8.2: The deep prediction network adopts a hybrid architecture based on Bi-LSTM and Transformer. Bi-LSTM captures short-term dependencies and Transformer captures long-term dependencies to generate prediction results.

[0045] Sp8.3: The multi-objective optimization loss function combines prediction accuracy, data noise and outlier effects to optimize the network output and ensure the robustness and noise resistance of the model.

[0046] Sp9: Receives new data fed back during the clinical trial process and inputs it into the real-time data evaluation submodule for quality control.

[0047] Sp9.1: Detect and remove abnormal data through the Isolation Forest algorithm to ensure high data quality.

[0048] Sp9.2: Evaluate data integrity, timestamp consistency, and noise levels to ensure that data that meets data quality standards is used to update the model.

[0049] Sp10: Input high-quality feedback data into the dynamic model fine-tuning module based on transfer learning and incremental learning.

[0050] Sp10.1: The dynamic model fine-tuning module automatically selects historical data with similar features to the new data, matches and fine-tunes the model’s weight parameters to keep the model in the best state.

[0051] Sp10.2: Monitor prediction deviation through adaptive feedback control layer, optimize the hierarchical configuration of Bi-LSTM and Transformer based on data deviation, and improve the prediction stability of the model under different stages of data distribution.

[0052] Sp11: Input the prediction results into the interpretability model and generate explanation information through the SHAP value or LIME algorithm.

[0053] Sp11.1: The interpretability model calculates the contribution of each modality data to the prediction results and forms an explanation report so that clinical staff can understand the impact of key features.

[0054] Sp12: Display the prediction results and explanation information through a visual dashboard.

[0055] Sp12.1: The visual dashboard displays the risk value, effectiveness probability, changing trend of sample characteristics and prediction stability of the prediction results in real time, enabling clinicians to quickly understand the test effects and risks and assist in decision making.

[0056] Beneficial Effects

[0057] The present invention provides a multimodal data-driven clinical trial result prediction model, which has the following beneficial effects:

[0058] 1. By integrating multimodal information such as images, genomes, laboratory data and text records, the present invention can more comprehensively obtain multi-dimensional health data of patients. The feature dynamic weight allocation module is based on the global and local attention mechanism to dynamically adjust the weights of different modal data, ensuring the outstanding contribution of key features in different prediction scenarios, thereby significantly improving the prediction accuracy and stability of the model.

[0059] 2. The present invention adopts an adaptive prediction model that combines Bi-LSTM and a hybrid network architecture based on Transformer, which is suitable for data dependency modeling at different time scales, and controls the influence of noise and outliers through a multi-objective optimization loss function to improve the robustness of the model under complex clinical data. At the same time, the adaptive learning and real-time fine-tuning mechanism ensures the dynamic optimization of the model under new data and feedback data, making it more adaptable in the progress of the trial.

[0060] 3. The present invention uses the isolation forest algorithm to achieve real-time anomaly detection and data quality control. The scheme ensures that the data input into the model is highly reliable and reduces the negative impact of low-quality data on prediction. In addition, the SHAP value and LIME algorithm provide interpretability support for the prediction results, intuitively showing the impact of each modality data on the results, helping clinical staff to quickly understand key features and potential risks, and assisting decision making. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the model structure of the present invention;

[0062] Figure 2 It is the model workflow diagram of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific embodiment one:

[0065] like Figure 1 As shown, the multimodal data-driven clinical trial result prediction model includes a data acquisition and preprocessing module, a feature extraction and fusion module, an adaptive prediction model, a feedback and update module, and a visualization module;

[0066] The data acquisition and preprocessing module is used to collect clinical trial data from multiple different sources, the clinical trial data includes image data, genomic data, laboratory test data, clinical text records and patient demographic information, and preprocess each data type separately, wherein the image data is processed by denoising and normalization, the genomic data is processed by feature encoding and dimensionality reduction, the text records are extracted by natural language processing technology, and different modal data are aligned according to timestamps to generate continuous time series data, the image denoising submodule further includes a generative adversarial network (GAN) structure based on content-aware denoising, the generative adversarial network structure includes a generator and a discriminator, the generator is used to enhance the key feature area of ​​the image data, the discriminator is used to screen the image with the best denoising effect, and high-resolution and noise-free images are generated through multiple rounds of iterative optimization, thereby improving the contribution of image data to model training;

[0067] The image denoising submodule mainly uses two denoising methods: an adaptive Gaussian filtering algorithm based on a multi-scale convolutional neural network (MSCNN) and a generative adversarial network (GAN) structure.

[0068] MSCNN performs convolution operations at different scales to extract features of different resolutions, thereby more effectively preserving key details and removing noise.

[0069] Multi-scale convolution operation, confirming that the input image is I(x,y), the convolution operation at the kth scale is defined as:

[0070]

[0071] f k (x, y): The feature value of the k-th scale convolution output at the pixel position (x, y), w k,i,j : weight parameter of the k-th scale convolution kernel, the weight is optimized during the training process, I(x+i,y+j): the pixel value of the original image at position (x+i,y+j), n k : The radius of the convolution kernel. For convolution kernels of different scales (such as 3x3, 5x5, 7x7), n k They are 1, 2, and 3, respectively, indicating the neighborhood range of the convolution operation;

[0072] Multi-scale feature fusion fuses convolutional features of different scales to more comprehensively preserve the information in the image.

[0073]

[0074] F(x,y): feature representation after multi-scale fusion, K: total number of scales, for example 3 (3x3, 5x5 and 7x7), α k : The weight coefficient of each scale feature. These weights are adaptively learned through training and are used to control the contribution of different scale features to the final feature;

[0075] The GAN structure consists of a generator G and a discriminator D, which achieves image generation with the best denoising effect through adversarial training. The generator G aims to generate images that are as close to real noise-free images as possible, and its loss function is defined as:

[0076]

[0077] Generator loss function, which represents the difference between the generated image and the real image, z: the noise vector input to the generator, from the distribution p z (z), G(z): noise-free image generated by the generator, D(G(z)): the discriminator’s score for the “realism” of the generated image.

[0078] The loss function of the discriminator, the goal of the discriminator D is to distinguish between real and generated images, and its loss function is defined as:

[0079]

[0080] The loss function of the discriminator includes the judgment error between the real and generated samples, x: real noise-free image, from the data distribution p data (x), D(x): the authenticity score of the discriminator for the real image.

[0081] The data acquisition and preprocessing module further includes an image denoising submodule and a genome data denoising submodule, wherein the image denoising submodule uses an adaptive Gaussian filtering algorithm based on a multi-scale convolutional neural network (MSCNN) to denoise the image data to remove image noise while retaining edge features of key lesion areas;

[0082] The adaptive Gaussian filter is used to further denoise the MSCNN feature map to ensure the preservation of edge details. The filter adjusts the standard deviation σ at each pixel to optimize the denoising effect. Gaussian filter formula:

[0083]

[0084] G(x,y): pixel value of the filtered image at position (x,y), representing the pixel brightness after denoising, I(x+i,y+j): pixel value of the input image at position (x+i,y+j), representing the original pixel brightness in the neighborhood window, H(i,j): value of the Gaussian filter kernel (i.e. weight), used to adjust the influence of neighboring pixels on the central pixel, m: window radius, usually half of the convolution kernel; the window size is generally an odd number (such as 3x3, 5x5), then m=1,2, corresponding to different filter windows.

[0085] The Gaussian kernel is defined as follows:

[0086]

[0087] H(i,j): Gaussian filter kernel (i.e. weight) at position (i,j), depends on the distance of the position relative to the center pixel, σ: standard deviation, adaptively adjusted to control the degree of smoothing.

[0088] Adaptive standard deviation σ(x,y) is used to make the filter sharper at the edge and smoother in the smooth area:

[0089]

[0090] σ(x,y): standard deviation of position (x,y), σ min and σ max : minimum and maximum standard deviation of the Gaussian kernel, Image gradient magnitude, used to represent edge strength, The maximum gradient magnitude of the image, used for normalization.

[0091] The genomic data denoising submodule uses a clustering algorithm combined with a principal component analysis (PCA) method to separate and compress the noise in the genomic data to maintain the biological relevance of the data and reduce the interference of irrelevant variables;

[0092] The genomic data denoising submodule mainly uses clustering algorithms and principal component analysis (PCA) methods to separate and compress the noise in genomic data to maintain biological relevance and reduce the interference of irrelevant variables.

[0093] Clustering algorithms are used to group different samples in genomic data to identify and eliminate noise data. Cluster center calculation: There are N genomic data samples, divided into K clusters, and the center (center of mass) of each cluster is μ k for:

[0094]

[0095] μ k : The center of the kth cluster, C k : The set of data points contained in the kth cluster, x i : Data sample.

[0096] Noise removal condition: Calculate the distance d(x,μ) from sample x to the center of its cluster k ), if the distance exceeds the preset threshold, the sample is considered to be noise:

[0097] d(x,μ k )=||x-μ k ||

[0098] d(x,μ k ): Sample x to cluster center μ k Distance, ||·||: distance metric, usually Euclidean distance is used.

[0099] Principal component analysis PCA compresses information and filters noise by reducing data dimensions while retaining the main features of the data. Covariance matrix: The genomic data matrix is ​​X, then the covariance matrix Σ is defined as:

[0100]

[0101] Σ: data covariance matrix, N: number of data samples, X T : Transpose of the data matrix.

[0102] Perform eigendecomposition on the covariance matrix Σ to obtain eigenvalues ​​and eigenvectors:

[0103] ∑v i =λi v i

[0104] λ i : The i-th eigenvalue, which indicates the variance of the data in the direction of the feature, v i : corresponds to the eigenvalue λ i The eigenvector of is used to define the direction of the principal component. The dimensionality reduction transformation selects the eigenvectors corresponding to the first M largest eigenvalues ​​to form the projection matrix V M , and then project the original data X into a low-dimensional space:

[0105] X′=XV M

[0106] X ′ : The data matrix after dimension reduction. V M : A matrix composed of the first M principal component eigenvectors, used to retain the information of maximum variance.

[0107] The feature extraction and fusion module is used to extract features from the preprocessed data after being processed by the data acquisition and preprocessing module, extract image features using a convolutional neural network, and extract text features based on a natural language processing model.

[0108] Image feature extraction Convolutional neural network (CNN) is used to extract image features and extract high-dimensional spatial features. The image is represented by high-level features after multiple layers of convolution and pooling operations. The convolution operation formula for the convolution output of the input image I at position (x, y) is:

[0109]

[0110] f(x,y): The feature value of the feature map after convolution at position (x,y), w i,j : weight parameters of the convolution kernel, these weights are optimized during the training process to capture different image features, I)x+i,y+j): the pixel value of the input image at position)x+i,y+j), k: the radius of the convolution kernel. Usually the convolution kernel size is 3x3, 5x5, etc.

[0111] Activation function (ReLU):

[0112] a(x,y)=max(0,f(x,y))

[0113] In order to introduce nonlinearity, the output of the convolution operation is processed by the ReLU activation function: a(x, y): output eigenvalue after activation, f(x, y): eigenvalue after convolution.

[0114] Pooling operation (max pooling):

[0115]

[0116] The pooling layer is used to reduce the dimension of the feature map and retain the most important feature information: p(x,y): the value of the feature map at position (x,y) after pooling, f(x+i,y+j): the value of the feature map at position (x+i,y+j) after convolution.

[0117] Text feature extraction natural language processing model (NLP), text data uses embedding and encoding techniques to generate semantic feature representations through natural language processing models (such as word embedding or Transformer).

[0118] Each word in the text is converted into a vector through the embedding matrix E:

[0119] v i =E·x i

[0120] v i : The embedding vector of the i-th word, E: word embedding matrix, generated by pre-training or training process, x i : One-hot encoding of the input word.

[0121] Self-Attention Mechanism in Transformer In order to capture the contextual relationship between words, a self-attention mechanism is used to define the query, key, and value matrices as follows:

[0122] Self-attention formula:

[0123]

[0124] Q: query matrix, generated by weighted transformation of input features, K: key matrix, calculating the similarity between query and key, V: value matrix, containing features matching the query, d k : The dimensions of the query and key to prevent vanishing or exploding gradients.

[0125] The key features of genomic data are extracted through graph neural networks or dimensionality reduction algorithms, and the weights of each modality feature are dynamically adjusted through a multimodal fusion network based on an attention mechanism to generate a high-dimensional feature representation to capture the potential correlation of multimodal data.

[0126] The genomic data is processed into a graph structure, and complex relationship features are extracted through graph neural networks. If the data dimension is high, principal component analysis (PCA) is used to reduce the dimension.

[0127] The graph neural network propagates the information of neighbor nodes to the central node through the graph convolution layer, thereby aggregating the graph structure information. The graph convolution formula is:

[0128]

[0129] The feature representation of node v at layer l+1, The feature representation of node v at layer l, Feature representation of node u at layer l, σ: activation function (ReLU), The neighbor set of node v, c vu : Normalization coefficient, used to balance the relationship between nodes, W (l) : The weight matrix of the current layer, Self-loop weight matrix.

[0130] The feature extraction and fusion module includes a feature dynamic weight allocation module based on a multi-layer attention mechanism, which calculates the global attention and local attention weights of each modal feature through a dual attention layer, wherein the global attention layer allocates modal weights based on data relevance, and the local attention layer performs weighting according to the feature sparsity of the sample data and the importance of rare features, so as to improve the discrimination of key features and dynamically balance the contribution of different modalities to the prediction results.

[0131] Multimodal fusion network based on attention mechanism This network assigns different weights to different modal data through the attention mechanism to capture the correlation between features.

[0132] Attention weight calculation in multimodal fusion network, for each modality feature, calculate its weight distribution α i :Global attention calculation:

[0133]

[0134] α i : The global weight of modal feature i, e i : Feature score, representing the importance of feature i, W: weight matrix, acting on the modal feature vector h i Up, h i : modal eigenvector.

[0135] Local attention calculation: Local attention weights are weighted according to the sparsity of modality features and the importance of rare features:

[0136]

[0137] e ij : The local similarity between features i and j, W q : Query matrix, used for the projection of feature i, W k : key matrix, used for the projection of feature j, d k : Dimension of the feature vector.

[0138] The feature dynamic weight allocation module further includes a weight update submodule based on a time window, which is used to monitor the impact of each modal feature on the prediction accuracy in different time windows in real time, and dynamically adjust the weight parameters of the modal features based on the reinforcement learning algorithm, so that the adaptability of the model under new data conditions is enhanced, ensuring that the prediction accuracy is continuously optimized as the experiment progresses;

[0139] The weight update submodule based on the time window monitors the feature performance within the time window and dynamically updates the feature weights according to the reinforcement learning algorithm, so that the model adapts to the new data.

[0140] The weight adjustment of features within the time window, the weight update formula within the time window: Assume that the performance of feature i monitored within the time window T is P i (T), then the new weight α i (T+1) Updated to:

[0141] α i (T+1)=α i (T)+η·ΔP i (T)

[0142] α i (T+1): Update weight within time window T+1, α i (T): original weights in time window T, η: learning rate, used to control the pace of weight update, ΔP i (T): The change in the performance of feature i within the time window T.

[0143] Weight update strategy based on reinforcement learning Reinforcement learning uses the reward function R to adjust the feature weights so that it is continuously optimized in new data. Assuming the reward function is R(α), the goal is to maximize R: Reward function maximization:

[0144]

[0145] α * : optimal feature weight distribution, R(α): reward function, defining the effect of feature weight adjustment, The expected value of the reward is used to guide the model to choose the optimal weight distribution.

[0146] The adaptive prediction model includes an adaptive learning module, a deep prediction network, and a loss function based on multi-objective optimization. The adaptive learning module dynamically adjusts the network structure and parameters according to the characteristics of clinical trial data and sample distribution. The core is to optimize the model configuration by monitoring the prediction deviation and model performance. The prediction output is The true value is y t, then the prediction deviation is:

[0147]

[0148] E t : Prediction bias, reflecting the prediction error of the current model.

[0149] The dynamic adjustment strategy updates the model parameters and structure by analyzing historical forecast deviations:

[0150]

[0151] Δθ: parameter update amount, η: learning rate, controls the update amplitude, The gradient of the loss function represents the effect of the bias on the model parameters.

[0152] The deep prediction network adopts a bidirectional long short-term memory network or a time series model based on Transformer to improve the prediction accuracy of data at different time points. The loss function combines prediction accuracy, data noise and outlier factors for multi-objective optimization to ensure the balance of the model in terms of accuracy, robustness and noise resistance. The deep prediction network of the adaptive prediction model adopts a bidirectional long short-term memory network (Bi-LSTM) and a hybrid neural network structure based on the Transformer architecture to achieve joint modeling of short-term dependency and long-term dependency. The Bi-LSTM is used to capture the short-term correlation characteristics of time series data, and the Transformer structure is used to extract long-term dependencies to achieve multi-dimensional modeling of data at different time scales, thereby improving the responsiveness of the prediction model to data trends and the accuracy of predictions.

[0153] The deep prediction network combines the bidirectional long short-term memory network (Bi-LSTM) and the Transformer structure to capture short-term and long-term dependency features. Bi-LSTM captures the short-term correlation features of time series data through forward and backward propagation. Its core unit is the LSTM unit, which has a memory unit and input, forget, and output gate mechanisms. LSTM unit formula:

[0154] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0155] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0156] o t =σ(Wo ·[h t-1 ,x t ]+b o )

[0157]

[0158] h t =o t tanh(C t )

[0159] i t : Input gate, which controls the influence of the current input on the unit state, f t : Forget gate, which controls the influence of the previous unit state on the current unit state. Candidate memory cell, representing the candidate value of the current input, c t : The current unit state, which integrates the previous state and the current input information, o t : Output gate, controls the influence of the current unit state on the output, h t : The current hidden state, as the output of the current moment, affects the calculation of the next moment, W i ,W f ,W c ,W o : Weight matrix, used to connect input x t and the hidden state h t-1 , b i ,b f ,b c ,b o : Bias term, adjusts the output of each gate, σ: Sigmoid activation function, the output is between (0,1).

[0160] The Transformer model uses a self-attention mechanism to capture long-term dependency features.

[0161] The input sequence is X, and the calculation steps of the self-attention mechanism are as follows:

[0162]

[0163] Q: query matrix, usually obtained by transforming the input sequence X through the weight matrix, K: key matrix, representing the characteristics of the input sequence, used to calculate the similarity with the query, V: value matrix, representing the information related to the input sequence, d k : The dimension of the key, used to normalize the calculation results to avoid too large values, softmax: normalization function, converting the similarity score into a probability distribution.

[0164] The deep prediction network further includes an adaptive feedback control layer, which has a built-in adaptive learning algorithm to monitor the prediction deviation of the model in real time, and adjust the hierarchical structure and parameter configuration of Bi-LSTM and Transformer according to the deviation, so that the prediction model is automatically optimized in a new data environment, thereby avoiding the decline in prediction performance of the model due to changes in data distribution during the progress of clinical trials;

[0165] The loss function based on multi-objective optimization is used to balance the prediction accuracy, data noise and outlier factors to ensure the robustness and noise resistance of the model. The multi-objective loss function formula is:

[0166]

[0167] Total loss function, λ1,λ2,λ3: weights of each loss term, controlling the importance of each target in the total loss, The loss of prediction accuracy is usually measured using the mean square error (MSE):

[0168]

[0169] Noise loss measures the robustness of the model to noise and may be expressed in the form of regularization terms. Outlier loss, which reduces the impact of outliers on model training, is usually calculated using weighted loss or specialized outlier detection methods;

[0170] The feedback and update module is used to receive feedback data from clinical trials in real time, incorporate new feedback data into model training using incremental learning or online learning mechanisms to improve the prediction effect of the model, and fine-tune model parameters and network levels in different test scenarios through transfer learning technology to achieve dynamic update and adaptive optimization of the model.

[0171] Transfer learning fine-tunes the model in a new scenario so that it performs better under the new data distribution. The goal of transfer learning is to transfer the knowledge of the existing model in the source domain to the target domain. The fine-tuning process of transfer learning usually adjusts the model parameters by minimizing the loss function. In the target scenario, the model parameters are θ source ,The goal of transfer learning is to solve the new parameter θ target :

[0172]

[0173] θ target : Parameters of the fine-tuned target domain model, θ source : Initial model parameters of the source domain, The loss function, such as mean squared error or cross entropy loss, x i: Data samples in the target domain, y i : The true label of the target domain data sample.

[0174] Incremental learning and online learning mechanisms are used to incorporate new feedback data into model training to improve the prediction effect of the model in a changing environment. The core idea is to gradually update the model parameters so that the model can adapt to the continuous changes in data distribution. The update formula of incremental learning is to achieve dynamic updates by minimizing the loss of new data samples to model parameters. Let the model's prediction be f(x; θ), and the new feedback data be The loss function of incremental learning is:

[0175]

[0176] The loss function of incremental learning represents the error of the new feedback data sample to the model parameters, x i : Input features of new feedback data samples, y i : The true label or target value of the new feedback data sample, θ: The parameter vector of the current model, Δθ: The incremental update amount, solved by minimizing the loss function, the parameter update rule optimizes the parameter update amount through the gradient descent method:

[0177]

[0178] η: learning rate, which controls the step size of parameter updates, The gradient of the loss function with respect to the parameters indicates the sensitivity of each parameter to the error.

[0179] The feedback and update module further includes a real-time data evaluation submodule, which includes anomaly detection and quality control algorithms. The real-time data evaluation submodule performs multi-dimensional quality evaluation on the received feedback data. The real-time data evaluation submodule mainly includes anomaly detection and quality control. The anomaly detection algorithm adopts an isolation forest (Isolation Forest). Isolation forest is an unsupervised anomaly detection algorithm that identifies anomalies based on the isolation of data. The isolation forest algorithm constructs a decision tree to segment data and analyzes the difficulty of data points being isolated to determine whether they are abnormal.

[0180] The anomaly score calculation of the isolation forest, the data sample is x, and the anomaly score s(x) of the isolation forest is defined as follows:

[0181]

[0182] s(x): anomaly score of sample x, close to 1 indicates anomaly, close to 0 indicates normality, E(h(x)): average isolation depth of sample x, that is, the average number of steps required to isolate the sample, c(n): regularization constant, which is the expected value of the average isolation depth, usually c(n) = 2ln(n-1) + 0.5772156649, where n is the number of data samples.

[0183] The anomaly detection algorithm uses a multivariate analysis technique based on isolation forest to identify and remove outliers in the data, and uses a quality control algorithm to evaluate data quality based on data integrity, timestamp consistency, and data noise level, ensuring that only high-quality data is used for model updates and reducing the impact of data noise on prediction results.

[0184] The quality control algorithm evaluates data quality based on data integrity, timestamp consistency, and data noise level. Data integrity check: If a dataset D contains m fields, then the integrity score Q integrity Defined as:

[0185]

[0186] Q integrity : Data integrity score, the value range is [0,1], δ j : The integrity indicator of the jth field, which is 1 if the data segment is not empty, otherwise it is 0.

[0187] Timestamp consistency ensures continuity by checking the time series order of the data, which is defined as follows: Time interval: If the data sequence {t1, t2, …, t n} is a timestamp sequence, then the expected interval is Δt expected , the actual interval is Δt i =t i+1 -t i . Consistency score:

[0188]

[0189] Q timestamp : Timestamp consistency score, the value range is [0,1], σ Δt : Standard deviation of the time interval.

[0190] The data noise level is evaluated by analyzing the variance of the data. The larger the variance, the more noise there is. The data sample is The noise level Q noise Defined as:

[0191]

[0192] Q noise: Data noise level. The larger the value, the more noise there is. The mean of the data is used to measure the central tendency of the data.

[0193] The feedback and update module further includes a dynamic model fine-tuning module, which automatically selects historical data similar to the new feedback data for matching, and fine-tunes the weight parameters of the model based on the data, so that the model can adapt to the continuous changes in data in clinical trials, so as to ensure the accuracy and consistency of the prediction model in different trial stages, thereby realizing adaptive dynamic optimization of the model;

[0194] The dynamic model fine-tuning module automatically selects historical data similar to the new feedback data to match the feature distribution and adjust the model's weight parameters to adapt the model to the continuous changes in data. Similar historical data matching, similarity measurement: use cosine similarity or Euclidean distance to measure the similarity between new feedback data and historical data. The new data is x, and the historical data is h, then the cosine similarity is defined as:

[0195]

[0196] Sim(x,h): the similarity between new data x and historical data h, x·h: the inner product of new data and historical data, ||x|| and ||h||: the norm of new data and historical data, indicating their length.

[0197] Weight parameter fine-tuning: Based on similar historical data samples, dynamically adjust the model parameters, and set the fine-tuned weight parameter as θ updated :

[0198] θ updated =θ current +α·Δθ

[0199] θ updated : Updated model parameters, θ current : Current model parameters, α: Fine-tuning rate, controlling the adjustment range of parameters, Δθ: Parameter update amount based on similar data samples. Through this fine-tuning method, the model can be automatically optimized in a new data environment to ensure the prediction accuracy and consistency in different test stages.

[0200] The visualization module includes an interpretable model and a visualization dashboard. The interpretable model uses the SHAP value / LIME algorithm to generate explanation information of the prediction results and display the contribution of different modal data to the prediction results. The visualization dashboard is used to display the prediction risk value, validity probability, sample feature change trend and other information in real time to assist clinical researchers and doctors to quickly understand the prediction results.

[0201] The SHAP (SHapley Additive ex Planations) value is based on cooperative game theory. It quantifies the contribution of each feature to the prediction result by assigning a Shapley value to each feature. Its calculation is based on the average marginal contribution of the feature combination. For a model with N features, the SHAP value φ of feature i is i It represents the contribution of feature i to the prediction and is defined as follows:

[0202]

[0203] φ i : SHAP value of feature i, i.e., the contribution of feature i to the model prediction result, S: feature subset, excluding feature i, N: size of feature set, f(S∪{i}): model prediction output under feature set S∪{i}, i.e., the prediction result of the model when feature i exists, f(S): model prediction output under feature set S, i.e., the prediction result of the model when feature i does not exist, |S|: size of feature set S, |S|! (|N|-|S|-1)! / |N|! : Shapley value distribution coefficient, representing all possible permutations of feature combinations. Calculation logic of SHAP value, calculate the marginal contribution of all feature combinations: for each feature i, calculate the marginal contribution f(S∪{i})-f(S) on all feature subsets S.

[0204] Weighted average: The marginal contributions of all combinations are weighted and summed to obtain the SHAP value φ of feature i i , represents the average contribution of the feature.

[0205] LIME (Local Interpretable Model-agnostic Explanations) is a model-independent explanation method that explains the prediction of a specific sample by building a local linear model to approximate the local behavior predicted by a nonlinear model. The LIME algorithm first generates some similar samples near the target sample and trains a linear model to approximate the behavior of the target model in this local area:

[0206] Perturbation sample generation: For the sample x to be explained, generate a set of perturbation samples {x′1, x′2, …, x′ m},These samples are near the original samples and are generated by adding noise.

[0207] Weight calculation: For each perturbation sample x′ i , calculate its similarity weight w(x,x′) with the original sample x i ):

[0208]

[0209] w(x,x′ i ): Original sample x and perturbation sample x′ i The similarity weight of ||xx′ i ||: Euclidean distance between the original sample and the perturbation sample, σ: distance scaling parameter, which controls the size of the local area.

[0210] Local linear model training: Fit the behavior of the original model in this local area through weighted linear regression. The goal is to minimize the weighted error:

[0211]

[0212] The weighted loss function of the local linear model, f(x′ i ): The original model is used to perturb the sample x′ i The predicted value of , θ: The coefficient of the local linear model, which indicates the contribution of the feature to the prediction.

[0213] Interpretation of linear model coefficients: The coefficient θ obtained by fitting is the contribution of the feature in the local area, which helps explain the model's decision on the sample.

[0214] The predicted risk value represents the uncertainty of the model prediction, and the risk is usually calculated by the distribution of the predicted probability. Let the predicted probability distribution be p(y|x), and the risk value is defined as:

[0215] Risk=1-p(y pred |x)

[0216] Risk: Prediction risk value, indicating the uncertainty of the model in the current prediction result, p(y pred |x): Model predicts category y pred probability.

[0217] The validity probability measures the credibility of the model prediction results, usually calculated by the prediction confidence interval or Bayesian estimation. Let the prediction confidence interval be [p low ,p high ], then the validity probability is:

[0218] Effectiveness = p high -p low

[0219] Effectiveness: effectiveness probability, indicating the confidence range of the prediction results, p low ,p high : The lower and upper limits of the confidence interval of the prediction results.

[0220] Sample feature change trend Sample feature change trend is used to show how sample features change over time, and displays the historical data and current trend of the features through a time series graph.

[0221] Feature change rate: Let the feature value of the sample at time t be x t , then the rate of change is:

[0222] Δx t =x t -x t-1

[0223] Δx t : The characteristic change rate between time t and t-1, x t : The characteristic value at time t.

[0224] Cumulative change trend: Cumulatively show the trend of feature changes over time, and add up the rate of change at each time step:

[0225]

[0226] X trend : Cumulative trend of feature changes, T: total time step.

[0227] The visual dashboard displays the above information in real time, so that doctors or researchers can monitor the prediction effect, assess risks and discover the trend of feature changes. Through charts and numerical displays, clinical staff can more intuitively understand the model prediction results and make more timely decisions.

[0228] The data processing module is responsible for the preliminary processing of the input multimodal data (such as images, genomes and text data), including data preprocessing, denoising, standardization and preliminary feature extraction. This module mainly relies on the collaborative work of high-performance general-purpose computing units (CPUs) and graphics processing units (GPUs) to accelerate the processing of various types of data.

[0229] How the CPU and GPU work together: The CPU is responsible for scheduling and control, distributing data in batches to the GPU for parallel processing. The GPU uses its multi-core architecture to perform parallel accelerated calculations on images and other high-dimensional data. The high parallelism of the GPU can efficiently complete operations such as image denoising, dimensionality reduction, and feature extraction. The CPU coordinates the data flow and transfers the processed data to the subsequent explanatory model module. The collaborative work of the CPU and GPU is connected through the PCIe channel to ensure the efficiency and stability of data transmission.

[0230] The interpretable model calculation module executes the SHAP and LIME algorithms, and uses specially designed acceleration hardware (such as FPGA or GPU) to efficiently calculate SHAP values ​​and LIME local linear regression, ensuring real-time generation of explanatory information.

[0231] FPGA accelerates SHAP calculations: In SHAP calculations, the logic units of the FPGA are configured to perform parallel calculations of feature combinations and marginal contributions. Through its programmable logic structure, the FPGA allocates different logic units to perform feature subset combinations and prediction value calculations, so that the contribution of each feature to the prediction result can be quickly solved. Each logic unit can calculate the SHAP value contribution of the feature combination in parallel within the clock cycle to complete the accumulation of marginal contributions. This hardware acceleration can significantly shorten the model interpretation time. The on-chip memory of the FPGA is used to store the intermediate results of each feature combination to avoid frequent external data access.

[0232] GPU accelerates LIME's local linear regression: The core of the LIME algorithm is to generate a large number of perturbation samples and perform weighted linear regression in local areas. The GPU processes these perturbation samples in batches in parallel through its stream processors. Each processor performs similarity weight calculations and generates a weighted loss function for the samples. The high parallelism of the GPU enables it to perform regression model training in parallel on multiple cores, accelerating the fitting process of the LIME local model.

[0233] The real-time data evaluation module is mainly used to perform multi-dimensional quality evaluation on real-time data, including anomaly detection, data quality control, and data synchronization check. This module is usually implemented in collaboration with the CPU and FPGA, where the FPGA is used to accelerate the calculation of the isolation forest algorithm for anomaly detection, while the CPU is used to comprehensively analyze quality control indicators.

[0234] FPGA for Isolation Forest Anomaly Detection: The Isolation Forest algorithm requires multiple random trees to build data segmentation decisions. The FPGA assigns the nodes of each tree to different logic units, so that each tree can segment samples in parallel. By assigning the tree structure to the FPGA's logic array, sample partitioning calculations are performed in parallel on each node, which speeds up anomaly detection. The segmentation depth of each tree is used to calculate the degree of isolation, and the logic unit outputs the anomaly score after accumulating the segmentation steps.

[0235] The CPU is used for quality control calculations and data evaluation: The CPU is responsible for managing data integrity, timestamp consistency, and noise level checks, and is particularly suitable for managing comprehensive evaluation tasks with lower computing requirements. Through logical operations and loop structures, the CPU can effectively calculate data integrity scores, time interval consistency, and noise levels. Quality control data is cached in local memory, and the CPU periodically calls this data for updates.

[0236] The dynamic model fine-tuning module is the core component of the entire system. It is used to receive new data feedback and adjust model parameters in real time to adapt to changes in data distribution. This module combines CPU and GPU to implement the incremental learning and fine-tuning process of the model.

[0237] The CPU and GPU work together for incremental learning and fine-tuning: During the model fine-tuning process, the CPU is responsible for the management of historical data and new feedback data, and matches historical data through similarity metrics to find sample sets with similar features to the new data to facilitate the fine-tuning process. The CPU transfers the sample set to the GPU, which performs back-propagation calculations for parameter updates. The GPU's stream processor parallelizes the calculation of weight adjustments and updates the model parameters through an incremental learning mechanism to adapt them to the new data distribution. The parallel computing capability of the GPU plays a significant acceleration role in back-propagation and parameter optimization.

[0238] The visualization dashboard module is used to display the explanatory information, predicted risk, validity probability and feature change trend of the model. This module consists of an image processor (such as DSP) and a display controller, which is responsible for real-time drawing and rendering of data.

[0239] DSP for data visualization: DSP (digital signal processor) is suitable for processing and accelerating graphic data. It can receive the model's predicted risk, effectiveness probability and other information in real time and convert the data into visual graphics. DSP uses hardware-accelerated linear interpolation, curve fitting and data change trend analysis to generate real-time and updateable curve graphs and trend graphs.

[0240] Display controller and graphics rendering: The display controller passes the processed data to the graphics display interface through the interface with the DSP. The display controller coordinates the refresh rate and resolution of the LCD or OLED screen to achieve real-time drawing. The graphics rendering hardware is responsible for drawing and refreshing graphic elements, including bar charts of risk indicators, dynamic range bars of effectiveness probability, and line charts of feature change trends. The display controller and DSP together ensure real-time updating and dynamic visualization of data to help doctors and researchers understand the prediction results. Specific embodiment 2:

[0242] like Figure 2 As shown, the working steps of the multimodal data-driven clinical trial outcome prediction model are as follows:

[0243] Sp1: Data collection: Collect data from different sources, including imaging data (such as MRI, CT, etc.), genomic data, laboratory test data, clinical text records and patient demographic information. These data are collected to the data center through standardized interfaces, stored uniformly, and provide support for subsequent processing steps. The data collection module verifies the legitimacy of the data source and the integrity of the data structure to ensure that the original data meets the expected format and content requirements.

[0244] Sp2: Image data preprocessing: Perform preliminary preprocessing on the image data, including format conversion (such as DICOM to JPEG, PNG and other standard formats) and size adjustment. The preprocessed data enters the image denoising submodule for denoising.

[0245] Sp2.1: MSCNN adaptive Gaussian filtering denoising: The image denoising submodule first applies a multi-scale convolutional neural network (MSCNN) and an adaptive Gaussian filtering algorithm. The multi-scale convolution structure of MSCNN captures features from multiple resolutions, and the adaptive Gaussian filtering algorithm is used to remove background noise and enhance edge features. The multi-layer convolution kernel of MSCNN can automatically extract important edge features in the image while suppressing noise.

[0246] Sp2.2: GAN denoising enhancement: Content-aware generative adversarial network (GAN) is used to further optimize the denoising effect. GAN enhances the key area details of the image data through the generator, and the discriminator selects the image with the best denoising effect. After multiple rounds of iterative training, the generator can remove noise and retain the details of the lesion area, providing clear image features for the model.

[0247] Sp3: Genomic data processing: Genomic data are denoised using principal component analysis (PCA) and clustering algorithms. PCA extracts the main eigenvectors of the data, and clustering algorithms are used to separate noise to ensure that biological relevance is preserved. The denoised genomic data is used for feature extraction to reduce the interference of irrelevant variables and improve the accuracy of features.

[0248] Sp4: Text record processing: Natural language processing (NLP) models are used to extract key information from clinical text records, such as disease descriptions, treatment plans, patient responses, etc. Through embedding technology (such as word embedding, BERT model, etc.), the model can convert text into structured data and extract key information for model analysis.

[0249] Sp5: Multimodal data alignment: Align multimodal data based on timestamp information to ensure the consistency of different types of data in the time dimension. The alignment module checks the timestamps of each data source, adjusts their order and time range, ensures the consistency and timing of the data, and provides support for subsequent analysis.

[0250] Sp6: Feature extraction module: The preprocessed data enters the feature extraction module, and features are extracted according to the modality of the data.

[0251] Sp6.1: CNN feature extraction: Convolutional neural network (CNN) is used for feature extraction of image data. The multi-layer convolution kernel of CNN extracts deep features of images through convolution and pooling operations to generate high-dimensional image feature vectors. These feature vectors can represent key areas and structures in the image, providing rich visual information for the model.

[0252] Sp6.2: NLP embedding coding: The NLP model embeds and codes text data to extract key words and contextual relationships. Through word embedding and context processing, text information is converted into numerical vectors to generate text features. The NLP model can automatically identify and encode important medical terms in the text and provide structured text information for the model.

[0253] Sp6.3: GNN or dimensionality reduction algorithm is used for genomic data analysis: Graph neural network (GNN) is used for genomic data analysis, structuring genomic data into a graph structure and extracting the correlation features between different genes; if the dimension of the genomic data is too high, a dimensionality reduction algorithm is used to process it to ensure that the feature vector can efficiently reflect the biological characteristics.

[0254] Sp7: Dynamic weight allocation module for features of multi-layer attention mechanism: Feature vectors of different modalities are input into the multi-layer attention mechanism for feature fusion, and the weight of each modality feature is dynamically allocated.

[0255] Sp7.1: Global attention layer: The global attention layer analyzes the correlation of each modality feature and dynamically assigns weights to optimize the feature contribution between different modalities. This step ensures that during the fusion process, more relevant modality features receive higher weights, generating more representative comprehensive features.

[0256] Sp7.2: Local attention layer: The local attention layer weights features based on feature sparsity and importance. By analyzing the sparsity of features, sparse and important features are enhanced to ensure the significance of key features to optimize the prediction effect of the model.

[0257] Sp7.3: Dynamic adjustment of reinforcement learning weights: The reinforcement learning algorithm monitors the contribution of different modal features within a specific time window, dynamically adjusts the fusion weights based on feedback, and generates a comprehensive feature vector. This step ensures that the model can adapt to different data scenarios and optimize the feature fusion effect.

[0258] Sp8: Adaptive prediction model: The comprehensive feature vector is input into the adaptive prediction model for prediction. The model includes an adaptive learning module, a deep prediction network, and a multi-objective optimization loss function.

[0259] Sp8.1: Adaptive learning module: The adaptive learning module adjusts the network structure and parameters according to the distribution characteristics of the data, so that the model can adapt to data changes and ensure prediction accuracy.

[0260] Sp8.2: Deep Prediction Network: The deep prediction network uses a hybrid architecture based on Bi-LSTM and Transformer. Bi-LSTM is used to capture short-term dependencies, and Transformer is used to capture long-term dependencies. This combined architecture enables the model to model data at different time scales in multiple dimensions, improving the accuracy and responsiveness of predictions.

[0261] Sp8.3: Multi-objective optimization loss function: The multi-objective optimization loss function combines the impact of prediction accuracy, data noise and outliers to ensure the robustness and noise resistance of the model and make the prediction results more stable.

[0262] Sp9: Real-time data evaluation submodule: Receives new data feedback and performs quality control to ensure the data quality of the input model.

[0263] Sp9.1: Isolation Forest Anomaly Detection: The Isolation Forest algorithm is used to detect and remove abnormal data to ensure high data quality. The algorithm constructs multiple isolated trees, calculates the isolation depth of samples, and filters out outliers.

[0264] Sp9.2: Data quality assessment: Evaluate the integrity, timestamp consistency, and noise level of the data to ensure that data quality standards are met. Quality control standards include data integrity, time consistency, and noise control to ensure that the data used to update the model is accurate and reliable.

[0265] Sp10: Dynamic model fine-tuning module: High-quality feedback data enters the dynamic model fine-tuning module to adjust the model weights.

[0266] Sp10.1: Similar historical data matching and fine-tuning: The dynamic model fine-tuning module matches the features of new data and historical data through similarity measurement, and fine-tunes the model weights based on the matching results to ensure that the model adapts to the latest data distribution.

[0267] Sp10.2: Adaptive feedback control: The adaptive feedback control layer monitors the prediction deviation and adjusts the hierarchical configuration of Bi-LSTM and Transformer based on the data deviation to improve the prediction stability of the model at different stages.

[0268] Sp11: Generate explanatory information of the interpretable model: Input the prediction results into the interpretable model, generate explanatory information through the SHAP value or LIME algorithm, quantify the impact of each modal feature on the prediction results, and form an explanation report.

[0269] Sp11.1: SHAP / LIME algorithm calculation: The SHAP value algorithm assigns a contribution to each feature, and the LIME algorithm generates a local linear model to explain specific prediction results, helping clinical staff understand the impact of each feature.

[0270] Sp12: Visual dashboard display: Prediction results and explanatory information are displayed through a visual dashboard to help clinicians quickly understand the prediction effects and risks.

[0271] Sp12.1: Real-time display of prediction information: The visual dashboard displays the predicted risk value, effectiveness probability, sample feature change trend and prediction stability, intuitively displays the prediction results of the model, and provides decision support for doctors. The dashboard contains real-time data display in various forms such as graphics, numerical values ​​and trend lines, making the information more visual.

[0272] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprising a reference structure" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0273] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal data-driven clinical trial outcome prediction model, characterized by: It includes data acquisition and preprocessing module, feature extraction and fusion module, adaptive prediction model, feedback and update module and visualization module; The data acquisition and preprocessing module is used to acquire clinical trial data from multiple different sources, the clinical trial data including imaging data, genomic data, laboratory test data, clinical text records and patient demographic information, and preprocess each data type separately; The feature extraction and fusion module is used to extract features from the preprocessed data after being processed by the data acquisition and preprocessing module, extract image features using a convolutional neural network, extract text features based on a natural language processing model, and extract key features from genomic data through a graph neural network or a dimensionality reduction algorithm to generate high-dimensional features; The adaptive prediction model includes an adaptive learning module, a deep prediction network and a loss function based on multi-objective optimization. The adaptive learning module dynamically adjusts the network structure and parameters according to the characteristics of clinical trial data and sample distribution. The deep prediction network adopts a bidirectional long short-term memory network or a Transformer-based time series model. The loss function combines prediction accuracy, data noise and outlier factors for multi-objective optimization. The feedback and update module is used to receive feedback data from clinical trials in real time, incorporate new feedback data into model training using incremental learning or online learning mechanisms, and fine-tune model parameters and network levels in different trial scenarios through transfer learning technology to achieve dynamic update and adaptive optimization of the model; The visualization module includes an interpretable model and a visualization dashboard. The interpretable model uses the SHAP value / LIME algorithm to generate explanatory information of the prediction results and display the contribution of different modal data to the prediction results. The visualization dashboard is used to display the predicted risk value, validity probability, and sample feature change trend information in real time.

2. The multimodal data-driven clinical trial outcome prediction model according to claim 1, characterized in that: The data acquisition and preprocessing module further includes an image denoising submodule and a genome data denoising submodule. The image denoising submodule uses an adaptive Gaussian filtering algorithm based on a multi-scale convolutional neural network (MSCNN) to denoise the image data, and the genome data denoising submodule uses a clustering algorithm combined with a principal component analysis (PCA) method to separate and compress the noise in the genome data.

3. The multimodal data-driven clinical trial outcome prediction model according to claim 2, characterized in that: The image denoising submodule further includes a generative adversarial network (GAN) structure based on content-aware denoising, and the generative adversarial network structure includes a generator and a discriminator.

4. The multimodal data-driven clinical trial outcome prediction model according to claim 1, characterized in that: The feature extraction and fusion module includes a feature dynamic weight allocation module based on a multi-layer attention mechanism, which calculates the global attention and local attention weights of each modal feature through a dual attention layer, wherein the global attention layer allocates modal weights based on data relevance, and the local attention layer performs weighting according to the feature sparsity of the sample data and the importance of rare features, so as to improve the discrimination of key features and dynamically balance the contribution of different modalities to the prediction results.

5. The multimodal data-driven clinical trial outcome prediction model according to claim 4, characterized in that: The feature dynamic weight allocation module further includes a time window-based weight update submodule for real-time monitoring of the impact of each modal feature on prediction accuracy within different time windows.

6. The multimodal data-driven clinical trial outcome prediction model according to claim 1, characterized in that: The deep prediction network of the adaptive prediction model adopts a hybrid neural network structure of a bidirectional long short-term memory network (Bi-LSTM) and a Transformer architecture to achieve joint modeling of short-term dependency and long-term dependency. The Bi-LSTM is used to capture the short-term correlation characteristics of time series data, and the Transformer structure is used to extract long-term dependency.

7. The multimodal data-driven clinical trial outcome prediction model according to claim 1, characterized in that: The depth prediction network further includes an adaptive feedback control layer, and the control layer includes an adaptive learning algorithm.

8. The multimodal data-driven clinical trial outcome prediction model according to claim 1, characterized in that: The feedback and update module further includes a real-time data evaluation submodule, which includes an anomaly detection and quality control algorithm. The real-time data evaluation submodule performs multi-dimensional quality evaluation on the received feedback data. The anomaly detection algorithm uses a multivariate analysis technology based on isolation forest to identify and eliminate outliers in the data.

9. The multimodal data-driven clinical trial outcome prediction model according to claim 1, characterized in that: The feedback and update module further includes a dynamic model fine-tuning module, which automatically selects historical data similar to new feedback data for matching.

10. The multimodal data driven clinical trial outcome prediction model according to any one of claims 1 to 9, characterized in that: The prediction process of the prediction model is as follows: Sp1: Collect clinical trial data from different data sources, including imaging data, genomic data, laboratory test data, clinical text records, and patient demographic information. Sp2: Perform preliminary processing on the image data, including format conversion, size adjustment, and input into the image denoising submodule for denoising. Sp2: The image denoising submodule removes background noise from the image and enhances edge features. Sp3: Processing genomic data. Sp4: Apply natural language processing (NLP) models to text records to generate structured data. Sp5: Alignment of data of different modalities based on the timestamp information of the data; Sp6: Input the preprocessed data into the feature extraction module to extract features from image data, text data, and genome data respectively; Sp7: Input feature vectors of different modalities into the feature dynamic weight allocation module of the multi-layer attention mechanism for feature fusion; Sp 8: Input the fused comprehensive feature vector into the adaptive prediction model; Sp9: Receives new data fed back during the clinical trial process and inputs it into the real-time data evaluation submodule for quality control. Sp10: Input high-quality feedback data into the dynamic model fine-tuning module based on transfer learning and incremental learning. Sp11: Input the prediction results into the interpretability model and generate explanation information through the SHAP value or LIME algorithm. Sp12: Display the prediction results and explanation information through a visual dashboard.

Images