Multi-modal medical database construction method and system

Abstract

The invention relates to the technical field of data information processing, in particular to a multi-modal medical database construction method and system, and the system comprises a data collection module, a data preprocessing module, a data storage module, and a multi-modal model construction module. According to the method, the deep learning diagnosis model based on multi-modal data fusion is constructed, so that accurate early warning of disease progress key nodes is realized; the model integrates multi-modal data such as clinical indexes, laboratory examination and iconography data, processing is performed through methods such as feature extraction, standardization and normalization, key information in each modal data is extracted, disease development rules and patient individual differences are captured, and high-risk patients are identified; particularly, the deep learning model is utilized to process and predict multi-time-point data, so that the timeliness and accuracy of the model are improved, early recognition and timely intervention are realized, and a better treatment opportunity is provided for a patient.

Description

Technical Field

[0001] The present invention relates to the technical field of data information processing, and particularly to a method and system for constructing a multi-modal medical database. Background Art

[0002] Infectious pulmonary diseases, as the fifth leading cause of death globally, have become a major public health challenge threatening human health; the latest report of the World Health Organization shows that the number of deaths from infectious pulmonary diseases globally exceeds 2.2 million annually; the annual incidence of infectious pulmonary diseases in China reaches 7.13 / 1000 person-years, which is higher than the average level of developed countries and has shown an increasing trend year by year in the past five years; this severe situation not only brings a huge disease burden, but also poses a major challenge to the allocation of medical resources; the three major challenges faced by the current diagnosis and treatment of infectious pulmonary diseases (time urgency, disease complexity, and individual differences), time urgency: the time window from the onset of the first symptom of the patient to the need for life support is extremely limited, with an average of only 10 days of the golden treatment period; within this short period of time, clinicians must complete a series of key decisions such as disease assessment, prognosis prediction, and treatment plan formulation. This time pressure not only tests the clinicians' clinical judgment ability, but also highlights the shortcoming of the insufficient response speed of existing prediction tools; disease complexity: infectious pulmonary diseases often trigger a systemic inflammatory response through cytokine storm and other cascade reactions, leading to multiple organ dysfunction; this complex pathophysiological process makes it difficult for a single-dimensional monitoring and prediction model to accurately grasp the disease progression law; individual differences: existing scoring systems such as APACHE II and SOFA adopt static comprehensive criteria and cannot fully consider the individual differences of patients. This "one-size-fits-all" assessment method not only has a time lag, but also is difficult to capture the unique disease characteristics of patients, seriously affecting the accuracy and timeliness of clinical decisions. Summary of the Invention

[0003] The purpose of the present invention is to provide a method and system for constructing a multi-modal medical database to solve the problems raised in the above background art.

[0004] To achieve the above purpose, the present invention provides the following technical solution: A method for constructing a multi-modal medical database, including the following steps:

[0005] S1. Obtain data, including the baseline information of the host, laboratory test data, imaging data, intervention measures, and clinical outcomes.

[0006] S2. Data preprocessing, cleaning and standardizing the collected data to ensure the consistency, integrity, and accuracy of the data, and recording the clinical data and dynamic changes of the condition of all patients through a standardized form.

[0007] S3. Data storage. A database is established in a hybrid mode, combining the advantages of the MySQL relational database and the NoSQL database, and structured and unstructured data are stored distributively.

[0008] For structured data, such as patient basic information and laboratory test results, the MySQL relational database is used, and the consistency and scalability of data storage are ensured through a standardized table structure and foreign key associations.

[0009] For unstructured data, such as imaging data and genomic data, the NoSQL database is used to support parallel processing and distributed storage of massive data, enhancing the flexibility of data expansion and retrieval.

[0010] S4. Data integration and modeling. Based on a multi-modal deep algorithm model, clinical data is integrated with laboratory test data, imaging data, and treatment responses according to patient information and medical visit information to construct a multi-modal fusion model. The specific steps are as follows:

[0011] a. Feature extraction of imaging data: The pixel values are adjusted to the standard range through normalization processing, reducing the differences between images. Further, deeper features of the preprocessed images are captured, and a pre-trained EfficientNet model is used to extract features through transfer learning:

[0012] F img = F EfficientNet (I norm )

[0013] where F img represents the features extracted from the EfficientNet model, F EfficientNet represents the EfficientNet model used to extract useful features from the images, and I norm represents the input image after normalization processing.

[0014] The features are further optimized through principal component analysis PCA:

[0015] F img-reduced = PCA(F img )

[0016] where F img-reduced represents the set of features or feature vectors after principal component analysis PCA processing, and F img represents the original set of features.

[0017] Dimensionality reduction is achieved by finding the eigenvectors of the feature covariance matrix:

[0018] F img-reduced = W T Fimg

[0019] Among them, W T is the eigenvector matrix of PCA;

[0020] By extracting the eigenvectors of the feature matrix, high-dimensional data is compressed into a lower-dimensional representation form;

[0021] b. Feature extraction of text data:

[0022] The feature extraction of text data covers medical records. The tool Python is used to preprocess the text, removing stop words and punctuation marks. The BioBERT model is used to perform deep semantic encoding on the preprocessed text to extract deep semantic features:

[0023] F txt = F BinBERT (Tokens)

[0024] Further unify the dimension of text features to facilitate the fusion with other modal data. The average pooling operation is used to convert the extracted semantic information into a fixed-length vector representation:

[0025]

[0026] Among them, F txt-fixed represents the fixed-length vector representation obtained after the average pooling operation, is the normalization factor in the average pooling operation to ensure that the average value of all elements is numerically stable, n is the total number of elements to be pooled, represents the summation of all extracted semantics F txt,i , where i ranges from 1 to n, and F txt,i represents the feature vector of the i-th semantic unit;

[0027] c. Structured data processing and feature extraction:

[0028] When processing structured data and feature extraction, in order to ensure that data with different dimensions can be compared on the same scale, the data is standardized:

[0029] X std = (X - μx) / σx

[0030] Among them, X represents the original data, μx represents the mean value, and σx represents the standard deviation;

[0031] The random forest model is used to extract features from the standardized data, identify the most important features, input the identified features into the random forest model, and extract and integrate the important features of the structured data for output:

[0032] Fstr-selected = SelectTopK(F RF (F ml ), k)

[0033] where SelectTopK refers to the SelectTopK function, a function for selecting the top k values, and F RF represents calling the random forest model, and F ml represents the most important features, and k is an integer parameter used to specify how many of the most important features the SelectTopK function should select;

[0034] d. Key clinical decision node regression and classification prediction model:

[0035] The gradient boosting machine GBM model is used for prediction. The GBM model is trained by minimizing a loss function such as logarithmic loss or mean squared error. Let the training set be X and the target variable be F. The prediction of the model is:

[0036]

[0037] where h m (X) is the m-th decision tree and α m is the weight;

[0038] e. Adaptive Fourier decomposition AFD time series data processing and feature extraction:

[0039] For daily statistical structured data such as time series data, adaptive Fourier decomposition AFD is used for feature extraction. The features extracted by AFD are further input into a long short-term memory network LSTM model for time series feature learning and prediction:

[0040] F afd = AFD(X time-series )、F afd-lstm = LSTM(F afd )

[0041] where F afd represents the set of features or results obtained after feature extraction of the time series data X time-series using the adaptive Fourier decomposition AFD algorithm. AFD(X time-series ) represents applying the adaptive Fourier decomposition algorithm to the time series data X time-series to extract its features, and F afd-lstm represents the final result or output obtained by further learning and prediction using the LSTM model with the features F afd extracted by the adaptive Fourier decomposition as the input. LSTM(F afd ) represents applying the LSTM model to Fafd Feature-wise, for learning and predicting time series features;

[0042] f. Feature extraction of text information and laboratory results based on large language models:

[0043] Adopt large language model technology to deeply mine key information in medical data, and the large language model technology automatically identifies and extracts key features from different data sources;

[0044] g. Multimodal fusion layer:

[0045] Adopt the self-attention mechanism to integrate image features, text features, structured data features, and the associated features extracted by the large language model into a feature matrix;

[0046] S5. Model training and optimization, adopt a multi-task learning framework, output the disease risk prediction part, and the total loss function of multi-task learning includes cross-entropy loss and regression loss:

[0047] L = aL1 + βL2

[0048] Among them, L1 and L2 represent cross-entropy loss, which is used for classification tasks:

[0049]

[0050] Use the Adam optimizer for training, and the parameter update formula is as follows:

[0051]

[0052] Among them, m t and v t are the first-order moment estimate and second-order moment estimate of the gradient respectively, and the update rules are as follows:

[0053] m t = β1m t-1 + (1 - β1)g t

[0054]

[0055] Among them, g t is the gradient, β1 and β2 are momentum terms, and h is the learning rate;

[0056] Comprehensively evaluate the model through accuracy, recall, precision, and F1 score;

[0057] A multimodal medical database construction system, the system includes: a data collection module, a data preprocessing module, a data storage module, and a multimodal model construction module, where:

[0058] S1. The data acquisition module obtains the host's baseline information, laboratory test data, imaging data, intervention measures, and clinical outcomes through the Spark interface, and aggregates them into the data storage module;

[0059] S2. The data preprocessing module uses the data cleaning tool Python to clean and preprocess the information, removing redundant, incorrect, or irrelevant data to ensure the reliability of the data quality finally incorporated into the database;

[0060] S3. The data storage module combines a MySQL relational database and a NoSQL database, and provides a RESTful API interface to facilitate external systems to access and share data, realizing flexible invocation and integration of data;

[0061] S4. The multimodal model construction module integrates clinical data with laboratory test data, imaging data, and treatment responses based on a multimodal deep algorithm model. The multimodal model construction module includes:

[0062] The imaging sub-module: extracts high-dimensional imaging features through normalization processing and the pre-trained EfficientNet model;

[0063] The text sub-module: uses the BioBERT model to perform deep semantic encoding on the preprocessed text, and uses average pooling operation to convert the extracted semantic information into a fixed-length vector representation;

[0064] The structured data sub-module: uses the random forest model to extract features from the standardized data, and inputs the identified features into the random forest model to extract and integrate the important features of the structured data;

[0065] Further, in the S4, the pre-trained EfficientNet model includes the following steps:

[0066] S41. Data preparation: divide the preprocessed image dataset into a training set, a validation set, and a test set;

[0067] S42. Load the pre-trained model: use the pre-trained EfficientNet model as the base model and adjust the output layer according to the task requirements;

[0068] S43. Model training: use the training set to train the model and train it according to the preset number of iterations;

[0069] S44. Model evaluation: use the validation set to evaluate the model and optimize the model according to the evaluation results;

[0070] S45. Model testing: Use the test set to test the model. Common test metrics include accuracy, recall rate, and F1 score.

[0071] Compared with the prior art, the beneficial effects of the present invention are as follows: By constructing a deep learning diagnosis model based on multi-modal data fusion, accurate early warning of key nodes in disease progression is achieved; A multi-modal data fusion strategy is proposed to fully exploit the complementary value of different types of medical data; By integrating patients' clinical data and advanced deep learning technologies, an early warning system is established to identify high-risk patients; In particular, using a deep learning model to process and predict multi-time point data will improve the timeliness and accuracy of the model, thereby achieving early identification and timely intervention, and providing better treatment opportunities for patients; The model integrates multi-modal data such as clinical indicators, laboratory tests, and imaging data, and processes them through methods such as feature extraction, standardization, and normalization, extracting key information from each modality of data, capturing the development laws of diseases and individual differences of patients; The model helps to deeply understand the progression laws and influencing factors of infectious pulmonary diseases, will significantly improve the prognostic prediction ability of infectious pulmonary diseases, provide more accurate AI-assisted decision-making support for the treatment of critically ill patients; And help clinicians achieve precise treatment, reduce the mortality rate; Optimize the allocation of medical resources and reduce medical costs; Provide decision-making support for hierarchical diagnosis and treatment; At the clinical application level, it can provide objective and timely decision-making support for doctors, optimize treatment plans, and improve the treatment effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is the method flow chart of the present invention;

[0073] Figure 2 is the system architecture diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0074] Next, the technical solutions of the present invention will be described in detail through specific embodiments.

[0075] Please refer to Figure 1-2 , the present invention provides a method for constructing a multi-modal medical database, which is characterized by including the following steps:

[0076] S1. Data acquisition: Acquire the host's baseline information, laboratory test data, imaging data, intervention measures, and clinical outcomes. Among them, the patient's laboratory test data, such as C-reactive protein, procalcitonin, and inflammatory factors, will be uniformly converted to international standard units, and the imaging data, such as CT and X-ray films, will be stored in DICOM format to ensure compatibility across hospital systems;

[0077] Metadata is attached to the data during the acquisition process, such as data source, acquisition time, and disease type, for subsequent data retrieval and correlation analysis;

[0078] S2. Data preprocessing, cleaning and standardizing the collected data to ensure the consistency, integrity and accuracy of the data;

[0079] Record the clinical data of all patients through a standardized form, including the scores on the day of admission, such as PSI, APACHEII, CURB-65 and SOFA scores, as well as the dynamic changes of the condition;

[0080] S3. Data storage, establish a database in a hybrid mode, combining the advantages of MySQL relational database and NoSQL database, and storing structured data and unstructured data distributively;

[0081] For structured data, such as patient basic information and laboratory test results, use the MySQL relational database to ensure the consistency and scalability of data storage through a standardized table structure and foreign key association;

[0082] For unstructured data, such as imaging data and genomic data, use the NoSQL database to support parallel processing and distributed storage of massive data, and improve the flexibility of data expansion and retrieval;

[0083] The database will ensure the security and efficient management of data through a distributed storage structure, and support simultaneous access by multiple users and multiple hospitals;

[0084] Through a distributed architecture, the database can handle the increasing multi-modal data in the future, provide efficient storage and retrieval capabilities, and support large-scale clinical cohort studies;

[0085] S4. Data integration and modeling, based on a multi-modal deep algorithm model, integrate clinical data with laboratory test data, imaging data and treatment responses according to patient information and visit information, and build a multi-modal fusion model;

[0086] The deep algorithm model focuses on two core diagnostic data of microbial detection and host information, adopts a hierarchical construction method, and uses multi-modal fusion technology to finally form a multi-task output of key clinical decision-making nodes for infectious pulmonary diseases. The specific steps are as follows:

[0087] a. Feature extraction of imaging data: Adjust the pixel values to the standard range through normalization processing, reducing the differences between images:

[0088]

[0089] Among them, I is the original imaging data, I min and I max are the minimum and maximum values of the imaging data respectively;

[0090] The image is processed by normalization to provide stable input data for subsequent analysis. The data augmentation technique is used to diversify the images, such as rotation, flipping, and cropping, which effectively expands the dataset and improves the model's performance on unknown data. For example, the image can be randomly rotated by a certain angle:

[0091] I rot = rotate(I,θ)

[0092] where θ is the rotation angle, and I is flipped horizontally flip = flip(I) and randomly cropped:

[0093] I crop = crop(I,crop_size)

[0094] where crop_size specifies the size of the cropping box;

[0095] The pre-trained EfficientNet model based on the preprocessed image data includes the following steps:

[0096] S41. Data preparation, dividing the preprocessed image dataset into a training set, a validation set, and a test set;

[0097] S42. Loading the pre-trained model, using the pre-trained EfficientNet model as the base model and adjusting the output layer according to the task requirements;

[0098] S43. Model training, training the model using the training set and training according to the preset number of iterations;

[0099] S44. Model evaluation, evaluating the model using the validation set and optimizing the model according to the evaluation results;

[0100] S45. Model testing, testing the model using the test set. Common testing metrics include accuracy, recall, and F1 score;

[0101] To further capture deeper features in the preprocessed image, a pre-trained EfficientNet model is used for feature extraction through transfer learning:

[0102] F img = F EfficientNet (I norm )

[0103] where, where, F img represents the features extracted from the EfficientNet model, F EfficientNetDenotes the EfficientNet model, which is used to extract useful features from images, I norm Denotes the normalized input image;

[0104] The features are further optimized through principal component analysis (PCA):

[0105] F img-reduced = PCA(F img )

[0106] where F img-reduced denotes the set of features or feature vectors after being processed by principal component analysis (PCA), and F img denotes the original set of features;

[0107] The image is further optimized to reduce the feature dimension, while retaining key image information, reducing the computational cost, and improving the training efficiency of the model;

[0108] The dimension is reduced by finding the eigenvectors of the feature covariance matrix:

[0109] F img-reduced = W T F img

[0110] where W T is the eigenvector matrix of PCA; by extracting the eigenvectors of the feature matrix, the high-dimensional data is compressed into a lower-dimensional representation form;

[0111] b. Feature extraction of text data:

[0112] The feature extraction of text data covers medical records, such as patients' symptoms, medical history, previous medications, and laboratory test results, such as blood routine, C-reactive protein (CRP), and procalcitonin (PCT);

[0113] Use the tool Python to preprocess the text, remove stop words and punctuation marks, and use the BioBERT model to perform deep semantic encoding on the preprocessed text to extract deep semantic features:

[0114] F txt = F BinBERT (Tokens)

[0115] Further unify the dimension of the text features to facilitate the fusion with other modal data, and use average pooling operation to convert the extracted semantic information into a fixed-length vector representation:

[0116]

[0117] where F txt-fixed denotes the fixed-length vector representation obtained after the average pooling operation, is the normalization factor in the average pooling operation, ensuring that the average value of all elements is numerically stable. n is the total number of elements being pooled. represents the summation over all the extracted semantics F txt,i , where the summation is performed from i = 1 to n, and F txt,i represents the feature vector of the i-th semantic unit;

[0118] c. Structured data processing and feature extraction:

[0119] When processing structured data and feature extraction, to ensure that data with different dimensions can be compared on the same scale, the data is normalized:

[0120] X std = (X - μx) / σx

[0121] where X represents the original data, μx represents the mean value, and σx represents the standard deviation;

[0122] By normalization, the quantization differences between different features are eliminated, enabling all data to be compared and modeled on a consistent basis;

[0123] A random forest model is used to extract features from the normalized data to identify the most important features:

[0124] F ml = MLFeatureExtraction(X std )

[0125] The identified features are input into the random forest model to extract and integrate the important features of the structured data for output:

[0126] F str-selected = SelectTopK(F RF (F ml ), k)

[0127] where SelectTopK refers to the SelectTopK function, a function for selecting the top k values. F RF represents the call to the random forest model, F ml represents the most important features, and k is an integer parameter used to specify how many of the most important features the SelectTopK function should select;

[0128] d. Key clinical decision node regression and classification prediction model:

[0129] The gradient boosting machine (GBM) model is used for prediction. The GBM model iteratively trains weak classifiers, such as decision trees, to gradually reduce the prediction error and improve the prediction performance of the model. In each round of training, the prediction ability of the model is optimized by fitting the residuals of the previous round of the model. Multitimepoint data from outpatient clinics, emergency departments, and ICUs are integrated, and after standardization or normalization processing, trend changes and time series features, such as heart rate changes and white blood cell counts, are extracted. The GBM model is trained by minimizing a loss function, such as logarithmic loss or mean squared error. Let the training set be X, the target variable be F, and the prediction of the model be:

[0130]

[0131] where h m (X) is the mth decision tree, and α m is the weight;

[0132] e. Adaptive Fourier decomposition (AFD) time series data processing and feature extraction:

[0133] For daily statistical structured data, such as time series data, adaptive Fourier decomposition (AFD) is used for feature extraction. The structured data is converted into a time series form, and the daily statistical results are used as the input of the time series:

[0134] X time-series = DailyStatistics(X)

[0135] AFD recursively decomposes the signal into multiple orthogonal basis functions. The basis functions are adaptively selected according to the spectral characteristics of the signal to ensure maximum extraction of the key features in the signal. Among them, each basis function is adaptively selected according to the specific spectral characteristics of the signal. Denote the standardized time series data as the signal f(t). First, follow the maximum selection principle of energy tracking matching to select the initial basis function g0(t) that best matches the signal characteristics. Subsequently, calculate the projection of the signal on the initial basis function <f, g0>. The inner product is the inner product in the Euclidean space, and the residual signal is calculated therefrom: r1(t) = f(t) - <f, g0>g0(t)

[0136] Repeat the above process to obtain a series of basis functions, calculate the projections and residuals respectively, until the stopping condition is met, such as, the energy of the residual signal is lower than the threshold:

[0137] r k+1 (t) = r k (t) - <r k , g k >g k (t) [[ID=�1]]

[0138] Sum up the projection results of all basis functions to obtain the AFD reconstruction result of the original signal:

[0139]

[0140] The AFD extracts the main spectral features of the signal, reflecting the hidden periodic and aperiodic components in the signal. The features extracted by the AFD are further input into the long short-term memory network LSTM model for learning and prediction of time series features:

[0141] F afd = AFD(X time-series ), F afd-lstm = LSTM(F afd )

[0142] Among them, F afd represents the feature set or result obtained after feature extraction from the time series data X time-series by the adaptive Fourier decomposition AFD algorithm. AFD(X time-series ) represents applying the adaptive Fourier decomposition algorithm to the time series data X time-series to extract its features. F afd-lstm represents the final result or output obtained after further learning and prediction through the LSTM model with the features F afd extracted by the adaptive Fourier decomposition as the input. LSTM(F afd ) represents applying the LSTM model to the F afd features for learning and prediction of time series features;

[0143] The LSTM model can effectively handle long-term dependencies and capture the dynamic change features in the time series, thus improving the prediction ability for complex time series data;

[0144] f. Feature extraction of text information and laboratory results based on large language models:

[0145] Using large language model technology to deeply mine the key information in medical data, the large language model technology automatically identifies and extracts the key features in different data sources, such as medical record text information, laboratory test results, and correlates this information to provide a high-dimensional and semantically rich feature representation for model training;

[0146] f1. Feature extraction and correlation of medical record text information

[0147] The large language model deeply understands the medical entities and their relationships in the text, speculates on their possible causes and development trends, and through combining with external medical knowledge bases, the model can automatically supplement the implicit associated information in the text to construct a feature representation with a deeper semantic level, such as inferring the potential impact of medical history on medication, thus generating comprehensive features. The output feature representation is:

[0148] F LLM-text = LLM(T medical , K external )

[0149] Wherein, T medical represents the medical record text data, and K external is the relevant information retrieved from external medical knowledge bases and the medical treatment experiences of multiple famous doctors; the generated feature vector reflects both the explicit information in the medical record and the implicit associations inferred by the model;

[0150] f2. Feature Extraction and Association of Laboratory Test Results

[0151] Laboratory test results, such as C-reactive protein CRP and procalcitonin PCT, are important bases for diagnosing and evaluating the condition. The large language model standardizes the data and associates it with the information in the medical record text to generate a composite feature representation;

[0152] By combining laboratory data and medical record text, the large language model can capture the trends and potential associations of pathological changes, providing support for subsequent condition prediction. The output feature representation form is:

[0153] F LLM-lab = LLM(F CRP , F PCT K external )

[0154] Wherein, F CRP and F PCT represent the data features of CRP and PCT, and K external is the relevant information retrieved from external medical knowledge bases and the medical treatment experiences of multiple famous doctors. The generated feature vector can integrate multiple data sources, providing a richer clinical feature description, and fusing the data features extracted by the two parts of the large model:

[0155] F LLM = F LLM-text + F LLM-lab

[0156] g. Multimodal Fusion Layer

[0157] Multimodal fusion is to integrate data from different modalities, such as images, text, time series, structured data, and time series data, in order to make full use of the complementarity of multi-source information and improve the overall performance of the task;

[0158] Specific steps to achieve effective fusion of multimodal data:[[ID=6l]]

[0159] The self-attention mechanism is adopted to integrate the image features, text features, and structured data features, and the associated features extracted by the large language model into a feature matrix:

[0160] F concat = [F img-reduced , F txt-fixed , F str-selected , F afd-lstm , F LLM

[0161] The process of feature fusion depends on the self-attention mechanism. The self-attention mechanism determines which modal features are most important for the current task by calculating the correlation weights between each feature and other features. The specific operations are as follows:

[0162] Generate query vectors, key vectors, and value vectors:

[0163] Q = W Q F concat , K = W K F concat , V = W V F concat

[0164] The query vector is used to represent how the features of the current modality are related to other modalities. The key vector is used to capture the feature representations of other modalities, and the value vector represents the weights and information of the features themselves;

[0165] Calculate the attention weights: The dot product operation between the query vector and the key vector reflects the correlation between different modal features. The dot product result will be normalized by the Softmax function to obtain the weights:

[0166]

[0167] Use the attention weight A to weight the value vector V to obtain the fused feature representation: Z = AV;

[0168] Multi-head attention mechanism: Use the multi-head attention mechanism. Each head processes the feature correlation from different perspectives and integrates the outputs of all heads:

[0169] MultiHead(Q, K, V) = [Z1, Z2,..., Z h W0

[0170] where, Z h represents the output of each attention head, h is the number of attention heads, and W0 is the linear transformation matrix used to integrate the outputs of all heads into a matrix;

[0171] Residual connection and layer normalization: To enhance the stability and training efficiency of feature learning, residual connection and layer normalization are used to process the fused features. The residual connection superimposes the original features and the self-attention output: ​

[0172] H res = F concat + Z

[0173] Normalize the features after residual connection through layer normalization:

[0174] H out = LayerNorm(H res )

[0175] Normalizing the features after residual connection effectively prevents the problem of gradient vanishing and accelerates model training;

[0176] Decoder and decision layer: After multi-modal fusion, the generated feature matrix is input into the fully connected layer to output the task prediction result;

[0177] By fusing data from multiple modalities, the system can capture more context and features, improve the accuracy and robustness of tasks such as classification and detection, and thus provide more reliable decision support in complex scenarios;

[0178] S5. Model training and optimization adopt a multi-task learning framework. For the part of disease risk prediction output, the total loss function of multi-task learning includes cross-entropy loss and regression loss:

[0179] L = aL1 + βL2

[0180] Among them, L1 and L2 represent cross-entropy loss, which is used for classification tasks:

[0181]

[0182] Use the Adam optimizer for training. The Adam optimizer adjusts parameter updates through adaptive learning rates, improving the convergence speed and stability of the model. The parameter update formula is as follows:

[0183]

[0184] Among them, m t and v t are the first-order moment estimate and second-order moment estimate of the gradient respectively, and the update rules are as follows:

[0185] m t = β1m t-1 + (1 - β1)g t

[0186]

[0187] Among them, g t is the gradient, β1 and β2 are momentum terms, and h is the learning rate;

[0188] Comprehensively evaluate the model through accuracy, recall, precision, and F1 score;

[0189] 2 A multi-modal medical database construction system, characterized in that the system includes: a data collection module, a data preprocessing module, a data storage module, and a multi-modal model construction module, where:

[0190] S1. The data collection module obtains the host's baseline information, laboratory examination data, imaging data, intervention measures, and clinical outcomes through the Spark interface, and converges them into the data storage module;

[0191] S2. The data preprocessing module uses the data cleaning tool Python to clean and preprocess the information, removing redundant, incorrect, or irrelevant data to ensure the reliable quality of the data finally incorporated into the database;

[0192] S3. The data storage module combines a MySQL relational database and a NoSQL database, and provides a RESTful API interface to facilitate external systems to access and share data, realizing flexible invocation and integration of data;

[0193] Regarding the permission management of data, the system will set different access levels according to different user roles, such as researchers, doctors, and data administrators, to ensure that the privacy protection of sensitive data complies with international privacy regulations; the personal privacy information of patients will be anonymized and encrypted technology will be used to ensure data security during storage and transmission;

[0194] S4. The multi-modal model construction module integrates clinical data with laboratory examination data, imaging data, and treatment responses based on a multi-modal deep algorithm model. The multi-modal model construction module includes:

[0195] Image sub-module: Extract high-dimensional image features through normalization processing and the pre-trained EfficientNet model;

[0196] Text sub-module: Use the BioBERT model to perform deep semantic encoding on the preprocessed text, and use average pooling operation to convert the extracted semantic information into a fixed-length vector representation;

[0197] Structured data sub-module: Use the random forest model to extract features from the standardized data, and input the identified features into the random forest model to extract and integrate the important features of the structured data;

[0198] The above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes should be covered within the protection scope of the present invention.

Claims

1. A method for constructing a multimodal medical database, characterized in that It includes the following steps: S1. Obtain data, including the baseline information, laboratory examination data, imaging data, intervention measures, and clinical outcomes of the host; S2. Data preprocessing, cleaning and standardizing the collected data to ensure the consistency, integrity, and accuracy of the data, and recording the clinical information and dynamic changes of the condition of all patients through a standardized form; S3. Data storage, establishing a database in a hybrid mode, combining the advantages of MySQL relational database and NoSQL database, and storing structured data and unstructured data distributively; S4. Data integration and modeling, based on a multi-modal deep algorithm model, integrating clinical data with laboratory examination data, imaging data, and treatment responses according to patient information and visit information to construct a multi-modal fusion model; S5. Model training and optimization, adopting a multi-task learning framework to output the disease risk prediction part. The total loss function of multi-task learning includes cross-entropy loss and regression loss: L = aL1 + βL2 where L1 and L2 represent cross-entropy loss for classification tasks: Use the Adam optimizer for training, and the parameter update formula is as follows: where m t and v t are the first and second moment estimates of the gradient, respectively, and the update rules are as follows: m t = β1m t-1 + (1 - β1)g t where g t is the gradient, β1 and β2 are momentum terms, and h is the learning rate; Comprehensively evaluate the model through accuracy, recall, precision, and F1 score.

2. A multimodal medical database construction system, characterized in that, The system includes: a data acquisition module, a data preprocessing module, a data storage module, and a multi-modal model construction module, where: S1. Data acquisition module, obtaining the baseline information, laboratory examination data, imaging data, intervention measures, and clinical outcomes of the host through the Spark interface and aggregating them into the data storage module; S2. Data preprocessing module, using the data cleaning tool Python to clean and preprocess the information, removing redundant, incorrect, or irrelevant data to ensure the reliability of the data quality finally incorporated into the database; S3. Data storage module, combining MySQL relational database and NoSQL database, and providing a RESTful API interface to facilitate external systems to access and share data, realizing flexible invocation and integration of data; S4. Multi-modal model construction module, based on a multi-modal deep algorithm model, integrating clinical data with laboratory examination data, imaging data, and treatment responses. The multi-modal model construction module includes: Image sub-module: Extracting high-dimensional image features through normalization processing and the pre-trained EfficientNet model; Text sub-module: Using the BioBERT model to perform deep semantic encoding on the preprocessed text, and using average pooling operation to convert the extracted semantic information into a fixed-length vector representation; Structured data sub-module: Using the random forest model to extract features from the standardized data, and inputting the identified features into the random forest model to extract and integrate the important features of the structured data.

3. A method for constructing a multimodal medical database according to claim 1, characterized in that: In the above S4, for structured data such as patient basic information and laboratory test results, a MySQL relational database is used to ensure data storage consistency and scalability through a normalized table structure and foreign key associations; for unstructured data such as imaging data and genomic data, a NoSQL database is used to support parallel processing and distributed storage of massive data.

4. A method for constructing a multimodal medical database according to claim 1, characterized in that: In the above S4, the pre-trained EfficientNet model includes the following steps: S41. Data preparation: Divide the preprocessed image dataset into a training set, a validation set, and a test set. S42. Load the pre-trained model: Use the pre-trained EfficientNet model as the base model and adjust the output layer according to the task requirements. S43. Model training: Train the model using the training set and perform training according to the preset number of iterations. S44. Model evaluation: Evaluate the model using the validation set and optimize the model according to the evaluation results. S45. Model testing: Test the model using the test set. Common test metrics include accuracy, recall rate, and F1 score.

Images