Medical algorithm model construction system based on multiple modes
By building a system based on multimodal medical algorithm model, integrating multiple data types and using self-attention mechanisms to fusion, the problems of data singularity and limitations of evaluation tools in traditional methods are solved, and accurate diagnosis and risk prediction of infectious lung diseases are achieved, and the accuracy and efficiency of clinical judgment are improved.
Patent Information
- Application Number
- CN202510413076.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-03
- Publication Date
- 2025-06-27
AI Technical Summary
Traditional infectious lung disease diagnosis and prediction methods are limited by the limitations of data singularity and evaluation tools, and it is difficult to meet the clinical needs for rapid and accurate diagnosis of diseases. It is difficult to integrate multimodal data, insufficient accuracy and robustness of prediction models, and difficult to identify early high-risk patients.
The system is constructed using a multimodal medical algorithm model based on multimodal, including multimodal data acquisition and preprocessing module, multimodal feature extraction module, multimodal fusion module and multitask learning and decision-making module. Through the self-attention mechanism, the deep learning diagnostic model is constructed to realize the accurate diagnosis and risk prediction of infectious lung diseases.
It has achieved the identification of early-stage high-risk patients with infectious lung diseases, improved the accuracy and robustness of disease prediction, reduced the risk of misdiagnosis and missed diagnosis, helped medical staff identify high-risk patients in the early stage of the disease and implemented timely interventions, and reduced the patient's mortality rate.
Smart Images

Figure CN120221056A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of medical algorithm models, and particularly relates to a multi-modal based medical algorithm model construction system. Background Art
[0002] Infectious pulmonary diseases, as a major global public health challenge, pose an increasingly severe threat to human health. Traditional diagnosis and prediction methods are limited by the singularity of data and the limitations of evaluation tools, and it is difficult to meet the clinical needs for rapid and accurate disease judgment. For example, existing scoring systems are lagging, single markers cannot comprehensively reflect the condition, and in the face of the time urgency, complexity of the condition, and individual differences in the diagnosis and treatment of infectious pulmonary diseases, traditional means are stretched. With the development of artificial intelligence technology, multi-modal data fusion shows potential, but there are still many key scientific problems to be solved in this field, such as difficult integration of multi-modal data, insufficient accuracy and robustness of prediction models, and difficult identification of early high-risk patients. The present invention aims to overcome these problems.
[0003] Therefore, the present invention provides a multi-modal based medical algorithm model construction system. Summary of the Invention
[0004] The object of the present invention is to provide a multi-modal based medical algorithm model construction system to solve the technical problems raised in the background art.
[0005] To achieve the above object, the specific technical solution of the present invention is as follows: A multi-modal based medical algorithm model construction system, comprising:
[0006] A multi-modal data collection and preprocessing module, responsible for collecting data related to infectious pulmonary diseases from multiple data sources, including but not limited to clinical indicators, laboratory test data, imaging data, and treatment responses, and collecting different modal data, such as text-based medical record information, numerical test indicators, and imaging data, using a standardized process. For example, laboratory test data is uniformly converted to international standard units, imaging data is stored in DICOM format, and detailed metadata is attached, covering data sources, collection times, disease types, etc. The collected data is subjected to cleaning and standardization processing to remove missing values, duplicate data, and outliers; ensure data consistency, integrity, and accuracy.
[0007] A multi-modal feature extraction module: performing deep feature extraction on different modal data, the multi-modal feature extraction module includes: an imaging feature extraction sub-module, a text feature extraction sub-module, a structured data processing sub-module, and a large language model assisted feature extraction sub-module;
[0008] The multi-modal fusion module uses the self-attention mechanism (Transformer) to integrate different modal features, including image, text, structured, time series, and associated features extracted by large language models.
[0009] The multi-task learning and decision-making module constructs a multi-task learning framework to concurrently execute two core tasks: predicting the risk of infectious pulmonary diseases and providing decision support based on this prediction.
[0010] Preferably, the specific method for processing the multi-modal data acquisition and preprocessing module is as follows: According to the degree of data structuring, a relational database (such as MySQL or PostgreSQL) is selected to store structured data, and a NoSQL database (such as MongoDB) is used to store unstructured data. A distributed storage structure is adopted to ensure data security, efficient management, and simultaneous access by multiple users and multiple hospitals. At the same time, a strict permission management and privacy protection mechanism is set up.
[0011] Preferably, the specific method of the imaging feature extraction sub-module is as follows:
[0012] For CT and X-ray image data, first perform normalization processing to adjust the pixel values to the standard range and reduce the differences between images:
[0013]
[0014] where I is the original image data, I min and I max are the minimum and maximum values of the image data respectively;
[0015] At the same time, data augmentation techniques such as rotation, flipping, and cropping are used to expand the dataset and improve the generalization ability of the model:
[0016] The image can be randomly rotated by a certain angle: I rot = rotate(I,θ), and perform horizontal flipping I flip = flip(I) and random cropping:
[0017] I crop = crop(I,crop_size);
[0018] To capture deeper features from the preprocessed images, we adopt a pre-trained EfficientNet model and perform feature extraction through transfer learning:
[0019] F img = F EffNet (I norm );
[0020] where F imgRepresents the extracted high-dimensional image features. Subsequently, these features are further optimized through principal component analysis (PCA): F img—reduced = PCA(F img );
[0021] The purpose is to reduce the feature dimension while retaining key image information, thereby reducing the computational cost and improving the training efficiency of the model. The dimension is reduced by finding the eigenvectors of the feature covariance matrix:
[0022] F img—reduced = WTF img
[0023] Here, W T is the eigenvector matrix of PCA. By extracting the eigenvectors of the feature matrix, the high-dimensional data is compressed into a lower-dimensional representation form.
[0024] Preferably, the text feature extraction sub-module includes medical record records and laboratory examination text results. During preprocessing, stop words and punctuation marks are removed, and the BioBERT model pre-trained with biomedical corpus is used for deep semantic encoding to extract semantic features: F txt = F BioBERT (Tokens), and then the average pooling operation is introduced to unify the text feature dimension, which is convenient for fusion with other modalities, and the extracted semantic information is converted into a fixed-length vector representation:
[0025]
[0026] Preferably, when the structured data processing sub-module processes structured data and feature extraction, it processes immune status data (such as inflammatory factor levels, flow cytometry analysis results) and microbial detection results (including metagenomic sequencing (mNGS), targeted gene sequencing (tNGS), microbial culture, and multiplex PCR, etc.). To ensure that data with different dimensions can be compared on the same scale, we perform standardization processing on these data:
[0027] X std = (X - μ X ) / σ X ;
[0028] Among them, X represents the original data, μ X represents the mean value, and σ X represents the standard deviation. This standardization process effectively eliminates the dimensional differences between different features, enabling all data to be compared and modeled on a consistent basis;
[0029] For the standardized data, we adopted a variety of advanced machine learning methods for feature extraction. Through algorithms such as the random forest model and support vector machine, we can identify the most important features:
[0030] F ml = MLFeatureExtraction(X std )
[0031] These features are further input into the random forest model to extract and integrate the important features of the structured data for output:
[0032] F str—selected= selectToPK(F RF (F ml ), k).
[0033] Preferably, the large language model-assisted feature extraction sub-module: Utilizing the powerful semantic understanding and reasoning ability of the large language model, deeply mines the medical record text information and laboratory test results, extracts symptoms, medical history, and medication information from the medical record text, and combines with the external medical knowledge base to associate implicit information to generate comprehensive features; for laboratory data, associates with the medical record text to capture the pathological change trend, and finally fuses the features extracted by the two parts of the large model to provide a high-dimensional and semantically rich feature representation for model training.
[0034] Preferably, the specific method of the multi-modal fusion module is as follows:
[0035] Integrate the image features, text features, structured data features, and associated features extracted by the large language model into a feature matrix:
[0036] F concat = [F img—reduced , F txt—fixed , F str—selected , F afd—lstm , F LLM
[0037] The process of feature fusion depends on the self-attention mechanism; this mechanism determines which modality features are most important for the current task by calculating the correlation weights between each feature and other features. The specific operations are as follows:
[0038] Generate query vectors, key vectors, and value vectors: Q = W Q F concat 、K = W K F concat 、V = W V F concat The query vector is used to represent how the features of the current modality are related to other modalities. The key vector is used to capture the feature representations of other modalities, and the value vector represents the weights and information of the features themselves;
[0039] Calculating attention weights: The dot product operation between the query vector and the key vector reflects the correlation between the features of different modalities. The result of the dot product is normalized by the Softmax function to obtain the weights:
[0040]
[0041] Then, the attention weights A are used to weight the value vector V to obtain the fused feature representation: Z = AV
[0042] Multi-Head Attention Mechanism: To enhance the feature interaction ability of the model between different modalities, the multi-head attention mechanism is used. Each head processes the feature correlation from different perspectives, and the outputs of all heads are integrated:
[0043] MultiHead(Q,K,V) = [Z1,Z2,…,Z h W O
[0044] Here, Z h represents the output of each attention head, h is the number of attention heads, and W O is the linear transformation matrix used to integrate the outputs of all heads into a matrix;
[0045] Residual Connection and Layer Normalization: To enhance the stability and training efficiency of feature learning, residual connection and layer normalization are used to process the fused features. The residual connection superimposes the original features and the self-attention output:
[0046] H res = F concat + Z
[0047] Then, the features after the residual connection are normalized through layer normalization: H out = LayerNorm(H res )
[0048] This can effectively prevent the problem of gradient vanishing and accelerate model training;
[0049] Decoder and Decision Layer: After multi-modal fusion, the generated feature matrix is input into the fully connected layer to output the task prediction result.
[0050] Preferably, in the multi-task learning and decision-making module, the risk prediction task determines the high-risk state of severe pneumonia in patients, whether ICU intervention is required, and the necessity of mechanical ventilation by fusing multi-modal features, combining imaging lesions, clinical symptoms, and laboratory data through a classifier; the decision support task is based on the risk prediction results, semantically analyzes them with the help of a large language model, and combines an expert knowledge graph to provide intuitive and interpretable decision-making suggestions to assist doctors in adjusting medications, allocating resources, and optimizing treatment plans.
[0051] The multi-modal based medical algorithm model construction system of the present invention has the following advantages:
[0052] 1. This multi-modal based medical algorithm model construction system integrates data at multiple time points in outpatient and emergency departments and chest radiograph imaging data to predict in advance the mortality rate, ICU admission rate, and ventilator requirements of patients with infectious pulmonary diseases, bringing forward key decision-making nodes, which helps medical staff identify high-risk patients at an early stage of the disease, implement interventions in a timely manner, change the disease process, and reduce the mortality rate of patients.
[0053] 2. This multi-modal based medical algorithm model construction system constructs a prediction model for ICU patients by integrating multi-modal data such as chest radiograph images, clinical indicators, and biomarkers, overcomes the one-sidedness of traditional single data source evaluation, accurately predicts the mortality rate and ventilator requirements, provides a reliable basis for intensive care, improves the monitoring accuracy of high-risk patients, and optimizes the allocation of medical resources.
[0054] 3. This multi-modal based medical algorithm model construction system, the deep learning diagnosis model based on multi-modal fusion constructed by the present invention, organically combines imaging, laboratory tests, and clinical data, breaks through the limitations of traditional diagnosis models, comprehensively captures the patient's condition information, realizes accurate and comprehensive diagnosis of infectious pulmonary diseases, and reduces the risks of misdiagnosis and missed diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative efforts.
[0056] Figure 1 It is the overall system block diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0057] In the following text, only some exemplary embodiments are briefly described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the accompanying drawings and the description are considered to be exemplary in nature rather than restrictive.
[0058] In the description of the embodiments of the present invention, it should be understood that the orientation or positional relationship indicated by terms such as "length", "vertical", "horizontal", "top", "bottom", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the embodiments of the present invention.
[0059] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, the meaning of "a plurality" is two or more unless otherwise specifically defined.
[0060] In the embodiments of the present invention, unless otherwise clearly specified and limited, terms such as "installed", "connected", "connected to", "fixed", etc. should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection, an electrical connection, or a communication connection; it can be directly connected, or indirectly connected through an intermediate medium, and can be the communication inside two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.
[0061] The following disclosure provides many different embodiments or examples for implementing different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the embodiments of the present invention. In addition, the embodiments of the present invention may repeat reference numerals and / or reference letters in different examples. This repetition is for the purpose of simplification and clarity and does not itself indicate the relationship between the various embodiments and / or settings discussed.
[0062] To better understand the purpose, structure, and function of the present invention, the following further describes the multimodal-based medical algorithm model construction system of the present invention in detail with reference to the accompanying drawings.
[0063] As Figure 1 shown, the multimodal-based medical algorithm model construction system of the present invention includes
[0064] The multi-modal data acquisition and preprocessing module is responsible for collecting data related to infectious pulmonary diseases from multiple data sources, including but not limited to clinical indicators, laboratory test data, imaging data, and treatment responses. For different modalities of data, such as text-based medical record information, numerical test indicators, and imaging data, a standardized process is used for acquisition. For example, laboratory test data is uniformly converted to international standard units, and imaging data is stored in DICOM format with detailed metadata attached, covering data sources, collection times, disease types, etc. The collected data is cleaned and standardized to remove missing values, duplicate data, and outliers; ensuring data consistency, integrity, and accuracy.
[0065] The specific method for the multi-modal data acquisition and preprocessing module to process data is as follows: According to the degree of data structuring, a relational database (such as MySQL or PostgreSQL) is selected to store structured data, and a NoSQL database (such as MongoDB) is used to store unstructured data. A distributed storage structure is used to ensure data security, efficient management, and simultaneous access by multiple users and multiple hospitals. At the same time, a strict permission management and privacy protection mechanism is set up.
[0066] The multi-modal feature extraction module: performs in-depth feature extraction on different modalities of data. The multi-modal feature extraction module includes: an imaging feature extraction sub-module, a text feature extraction sub-module, a structured data processing sub-module, and a large language model-assisted feature extraction sub-module;
[0067] The specific method of the imaging feature extraction sub-module is as follows:
[0068] For CT and X-ray imaging data, first perform normalization processing to adjust the pixel values to the standard range and reduce the differences between images:
[0069]
[0070] where I is the original imaging data, I min and I max are the minimum and maximum values of the imaging data respectively;
[0071] At the same time, data augmentation techniques are used, such as rotation, flipping, cropping, etc. operations to expand the dataset and improve the generalization ability of the model:
[0072] The image can be randomly rotated by a certain angle: I rot = rotate(I,θ), and perform horizontal flipping I flip = flip(I) and random cropping:
[0073] I crop= crop(I, crop_size);
[0074] To capture deeper features from the preprocessed image, we adopted a pre-trained EfficientNet model for feature extraction through transfer learning:
[0075] Fimg = FEffNet(Inorm);
[0076] where Fimg represents the extracted high-dimensional image features. Subsequently, these features are further optimized through principal component analysis (PCA): F img—reduced = PCA(F img );
[0077] The purpose is to reduce the feature dimension while retaining key image information, thereby reducing the computational cost and improving the training efficiency of the model. The dimension is reduced by finding the eigenvectors of the feature covariance matrix:
[0078] F img—reduced = W T F img
[0079] Here, W T is the eigenvector matrix of PCA. By extracting the eigenvectors of the feature matrix, the high-dimensional data is compressed into a lower-dimensional representation form.
[0080] The multi-modal fusion module uses the self-attention mechanism (Transformer) to integrate different modal features, including image, text, structured, time series, and associated features extracted by the large language model;
[0081] The specific method of the multi-modal fusion module is as follows:
[0082] Integrate the image features, text features, structured data features, and associated features extracted by the large language model into a feature matrix:
[0083] F concat = [F img—reduced , F txt—fixed , F str—selected , F afd—lstm , F LLM
[0084] The process of feature fusion depends on the self-attention mechanism (Self-Attention); this mechanism determines which modal features are most important for the current task by calculating the correlation weights between each feature and other features. The specific operations are as follows:
[0085] Generate query vectors, key vectors, and value vectors: Q = W Q Fconcat , K = W K F concat , V = W V F concat The query vector is used to represent how the features of the current modality are related to other modalities. The key vector is used to capture the feature representations of other modalities, and the value vector represents the weights and information of the features themselves;
[0086] Calculate the attention weights: The dot product operation between the query vector and the key vector reflects the correlation between the features of different modalities. The dot product result is normalized by the Softmax function to obtain the weights:
[0087]
[0088] Then, use the attention weights A to weight the value vector V to obtain the fused feature representation: Z = AV
[0089] Multi-Head Attention Mechanism: To improve the model's feature interaction ability between different modalities, the multi-head attention mechanism is used. Each head processes the feature correlation from a different perspective, and the outputs of all heads are integrated:
[0090] MultiHead(Q, K, V) = [Z1, Z2,..., Z h W O
[0091] Here, Z h represents the output of each attention head, h is the number of attention heads, and W O is a linear transformation matrix used to integrate the outputs of all heads into a matrix;
[0092] Residual Connection and Layer Normalization: To enhance the stability and training efficiency of feature learning, residual connection and layer normalization are used to process the fused features. The residual connection superimposes the original features and the self-attention output:
[0093] H res = F concat + Z
[0094] Then, the features after the residual connection are normalized by layer normalization: H out = LayerNorm(H res )
[0095] This can effectively prevent the problem of gradient vanishing and accelerate the model training;
[0096] Decoder and Decision Layer: After multi-modal fusion, the generated feature matrix is input into the fully connected layer to output the task prediction result.
[0097] The multi-task learning and decision-making module constructs a multi-task learning framework to parallelly execute two core tasks: the risk prediction of infectious pulmonary diseases and the decision support based on this prediction.
[0098] The text feature extraction sub-module includes medical record records and laboratory examination text results. During preprocessing, stop words and punctuation marks are removed, and the BioBERT model pre-trained on biomedical corpora is used for deep semantic encoding to extract semantic features: F txt = FBioBERT(Tokens). Then, an average pooling operation is introduced to unify the text feature dimensions for easy fusion with other modalities, and the extracted semantic information is converted into a fixed-length vector representation:
[0099]
[0100] When processing structured data and feature extraction, the structured data processing sub-module deals with immune status data (such as inflammatory factor levels, flow cytometry analysis results) and microbial detection results (including metagenomic sequencing (mNGS), targeted gene sequencing (tNGS), microbial culture, and multiplex PCR, etc.). To ensure that data with different dimensions can be compared on the same scale, we standardized these data:
[0101] X std = (X - μ X ) / σ X ;
[0102] Among them, X represents the original data, μ X represents the mean value, and σ X represents the standard deviation. This standardization process effectively eliminates the dimensional differences between different features, enabling all data to be compared and modeled on a consistent basis;
[0103] For the standardized data, we used a variety of advanced machine learning methods for feature extraction. Through algorithms such as the random forest model and support vector machine, we can identify the most important features:
[0104] F ml = MLFeatureExtraction(X std )
[0105] These features are further input into the random forest model to extract and integrate the important features of the structured data for output:
[0106] F str—selected = selectToPK(F RF (F ml ), k).
[0107] The large language model-assisted feature extraction sub-module: Utilize the powerful semantic understanding and reasoning capabilities of the large language model to deeply mine the medical record text information and laboratory test results, extract symptoms, medical history, and medication information from the medical record text, and combine with external medical knowledge bases to associate implicit information and generate comprehensive features. For laboratory data, associate with the medical record text to capture the pathological change trend, and finally fuse the features extracted by the two parts of the large model to provide high-dimensional and semantically rich feature representations for model training.
[0108] In the multi-task learning and decision-making module, the risk prediction task is based on the fused multi-modal features, combined with imaging lesions, clinical symptoms, and laboratory data, and determines the high-risk state of severe pneumonia in patients, whether ICU intervention is required, and the necessity of mechanical ventilation through a classifier; the decision support task is based on the risk prediction results, uses the large language model to parse its semantics, and combines with the expert knowledge graph to provide intuitive and interpretable decision-making suggestions to assist doctors in adjusting medications, allocating resources, and optimizing treatment plans.
[0109] The specific implementation directions are as follows:
[0110] I. Implementation Environment Setup
[0111] 1. Hardware configuration: Equip with a high-performance server cluster, including sufficient CPU cores, a large amount of memory, and high-speed storage devices to meet the needs of multi-modal data storage, processing, and model training. At the same time, configure professional GPU acceleration cards to accelerate the deep learning model training process and ensure the system operation efficiency. For example, select NVIDIA series GPUs and build a parallel computing environment in combination with the server motherboard to ensure the data processing speed.
[0112] 2. Software environment: Install an operating system, such as a Linux distribution, build a stable database management system, and select a combination of MySQL or PostgreSQL and MongoDB according to the above description to ensure data storage and management. Deploy a deep learning framework, such as TensorFlow or PyTorch, for building, training, and optimizing multi-modal algorithm models. At the same time, install necessary auxiliary tool software for preprocessing, visualization, etc., such as OpenCV for image processing, NLTK for text processing, etc., to build a complete software development toolchain.
[0113] II. Data Collection Implementation
[0114] 1. Interface with the information systems of partner hospitals: Establish secure and efficient data interfaces with the medical information systems (such as HIS, LIS, PACS, etc.) of partner units like the Second Affiliated Hospital of Guangzhou Medical University to ensure real-time and accurate acquisition of various data of patients with infectious pulmonary diseases. Adopt standardized data transmission protocols, such as the HL7 protocol or a custom interface protocol that complies with medical data security specifications, to guarantee data integrity and consistency, and prevent data leakage and transmission errors.
[0115] 2. Multimodal data acquisition process: According to the predefined data acquisition system, collect corresponding data from different systems at each link of the patient's medical treatment. At the outpatient stage, collect the patient's basic information, symptom description, and preliminary examination results; at the emergency stage, supplement emergency detection data and preliminary imaging examination results; during the hospitalization period, continuously collect detailed laboratory examinations, daily condition monitoring data, and treatment process feedback; at the ICU stage, focus on obtaining key data such as vital sign monitoring and advanced imaging reexamination results. For the collected image data, convert the format and store it in real time through a DICOM gateway, and after cleaning and verification, the text and numerical data are stored in the corresponding database tables to ensure data quality.
[0116] III. Model construction and training implementation
[0117] 1. Multimodal feature extraction: According to the system design, call data of each modality from the database to the feature extraction module. The image data is normalized and data-augmented, then input into the EfficientNet model to extract features, and PCA is used for dimensionality reduction; the text data removes stop words and punctuation, is encoded by BioBERT and averaged for pooling; the structured data is standardized and feature screening is performed using machine learning algorithms; the time series data is processed by AFD and sent to LSTM for learning; the large language model docks with an external knowledge base to deeply mine the correlation features of medical record texts and laboratory results, and the features of each modality are gathered for fusion.
[0118] 2. Multimodal fusion and model training: Input the extracted multimodal features into a fusion module based on Transformer, calculate the feature weights and fuse the features according to the self-attention mechanism, and construct a multi-task learning framework with the risk prediction and decision support of infectious pulmonary diseases as the task objectives. Adopt a total loss function composed of cross-entropy loss and regression loss, use the Adam optimizer, and iteratively train the model based on a large amount of clinical sample data, continuously adjust the model parameters, and improve performance indicators such as the accuracy, recall rate, and precision rate of the model until the model converges and meets the requirements of clinical practical accuracy.
[0119] It is understood that the present invention is described by way of some embodiments. Those skilled in the art will appreciate that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Additionally, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application belong to the scope protected by the present invention.
Claims
1. A multimodal medical algorithm model construction system, characterized by: include: The multimodal data collection and preprocessing module is responsible for collecting data related to infectious lung diseases from multiple data sources, including but not limited to clinical indicators, laboratory test data, imaging data, and treatment response. Different modal data are collected using standardized processes. The collected data are cleaned and standardized to remove missing values, duplicate data, and outliers. Multimodal feature extraction module: performs deep feature extraction on different modal data. The multimodal feature extraction module includes: an imaging feature extraction submodule, a text feature extraction submodule, a structured data processing submodule and a large language model auxiliary feature extraction submodule; The multimodal fusion module uses the self-attention mechanism (Transformer) to integrate different modal features, including images, text, structure, time series, and related features extracted by large language models; The multi-task learning and decision-making module builds a multi-task learning framework to perform two core tasks in parallel: infectious lung disease risk prediction and decision support based on the prediction.
2. The multimodal medical algorithm model construction system according to claim 1, characterized in that: The specific method of multimodal data acquisition and preprocessing module is as follows: according to the degree of data structuring, relational database is selected to store structured data, and NoSQL database is used to store unstructured data. Data security, efficient management and simultaneous access by multiple users and multiple hospitals are ensured through distributed storage structure, and strict permission management and privacy protection mechanisms are set up at the same time.
3. The multimodal medical algorithm model construction system according to claim x, characterized in that: The specific method of the imaging feature extraction submodule is as follows: For CT and X-ray image data, normalization is first performed to adjust the pixel values to the standard range to reduce the differences between images: Where I is the original image data, I min and I max are the minimum and maximum values of the image data respectively; At the same time, data enhancement techniques such as rotation, flipping, and cropping are used to expand the data set and improve the generalization ability of the model: The image can be randomly rotated by a certain angle: I rot =rotate(I,θ), and flip horizontally I flip = flip(I) and random cropping: I crop =crop(I,crop_size); In order to capture deeper features from the preprocessed images, we used the pre-trained EfficientNet model to extract features through transfer learning: F img =F EffNet (I norm ); where F img represents the extracted high-dimensional image features. Subsequently, these features are further optimized by principal component analysis (PCA): img —reduced=PCA(F img ); The purpose is to reduce the feature dimension while retaining key image information, thereby reducing computational costs and improving model training efficiency. Dimensionality reduction is achieved by finding the eigenvectors of the feature covariance matrix: F img—reduced =WTF img Here, W T is the eigenvector matrix of PCA. By extracting the eigenvectors of the feature matrix, the high-dimensional data is compressed into a lower-dimensional expression.
4. The multimodal medical algorithm model construction system according to claim 1, characterized in that: The text feature extraction submodule includes medical records and laboratory test text results. Stop words and punctuation marks are removed during preprocessing. The BioBERT model pre-trained with biomedical corpus is used for deep semantic encoding to extract semantic features: F txt =F BioBERT (Tokens), and then introduce the average pooling operation to unify the text feature dimension, which is convenient for fusion with other modalities and converts the extracted semantic information into a vector representation of fixed length:
5. The multimodal medical algorithm model construction system according to claim 1, characterized in that: The structured data processing submodule processes immune status data and microbial test results when processing structured data and feature extraction. In order to ensure that data of different dimensions are compared on the same scale, we standardize these data: X std =(X-μ X ) / s X ; Among them, X represents the original data, μ X represents the mean, σ X Represents the standard deviation. This standardization process effectively eliminates the dimensional differences between different features, allowing all data to be compared and modeled on a consistent basis. We use a variety of advanced machine learning methods to extract features from the standardized data. Through algorithms such as random forest models and support vector machines, we can identify the most important features: F ml =MLFeatureExtraction(X std ) These features are further input into the random forest model to extract the important features of the structured data for integration and output: Fs tr—selected =selectToPK(F RF (F ml ),k)。 6. The multimodal medical algorithm model construction system according to claim 1, characterized in that: The large language model-assisted feature extraction submodule utilizes the powerful semantic understanding and reasoning capabilities of the large language model to deeply mine medical record text information and laboratory test results, extract symptoms, medical history, and medication information from the medical record text, and associate implicit information with an external medical knowledge base to generate comprehensive features; for laboratory data, associate medical record text to capture pathological change trends, and finally fuse the two parts of the large model to extract features to provide high-dimensional, semantically rich feature representation for model training.
7. The multimodal medical algorithm model construction system according to claim 1, characterized in that: The specific method of the multimodal fusion module is as follows: Integrate image features, text features, structured data features, and related features extracted by the large language model into a feature matrix: F concat =[F img—reduced ,F txt—fixed ,F str—selected ,F afd—lstm ,F LLM ] The process of feature fusion relies on the self-attention mechanism; this mechanism determines which modal features are most important for the current task by calculating the correlation weight between each feature and other features. The specific operation is as follows: Generate query vector, key vector and value vector: Q = W Q F concat , K=W K F concat 、V=W VFconcat The query vector is used to represent how the features of the current modality are related to other modalities, the key vector is used to capture the feature representations of other modalities, and the value vector represents the weight and information of the features themselves; Calculate attention weights: The dot product operation between the query vector and the key vector reflects the correlation between features of different modalities. The dot product result is normalized by the Softmax function to obtain the weights: Then use the attention weight A weighted value vector V to get the fused feature representation: Z = AV Multi-head attention mechanism: In order to improve the model's feature interaction ability between different modalities, a multi-head attention mechanism is used. Each head processes feature correlation from a different perspective and integrates the outputs of all heads: MultiHead(Q,K,V)=[Z1,Z2,…,Z h ]IN O Here, Z h represents the output of each attention head, h is the number of attention heads, and W O is the linear transformation matrix used to integrate the outputs of all heads into one matrix; Residual connection and layer normalization: In order to enhance the stability of feature learning and training efficiency, residual connection and layer normalization are used to process the fused features. Residual connection superimposes the original features with the self-attention output: H res =F concat +Z Then, the features after residual connection are normalized by layer normalization: H out =LayerNorm(H res ) This can effectively prevent the gradient vanishing problem and speed up model training; Decoder and decision layer: After multimodal fusion, the generated feature matrix will be input into the fully connected layer to output the task prediction result.
8. The multimodal medical algorithm model construction system according to claim 1, characterized in that: The risk prediction task in the multi-task learning and decision-making module is based on the fusion of multimodal features, combined with imaging lesions, clinical symptoms, and laboratory data, and uses a classifier to determine the patient's high-risk status for severe pneumonia, whether ICU intervention is required, and the necessity of mechanical ventilation; the decision support task is based on the risk prediction results, with the help of a large language model for semantic analysis, combined with expert knowledge graphs, to provide intuitive and explainable decision-making suggestions to assist doctors in adjusting medication, allocating resources, and optimizing treatment plans.
Citation Information
Cited By
Disease prediction and auxiliary diagnosis system construction method and system based on multi-modal large model
CN120565123A
Severe patient acute lung injury assessment system and method based on machine learning
CN120747015A
