Medical data depth algorithm model construction method based on multiple modes
Through multimodal fusion model and federated learning verification, individual differences and accuracy problems in the prediction of severe pneumonia are solved, and high-precision disease prediction and data privacy protection are achieved.
Patent Information
- Application Number
- CN202510388893.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-31
- Publication Date
- 2025-08-08
AI Technical Summary
The prior art has problems such as large individual differences and insufficient prediction accuracy in predicting severe pneumonia, especially in the lack of effective methods in multimodal data fusion.
A multimodal fusion model is adopted, combining image data, text data, structured data and time series data, feature extraction and fusion is performed through adaptive Fourier decomposition and self-attention mechanism, model training and evaluation is used for model training and evaluation, and external verification is combined with federated learning to ensure data privacy and security.
It improves the accuracy and robustness of the prediction of severe pneumonia, ensuring the application effect of the model in a diverse medical environment and data privacy security.
Smart Images
Figure CN120452815A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep algorithm model construction, and in particular to a method for constructing a deep algorithm model of medical data based on multimodality. Background Art
[0002] A new method of adaptive Fourier decomposition model (AFD) is used to predict epidemic trends. Currently, this method has been applied to clinical cohorts of severe avian influenza patients to predict the relationship between the severity of the disease and cytokine levels in severe patients. AFD technology can effectively capture and analyze the complex dynamic characteristics of the disease progression, thereby providing more accurate predictions and assessments. By performing spectral analysis on the data, the model can identify specific frequency components that are highly correlated with the severity of the disease. These frequency components reflect the fluctuation patterns and change patterns of cytokines, which help to reveal the role and mechanism of cytokines in the progression of pneumonia. Secondly, the model can dynamically adapt to the individual differences of different patients through adaptive adjustment algorithms, realize personalized disease assessment and thus improve the accuracy of predictions. The adaptive Fourier decomposition model (AFD) constructs a multimodal fusion model by combining other modules to predict severe pneumonia, and the accuracy of the model is determined by deep calculations. Summary of the Invention
[0003] The purpose of the present invention is to propose a method for constructing a deep algorithm model of medical data based on multimodality to solve one or more technical problems existing in the prior art and at least provide a beneficial option or create conditions.
[0004] A method for constructing a deep algorithm model for multimodal medical data, which fuses multimodal medical data according to a multimodal fusion model, obtains medical data processed by the multimodal fusion model, performs deep calculations on the output results of the multimodal fusion model combined with the medical data, and determines the diagnostic accuracy of the output of the multimodal fusion model based on the calculation results.
[0005] Furthermore, the medical data deep algorithm model is based on a multimodal fusion model. The method for constructing the multimodal fusion model is:
[0006] S1: Image Data Feature Extraction: The image data includes CT and X-ray image data. Key image features of lung infection are extracted and preprocessed. The preprocessing includes normalization and data enhancement. The normalization process is to normalize the original image pixel values to the range of [0, 1] to eliminate brightness differences between different images. Where is the original image data, I min and I max are the minimum and maximum values of the image data, respectively.
[0007] S2: Text data feature extraction: The text data includes medical records and laboratory test reports. Word segmentation is used to convert the medical record text into a word sequence to facilitate subsequent feature extraction: Tokens = tokenize(Text), remove stop words and punctuation Tokens = remove_stopwords(Tokens), Tokens = remove_punctuation(Tokens). The pre-trained BioBERT model is used to encode the pre-processed text and extract deep semantic features: F txt =F BioBERT The BioBERT model is pre-trained specifically for biomedical text and can effectively capture the complex semantic relationships in medical text. To represent these features as fixed-length vectors for easy fusion with other modal features, we use an average pooling method to output the text features:
[0008] S3. Structured data processing and feature extraction: The structured data includes immune status and microbial test results. The structured data is standardized: X std =(X-μ X ) / σ X ;
[0009] Among them, X represents the original data, μ X represents the mean, σ X Represents the standard deviation. The standardization process eliminates the dimensional differences between different indicators, so that each feature can be compared on the same scale.
[0010] S4. Time series data processing and feature extraction using adaptive Fourier decomposition, converting structured data into time series form, with daily statistical results as time series input: X time-series = DailyStatistics(X), AFD captures the main features of the signal by recursively decomposing the signal into a set of non-orthogonal basis functions. The AFD is adaptive Fourier decomposition, and each basis function is adaptively selected according to the specific spectral characteristics of the signal. AFD first represents the normalized time series data as a signal f(t), then selects the initial basis function g0(t) that best matches the signal characteristics, calculates the projection of the signal on the basis function 〈f,g0〉, and calculates the residual signal: r1(t)=f(t)-〈f,g0〉g0(t). This process is repeated to select a new basis function g k (t), calculate the projection and residual until the stopping condition is met (for example, the residual signal energy is lower than the threshold): r k+1 (t) = r k (t)-〈r k ,gk 〉g k (t) Finally, the projection results of all basis functions are added together to reconstruct the original signal:
[0011] AFD extracts the main spectral features of the signal, reflecting the hidden periodic and non-periodic components in the signal. The features extracted by AFD are further input into the long short-term memory network (LSTM) model to learn and predict time series features: F afd =AFD(X time-series ), F afd-lstm =LSTM(F afd ).
[0012] Furthermore, the bookstore in the multimodal fusion model is subjected to multimodal fusion: image data features, text data features, time series, structured data, and time series data are integrated to fully utilize the complementarity of multi-source information and improve the overall performance of the task. By fusing data from multiple modalities, the system can capture more context and features, improving the accuracy and robustness of tasks such as classification and detection.
[0013] The self-attention mechanism is used to integrate image features, text features, and structured data features into a feature matrix:
[0014] F concat =[F img-reduced ,F txt-fixed ,F str-selected ,F afd-lstm ];
[0015] Multi-head Attention is used to calculate the correlation between features and calculate the query Q, key K, and value V:
[0016] Q=W Q F concat 、K=W K F concat 、V=W V F concat Then, calculate the attention weight Output:
[0017]
[0018] The self-attention mechanism effectively integrates multimodal features by calculating the correlation between features, thereby enhancing the predictive ability of the model.
[0019] Furthermore, the accuracy of medical data is determined by constructing a multimodal fusion model, and the prediction results output by the multimodal fusion model are combined with the medical data for calculation. The calculation method is: model training and evaluation
[0020] The medical data is: sample characteristic data, including nucleic acid extraction concentration, amplification efficiency, host ratio and the number of normalized sequences detected by species, based on which a random forest regression model is trained;
[0021] A multi-task learning framework is used to output the pathogen category and predict the probability of severe mortality. The total loss function of multi-task learning includes cross entropy loss and regression loss:
[0022] L=αL CE +βL Reg Among them, L CE Represents the cross entropy loss, which is used for classification tasks: L Reg Represents regression loss, used for positioning tasks: We use the Adam optimizer for training. The Adam optimizer adjusts parameter updates through adaptive learning rates, improving the convergence speed and stability of the model. The parameter update formula is as follows:
[0023] Among them, m t and v t are the first-order moment estimate and the second-order moment estimate of the gradient, respectively. The update rules are as follows:
[0024] Among them, g t is the gradient, β1 and β1 are momentum terms, and η is the learning rate;
[0025] Finally, the model is comprehensively evaluated by accuracy, recall, precision, and F1 score, which comprehensively reflect the classification and prediction performance of the model.
[0026] Furthermore, the multi-task learning output layer integrates multimodal fusion and pathogen detection model output to filter the data through the pathogen detection pollution noise elimination model, filtering out data sources that pollute the model. Then, through the multimodal fusion model, a multi-task model is constructed to identify the pathogens of severe pneumonia and predict the risk of severe disease aggravation.
[0027] Optimally, we designed a federated learning-based validation system that combines external validation with federated learning algorithms to conduct real-world clinical data validation, ensuring data privacy and security. In this system, external validation is performed first, and then the model is collaboratively trained within the federated learning architecture to fully verify and improve model performance. External validation utilizes data not previously used in model development to evaluate the model's performance on new data, focusing on the consistency of the model across different time periods, regions, or populations. External validation includes:
[0028] 1. Timeframe Validation: Validate the model's performance using data from the same cohort as the model development process, but over a different timeframe. This typically involves collecting data after model development is complete for timeframe validation.
[0029] 2. Spatial validation: Validating the model's performance in data from other centers or countries, which may use different inclusion / exclusion criteria or measurement methods than the development cohort. Spatial validation helps test the transferability and generalizability of the model.
[0030] 3. Domain Validation: Validate the model in different clinical scenarios. For example, based on hospital data, use community resident data to test the performance of the model in different populations.
[0031] After external validation, we leveraged a federated learning architecture, allowing multiple participants to collaboratively train the model while maintaining data locality and ensuring data privacy and security. Leveraging data from multiple centers, we comprehensively evaluated the model's performance in real-world clinical settings, including metrics such as accuracy, sensitivity, and specificity. We collected validation results from each center, analyzed the model's performance in various medical scenarios, and continuously improved its robustness and practicality. Cross-validation was used to ensure the model's generalizability.
[0032] Through a federated learning-style external multi-center validation system, we are able to fully verify the reliability and versatility of the model. This approach not only protects data privacy but also, through multi-center validation, ensures the model's effectiveness in diverse medical settings, enabling it to better serve clinical decision-making and patient treatment.
[0033] The beneficial effects of the present invention are: obtaining a high-quality multi-dimensional clinical cohort data system covering thousands of cases of severe pneumonia with different causes; determining the high prediction accuracy of the multimodal fusion model based on model training pre-evaluation, realizing real-world multi-center clinical verification and practice, and determining the authenticity of the data source and the accuracy of the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The above and other features of the present invention will become more apparent through a detailed description of the embodiments shown in conjunction with the accompanying drawings. The same reference numerals in the drawings of the present invention represent the same or similar elements. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0035] In the picture:
[0036] Figure 1 A flowchart of a method for building a deep algorithm model based on multimodal medical data. DETAILED DESCRIPTION
[0037] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0038] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.
[0039] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand the advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.
[0040] The purpose of the present invention is to propose a method for constructing a deep algorithm model of medical data based on multimodality to solve one or more technical problems existing in the prior art and at least provide a beneficial option or create conditions.
[0041] A method for constructing a deep algorithm model for multimodal medical data, which fuses multimodal medical data according to a multimodal fusion model, obtains medical data processed by the multimodal fusion model, performs deep calculations on the output results of the multimodal fusion model combined with the medical data, and determines the diagnostic accuracy of the output of the multimodal fusion model based on the calculation results.
[0042] Furthermore, the medical data deep algorithm model is based on a multimodal fusion model. The method for constructing the multimodal fusion model is:
[0043] S1: Image Data Feature Extraction: The image data includes CT and X-ray image data. Key image features of lung infection are extracted and preprocessed. The preprocessing includes normalization and data enhancement. The normalization process is to normalize the original image pixel values to the range of [0, 1] to eliminate brightness differences between different images. Where is the original image data, I min and I maxare the minimum and maximum values of the image data, respectively.
[0044] S2: Text data feature extraction: The text data includes medical records and laboratory test reports. Word segmentation is used to convert the medical record text into a word sequence to facilitate subsequent feature extraction: Tokens = tokenize(Text), remove stop words and punctuation Tokens = remove_stopwords(Tokens), Tokens = remove_punctuation(Tokens). The pre-trained BioBERT model is used to encode the pre-processed text and extract deep semantic features: F txt =F BioBERT The BioBERT model is pre-trained specifically for biomedical text and can effectively capture the complex semantic relationships in medical text. To represent these features as fixed-length vectors for easy fusion with other modal features, we use an average pooling method to output the text features:
[0045] S3. Structured data processing and feature extraction: The structured data includes immune status and microbial test results. The structured data is standardized: X std =(X-μ X ) / σ X ;
[0046] Among them, X represents the original data, μ X represents the mean, σ X Represents the standard deviation. The standardization process eliminates the dimensional differences between different indicators, so that each feature can be compared on the same scale.
[0047] S4. Time series data processing and feature extraction using adaptive Fourier decomposition, converting structured data into time series form, with daily statistical results as time series input: X time-series = DailyStatistics(X), AFD captures the main features of the signal by recursively decomposing the signal into a set of non-orthogonal basis functions. The AFD is adaptive Fourier decomposition, and each basis function is adaptively selected according to the specific spectral characteristics of the signal. AFD first represents the normalized time series data as a signal f(t), then selects the initial basis function g0(t) that best matches the signal characteristics, calculates the projection of the signal on the basis function 〈f,g0〉, and calculates the residual signal: r1(t)=f(t)-〈f,g0〉g0(t). This process is repeated to select a new basis function g k (t), calculate the projection and residual until the stopping condition is met (for example, the residual signal energy is lower than the threshold): r k+1(t) = r k (t)-〈r k ,g k 〉g k (t) Finally, the projection results of all basis functions are added together to reconstruct the original signal:
[0048] AFD extracts the main spectral features of the signal, reflecting the hidden periodic and non-periodic components in the signal. The features extracted by AFD are further input into the long short-term memory network (LSTM) model to learn and predict time series features: F afd =AFD(X time-series ), F afd-lstm =LSTM(F afd ).
[0049] Furthermore, the bookstore in the multimodal fusion model is subjected to multimodal fusion: image data features, text data features, time series, structured data, and time series data are integrated to fully utilize the complementarity of multi-source information and improve the overall performance of the task. By fusing data from multiple modalities, the system can capture more context and features, improving the accuracy and robustness of tasks such as classification and detection.
[0050] The self-attention mechanism is used to integrate image features, text features, and structured data features into a feature matrix:
[0051] F concat =[F img-reduced ,F txt-fixed ,F str-selected ,F afd-lstm ];
[0052] Multi-head Attention is used to calculate the correlation between features and calculate the query Q, key K, and value V:
[0053] Q=W Q F concat 、K=W K F concat 、V=W V F concat Then, calculate the attention weight Output:
[0054]
[0055] The self-attention mechanism effectively integrates multimodal features by calculating the correlation between features, thereby enhancing the predictive ability of the model.
[0056] Furthermore, the accuracy of medical data is determined by constructing a multimodal fusion model, and the prediction results output by the multimodal fusion model are combined with the medical data for calculation. The calculation method is: model training and evaluation
[0057] The medical data is: sample characteristic data, including nucleic acid extraction concentration, amplification efficiency, host ratio and the number of normalized sequences detected by species, based on which a random forest regression model is trained;
[0058] A multi-task learning framework is used to output the pathogen category and predict the probability of severe mortality. The total loss function of multi-task learning includes cross entropy loss and regression loss:
[0059] L=αL CE +βL Reg Among them, L CE Represents the cross entropy loss, which is used for classification tasks: L Reg Represents regression loss, used for positioning tasks: We use the Adam optimizer for training. The Adam optimizer adjusts parameter updates through adaptive learning rates, improving the convergence speed and stability of the model. The parameter update formula is as follows:
[0060] Among them, m t and v t are the first-order moment estimate and the second-order moment estimate of the gradient, respectively. The update rules are as follows:
[0061] Among them, g t is the gradient, β1 and β1 are momentum terms, and η is the learning rate;
[0062] Finally, the model is comprehensively evaluated by accuracy, recall, precision, and F1 score, which comprehensively reflect the classification and prediction performance of the model.
[0063] Furthermore, the multi-task learning output layer integrates multimodal fusion and pathogen detection model output to filter the data through the pathogen detection pollution noise elimination model, filtering out data sources that pollute the model. Then, through the multimodal fusion model, a multi-task model is constructed to identify the pathogens of severe pneumonia and predict the risk of severe disease aggravation.
[0064] Optimally, we designed a federated learning-based validation system that combines external validation with federated learning algorithms to conduct real-world clinical data validation, ensuring data privacy and security. In this system, external validation is performed first, and then the model is collaboratively trained within the federated learning architecture to fully verify and improve model performance. External validation utilizes data not previously used in model development to evaluate the model's performance on new data, focusing on the consistency of the model across different time periods, regions, or populations. External validation includes:
[0065] 1. Timeframe Validation: Validate the model's performance using data from the same cohort as the model development process, but over a different timeframe. This typically involves collecting data after model development is complete for timeframe validation.
[0066] 2. Spatial validation: Validating the model's performance in data from other centers or countries, which may use different inclusion / exclusion criteria or measurement methods than the development cohort. Spatial validation helps test the transferability and generalizability of the model.
[0067] 3. Domain Validation: Validate the model in different clinical scenarios. For example, based on hospital data, use community resident data to test the performance of the model in different populations.
[0068] After external validation, we leveraged a federated learning architecture, allowing multiple participants to collaboratively train the model while maintaining data locality and ensuring data privacy and security. Leveraging data from multiple centers, we comprehensively evaluated the model's performance in real-world clinical settings, including metrics such as accuracy, sensitivity, and specificity. We collected validation results from each center, analyzed the model's performance in various medical scenarios, and continuously improved its robustness and practicality. Cross-validation was used to ensure the model's generalizability.
[0069] Through a federated learning-style external multi-center validation system, we are able to fully verify the reliability and versatility of the model. This approach not only protects data privacy but also, through multi-center validation, ensures the model's effectiveness in diverse medical settings, enabling it to better serve clinical decision-making and patient treatment.
[0070] Although the present invention has been described in considerable detail and with particularity with respect to several embodiments, it is not intended to limit the present invention to any of these details or embodiments or any particular embodiment, so as to effectively encompass the intended scope of the present invention. In addition, the present invention has been described above with respect to embodiments foreseen by the inventors for the purpose of providing a useful description, and those insubstantial modifications of the present invention that are not currently foreseen may still represent equivalent modifications of the present invention.
Claims
1. A method for constructing a deep algorithm model for multimodal medical data, characterized in that: The multimodal medical data is fused according to the multimodal fusion model to obtain the medical data processed by the multimodal fusion model. The output results of the multimodal fusion model are combined with the medical data for in-depth calculation, and the diagnostic accuracy of the output of the multimodal fusion model is determined based on the calculation results.
2. The method for constructing a multimodal medical data deep algorithm model according to claim 1, wherein: The medical data deep algorithm model is based on a multimodal fusion model. The method for constructing the multimodal fusion model is as follows: S1: Image Data Feature Extraction: The image data includes CT and X-ray image data. Key image features of lung infection are extracted and preprocessed. The preprocessing includes normalization and data enhancement. The normalization process is to normalize the original image pixel values to the range of [0, 1] to eliminate brightness differences between different images. Where is the original image data, I min and I max are the minimum and maximum values of the image data, respectively. S2: Text data feature extraction: The text data includes medical records and laboratory test reports. Word segmentation is used to convert the medical record text into a word sequence to facilitate subsequent feature extraction: Tokens = tokenize(Text), remove stop words and punctuation Tokens = remove_stopwords(Tokens), Tokens = remove_punctuation(Tokens). The pre-trained BioBERT model is used to encode the pre-processed text and extract deep semantic features: F txt =F BioBERT The BioBERT model is pre-trained specifically for biomedical text and can effectively capture the complex semantic relationships in medical text. To represent these features as fixed-length vectors for easy fusion with other modal features, we use an average pooling method to output the text features: S3. Structured data processing and feature extraction: The structured data includes immune status and microbial test results. The structured data is standardized: X std =(X-μ X ) / σ X ; Among them, X represents the original data, μ X represents the mean, σ X Represents the standard deviation. The standardization process eliminates the dimensional differences between different indicators, so that each feature can be compared on the same scale. S4. Time series data processing and feature extraction using adaptive Fourier decomposition, converting structured data into time series form, with daily statistical results as time series input: X time-series = DailyStatistics(X), AFD captures the main features of the signal by recursively decomposing the signal into a set of non-orthogonal basis functions. The AFD is adaptive Fourier decomposition, and each basis function is adaptively selected according to the specific spectral characteristics of the signal. AFD first represents the normalized time series data as a signal f(t), then selects the initial basis function g0(t) that best matches the signal characteristics, calculates the projection of the signal on the basis function 〈f,g0〉, and calculates the residual signal: r1(t)=f(t)-〈f,g0〉g0(t). This process is repeated to select a new basis function g k (t), calculate the projection and residual until the stopping condition is met (for example, the residual signal energy is lower than the threshold): r k+1 (t) = r k (t)-〈r k ,g k 〉g k (t) Finally, the projection results of all basis functions are added together to reconstruct the original signal: AFD extracts the main spectral features of the signal, reflecting the hidden periodic and non-periodic components in the signal. The features extracted by AFD are further input into the long short-term memory network (LSTM) model to learn and predict time series features: F afd =AFD(X time-series ), F afd-lstm =LSTM(F afd ).
3. The method for constructing a multimodal medical data deep algorithm model according to claim 2, wherein: Perform multimodal fusion on the bookstore in the multimodal fusion model: Integrate image data features, text data features, time series, structured data, and time series data to fully utilize the complementarity of multi-source information and improve the overall performance of the task. By fusing data from multiple modalities, the system can capture more context and features, improving the accuracy and robustness of tasks such as classification and detection; The self-attention mechanism is used to integrate image features, text features, and structured data features into a feature matrix: F concat =[F img-reduced ,F txt-fixed ,F str-selected ,F afd-lstm ]; Multi-head Attention is used to calculate the correlation between features and calculate the query Q, key K, and value V: Q=W Q F concat 、K=W K F concat 、V=W V F concat Then, calculate the attention weight Output: The self-attention mechanism effectively integrates multimodal features by calculating the correlation between features, thereby enhancing the predictive ability of the model.
4. The method for constructing a multimodal medical data deep algorithm model according to claim 1, wherein: The accuracy of medical data is determined by constructing multimodal fusion, and the prediction results output by the multimodal fusion model are combined with medical data for calculation. The calculation method is: model training and evaluation The medical data is: sample characteristic data, including nucleic acid extraction concentration, amplification efficiency, host ratio and the number of normalized sequences detected by species, based on which a random forest regression model is trained; A multi-task learning framework is used to output the pathogen category and predict the probability of severe mortality. The total loss function of multi-task learning includes cross entropy loss and regression loss: L=αL CE +βL Reg Among them, L CE Represents the cross entropy loss, which is used for classification tasks: L Reg Represents regression loss, used for positioning tasks: We use the Adam optimizer for training. The Adam optimizer adjusts parameter updates through adaptive learning rates, improving the convergence speed and stability of the model. The parameter update formula is as follows: Among them, m t and v t are the first-order moment estimate and the second-order moment estimate of the gradient, respectively. The update rules are as follows: m t =β1m t-1 +(1-β1)g t 、 Among them, g t is the gradient, β1 and β1 are momentum terms, and η is the learning rate; Finally, the model is comprehensively evaluated by accuracy, recall, precision, and F1 score, which comprehensively reflect the classification and prediction performance of the model.
5. The method for constructing a multimodal medical data deep algorithm model according to claim 1, wherein: The multi-task learning output layer integrates multimodal fusion and pathogen detection model outputs. The pathogen detection pollution noise elimination model filters the data and filters out the data sources that pollute the model. Then, through the multimodal fusion model, a multi-task model is constructed to identify the pathogens of severe pneumonia and predict the risk of severe disease aggravation.