Image analysis method based on double-branch multi-modal sequence learning
Through the dual branch multimodal sequence learning method, the multimodal embedding module and static and dynamic information analysis module are used to fill in missing data and fuse features, which solves the insufficient utilization of multimodal information in Alzheimer's disease image analysis The problem improves the accuracy and robustness of image analysis.
Patent Information
- Application Number
- CN202411032784.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
AI Technical Summary
The existing Alzheimer's disease imaging analysis method ignores the correlation between individual's inherent attributes and data when processing longitudinal data, resulting in misleading analysis results and it is difficult to effectively utilize multimodal information and timing evolution information.
Using the dual branch multimodal sequence learning method, by constructing multimodal embedded modules and static and dynamic information analysis modules, using learnable parameters to fill in missing data, fuse multimodal features, decompose complex image evolution information, and combine demographic statistics. Longitudinal image analysis was performed for the academic and cognitive assessment data.
It improves the accuracy and robustness of image analysis, effectively utilizes multimodal information and timing evolution information, reduces the risk of overfitting, and enhances the generalization ability and interpretability of the model.
Smart Images

Figure CN118948212A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to an image analysis method based on dual-branch multimodal sequence learning. Background Art
[0002] Alzheimer's disease (AD) is the leading cause of dementia, causing a huge social and economic burden. Alzheimer's disease imaging analysis can assist in pathological research and treatment effect evaluation, greatly improve patients' quality of life, and allow more efficient allocation of medical resources. Alzheimer's disease imaging analysis faces multiple challenges: in brain images of early Alzheimer's disease, changes in brain structure and function may be very subtle; data heterogeneity, such as inconsistent imaging data standards between different medical centers and imaging equipment; complex data processing steps, etc. These factors increase the complexity of medical imaging analysis and make Alzheimer's disease imaging analysis difficult. At the same time, in the long-term longitudinal data collection process of Alzheimer's disease subjects, individuals often interrupt follow-up due to factors such as expensive imaging examinations, physical discomfort, disease worsening, or withdrawal from the study, resulting in a large amount of missing disease data.
[0003] Existing Alzheimer's disease image analysis methods can be divided into two categories: Alzheimer's disease image analysis based on classification methods and Alzheimer's disease image analysis based on sequence learning. Alzheimer's disease image analysis methods based on classification methods usually process cross-sectional data and ignore the temporal evolution information of multimodal data. Alzheimer's disease image analysis based on sequence learning enhances the model's ability to capture image changes by modeling the temporal changes of image changes, but it is still plagued by the complexity of modeling. Some methods attempt to use ordinary differential equations (ODEs) for image analysis to obtain image analysis in the form of solving equations, but this method uses overly simplified assumptions, puts higher requirements on data integrity, and it is difficult to find feasible solutions for complex problems. There are also studies that integrate long short-term memory networks (LSTMs) with modal fusion modules, using complementary information between modalities and learning potential evolution representations to achieve medical image analysis of Alzheimer's disease.
[0004] However, this method ignores the inherent attributes of patients in the longitudinal data analysis process, such as age, education level, and gender. To accurately analyze Alzheimer's disease imaging data, it is necessary to combine the inherent information and evolutionary information of the subjects. The inherent attributes provide a basis for understanding individual susceptibility, and the evolutionary information enables the model to understand the trajectory of image changes, thereby improving the accuracy of medical image analysis. In addition, existing methods use fixed value completion, interpolation completion and other methods to complete missing data, which rely on the order of data or the continuity of time series, ignore the correlation between data and potential complex patterns, and can lead to misleading analysis results. Summary of the invention
[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides an image analysis method based on dual-branch multimodal sequence learning, which utilizes other modal data to assist in analyzing longitudinal image data and analyze the image features in longitudinal queue data, thereby solving the problem of insufficient utilization of multimodal information and time series evolution information by multimodal longitudinal queue image analysis methods.
[0006] In order to achieve the above objectives, the technical solution adopted by the present invention is: an image analysis method based on dual-branch multimodal sequence learning, comprising the following steps:
[0007] S1. Construct a multimodal embedding module and use learnable parameters to fill in the missing multimodal disease data, fuse the filled multimodal features, and obtain the encoding features of the subjects;
[0008] S2, analyzing the coding features using a dual-branch structure including a static information analysis module and a dynamic information analysis module to obtain dual-branch multimodal image analysis features;
[0009] S3, fusing the dual-branch multimodal image analysis features obtained in step S2 to obtain image features in the longitudinal queue data;
[0010] S4. Repeat steps S1 to S3, stack the image features in all longitudinal cohort data, and use the multimodal data except the padded multimodal disease data to analyze the image features in all stacked longitudinal cohort data to obtain image analysis results, thereby completing the image analysis based on dual-branch multimodal sequence learning.
[0011] The beneficial effects of the present invention are as follows: the present invention uses magnetic resonance images, demographic data and cognitive assessment scores as key inputs to analyze image changes, uses learnable parameters to fill in multimodal missing data, realizes effective use of missing data, proposes to decompose complex image evolution information, and uses a dual-branch structure of a static information analysis module and a dynamic information analysis module to analyze complex information in longitudinal image data, wherein the static information analysis module is a multi-layer perceptron, and the dynamic information analysis module adopts a classic LSTM to obtain image features of longitudinal queue data, uses other modal data to assist in analyzing longitudinal image data, and analyzes image features in longitudinal queue data. The present invention solves the problem of insufficient use of multimodal information and time series evolution information by multimodal longitudinal queue image analysis methods.
[0012] Furthermore, the step S1 includes the following steps:
[0013] S101. Build a multimodal embedding module to fill missing multimodal disease data including demographics, MRI images, and cognitive assessment data using learnable parameters.
[0014] S102, using the fully connected layer as a multimodal feature encoder, and according to the filling result obtained by the multimodal embedding module in step S101, using the multimodal feature encoder to fuse the multimodal features to obtain the encoding features of the subject.
[0015] The beneficial effect of the above further scheme is: the present invention realizes the utilization and fusion of multimodal data in the case of partial modality missing in a new method, avoids the information loss caused by data missing, makes the model better adapt to different data sets, and enhances its generalization ability.
[0016] Furthermore, the expression of the coding feature is as follows:
[0017] X encoder =E encoder (X fill )
[0018]
[0019] Among them, X encoder represents the encoding feature, E encoder () represents the multimodal feature encoder, X fill represents the multimodal data after being filled with learnable parameters, X represents the longitudinal multimodal data obtained by connecting the three modal data of demographics, MRI images, and cognitive assessment data, and X t v represents the modal data of the vth mode at the tth time point, NaN indicates that the data point does not exist here, and X' represents the learnable parameters.
[0020] The beneficial effect of the above further scheme is that the present invention performs feature selection and extraction through an encoder, retains useful information, removes irrelevant or redundant features, and improves the performance and efficiency of the model.
[0021] Furthermore, step S2 includes the following steps:
[0022] S201. In the static information analysis module, the encoding features are analyzed using a fully connected layer according to the following formula:
[0023] X static =E static (X encoder )
[0024] Among them, X static represents the static information representation obtained by the static information analysis module, X encoder represents the encoding feature, E static () represents a fully connected layer;
[0025] S202, according to the coding features, using the following formula, using the dynamic information analysis module to analyze the dynamic change features in the image evolution:
[0026] X dynamic =LSTM dynamic (X encoder )
[0027] Among them, X dynamic Representing dynamic changes in longitudinal cohort data, LSTM dynamic () indicates dynamic information analysis operation;
[0028] S203. According to the analysis results of step S201 and step S202, dual-branch multimodal image analysis features are obtained.
[0029] The beneficial effects of the above further scheme are: reducing the difficulty of complex multimodal feature extraction through multimodal information decomposition, making full use of inherent information and time series evolution information, enhancing the interpretability and robustness of the model, and reducing the risk of overfitting.
[0030] Furthermore, the step S3 is specifically as follows:
[0031] The dual-branch multimodal image analysis features obtained in step S2 are fused in a point-to-point addition manner using the following formula to obtain the image features in the longitudinal queue data:
[0032] Y sd =X static +X dynamic
[0033] Among them, Y sd represents the image features in the longitudinal cohort data, X dynamic Indicates the dynamic changes in image evolution, X static Represents the static information representation obtained by the static information analysis module.
[0034] The beneficial effect of the above further scheme is that the present invention obtains richer information representation by fusing information of different categories, captures the interactive relationship between different features, and improves model performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0036] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0037] Example
[0038] like Figure 1 As shown, the present invention provides an image analysis method based on dual-branch multimodal sequence learning, and its implementation method is as follows:
[0039] S1. Construct a multimodal embedding module, use learnable parameters to fill in the missing multimodal disease data, fuse the filled multimodal features, and obtain the encoding features of the subject. The implementation method is as follows:
[0040] S101. Build a multimodal embedding module to fill missing multimodal disease data including demographics, MRI images, and cognitive assessment data using learnable parameters.
[0041] S102, using the fully connected layer as a multimodal feature encoder, and according to the filling result obtained by the multimodal embedding module in step S101, using the multimodal feature encoder to fuse the multimodal features to obtain the encoding features of the subject;
[0042] S2. Analyze the coding features using a dual-branch structure including a static information analysis module and a dynamic information analysis module to obtain dual-branch multimodal image analysis features, and the implementation method is as follows:
[0043] S201, in a static information analysis module, using a fully connected layer to analyze coding features;
[0044] S202, analyzing the dynamic change characteristics of the image evolution using a dynamic information analysis module according to the coding characteristics;
[0045] S203: According to the analysis results of step S201 and step S202, obtain the dual-branch multi-modal image analysis feature
[0046] S3, fusing the dual-branch multimodal image analysis features obtained in step S2 to obtain image features in the longitudinal queue data, which are specifically:
[0047] The dual-branch multimodal image analysis features obtained in step S2 are fused in a point-to-point addition manner to obtain image features in the longitudinal queue data;
[0048] S4. Repeat steps S1 to S3, stack the image features in all longitudinal cohort data, and use the multimodal data except the padded multimodal disease data to analyze the image features in all stacked longitudinal cohort data to obtain image analysis results, thereby completing the image analysis based on dual-branch multimodal sequence learning.
[0049] In this embodiment, the long short-term memory network (LSTM) is a classic time series model. By introducing a gating mechanism, the gradient vanishing problem of the traditional recurrent neural network is effectively alleviated. It can effectively process input sequences of different lengths and capture long-term dependencies. Therefore, in the embodiment of the present invention, the time series model LSTM is also used as a baseline model to realize Alzheimer's disease image analysis.
[0050] In this embodiment, the image analysis method provided by the present invention is as follows: constructing a multimodal embedding module (the module is a learnable parameter), using learnable parameters (learnable parameters refer to parameters that are optimized by a learning algorithm during the training process of the model. This part of the learnable parameter filling operation is defined as a multimodal embedding module, which has no essential difference from other modules composed of linear layers and convolutional layers) to fill in missing multimodal disease data (including demographics, magnetic resonance imaging, and cognitive assessment data) to achieve efficient use of existing data; constructing an information classification image analyzer, using a dual-branch structure of a static information analysis module and a dynamic information analysis module to model the inherent attributes and evolution information of the subject respectively, and decompose complex longitudinal image information; integrating the inherent attributes and the decomposition representation of the image dynamics, and analyzing the image features in the longitudinal cohort data; based on the image analysis model based on dual-branch multimodal sequence learning (the construction process of the model is: the construction process of steps S1 to S3), using other modality data to assist in analyzing the longitudinal image data to obtain image analysis results.
[0051] In this embodiment, assuming that the total number of people is N, each subject contains T historical access data, and the vth modal data is in, Represents the feature matrix of the vth modality visited at the tth time, composed of n t Personal and v Since the present invention is based on magnetic resonance imaging (MRI), cognitive assessment data and demographic data, they are represented as X (1) ,X (2) ,X (3) By connecting the three modal data, we get the longitudinal multimodal data X∈R N×T×D , where D represents the number of input features. Let Y∈R N×F×D′ ,Y={Y1,Y2,...,YF}, D' represents the amount of information corresponding to the decoding of the image features, Y represents the actual image analysis results at all historical moments, and Y F Indicates the true situation of the image analysis result at the Fth historical moment. The present invention is given three longitudinal multimodal data with T visits, and analyzes the longitudinal image data frame by frame.
[0052] In this embodiment, a multimodal embedding module is constructed. For the longitudinal multimodal data X, a learnable parameter X'∈R is used. NxTxD Automatically fill in missing values in longitudinal multimodal data X using the following rules:
[0053]
[0054] In this embodiment, different modal data provide different information about the disease, and other modal data can make up for the missing information of the image data and assist in image analysis. In order to make full use of the complementary information of multimodal data, a fully connected layer is used as a multimodal feature encoder E. encoder To fuse multimodal features and obtain the historical comprehensive features of the subject (i.e., encoding features) Among them, D e represents the dimension size encoded by the multimodal feature encoder, and T′ represents the time dimension size. Since the present invention adopts an autoregressive time series model, T′=1. This process can be abstracted into the following formula:
[0055] X encoder =E encoder (X fill )
[0056] Among them, X encoder represents the encoding feature, E encoder () represents the multimodal feature encoder, X fill represents the multimodal data after being filled with learnable parameters, X represents the longitudinal multimodal data obtained by connecting the three modal data of demographics, MRI images, and cognitive assessment data, and X t v represents the modal data of the vth mode at the tth time point, NaN indicates that the data point does not exist here, and X' represents the learnable parameters.
[0057] In this embodiment, an image analyzer for information classification is constructed, and the coding feature X obtained in step S1 is converted into encoder Input it, use the dual-branch structure of static information analysis module and dynamic information analysis module to perform overall analysis modeling, and obtain image analysis features Among them, D p Represent feature dimensions to decompose complex longitudinal image information.
[0058] In this embodiment, the information classification image analyzer is a dual-branch structure consisting of a static information analysis module and a dynamic information analysis module, which decomposes complex longitudinal image information.
[0059] In this embodiment, in the static information analysis module, the fully connected layer E is used. static analyze It represents the factors that remain stable in the image but have a significant impact on the analysis results. This process can be abstracted into the following formula:
[0060] X static =E static (X encoder )
[0061] Among them, X static represents the static information (such as gender, education level, etc.) obtained by the static information analysis module, X encoder represents the encoding feature, E static () indicates a fully connected layer.
[0062] In this embodiment, the classic time series model is used in the dynamic information analysis module to track and analyze the dynamic changes of the image, so that it can characterize the temporal evolution of the image. Specifically, the present invention uses LSTM as the dynamic information analysis module LSTM dynamic To analyze dynamic changes in image evolution Update the hidden unit to record historical information. This process can be abstracted into the following formula:
[0063] X dynamic =LSTM dynamic (X encoder )
[0064] Among them, X dynamic Represents the dynamic changes in image evolution, LSTM dynamic () indicates dynamic information analysis operation.
[0065] In this embodiment, the present invention combines the results of the static information analysis module and the dynamic information analysis module to obtain the image feature Y in the longitudinal queue data. sd ∈R N×T×D Specifically, the present invention fuses different types of analysis information in the dual branches in a point-to-point addition manner to extract the features of image data at historical moments. This process can be abstracted into the following formula:
[0066] Y sd =X static +X dynamic
[0067] Among them, Ysd represents the image features in the longitudinal cohort data, X dynamic Indicates the dynamic changes in image evolution, X static It represents the static information (such as gender, education level, etc.) obtained by the static information analysis module.
[0068] In this embodiment, the multimodal data X of the subject is T} Input the image analysis model based on dual-branch multimodal sequence learning in chronological order, according to the multimodal data X at time t t , analyze the static information and dynamic information of historical moments, and fuse the two to obtain the image characteristics of the longitudinal queue data. Repeat the above process and use other modal data to assist in the analysis of longitudinal image data to support the discovery of image time series evolution laws.
[0069] In this embodiment, all longitudinal image data features Y are stacked. sd (i.e., the image features in the longitudinal queue data), and use other modality data to assist in analyzing the longitudinal image data to obtain the analysis result Y'∈R N×F×D' ,Y'={Y'1,Y'2,...,Y' F}, supporting the discovery of imaging biomarkers. The mean square error loss (MSE) is used to guide the learning process of image analysis, and the calculation formula is as follows:
[0070]
[0071] Among them, l mse represents the loss function, Y'={Y'1,Y'2,...,Y' F} represents the image analysis results of F historical moments. Correspondingly, Y represents the true image analysis results of all historical moments. N represents the number of subjects. Y' F Represents the image analysis result at the Fth historical moment.
[0072] In this embodiment, in order to verify the actual effect of the image analysis method provided by the embodiment of the present invention, the inventor carried out verification on the Tadpole dataset. The Tadpole dataset is extracted from the Alzheimer's Disease Neuroimaging Initiative (ADNI) and is used for the Tadpole challenge after preprocessing. The dataset contains 12,742 records from 1,737 individuals, which are classified as follows at baseline: 417 individuals in a cognitively normal (CN) state, 872 individuals with mild cognitive impairment (MCI), 342 individuals diagnosed with Alzheimer's disease (AD), and 106 individuals showing subjective memory complaints (SMC). The present invention focuses on using image analysis to obtain image features at a specific time point, and using other modal data such as cognitive assessment data and demographics to assist in the analysis of longitudinal imaging data.
[0073] Since the changes in brain structure of AD patients are only in specific locations, the inventors used data from the hippocampus, whole brain, entorhinal cortex, inferior temporal gyrus and middle temporal gyrus instead of whole brain MRI data. Demographics consider race, age, marital status and education. Only the Alzheimer's Disease Assessment Scale ADAS-13 was used as cognitive assessment data. The method uses the initial 11 visits as historical input to learn the image features of historical moments, with a 6-month interval between each two data records. The multimodal feature encoder consists of a fully connected layer, the static information analysis module consists of a multilayer perceptron (MLP), and the dynamic information analysis module is constructed as a two-layer LSTM network. As shown in Table 1, the present invention is compared with three classic time series models: recurrent neural network (RNN), long short-term memory network (LSTM) and gated recurrent unit (GRU). LSTM and GRU are both variants of RNN, designed to solve the problem of gradient disappearance when processing long sequence analysis. Through the forgetting and memory mechanisms, LSTM and GRU can effectively capture the temporal dependencies in time series data, and have enhanced performance compared with RNN. The present invention demonstrates its superior performance by making full use of incomplete multimodal disease data and accurately modeling complex image evolution characteristics.
[0074] Table 1
[0075] Model MSE(↓) MAE(↓) LSTM 0.6921 1.1376 GRU 0.7056 1.2265 RNN 0.7744 1.1855 The present invention 0.5767 0.8862
[0076] The information classification image analyzer improves the ability of the present invention to learn longitudinal image features, and can also learn parameters to fill in missing data instead of discarding them prematurely, thereby enhancing the full utilization of missing data by the present invention. As can be seen from Table 1, the present invention can achieve relatively excellent performance in the Alzheimer's disease image analysis task scenario and has good practical value.
[0077] Compared with the prior art, the image analysis method based on dual-branch multimodal sequence learning provided by the present invention uses learnable parameters to complete multimodal missing data, proposes an information classification image analyzer to decompose complex image evolution information, and uses a dual-branch structure to analyze the static composition and dynamic evolution information in longitudinal image data. It improves the stability and robustness of the method when processing incomplete or abnormal data, simplifies the longitudinal image feature learning process through information decomposition, and effectively improves the accuracy of image analysis compared with the baseline model during the reasoning process.
[0078] The above is a detailed description of the image analysis method based on dual-branch multimodal sequence learning provided by the present invention. For those skilled in the art, any obvious changes made to it without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.
Claims
1. An image analysis method based on dual-branch multimodal sequence learning, characterized in that: The following steps are involved: S1. Construct a multimodal embedding module and use learnable parameters to fill in the missing multimodal disease data, fuse the filled multimodal features, and obtain the encoding features of the subjects; S2, analyzing the coding features using a dual-branch structure including a static information analysis module and a dynamic information analysis module to obtain dual-branch multimodal image analysis features; S3, fusing the dual-branch multimodal image analysis features obtained in step S2 to obtain image features in the longitudinal queue data; S4. Repeat steps S1 to S3, stack the image features in all longitudinal cohort data, and use the multimodal data except the padded multimodal disease data to analyze the image features in all stacked longitudinal cohort data to obtain image analysis results, thereby completing the image analysis based on dual-branch multimodal sequence learning.
2. The image analysis method based on dual-branch multimodal sequence learning according to claim 1, characterized in that: The step S1 comprises the following steps: S101. Build a multimodal embedding module to fill missing multimodal disease data including demographics, MRI images, and cognitive assessment data using learnable parameters. S102, using the fully connected layer as a multimodal feature encoder, and according to the filling result obtained by the multimodal embedding module in step S101, using the multimodal feature encoder to fuse the multimodal features to obtain the encoding features of the subject.
3. The image analysis method based on dual-branch multimodal sequence learning according to claim 2, characterized in that: The expression of the encoding feature is as follows: X encoder =E encoder (X fill ) Among them, X encoder represents the encoding feature, E encoder () represents the multimodal feature encoder, X fill represents the multimodal data after being filled with learnable parameters, X represents the longitudinal multimodal data obtained by connecting the three modal data of demographics, MRI images, and cognitive assessment data, and X t v represents the modal data of the vth mode at the tth time point, NaN indicates that the data point does not exist here, and X' represents the learnable parameters.
4. The image analysis method based on dual-branch multimodal sequence learning according to claim 1, characterized in that: The step S2 comprises the following steps: S201. In the static information analysis module, the encoding features are analyzed using a fully connected layer according to the following formula: X static =E static (X encoder ) Among them, X static represents the static information representation obtained by the static information analysis module, X encoder represents the encoding feature, E static () represents a fully connected layer; S202, according to the coding features, using the following formula, using the dynamic information analysis module to analyze the dynamic change features in the image evolution: X dynamic =LSTM dynamic (X encoder ) Among them, X dynamic Representing dynamic changes in longitudinal cohort data, LSTM dynamic () indicates dynamic information analysis operation; S203. According to the analysis results of step S201 and step S202, dual-branch multimodal image analysis features are obtained.
5. The image analysis method based on dual-branch multimodal sequence learning according to claim 1, characterized in that: The step S3 is specifically as follows: The dual-branch multimodal image analysis features obtained in step S2 are fused in a point-to-point addition manner using the following formula to obtain the image features in the longitudinal queue data: Y sd =X static +X dynamic Among them, Y sd represents the image features in the longitudinal cohort data, X dynamic Indicates the dynamic changes in image evolution, X static Represents the static information representation obtained by the static information analysis module.
