Methods, systems, devices, processors, and computer-readable storage media for multimodal data fusion addressing modal gaps

Abstract

This invention relates to a multimodal data fusion method for modality missing, wherein the method includes the following steps: (1) acquiring multimodal data, including at least two of neuroimaging data, text data, and numerical data, wherein modality missing exists in the multimodal data; (2) inputting the multimodal data into a pre-trained multimodal data fusion model for feature encoding and cross-modal feature fusion processing to obtain fused features containing multimodal information; (3) outputting personalized risk prediction results through a risk prediction module using the acquired multimodal fusion features. This invention also relates to a corresponding system, device, processor, and computer-readable storage medium thereof. Using the multimodal data fusion method, system, device, processor, and computer-readable storage medium of this invention for modality missing, cross-modal adaptive modeling of patient multimodal data is performed to achieve accurate efficacy prediction.

Description

Technical Field

[0001] This invention relates to the field of medical image data information processing, and more particularly to multimodal data processing, specifically to a method, system, device, processor, and computer-readable storage medium for multimodal data fusion with missing modalities. Background Technology

[0002] In clinical applications, multimodal data (including patient basic information, blood routine data, neurobiochemical data, behavioral scales, chief complaint text, hospital admission records, medication information, and neuroimaging modalities such as structure / function) obtained through different acquisition methods can be used to model the patient's health status from multiple perspectives, thereby providing effective information from different dimensions for establishing precise diagnosis and treatment of patients.

[0003] Multimodal clinical data (including neuroimaging data, electroencephalogram data, neurochemical indicators, cognitive and behavioral data, etc.) originates from diverse data sources and is highly heterogeneous and complex. Conventional deep learning techniques face challenges in data fusion and interpretability when processing multimodal clinical data. Furthermore, in clinical practice, high-quality paired multimodal data is difficult to obtain due to limitations in data recording and acquisition costs. In real-world clinical scenarios, multimodal data often suffers from modality gaps, while paired multimodal data is often limited in volume, thus restricting the performance of precision diagnostic models. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system, device, processor and computer-readable storage medium for multimodal data fusion with missing modalities.

[0005] To achieve the above objectives, the present invention provides a multimodal data fusion method, system, apparatus, processor, and computer-readable storage medium for modality missing data, as follows:

[0006] The main feature of this multimodal data fusion method for addressing modal missing features is that the method includes the following steps:

[0007] (1) Acquire multimodal data, wherein the multimodal data includes at least two of the following: neuroimaging data, text data, and numerical data, and wherein there is modality missing in the multimodal data;

[0008] (2) Input the multimodal data into a pre-trained multimodal data fusion model for feature encoding and cross-modal feature fusion to obtain fused features containing multimodal information;

[0009] (3) The acquired multimodal fusion features are output as personalized risk prediction results through the risk prediction module.

[0010] Preferably, the multimodal data fusion model specifically comprises: encoding the data of each input modality to obtain the corresponding feature encoding vector; outputting an empty encoding for missing modalities; and performing bimodal and multimodal fusion processing on all output feature encoding vectors.

[0011] Preferably, the bimodal fusion process specifically involves: combining all feature encoding vectors in pairs; for each pair of non-empty feature encoding vector combinations, using a cross-attention mechanism to perform bimodal feature fusion to obtain the corresponding bimodal fusion feature; and skipping the processing for combinations containing empty encodings.

[0012] The multimodal fusion process specifically involves aggregating all bimodal fusion features to obtain multimodal fusion features.

[0013] Preferably, the multimodal data fusion model is trained using a loss function, which includes a task prediction loss function and a fusion feature consistency constraint loss function; wherein the fusion feature consistency constraint loss function is used to constrain the distribution consistency between different bimodal fusion features.

[0014] Preferably, the cross-attention mechanism is specifically as follows:

[0015] For the feature encoding from mode A and feature encoding from mode B The features of mode A obtained after its fusion and the characteristics of mode B Calculate as follows:

[0016] ;

[0017] ;

[0018] Where Q and V are the query matrix and value matrix in the Transformer's self-attention mechanism, respectively, and d is the length of the feature encoding vector.

[0019] Preferably, the multimodal fusion process further includes: aggregating all bimodal fusion features by adding and averaging them element by element.

[0020] Preferably, the task prediction loss function is specifically: when risk prediction is a classification task, the cross-entropy loss function is used, and when risk prediction is a regression task, the mean squared error loss function is used.

[0021] Preferably, the loss function for consistency constraint of fusion features is specifically: calculating the distribution difference between each pair of different bimodal fusion features using mean square error or KL divergence.

[0022] The multimodal data fusion system for implementing the above-described method, which addresses modality loss, is characterized in that the system comprises:

[0023] A multimodal clinical data acquisition module collects at least two types of data from patients during clinical practice, including neuroimaging data, text data, and numerical data, and inputs them into a pre-trained multimodal data fusion model for processing; wherein the multimodal data fusion model includes:

[0024] The feature encoding module is used to encode the data for each input modality to obtain the corresponding feature encoding vector; for missing modalities, an empty encoding is output.

[0025] A hierarchical cross-modal feature fusion module is used to fuse all feature encoding vectors output by the feature encoding module; and the hierarchical cross-modal feature fusion module includes a bimodal fusion layer and a multimodal fusion layer;

[0026] The risk prediction module is used to output risk prediction results based on the acquired multimodal fusion features.

[0027] Preferably, the dual-modal fusion layer is used to combine all feature encoding vectors in pairs, and for each pair of non-empty feature encoding vectors, a cross-attention mechanism is used to perform dual-modal feature fusion to obtain the corresponding dual-modal fused features; combinations containing empty encodings are skipped.

[0028] The multimodal fusion layer is used to aggregate the bimodal fusion features output by all bimodal fusion layers to obtain multimodal fusion features.

[0029] The main feature of this multimodal data fusion device for addressing modal missing features is that the device comprises:

[0030] A processor is configured to execute computer-executable instructions;

[0031] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the multimodal data fusion method for modality loss described above.

[0032] The main feature of this multimodal data fusion processor for modality missing is that the processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the multimodal data fusion method for modality missing described above.

[0033] The computer-readable storage medium is characterized in that it stores a computer program that can be executed by a processor to implement the steps of the multimodal data fusion method for modality missing described above.

[0034] The present invention employs a multimodal data fusion method, system, device, processor, and computer-readable storage medium for modality missing data. Based on a multimodal deep learning network with cross-modal self-attention technology, and combined with a multimodal loss function designed for missing data for model optimization, it achieves full utilization of large-scale multimodal data with modality missing data. By performing cross-modal modeling on the data, it enables accurate risk prediction of mental illness. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the multimodal data fusion method for modality loss according to the present invention. Detailed Implementation

[0036] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.

[0037] Before describing the embodiments of the present invention in detail, it should be noted that, in the following, the terms “comprising,” “including,” or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0038] This multimodal data fusion method for addressing modality loss employs a semi-supervised learning strategy, whereby its optimized loss function combines label-based supervised training with feature similarity-based unsupervised training without training labels.

[0039] Please see Figure 1As shown, the overall technical framework of this solution includes a multimodal data feature encoding part and a cross-modal feature fusion part. Based on the data organization, multimodal clinical data can be divided into neuroimaging data, text data, and numerical data. For input data of different modalities, it is first input into the corresponding feature encoder to obtain the feature code for each modality. Next, the features of different modalities are fused through a hierarchical cross-modal feature fusion module. The cross-modal feature fusion module adaptively and dynamically models the feature codes of different modalities, ultimately obtaining fused features containing multimodal information. This fused feature is then used by a risk prediction module to achieve individualized risk prediction for mental illnesses.

[0040] The following will further explain the processing procedure of this technical solution with reference to specific implementation methods:

[0041] For image data, the image feature encoder can be independently trained using a convolutional neural network (or a model like ViT). The loss function used during training depends on the risk prediction category; it can be a binary classification loss function such as cross-entropy (if risk prediction is considered a classification task), or a regression loss function such as mean squared error (if risk prediction is considered a regression task for continuous indicators). After this independent training, the resulting neural network model (i.e., the image feature encoder) can encode the input image features into fixed-dimensional vectors (i.e., image feature encoding). The encoded image features contain effective information related to mental illness risk prediction. For i neuroimaging modalities, the final image data encoding is i sets (considering the possibility of missing modalities, there may be empty encodings in the i sets).

[0042] The data obtained above also includes data preprocessing: non-uniform bias correction, denoising, grayscale normalization, and registration to a standard space (such as MNI space) are performed on both calibrated and uncalibrated brain images.

[0043] For text-based data, the text feature encoder is implemented by loading a pre-trained large language model using a Transformer network architecture. Based on pre-trained models such as RoBERTa and Baichuan2-13B, it is possible to encode text-based data such as chief complaints and medical histories, encoding the text data into fixed-dimensional vectors (i.e., text data encoding). For j types of text-based data, the final text data encoding is j groups (considering the possibility of missing modalities, there may be empty encodings in the j groups).

[0044] For numerical data, the numerical data encoder is independently trained using a fully connected multilayer neural network. The loss function used during training can be a binary classification loss function such as cross-entropy (if risk prediction is considered a classification task), or a regression loss function such as mean squared error (if risk prediction is considered a regression task for continuous indicators), depending on the category of risk prediction. After this independent training, the resulting neural network model (i.e., the numerical feature encoder) can encode the input numerical features into fixed-dimensional vectors (i.e., image feature encoding). The encoded numerical features contain effective information related to mental illness risk prediction. Similarly, for k categories of numerical data, the final numerical data encoding consists of k groups (considering the possibility of modality loss, there may be empty encodings in the k groups).

[0045] For the three extracted modalities and a total of i+j+k sets of features (including possible missing modal empty codes), the hierarchical cross-modal feature fusion module achieves dynamic modeling between modalities through a dual-modal + multi-modal hierarchical feature fusion strategy, and finally obtains features that fuse all modal information.

[0046] Specifically: combine the i+j+k groups of multimodal features in pairs to obtain Combination of dual-modal features;

[0047] For each combination of bimodal features, train independently. A dual-modal feature fusion module enables dynamic modeling within each pair of modalities.

[0048] In a preferred embodiment of the present invention, the dual-modal feature fusion module employs a deep neural network based on a stacked cross-attention strategy, referencing the self-attention mechanism in Transformer, for feature encoding from modality A... Feature encoding from mode B The feature encoding of mode A obtained after processing by the cross-attention mechanism Feature encoding of mode B They are respectively:

[0049] ;

[0050] ;

[0051] Where Q and V are the query matrix and value matrix in the Transformer's self-attention mechanism, and d is the length of the feature encoding vector. The above calculation can dynamically model the correlation between modalities A and B, yielding... , Features related to mental illness risk from modalities A and B are fused separately. Through a stacked cross-attention mechanism, fused features 1, fused feature 2, ... are obtained after pairwise fusion. .

[0052] Understandably, in practical applications, bimodal feature fusion can also be achieved through other multi-layer neural network architectures. The loss function used for training the bimodal feature fusion module can be a binary classification loss function such as cross-entropy (if risk prediction is regarded as a classification task) or a regression loss function such as mean squared error (if risk prediction is regarded as a regression task for continuous indicators), depending on the category of risk prediction.

[0053] The result obtained after pairwise fusion (bimodal hierarchy) The system integrates features by adding and averaging each feature element to achieve feature fusion across all modalities (multimodal level). The fused features are then input into the risk prediction module to enable individualized risk prediction for mental illnesses.

[0054] In a preferred embodiment of the present invention, the risk prediction module is a multi-layer fully connected neural network, but other network model architectures can also be used. The loss function used for training can be a binary classification loss function such as cross-entropy (if risk prediction is regarded as a classification task), or a regression loss function such as mean squared error (if risk prediction is regarded as a regression task for continuous indicators), depending on the category of risk prediction.

[0055] Based on this, considering that potential modality omissions leading to empty encodings could impact hierarchical cross-modal feature fusion, this hierarchical cross-modal feature fusion module needs to possess two characteristics to address the modality omission problem: 1) fully utilize all existing modality information; 2) modality omissions should not affect feature fusion. To achieve these characteristics, this technical solution adds a feature consistency constraint module to the hierarchical cross-modal feature fusion module, specifically:

[0056] for If there is an empty code (missing modality) in the feature encoding combination of a pairwise modality combination, the fused feature generated by the feature encoding combination of that pairwise modality combination will not participate in the feature fusion of the subsequent multimodal level (i.e., the feature elements are added and averaged one by one).

[0057] For the fused features output by different bimodal feature fusion modules, there are significant differences in the numerical values ​​of the fused features due to the differences in modality combinations. Therefore, the lack of fused features caused by modality omissions will have a significant impact on the feature fusion (averaging) at the multimodal level. Therefore, this invention constrains all fused features to have similar data distributions.

[0058] In a preferred embodiment of the present invention, the mean square error between any two fused features is used as a consistency constraint index, and this index is incorporated as part of the loss function of the hierarchical cross-modal feature fusion module. Alternatively, differentiable functions such as KL divergence can be used as a measure of fused feature consistency. By adding a fused feature consistency constraint to the loss function, the impact of modality loss on cross-modal feature fusion is weakened through a data-driven approach, thereby achieving full utilization of existing modal data.

[0059] In practical applications, this technical solution uses the Adam optimizer for training relevant models, with a learning rate set to [value missing]. The training epochs are set to 300, and the batch size is set to 4-8 based on the available GPU memory. Specifically, during model training, an existence regularization strategy is employed: 1. Batch normalization layers are added after all fully connected and convolutional layers in the entire model architecture to standardize the data distribution between layers; 2. During the optimization of the model by the Adam optimizer, an L2 regularization strategy is used, with a regularization decay coefficient set to 0.01. In addition, an early stopping strategy is adopted during model training. If the loss function value on the model validation set data does not improve for five consecutive epochs, model training is stopped. By limiting the number of training iterations, the complexity of the model is indirectly controlled, and overfitting is avoided.

[0060] As a specific embodiment of the present invention, the performance comparison experiment under different modality missing ratios is used as a test of multimodal data. Complete multimodal data (including 300 T1 structural magnetic resonance imaging data, 300 chief complaint data, and 300 blood routine data) and 215 healthy control data were collected from Shanghai Mental Health Center. The experiment adopted a ten-fold cross-validation strategy and achieved an average prediction accuracy of 80%. In comparison, when 50% of the T1 data is missing and other modalities are complete, a prediction accuracy of 74% can be achieved; when 20% of the T1 data is missing and other modalities are complete, a prediction accuracy of 77% can be achieved; when 50% of the chief complaint data is missing and other modalities are complete, an accuracy of 71% can be achieved; when 20% of the chief complaint data is missing and other modalities are complete, a classification accuracy of 78% can be achieved; when 50% of the blood routine data is missing and other modalities are complete, a prediction accuracy of 76% can be achieved; and when 20% of the blood routine data is missing and other modalities are complete, a prediction accuracy of 79% can be achieved.

[0061] In another embodiment of the present invention, for comparison, the bimodal-multimodal feature fusion module of this technical solution is directly replaced with a multilayer fully connected neural network with the same number of neurons and network layers, and the modified model is used as a comparison method. As a test of multimodal data, complete multimodal data was used, including data from 300 patients with high-risk clinical syndromes of mental illness (300 T1 structural MRI images, 300 chief complaints, and 300 blood routine data) and 215 healthy controls collected from the Shanghai Mental Health Center. A ten-fold cross-validation strategy was employed, achieving an average prediction accuracy of 80%. For comparison, when 50% of the T1 data was missing and other modalities were complete, this technical solution achieved a prediction accuracy of 74%, while the comparison method achieved 61%; when 20% of the T1 data was missing and other modalities were complete, this technical solution achieved a prediction accuracy of 77%, while the comparison method achieved 65%.

[0062] In summary, this technical solution employs a hierarchical feature fusion strategy. First, features from different modalities are fused pairwise. During this fusion stage, the similarity of the pairwise modal features is constrained to minimize the loss of each modal feature without introducing data noise, while also reducing the impact of missing modal features on downstream tasks. Next, in the multimodal feature fusion stage, the fused pairwise modal features are further fused to make full use of all effective information in the existing modal data and to model the potential correlations between modalities.

[0063] This multimodal data fusion device for addressing modal missing features includes:

[0064] A processor is configured to execute computer-executable instructions;

[0065] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the multimodal data fusion method for modality loss described above.

[0066] The multimodal data fusion processor for modality missing is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the multimodal data fusion method for modality missing described above.

[0067] The computer-readable storage medium contains a computer program that can be executed by a processor to implement the steps of the multimodal data fusion method for modality loss described above.

[0068] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0069] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution device.

[0070] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0071] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0072] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0073] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

[0074] The present invention employs a multimodal data fusion method, system, device, processor, and computer-readable storage medium for modality missing data. Based on a multimodal deep learning network with cross-modal self-attention technology, and combined with a multimodal loss function designed for missing data for model optimization, it achieves full utilization of large-scale multimodal data with modality missing data. By performing cross-modal modeling on the data, it enables accurate risk prediction of mental illness.

[0075] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method for multi-modal data fusion against missing modalities, characterized in that, The method includes the following steps: (1) Acquire multimodal data, wherein the multimodal data includes at least two of the following: neuroimaging data, text data, and numerical data, and wherein there is modality missing in the multimodal data; (2) Input the multimodal data into a pre-trained multimodal data fusion model for feature encoding and cross-modal feature fusion to obtain fused features containing multimodal information; (3) The acquired multimodal fusion features are used to output personalized risk prediction results through the risk prediction module; The multimodal data fusion model specifically involves: encoding the input data for each modality to obtain the corresponding feature encoding vector; outputting an empty encoding for missing modalities; and performing bimodal and multimodal fusion processing on all output feature encoding vectors. The bimodal fusion processing specifically involves: combining all feature encoding vectors pairwise; for each pair of non-empty feature encoding vectors, using a cross-attention mechanism to perform bimodal feature fusion to obtain the corresponding bimodal fused features; skipping combinations containing empty encodings. The multimodal fusion process specifically involves aggregating all bimodal fusion features to obtain multimodal fusion features. The multimodal data fusion model is trained using a loss function, which includes a task prediction loss function and a fusion feature consistency constraint loss function; wherein, the fusion feature consistency constraint loss function is used to constrain the distribution consistency between different bimodal fusion features. The multimodal fusion processing further includes: aggregating all bimodal fusion features by adding and averaging them element by element.

2. The multi-modal data fusion method for missing modality according to claim 1, wherein, The cross-attention mechanism is specifically as follows: For feature encoding from modality A and feature encoding from modality B whose fusion results in features of modality A and features of modality B are computed as follows: Where Q and V are the query matrix and value matrix in the Transformer's self-attention mechanism, respectively, and d is the length of the feature encoding vector.

3. The multi-modal data fusion method for missing modality according to claim 1, wherein, The task prediction loss function is specifically as follows: when risk prediction is a classification task, the cross-entropy loss function is used; when risk prediction is a regression task, the mean squared error loss function is used.

4. The multi-modal data fusion method for missing modality according to claim 1, wherein, The aforementioned loss function for consistency constraint of fusion features is specifically: calculating the distribution differences between pairs of different bimodal fusion features using mean squared error or KL divergence.

5. A multi-modal data fusion system for implementing the method of any one of claims 1 to 4 for missing modalities, characterized by The system includes: A multimodal clinical data acquisition module collects at least two types of data from patients during clinical practice, including neuroimaging data, text data, and numerical data, and inputs them into a pre-trained multimodal data fusion model for processing; wherein the multimodal data fusion model includes: The feature encoding module is used to encode the data for each input modality to obtain the corresponding feature encoding vector; for missing modalities, an empty encoding is output. A hierarchical cross-modal feature fusion module is used to fuse all feature encoding vectors output by the feature encoding module; and the hierarchical cross-modal feature fusion module includes a bimodal fusion layer and a multimodal fusion layer; The risk prediction module is used to output risk prediction results based on the acquired multimodal fusion features.

6. The multimodal data fusion system for modality missing data according to claim 5, characterized in that, The dual-modal fusion layer is used to combine all feature encoding vectors in pairs. For each pair of non-empty feature encoding vectors, a cross-attention mechanism is used to perform dual-modal feature fusion to obtain the corresponding dual-modal fused features. For combinations containing empty encodings, the processing is skipped. The multimodal fusion layer is used to aggregate the bimodal fusion features output by all bimodal fusion layers to obtain multimodal fusion features.

7. A multi-modal data fusion apparatus against modal missing, characterized by, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the multimodal data fusion method for modality missing according to any one of claims 1 to 4.

8. A multi-modal data fusion processor for modalities missing, characterized by, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the multimodal data fusion method for modality missing as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the multimodal data fusion method for modality missing as described in any one of claims 1 to 4.

Images