Multi-modal data fusion method and device, computer equipment and storage medium

Abstract

The invention relates to a multi-modal data fusion method and device, computer equipment and a storage medium. The method comprises the following steps: carrying out high-order feature extraction on multi-modal data to be fused by utilizing a special modal encoder to obtain high-order embedding features of the multi-modal data; the high-order embedded features are decoupled in combination with expert knowledge constraints and a multi-head attention mechanism, and high-frequency features representing local information and low-frequency features representing global information are obtained respectively; and aligning the low-frequency features by using a cosine alignment algorithm, performing orthogonal projection on the high-frequency features by using an orthogonal projection technology, and performing multi-level fusion on the aligned low-frequency features and the orthogonally projected high-frequency features through an adaptive deep fusion network to obtain fused multi-modal joint features. According to the method, the problems of heterogeneity and isomerism of the data are solved, effective fusion of different modal data is ensured, and the accuracy and individuation capability of disease risk early warning can be remarkably improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing and medical health technology, and in particular, relates to a multimodal data fusion method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of medical information technology, more and more medical data are collected and stored through different channels. Especially in the treatment of multi-center perioperative patients, multiple modal data such as structured electronic health records (EHRs), preoperative electrocardiograms, medical images (chest X-rays, abdominal CT films) and intraoperative time-series physiological monitoring are involved. By fusing multimodal data to obtain rich patient health information, it provides a reference for clinical decision-making. In the existing technology, multimodal data fusion methods mostly rely on rule-based feature extraction and manually designed fusion strategies. Although data fusion can be achieved in some scenarios, due to the significant differences in structure, quality, and spatiotemporal scales of different modal data, multimodal data fusion faces many technical challenges, especially in dealing with data heterogeneity. There are great limitations, which are specifically manifested as follows: (1) It is difficult to effectively handle highly heterogeneous and heterogeneous data: Multimodal data comes from various sources. Due to the different acquisition equipment and environments of multiple centers, the same modality data not only has large differences in data formats such as quality, resolution, and scale, but also has significant heterogeneity and heterogeneity in data type, quality, and scale. Existing multimodal data fusion methods are usually unable to effectively handle these differences, resulting in poor fusion effects and the inability to extract effective complementary information from each modality. Especially in the task of perioperative cardiovascular event warning, the heterogeneity of data is more prominent, affecting the accuracy and reliability of the warning system.

[0003] (2) Poor information extraction capability: Existing technologies usually rely on traditional rule-based feature extraction methods and fail to fully consider the noise and redundant information in the data. Especially in intraoperative temporal physiological monitoring and imaging data, noise and redundant information account for a large proportion, resulting in a low signal-to-noise ratio and difficulty in effectively extracting valuable medical information, which in turn affects the risk assessment accuracy and early warning accuracy of cardiovascular events.

[0004] (3) Unable to fully utilize the correlation between multimodal data: In multimodal data, there is often a complex horizontal coupling relationship between the modalities, especially the high coupling between imaging data and structured electronic health records. Existing data fusion methods are often unable to effectively analyze the relationship between different data modalities, and it is difficult to fully utilize the complementary information in multimodal data for accurate prediction.

[0005] (4) Insufficient medical interpretability: Most existing multimodal data fusion methods rely on black-box models and lack medical interpretability of the fusion process, which not only reduces the credibility of the model but also limits its promotion and application in clinical practice.

[0006] Therefore, in order to better deal with the heterogeneity problem of multimodal data and improve the accuracy of perioperative cardiovascular event warning, a new multimodal data fusion algorithm is urgently needed to provide clinicians with more accurate and personalized decision support. Summary of the invention

[0007] The present application provides a multimodal data fusion method, apparatus, computer device and storage medium, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0008] In order to solve the above problems, this application provides the following technical solutions: A multimodal data fusion method, comprising: A dedicated modal encoder is used to extract high-order features from the multimodal data to be fused, and high-order embedding features of the multimodal data are obtained; Combining expert knowledge constraints and a multi-head attention mechanism to decouple the high-order embedded features, high-frequency features representing local information and low-frequency features representing global information are obtained respectively; The low-frequency features are aligned using a cosine alignment algorithm, the high-frequency features are orthogonally projected using an orthogonal projection technique, and the aligned low-frequency features and the orthogonally projected high-frequency features are multi-level fused using an adaptive deep fusion network to obtain fused multi-modal joint features; The fused multimodal features are input into a pre-trained risk prediction model, and the disease risk warning probability is output through the risk prediction model.

[0009] The technical solution adopted by the embodiment of the present application also includes: the multimodal data includes structured electronic health records, preoperative electrocardiograms, medical imaging data, and intraoperative physiological monitoring, and the dedicated modal encoder is used to extract high-order features from the multimodal data to be fused to obtain high-order embedded features of the multimodal data, specifically: For the structured electronic health record, high-order feature extraction is performed by an encoder based on a combination of a multi-layer perceptron and a decision tree; For the preoperative electrocardiogram and intraoperative physiological monitoring data, Fourier transform and frequency domain attention mechanism are used to extract high-order features in the frequency domain; For medical imaging data, detailed features are extracted through deep convolutional networks.

[0010] The technical solution adopted by the embodiment of the present application also includes: combining the expert knowledge constraint and the multi-head attention mechanism to decouple the high-order embedded features, and respectively obtain high-frequency features representing local information and low-frequency features representing global information, specifically: The expert knowledge of the relevant disease risk warning is encoded into an embedding vector through an embedding layer, and the embedding vector is concatenated with the high-order embedding feature and then input into the feature decoupling module; The feature decoupling module processes the high-order embedded features through a normalization layer, a multi-head attention mechanism layer, and a feed-forward fully connected layer in sequence to capture the potential relationship and feature relationship between the multimodal data; Based on the potential relationship and feature relationship between the multimodal data, the high-order embedded features are parsed through a low-frequency feature common decoupling layer and a high-frequency feature specific decoupling layer to extract the low-frequency features and high-frequency features of the multimodal data; The mean and variance of the low-frequency features and the high-frequency features are calculated, and the mean and variance are sampled by a Gaussian sampling method to obtain the final low-frequency features and high-frequency features.

[0011] The technical solution adopted in the embodiment of the present application also includes: the feature decoupling module processes the high-order embedded features through a normalization layer, a multi-head attention mechanism layer and a feedforward fully connected layer in sequence to capture the potential relationship and feature relationship between the multimodal data, specifically: The high-order embedding features of each modality data are standardized by the normalization layer; given the embedding features , the normalization layer normalizes it by the following formula:

[0012] in and They are The mean and standard deviation of Through the multi-head attention mechanism layer, the potential relationship between the high-order embedded features of each modal data is learned to mine the semantic information of each modal data; The output features of the multi-head attention mechanism layer are further extracted by introducing nonlinear transformation through the feed-forward fully connected layer to capture the characteristic relationship of each modality data.

[0013] The technical solution adopted in the embodiment of the present application also includes: based on the potential relationship and feature relationship between the multimodal data, the high-order embedded features are parsed through the low-frequency feature common decoupling layer and the high-frequency feature specific decoupling layer respectively, and the low-frequency features and high-frequency features of the multimodal data are extracted, specifically: The low-frequency feature commonality decoupling layer uses the spectrum modulation attention mechanism of the weighted spectrum matrix to adaptively adjust the low-frequency components in the high-order embedded features, thereby enhancing the model's attention to and extraction of low-frequency features:

[0014]

[0015] in It is The absolute value of the spectrum of the input features, is the feature dimension, is the total number of features; represents the extracted low-frequency features, The feature vector output by the previous layer is obtained through three different linear transformations. is the dimension of the key vector; The high-frequency feature-specific decoupling layer uses a spectral modulation attention mechanism to adaptively adjust the high-frequency components in the high-order embedded features, emphasizing the modality-specific detail features:

[0016]

[0017] in, is the square of the spectral intensity in the input high-order embedded feature, indicating the size of the high-frequency component. Represents the extracted high-frequency features.

[0018] The technical solution adopted by the embodiment of the present application also includes: aligning the low-frequency features using the cosine alignment algorithm, and performing orthogonal projection on the high-frequency features using the orthogonal projection technology, specifically: The cosine similarity between the low-frequency features is calculated by the cosine alignment algorithm, and the alignment of the low-frequency features is constrained by the strategy of minimizing the cosine distance:

[0019] in and Respectively represent and The low-frequency characteristics of the modal data, is the Euclidean norm of the vector; Combined with expert knowledge, the aligned low-frequency features are weighted by a multilinear adaptive weighting algorithm to obtain weighted low-frequency features. ; The weighted formula is:

[0020] in, Based on expert knowledge Adaptively calculated weighting coefficients; All weighted low-frequency features are fused to obtain the fused low-frequency feature representation for:

[0021] in represents the weight coefficient of low-frequency features, Indicates the number of modes; The high-frequency features are To standardize:

[0022] in It is The high-frequency eigenvector of the normalized modal data, Represents the Euclidean norm of a vector; For high frequency features , projecting it outside the space of other modal features:

[0023] in, It is The high-frequency features of the modal data after projection are represented by represents the inner product operation; The high-frequency features after orthogonal projection are linearly weighted Fusion is performed to obtain the final high-frequency feature representation :

[0024] in is the weight coefficient of high-frequency features, Indicates the number of modes.

[0025] The technical solution adopted in the embodiment of the present application also includes: the aligned low-frequency features and the orthogonally projected high-frequency features are multi-level fused through the adaptive deep fusion network to obtain the fused multi-modal joint features, specifically: The low-frequency and high-frequency spatiotemporal information collaborative sub-network performs one-dimensional temporal convolution on the low-frequency features and high-frequency features and then performs weighted fusion to obtain the temporal features. ; Calculate the spatial interaction relationship between the low-frequency features and the high-frequency features by point-by-point multiplication to obtain the spatial features ; The time feature and spatial characteristics Perform spatial consistency projection and normalization to obtain normalized time features and spatial characteristics and ; The normalized time feature and spatial characteristics and Perform weighted fusion to obtain unified multi-modal joint features :

[0026] in is the learnable fusion weight coefficient.

[0027] Another technical solution adopted by the embodiment of the present application is: a multimodal data fusion device, comprising: Feature extraction module: used to extract high-order features from the multimodal data to be fused using a dedicated modal encoder to obtain high-order embedded features of the multimodal data; Feature decoupling module: used to decouple the high-order embedded features by combining expert knowledge constraints and multi-head attention mechanism, and obtain high-frequency features representing local information and low-frequency features representing global information respectively; Feature fusion module: used to align the low-frequency features using the cosine alignment algorithm, perform orthogonal projection on the high-frequency features using the orthogonal projection technology, and perform multi-level fusion of the aligned low-frequency features and the orthogonally projected high-frequency features through an adaptive deep fusion network to obtain fused multi-modal joint features; Risk prediction module: used to input the fused multimodal features into a pre-trained risk prediction model, and output the disease risk warning probability through the risk prediction model.

[0028] Another technical solution adopted by the embodiment of the present application is: a computer device, the computer device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the multimodal data fusion method; The processor is used to execute the program instructions stored in the memory to control a multimodal data fusion method.

[0029] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the multimodal data fusion method.

[0030] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: the multimodal data fusion method, device, computer equipment and storage medium of the embodiments of the present application effectively process the differences of different modal data by designing a dedicated encoder for each modal data, improve the signal-to-noise ratio of the data, solve the heterogeneity and heterogeneity of the data, and ensure that different modal data can be effectively fused; the high-order features of each modal data are decomposed into low-frequency features and high-frequency features by expert knowledge, effectively eliminating redundant information and noise, reducing the influence of high coupling between modalities, and ensuring the independence of each modal information; the low-frequency features and high-frequency features are fused through a multi-level interpretable feature fusion module, realizing the spatiotemporal correlation modeling of the data, providing a transparent explanation of data fusion, and improving the credibility and availability of the results. This application solves the problem of information sharing and differential extraction between multimodal data through the decoupling of high and low frequency features, the extraction of commonality and uniqueness of features between modalities, and the fusion of multi-level interpretable features, which can significantly improve the accuracy and personalization of disease risk warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of a multimodal data fusion method according to an embodiment of the present application; Figure 2 This is a schematic diagram of the structure of a feature decoupling module based on expert knowledge constraints and a multi-head attention mechanism in an embodiment of the present application; Figure 3 This is a schematic diagram of the structure of a multimodal data fusion device according to an embodiment of the present application; Figure 4 A schematic diagram of the computer device structure of an embodiment of the present application; Figure 5 A schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The terms "first", "second" and "third" in this application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of this application are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or computer device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or computer devices.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] Specifically, see Figure 1 , is a flow chart of a multimodal data fusion method according to an embodiment of the present application. The multimodal data fusion method according to an embodiment of the present application comprises the following steps: S100: Acquire multimodal data of a target patient; In this step, the multimodal data obtained include but are not limited to the patient's structured electronic health record, preoperative electrocardiogram, medical imaging data, and intraoperative physiological monitoring and other medical data. Among them, the medical imaging data includes but is not limited to chest X-rays or abdominal CT films, and the intraoperative physiological monitoring data includes but is not limited to the patient's heart rate, blood oxygen saturation, respiratory rate and other real-time physiological signals.

[0036] S110: designing a corresponding dedicated modal encoder for each modal data, and using the dedicated modal encoder to extract high-order features of the multimodal data to be fused, so as to obtain high-order embedding features of the multimodal data; In this step, in order to effectively learn the invariant features between each modal data and fully mine the key information in each modal data, the present application designs corresponding dedicated modal encoders according to the properties of different modal data, and uses dedicated modal encoders to extract high-order features from different modal data using different technologies. The output feature dimension of each modal encoder remains consistent, thereby providing a unified feature space for subsequent multimodal data fusion and avoiding fusion difficulties caused by inconsistent feature dimensions.

[0037] Specifically, the high-order feature extraction process of multimodal data includes: S111: High-order feature extraction for structured electronic health records using an encoder based on a combination of multilayer perceptron and decision tree; Among them, the data in structured electronic health records are usually presented in structured numerical and categorical forms, including important information such as the patient's medical history, diagnosis, and laboratory test results. In order to extract high-level semantic features from this information, this application designs an encoder that combines a multi-layer perceptron (MLP) and a decision tree model (LightGBM). Through MLP, numerical data such as age, body mass index, and preoperative blood sugar are embedded and mapped and nonlinear feature learning is performed to capture single variable information and simple variable relationships; through LightGBM, deep learning is performed on categorical data such as gender, surgical site, and history of hypertension to capture complex variable interactions and nonlinear features. The features extracted by MLP and LightGBM are then spliced ​​to obtain high-order features of structured electronic health records, denoted as .

[0038] S112: For preoperative electrocardiogram and intraoperative physiological monitoring data, high-order features in the frequency domain are extracted using Fourier transform and frequency domain attention mechanism; Among them, the preoperative electrocardiogram and intraoperative physiological monitoring data are both time series data, which have strong time dependence and dynamic changes. In order to better process time series data, this application designs a feature extraction network that combines Fourier transform (FFT) and frequency domain attention mechanism. First, the time domain signal is converted into the frequency domain through Fourier transform, and frequency domain features such as power spectrum density are extracted to enhance the ability to capture key frequency components; then, the key frequency components are selected based on the frequency domain attention mechanism to further enhance the expression ability of frequency domain features, thereby extracting high-order features of preoperative electrocardiogram and intraoperative physiological monitoring data, which are recorded as , .

[0039] S113: Extract detailed features from medical imaging data through deep convolutional networks; Among them, medical imaging data provides a detailed view of the internal structure of the patient's body and can reflect the potential lesion area of ​​the disease. In order to extract effective features from medical imaging data, this application uses a densely connected network (DenseNet) as an encoder for image features. Through the dense connection design between layers of DenseNet, the multi-scale features in medical imaging data are efficiently captured, denoted as , which contains local and global information of medical imaging data.

[0040] It can be understood that by utilizing the above-mentioned feature extraction method, the present application can effectively extract high-order embedded features for disease risk warning from each modal data, thereby improving the signal-to-noise ratio of the data, optimizing the feature extraction capability, and providing rich basic information for subsequent multimodal data fusion.

[0041] S120: Combining expert knowledge constraints and multi-head attention mechanism to decouple high-order embedded features, respectively obtaining high-frequency features representing local information and low-frequency features representing global information; In this step, in order to solve the problem of high lateral structural coupling of multimodal data, this application introduces a feature decoupling module based on expert knowledge constraints and multi-head attention mechanism to parse and separate the highly coupled parts between each modal data, thereby achieving effective data decoupling and better utilizing the potential correlation between different modal data. Figure 2 As shown, it is a schematic diagram of the feature decoupling module structure based on expert knowledge constraints and multi-head attention mechanism in the embodiment of the present application. The feature decoupling module encodes the high-order embedded features of the single modality and decomposes it into low-frequency features and high-frequency features. The low-frequency features capture the global pattern, reveal the common information of different modal data, and reflect the long-term pattern of disease risk warning; the high-frequency features extract local detail information to reflect the sudden changes of disease risk warning. Through this decoupling process, it is possible to effectively separate the complementary information of multimodal data, remove the influence of redundancy and noise, enhance the expression ability of meaningful patterns in multimodal data, and realize the efficient parsing and explainable separation of shared information and unique information between multimodal data.

[0042] Furthermore, if Figure 2 As shown, the feature decoupling process of the feature decoupling module based on expert knowledge constraints and multi-head attention mechanism in the embodiment of the present application includes the following steps: S121: Encode the expert knowledge of relevant disease risk warning (such as disease characteristics, clinical experience, medical guidelines, etc.) into an embedding vector C through an embedding layer, and concatenate the embedding vector C with the high-order embedding feature and input it into the feature decoupling module; S122: The feature decoupling module processes the high-order embedded features through the normalization layer, the multi-head attention mechanism layer and the feed-forward fully connected layer in turn to capture the potential relationship and feature relationship between multimodal data; Among them, the normalization layer, multi-head attention mechanism layer and feed-forward fully connected layer process the high-order embedding features as follows: The high-order embedding features of each modality are normalized through the normalization layer; given the embedded features (No. features), and the normalization layer normalizes them using the following formula:

[0043] in and They are The mean and standard deviation of .

[0044] Through the multi-head attention mechanism layer, the potential relationship between the features of multimodal data is learned from multiple different "perspectives" in parallel, capturing the long-distance dependencies within the multimodal data and between modalities, thereby fully mining the semantic information of each modal data.

[0045] By introducing nonlinear transformation through the feedforward fully connected layer, the output features of the multi-head attention mechanism layer are further extracted, so that it can better capture the characteristic relationship of each modal data and enhance the model's ability to express disease risk warning.

[0046] S123: Based on the potential relationship and feature relationship between multimodal data, the high-order embedded features are analyzed through the low-frequency feature common decoupling layer and the high-frequency feature specific decoupling layer to extract the low-frequency features and high-frequency features of the multimodal data respectively; Among them, the task of the low-frequency feature commonality decoupling layer is to extract the low-frequency features shared between multimodal data by analyzing the input high-order embedded features. The low-frequency features reflect the relatively stable common information in different modal data, usually representing the long-term and common patterns in disease risk warning, and can effectively reveal the potential risk patterns of the disease. Specifically, the low-frequency feature commonality decoupling layer adopts the spectral modulation attention mechanism of the weighted spectral matrix, and enhances the model's attention and extraction of low-frequency features by adaptively adjusting the low-frequency components in the input high-order embedded features. The specific algorithm is as follows:

[0047]

[0048] in It is The absolute value of the spectrum of the input feature reflects the intensity of the low-frequency component in the spectrum. is the feature dimension, is the total number of features. represents the extracted low-frequency features, The feature vector output by the previous layer is obtained through three different linear transformations. is the dimension of the key vector.

[0049] The high-frequency feature-specific decoupling layer focuses on extracting high-frequency features of multimodal data, namely local details and sudden changes in the data, which can help distinguish subtle differences between different modal data. Specifically, the high-frequency feature-specific decoupling layer uses the spectral modulation attention mechanism to adaptively adjust the high-frequency components in the input high-order embedded features, emphasizing the modality-specific detail features. The specific calculation formula is as follows:

[0050]

[0051] in, is the square of the spectral intensity in the input high-order embedded feature, indicating the size of the high-frequency component. Represents the extracted high-frequency features.

[0052] S124: Calculate the mean of low-frequency features and high-frequency features ( ) and variance ( ), by Gaussian sampling from the mean ( ) and variance ( ) to obtain the final low-frequency features and high-frequency features; Among them, the low-frequency features and high-frequency features of different modal data can reconstruct the high-order embedded features of the original input after passing through the decoder module ( ), this reconstruction process ensures the stability of model training and avoids the problem of training instability caused by feature loss or deformation. Finally, the decoder module recovers the embedded representation that matches the input high-order embedded features by decoding the combination of low-frequency features and high-frequency features.

[0053] It can be understood that through the above-mentioned feature decoupling process, the model can efficiently separate the common information and unique information in multimodal data, and through the alternating action of the encoder and decoder modules, it ensures the stability and effectiveness of different modal features, providing a solid foundation for subsequent multimodal data fusion, which is conducive to the model to more accurately identify and evaluate the potential risks of disease risk warning, and improve the accuracy and personalization level of prediction.

[0054] S130: aligning the decoupled low-frequency features using a cosine alignment algorithm, and orthogonally projecting the high-frequency features using an orthogonal projection technique, and then performing multi-level fusion of the low-frequency features and the high-frequency features through an adaptive deep fusion network to obtain fused multi-modal joint features; In this step, at the feature fusion stage, the present application fuses the decoupled low-frequency features and high-frequency features through a multi-level interpretable feature fusion module to achieve effective joint representation of multimodal data and provide more accurate support for disease risk warning of target patients. First, the interpretable feature fusion module captures the commonalities between modalities by cosine alignment of low-frequency features, then orthogonally projects high-frequency features to maintain the uniqueness of different modalities, and finally, fuses all modal features through an adaptive deep fusion network, which can clearly parse the relationship between shared features and unique features, and ensure that the model has good interpretability, thereby improving the accuracy and personalization of cardiovascular event risk assessment.

[0055] Specifically, the feature fusion process of the interpretable feature fusion module includes the following steps: S131: performing cosine alignment on low-frequency features by using a cosine alignment algorithm; Among them, low-frequency features usually contain common information in multimodal data and can reflect the global pattern of cardiovascular events. In order to enhance the correlation between low-frequency features between multimodal data, this application first projects the low-frequency representation of each modal data to the same subspace through a cosine alignment algorithm. In this way, the shared information in different modal data can be uniformly represented to ensure that the low-frequency representations in different modal data remain consistent in space, thereby better revealing the potential risk patterns of the disease. Specifically, the cosine alignment algorithm calculates the cosine similarity between low-frequency features and constrains the alignment of low-frequency features through a strategy that minimizes the cosine distance, thereby further optimizing the collaborative representation of low-frequency features between different modal data. The cosine alignment formula is as follows:

[0056] in and Respectively represent and The low-frequency characteristics of the modal data, is the Euclidean norm of the vector.

[0057] Defining the loss function , minimizing the cosine distance between low-frequency features is:

[0058] in, and Respectively represent the number of modes of multimodal data.

[0059] Through the above alignment operation, the low-frequency features of each modal data can work together better, providing a unified representation space for subsequent multimodal data fusion, helping to discover key risk factors for disease risk warning, and providing a quantitative basis for clinical decision-making.

[0060] S132: combining expert knowledge, weighting the aligned low-frequency features through a multilinear adaptive weighting algorithm to obtain weighted low-frequency features; Among them, in order to further enhance the clinical applicability of low-frequency features, this application combines expert knowledge and uses a multilinear adaptive weighting algorithm to weight low-frequency features, giving higher weights to information related to disease risk warning, and obtaining weighted low-frequency features. , so that the model can more effectively focus on the characteristics that are closely related to the patient's perioperative risk. The specific weighting formula is as follows:

[0061] in, Based on expert knowledge The adaptively calculated weighting coefficient is calculated by the following formula:

[0062] is a learnable weight vector.

[0063] S133: Fusing all weighted low-frequency features to obtain a fused low-frequency feature representation; wherein the final fused low-frequency feature representation for:

[0064] in represents the weight coefficient of low-frequency features, Indicates the number of modes.

[0065] Through this process, the model is able to effectively integrate low-frequency features in multimodal data, thereby more accurately identifying perioperative risk factors.

[0066] S134: performing weighted fusion after performing redundancy removal on the high-frequency features of the multimodal data by using an orthogonal projection algorithm to obtain a fused high-frequency feature representation; Among them, in the perioperative disease risk warning, high-frequency features represent the local details and unique information in different modal data, which is of great significance for predicting sudden changes in diseases and their risks. However, since high-frequency features in multimodal data may contain redundant information and noise, in order to improve the robustness and accuracy of the model, these high-frequency features must be effectively extracted and fused. To this end, this application designs a high-frequency orthogonal projection layer, which uses an orthogonal projection algorithm to deduplicate the high-frequency features of multimodality, thereby maximally retaining the modality-specific detail information and reducing noise interference. Specifically, first, the high-frequency features Normalize to ensure that the feature vectors have the same length:

[0067] in It is The high-frequency eigenvector of the normalized modal data, Represents the Euclidean norm of a vector.

[0068] In order to further remove the redundant information between the high-frequency features of different modal data, the independence of the high-frequency features of each modal data is enhanced by orthogonal projection. High-frequency features of modal data , projecting it outside the space of other modal features, the formula is as follows:

[0069] in, It is The high-frequency features of the modal data after projection are represented by Represents the inner product operation.

[0070] Finally, the high-frequency features after orthogonal projection are linearly weighted Fusion is performed to obtain the final high-frequency feature representation :

[0071] in is the weight coefficient of high-frequency features, Indicates the number of modes, weight coefficient It can be adjusted according to the importance of each modality data to ensure that the high-frequency features of each modality data receive appropriate attention during the final fusion.

[0072] Through the above process, the model can efficiently fuse the high-frequency features of each modality data, remove redundancy and noise, and enhance the accurate assessment of perioperative disease risk warning.

[0073] S135: weighted fusion of the fused low-frequency feature representation and high-frequency feature representation is performed through an adaptive deep fusion network to obtain a unified multi-modal joint feature; Among them, in order to further improve the fusion effect of low-frequency features and high-frequency features, this application performs feature fusion by designing an adaptive deep fusion network. The network performs spatiotemporal coordination and spatial consistency projection on low-frequency features and high-frequency features through two-layer sub-networks, and realizes dynamic modeling of low-frequency features and high-frequency features in time and space dimensions, thereby ensuring that multimodal data is fully integrated with low-frequency features and high-frequency features in time and space dimensions, and enhancing the sensitivity and robustness of the model to disease risk warning. The high-frequency features are mapped to a low-dimensional space through orthogonal projection technology to remove the interference of noise and redundant information, improve the signal-to-noise ratio of the data, and enhance the independence of the modal features. At the same time, the feature contribution weights of each modality data are dynamically adjusted through a multilinear adaptive weighted algorithm combined with clinical expert knowledge to improve the interpretability of the model. The specific fusion process includes: S1351: Use the low-frequency-high-frequency spatiotemporal information collaborative subnetwork to perform spatiotemporal collaboration on low-frequency features and high-frequency features to explore the dynamic relationship between low-frequency features and high-frequency features, and ensure that low-frequency features and high-frequency features can effectively interact and correlate synchronously in the temporal and spatial dimensions; the specific operation is: perform one-dimensional temporal convolution on low-frequency features and high-frequency features, and then perform weighted fusion to obtain temporal features ; The spatial interaction relationship between low-frequency features and high-frequency features is calculated by point-by-point multiplication to ensure the spatial coordination of features and obtain spatial features Through spatiotemporal collaborative operations, we can not only capture the laws of disease risk warning in time changes, but also take into account the detailed information in spatial data.

[0074] S1352: Time feature and spatial characteristics Perform spatial consistency projection and normalization; Among them, by and spatial characteristics Perform spatial consistency projection to ensure the consistency and scale of low-frequency and high-frequency features when fused, avoiding fusion problems caused by inconsistent feature scales. The specific operation is as follows: First, linearly project the temporal features and spatial features respectively, and map them into a unified feature space:

[0075] in, is the projection weight matrix of the joint space, and are the dimensions of the original feature and projection space respectively.

[0076] Then, the projected time features and spatial characteristics Perform normalization to obtain normalized time features and spatial characteristics and .

[0077] Finally, the normalized time features and spatial characteristics and Perform weighted fusion to obtain unified multi-modal joint features :

[0078] in is the learnable fusion weight coefficient.

[0079] Through the fusion processing of the above-mentioned adaptive deep fusion network, effective multimodal joint features can be obtained. The multimodal joint features not only contain the common information in the multimodal data, but also retain the unique information of each modality data, which is helpful to help clinicians make more accurate diagnosis and treatment.

[0080] S140: inputting the fused multimodal features into a pre-trained risk prediction model, and outputting the disease risk warning probability of the target patient through the risk prediction model; In this step, in order to optimize the performance of the model in the risk prediction task, this application designs a high-low frequency collaborative contrast loss function By jointly optimizing the two losses of similarity constraints between low-frequency features and contrast between low-frequency and high-frequency features, while maintaining the shared information between low-frequency features, the uniqueness and time-varying information of high-frequency features are strengthened, the synergy of low-frequency and high-frequency features is further enhanced, and the model's ability to capture long-term patterns and sudden changes is enhanced, thereby improving the robustness and accuracy of the model in perioperative disease risk warning.

[0081] Specifically, the similarity constraint between low-frequency features is as follows: first, the Euclidean distance between low-frequency features of different modal data is calculated to ensure their similarity in the feature space and reduce the difference between low-frequency features. The calculation formula is as follows:

[0082] in and They are and The low-frequency characteristics of the modal data.

[0083] In order to ensure the complementarity of low-frequency features and high-frequency features in multimodal data learning, this application designs a comparison mechanism between low-frequency and high-frequency features to encourage them to maintain their differences, so that low-frequency features and high-frequency features can complement each other in the feature space, avoiding information redundancy and insufficient feature decoupling. The formula for the comparison mechanism between low-frequency and high-frequency features is as follows:

[0084] in, and Respectively represent The low-frequency and high-frequency features of each modal data, and Represents the number of modalities for multimodal data.

[0085] Combining the similarity constraints between low-frequency features and the contrast between low-frequency and high-frequency features is the final high-low-frequency collaborative contrast loss. :

[0086] It can be understood that the present application can ensure that low-frequency features and high-frequency features can complement each other well in the feature space, avoid information redundancy, and improve the accuracy of feature fusion by jointly optimizing the similarity constraints between low-frequency features and the comparison between low-frequency and high-frequency features.

[0087] In addition, the risk prediction model optimizes the risk probability of the model output through cross entropy loss. By calculating the above losses and back-propagating optimization, the model training is optimized. According to the risk warning probability output by the model, clinicians can timely understand the disease risk of patients in the perioperative period and take corresponding intervention measures. It can not only provide important risk warnings for surgery and guide the smooth progress of surgery, but also improve the success rate of surgery and the survival rate of patients, which has important clinical application value.

[0088] It can be understood that in the above embodiments, this application uses perioperative cardiovascular event risk warning as an example for explanation, and the applicant can also apply it to other types of disease risk warnings such as chronic disease monitoring. For different disease areas, this application does not need to adjust the model structure. It only needs to replace the multimodal data related to the target disease type as a training sample during the training phase to achieve efficient data fusion and feature extraction, and output the corresponding disease risk warning probability. In addition, the feature decoupling technology and adaptive deep fusion network constrained by expert knowledge designed in this application can also be applied to data fusion tasks in the industrial field, such as multi-sensor data anomaly detection, complex system fault prediction, etc. By adjusting the input data type and task objectives, this application can achieve a wider range of data fusion applications with high versatility and adaptability.

[0089] The multimodal data fusion method of the embodiment of the present application effectively processes the differences of each modal data by designing a corresponding dedicated encoder for each modal data, improves the signal-to-noise ratio of the data, solves the heterogeneity and heterogeneity problems of multimodal data, and ensures that multimodal data can be effectively fused; the high-order features of each modal data are decomposed into low-frequency features and high-frequency features by expert knowledge, effectively eliminating redundant information and noise, reducing the impact of high coupling between modalities, and ensuring the independence of each modal information; through the multi-level interpretable feature fusion module, the low-frequency features and high-frequency features are fused, and the spatiotemporal correlation modeling of the data is realized, and a transparent explanation of data fusion is provided, which improves the credibility and availability of the results. This application solves the problem of information sharing and differentiated extraction between multimodal data through the decoupling of high and low frequency features, the extraction of commonality and uniqueness of features between modalities, and the fusion of multi-level interpretable features, which can significantly improve the accuracy and personalization of disease risk warning.

[0090] See also Figure 3 , is a schematic diagram of the structure of a multimodal data fusion device according to an embodiment of the present application. The multimodal data fusion method device 40 according to an embodiment of the present application comprises: Feature extraction module 41: used to extract high-order features from the multimodal data to be fused using a dedicated modal encoder to obtain high-order embedded features of the multimodal data; Feature decoupling module 42: used to decouple the high-order embedded features by combining expert knowledge constraints and a multi-head attention mechanism, and obtain high-frequency features representing local information and low-frequency features representing global information respectively; Feature fusion module 43: used to align the low-frequency features using a cosine alignment algorithm, perform orthogonal projection on the high-frequency features using an orthogonal projection technique, and perform multi-level fusion of the aligned low-frequency features and the orthogonally projected high-frequency features through an adaptive deep fusion network to obtain fused multi-modal joint features; The risk prediction module 44 is used to input the fused multimodal features into a pre-trained risk prediction model, and output the disease risk warning probability through the risk prediction model.

[0091] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0092] The device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0093] See also Figure 4, is a schematic diagram of the computer device structure of an embodiment of the present application. The computer device 50 includes: A memory 51 storing executable program instructions; A processor 52 connected to the memory 51; The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: use a dedicated modal encoder to extract high-order features of the multimodal data to be fused, and obtain high-order embedded features of the multimodal data; decouple the high-order embedded features in combination with expert knowledge constraints and a multi-head attention mechanism to obtain high-frequency features representing local information and low-frequency features representing global information respectively; use a cosine alignment algorithm to align the low-frequency features, use an orthogonal projection technique to orthogonally project the high-frequency features, and perform multi-level fusion of the aligned low-frequency features and the orthogonally projected high-frequency features through an adaptive deep fusion network to obtain fused multimodal joint features; input the fused multimodal features into a pre-trained risk prediction model, and output the disease risk warning probability through the risk prediction model.

[0094] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having the ability to process signals. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0095] See also Figure 5, is a schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores a program instruction 61 that can implement the following steps: using a dedicated modal encoder to extract high-order features from the multimodal data to be fused, and obtaining high-order embedded features of the multimodal data; combining expert knowledge constraints and multi-head attention mechanisms to decouple the high-order embedded features, and respectively obtain high-frequency features representing local information and low-frequency features representing global information; using a cosine alignment algorithm to align the low-frequency features, using an orthogonal projection technique to orthogonally project the high-frequency features, and through an adaptive deep fusion network, the aligned low-frequency features and the high-frequency features after orthogonal projection are multi-level fused to obtain fused multimodal joint features; the fused multimodal features are input into a pre-trained risk prediction model, and the disease risk warning probability is output through the risk prediction model. Among them, the program instruction 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network computer device, etc.) or a processor (processor) to perform all or part of the steps of the various implementation methods of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program instructions, or terminal computer devices such as computers, servers, mobile phones, tablets, etc. Among them, the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0096] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0097] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A multimodal data fusion method, characterized in that: include: A dedicated modal encoder is used to extract high-order features from the multimodal data to be fused, and high-order embedding features of the multimodal data are obtained; Combining expert knowledge constraints and a multi-head attention mechanism to decouple the high-order embedded features, high-frequency features representing local information and low-frequency features representing global information are obtained respectively; The low-frequency features are aligned using a cosine alignment algorithm, the high-frequency features are orthogonally projected using an orthogonal projection technique, and the aligned low-frequency features and the orthogonally projected high-frequency features are multi-level fused using an adaptive deep fusion network to obtain fused multi-modal joint features; The fused multimodal features are input into a pre-trained risk prediction model, and the disease risk warning probability is output through the risk prediction model.

2. The multimodal data fusion method according to claim 1, wherein the multimodal data includes structured electronic health records, preoperative electrocardiograms, medical imaging data, and intraoperative physiological monitoring, characterized in that: The dedicated modal encoder is used to extract high-order features from the multimodal data to be fused, so as to obtain high-order embedding features of the multimodal data, specifically: For the structured electronic health record, high-order feature extraction is performed by an encoder based on a combination of a multi-layer perceptron and a decision tree; For the preoperative electrocardiogram and intraoperative physiological monitoring data, Fourier transform and frequency domain attention mechanism are used to extract high-order features in the frequency domain; For medical imaging data, detailed features are extracted through deep convolutional networks.

3. The multimodal data fusion method according to claim 2, characterized in that: The high-order embedding features are decoupled by combining expert knowledge constraints and multi-head attention mechanisms to obtain high-frequency features representing local information and low-frequency features representing global information, respectively. Specifically, The expert knowledge of the relevant disease risk warning is encoded into an embedding vector through an embedding layer, and the embedding vector is concatenated with the high-order embedding feature and then input into the feature decoupling module; The feature decoupling module processes the high-order embedded features through a normalization layer, a multi-head attention mechanism layer, and a feed-forward fully connected layer in sequence to capture the potential relationship and feature relationship between the multimodal data; Based on the potential relationship and feature relationship between the multimodal data, the high-order embedded features are parsed through a low-frequency feature common decoupling layer and a high-frequency feature specific decoupling layer to extract the low-frequency features and high-frequency features of the multimodal data; The mean and variance of the low-frequency features and the high-frequency features are calculated, and the mean and variance are sampled by a Gaussian sampling method to obtain the final low-frequency features and high-frequency features.

4. The multimodal data fusion method according to claim 3, characterized in that: The feature decoupling module processes the high-order embedded features through a normalization layer, a multi-head attention mechanism layer, and a feed-forward fully connected layer in sequence to capture the potential relationship and feature relationship between the multimodal data, specifically: The high-order embedding features of each modality data are standardized by the normalization layer; given the embedding features , the normalization layer normalizes it by the following formula: in and They are The mean and standard deviation of Through the multi-head attention mechanism layer, the potential relationship between the high-order embedded features of each modal data is learned to mine the semantic information of each modal data; The output features of the multi-head attention mechanism layer are further extracted by introducing nonlinear transformation through the feed-forward fully connected layer to capture the characteristic relationship of each modality data.

5. The multimodal data fusion method according to claim 3, characterized in that: Based on the potential relationship and feature relationship between the multimodal data, the high-order embedded features are parsed through the low-frequency feature common decoupling layer and the high-frequency feature specific decoupling layer respectively to extract the low-frequency features and high-frequency features of the multimodal data, specifically: The low-frequency feature commonality decoupling layer uses the spectrum modulation attention mechanism of the weighted spectrum matrix to adaptively adjust the low-frequency components in the high-order embedded features, thereby enhancing the model's attention to and extraction of low-frequency features: in It is The absolute value of the spectrum of the input features, is the feature dimension, is the total number of features; represents the extracted low-frequency features, The feature vector output by the previous layer is obtained through three different linear transformations. is the dimension of the key vector; The high-frequency feature-specific decoupling layer uses a spectral modulation attention mechanism to adaptively adjust the high-frequency components in the high-order embedded features, emphasizing the modality-specific detail features: in, is the square of the spectral intensity in the input high-order embedded feature, indicating the size of the high-frequency component. Represents the extracted high-frequency features.

6. The multimodal data fusion method according to any one of claims 1 to 5, characterized in that: The cosine alignment algorithm is used to align the low-frequency features, and the orthogonal projection technology is used to perform orthogonal projection on the high-frequency features, specifically: The cosine similarity between the low-frequency features is calculated by the cosine alignment algorithm, and the alignment of the low-frequency features is constrained by the strategy of minimizing the cosine distance: in and Respectively represent and The low-frequency characteristics of the modal data, is the Euclidean norm of the vector; Combined with expert knowledge, the aligned low-frequency features are weighted by a multilinear adaptive weighting algorithm to obtain weighted low-frequency features. ; The weighted formula is: in, Based on expert knowledge Adaptively calculated weighting coefficients; All weighted low-frequency features are fused to obtain the fused low-frequency feature representation for: in represents the weight coefficient of low-frequency features, Indicates the number of modes; The high-frequency features are To standardize: in It is The high-frequency eigenvector of the normalized modal data, Represents the Euclidean norm of a vector; For high frequency features , projecting it outside the space of other modal features: in, It is The high-frequency features of the projected modal data are represented by represents the inner product operation; The high-frequency features after orthogonal projection are linearly weighted Fusion is performed to obtain the final high-frequency feature representation : in is the weight coefficient of high-frequency features, Indicates the number of modes.

7. The multimodal data fusion method according to claim 6, characterized in that: The adaptive deep fusion network is used to perform multi-level fusion of the aligned low-frequency features and the orthogonally projected high-frequency features to obtain the fused multi-modal joint features, specifically: The low-frequency and high-frequency spatiotemporal information collaborative sub-network performs one-dimensional temporal convolution on the low-frequency features and high-frequency features and then performs weighted fusion to obtain the temporal features. ; Calculate the spatial interaction relationship between the low-frequency features and the high-frequency features by point-by-point multiplication to obtain the spatial features ; The time feature and spatial characteristics Perform spatial consistency projection and normalization to obtain normalized time features and spatial characteristics and ; The normalized time feature and spatial characteristics and Perform weighted fusion to obtain unified multi-modal joint features : in is the learnable fusion weight coefficient.

8. A multimodal data fusion device, characterized in that: include: Feature extraction module: used to extract high-order features from the multimodal data to be fused using a dedicated modal encoder to obtain high-order embedded features of the multimodal data; Feature decoupling module: used to decouple the high-order embedded features by combining expert knowledge constraints and multi-head attention mechanism, and obtain high-frequency features representing local information and low-frequency features representing global information respectively; Feature fusion module: used to align the low-frequency features using the cosine alignment algorithm, perform orthogonal projection on the high-frequency features using the orthogonal projection technology, and perform multi-level fusion of the aligned low-frequency features and the orthogonally projected high-frequency features through an adaptive deep fusion network to obtain fused multi-modal joint features; Risk prediction module: used to input the fused multimodal features into a pre-trained risk prediction model, and output the disease risk warning probability through the risk prediction model.

9. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the multimodal data fusion method according to any one of claims 1 to 7; The processor is used to execute the program instructions stored in the memory to control a multimodal data fusion method.

10. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the multimodal data fusion method according to any one of claims 1 to 7.

Images