Diagnosis auxiliary system based on cross-modal attention mechanism
By achieving precise alignment between physiological time-series data and imaging data through a cross-modal attention mechanism, the problem of inaccurate diagnosis in existing technologies is solved, and the accuracy and interpretability of multimodal diagnosis are improved.
Patent Information
- Application Number
- CN202511777177.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-02-27
AI Technical Summary
Existing multimodal medical diagnostic systems cannot effectively capture and utilize the dynamic changes in high-frequency physiological signals, leading to inaccurate diagnoses and an inability to establish a causal mapping between physiological time-series data and imaging data.
By employing a cross-modal attention mechanism, a spatiotemporal correlation fusion feature tensor is generated through multimodal data acquisition, spatiotemporal feature extraction, anomaly capture, and cross-modal attention fusion, thereby achieving accurate alignment and causal mapping between physiological time-series data and image data.
It improves the accuracy of multimodal diagnosis, achieves precise alignment between physiological time-series data and imaging data, establishes a causal mapping between signal events and anatomical locations, and enhances the accuracy and interpretability of diagnosis.
Smart Images
Figure CN121583508A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical diagnostic system technology, and in particular to a diagnostic assistance system based on a cross-modal attention mechanism. Background Technology
[0002] With the development of precision medicine, clinical diagnosis is no longer satisfied with single static images or discrete laboratory indicators, but aims to integrate multimodal information to achieve accurate early, localized, and qualitative judgments. Among them, high-sampling-rate physiological time-series signals (such as electrocardiogram and electroencephalogram) contain dynamic pathological features at the second to millisecond level, and their fusion with high-resolution anatomical images is regarded as a key direction to overcome the bottlenecks in the diagnosis and treatment of various diseases (such as arrhythmia and epilepsy).
[0003] In existing multimodal medical diagnostic systems, the first step is to perform simple timestamp alignment on physiological time-series data and image data. Then, features are extracted from the timestamp-aligned physiological time-series and image data separately, and these early features are concatenated to form a long vector before being fed into the recognition model for analysis. Alternatively, the timestamp-aligned physiological time-series features and image features are fed into the recognition model separately for analysis, and the results are then fused later. Some medical diagnostic systems even reduce computational effort by downsampling the physiological time-series data to a few hertz or compressing it into mean, variance, and peak values before feature extraction.
[0004] The core characteristics of many diseases (such as arrhythmia, epilepsy, and sleep apnea) are hidden in the dynamic changes of physiological signals at the millisecond or second level. The existing technology, which reduces the frequency of using physiological time series and compresses the values into mean, variance, and peak values before feature extraction, cannot capture and utilize this high-frequency information, resulting in inaccurate subsequent diagnosis.
[0005] In addition, the use of early splicing or late fusion methods does not effectively align physiological time-series data and imaging data at specific time points with imaging data of specific regions, making it impossible to establish a causal mapping between signal events and anatomical locations, resulting in inaccurate subsequent diagnoses. Summary of the Invention
[0006] Based on this, and to address the aforementioned technical problems, a diagnostic assistance system based on a cross-modal attention mechanism is provided to solve the problem that existing technologies only align timestamps of physiological time-series data and image data, thus reducing diagnostic accuracy.
[0007] A diagnostic assistance system based on a cross-modal attention mechanism, the system comprising:
[0008] Multimodal data acquisition module: used to acquire image data and physiological time-series data of the same location of the patient; and to preprocess the image data and physiological time-series data, and to perform timestamp alignment;
[0009] Spatiotemporal feature extraction module: used to extract image spatial features from the image data and extract physiological time series information features from the physiological time series data;
[0010] Anomaly capture module: used to acquire physiological time-series information features of abnormal time points determined based on the physiological time-series information feature sequence, or image spatial features of abnormal regions determined based on the image spatial features;
[0011] A cross-modal attention fusion module is used to perform cross-modal attention calculations using the physiological temporal information features of the abnormal time points as query vectors and the image spatial features as key vectors, generating a first attention weight map indicating the correlation between the temporal information features of the abnormal time points and each region of the image spatial features; using the image spatial features of the abnormal regions as query vectors and the physiological temporal data as key vectors, it performs cross-modal attention calculations to generate a second attention weight map indicating the correlation between the image spatial features of the abnormal regions and each time period of the physiological temporal data; and obtains a spatiotemporal correlation fusion feature tensor by weighted averaging of the first and second attention weight maps.
[0012] Risk assessment module: used to output diagnostic results based on the spatiotemporal correlation fusion feature tensor.
[0013] Optionally, in the above scheme, the system further includes: an interpretive output module; the interpretive output module includes:
[0014] Spatiotemporal correlation view unit: used to dynamically mark the attention weight distribution corresponding to the physiological time sequence information features on the image data according to the spatiotemporal correlation fusion feature tensor, and generate change animation;
[0015] Fusion Reporting Unit: Used to generate a natural language report based on the temporal information features of the abnormal time points and the image of the temporal information feature region of the abnormal time points.
[0016] Optionally, in the above scheme, the risk assessment module is further used for:
[0017] Based on the obtained spatiotemporal correlation fusion feature tensors, a risk time series curve is predicted to show the future occurrence probability over time. When the risk time series curve exceeds a preset threshold, an early warning signal is output.
[0018] Optionally, in the above scheme, preprocessing of the image data and physiological time series data includes: performing bandpass filtering and power frequency notch filtering on the physiological time series data.
[0019] In the above scheme, optionally, the spatiotemporal feature extraction module is specifically used to extract image spatial features from the image data using a three-dimensional convolutional neural network, and to extract physiological time-series information features containing timestamps from the physiological time-series data using a one-dimensional convolutional neural network.
[0020] Optionally, in the above scheme, the system further includes an anomaly identification module: the anomaly identification module is used to determine the physiological time sequence information features of the abnormal time point based on the physiological time sequence information feature sequence, or to determine the image spatial features of the abnormal region based on the image spatial features;
[0021] Used to determine the results of anomaly identification by the anomaly identification module or to provide instructions from external input.
[0022] In the above scheme, optionally, the anomaly capture module is used to obtain physiological time sequence information features of the abnormal time point or image spatial features of the abnormal region from the anomaly recognition module or external input instructions.
[0023] Optionally, in the above scheme, the cross-modal attention fusion module generates an attention weight map through the following steps:
[0024] The original similarity score is obtained by calculating the similarity between the query vector and the key vector;
[0025] The original similarity scores are normalized to generate an attention weight map.
[0026] This application has at least the following beneficial effects:
[0027] This application acquires high-frequency physiological time-series data and image data, performs preprocessing and timestamp alignment, extracts image spatial features from the image data, and extracts physiological time-series information features from the physiological time-series data; acquires physiological time-series information features of abnormal time points determined based on the sequence of physiological time-series information features, or image spatial features of abnormal regions determined based on image spatial features; performs cross-modal attention calculation using the physiological time-series information features of abnormal time points as query vectors and image spatial features as key vectors to generate a first attention weight map indicating the correlation between the time-series information features of abnormal time points and each region of image spatial features; simultaneously performs cross-modal attention calculation using the image spatial features of abnormal regions as query vectors and physiological time-series data as key vectors to generate a second attention weight map indicating the correlation between the image spatial features of abnormal regions and each time period of physiological time-series data; and obtains a spatiotemporal correlation fusion feature tensor by weighted averaging of the first and second attention weight maps; and outputs diagnostic results based on the spatiotemporal correlation fusion feature tensor. Therefore, the cross-modal attention fusion module of this application can be used to align physiological time-series data at specific time points and imaging data of specific regions, establish a causal mapping between signal events and anatomical locations, thereby achieving a good alignment relationship and improving the accuracy of subsequent diagnosis. Attached Figure Description
[0028] Figure 1 This application provides a modular framework diagram of a diagnostic assistance system based on a cross-modal attention mechanism as one embodiment;
[0029] Figure 2 A flowchart of the execution of a cross-modal attention fusion module is provided for one embodiment of this application;
[0030] Figure 3 This is a modular framework diagram of a diagnostic assistance system based on a cross-modal attention mechanism in a scenario for locating epileptogenic foci in one embodiment of this application;
[0031] Figure 4 This is a flowchart of the multimodal data fusion steps in one embodiment of this application. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0033] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).
[0034] In one embodiment, such as Figure 1 As shown, a diagnostic assistance system based on a cross-modal attention mechanism is provided, the system comprising:
[0035] Multimodal data acquisition module: used to acquire image data and physiological time-series data of the same location of the patient; and to preprocess the image data and physiological time-series data, and to perform timestamp alignment.
[0036] The multimodal data acquisition module is used to simultaneously or alignedly acquire high-resolution medical images (such as MRI or ultrasound images) and high-sampling-rate clinical physiological time-series data of patients (such as EEG signals with a sampling rate of 500Hz-1000Hz and ECG signals with a sampling rate of 256Hz-512Hz). This module includes signal denoising, timestamp alignment, and segmentation functions. Preprocessing of the image data and physiological time-series data includes bandpass filtering and power frequency notch filtering for the physiological time-series data.
[0037] Spatiotemporal feature extraction module: used to extract image spatial features from the image data and extract physiological time series information features from the physiological time series data.
[0038] Spatiotemporal Feature Extraction Module: Connected to the multimodal data acquisition module, this module extracts high-level features with spatiotemporal semantics from data of different modalities. Specifically, it includes:
[0039] (1) Image spatial feature extraction unit: Three-dimensional convolutional neural network is used to extract three-dimensional spatial structural features from medical images.
[0040] (2) Time-series signal feature extraction unit: A one-dimensional convolutional neural network is used to extract the dynamic features in the time and frequency domains from the original high-frequency physiological signals and retain their accurate time positioning information.
[0041] Anomaly capture module: used to acquire physiological time sequence information features of abnormal time points determined based on the physiological time sequence information feature sequence, or image spatial features of abnormal regions determined based on the image spatial features.
[0042] Anomaly capture module: used to obtain physiological time-series information features of abnormal time points or image spatial features of abnormal areas from the anomaly recognition module or external input instructions.
[0043] Anomaly identification module: used to determine the physiological time sequence features of abnormal time points based on the physiological time sequence features, or to determine the image spatial features of abnormal regions based on the image spatial features.
[0044] The cross-modal attention fusion module is used to perform cross-modal attention calculation using the physiological temporal information features of the abnormal time point as the query vector and the image spatial features as the key vector, generating a first attention weight map indicating the correlation between the temporal information features of the abnormal time point and each region of the image spatial features; using the image spatial features of the abnormal region as the query vector and the physiological temporal data as the key vector, performing cross-modal attention calculation, generating a second attention weight map indicating the correlation between the image spatial features of the abnormal region and each time period of the physiological temporal data; and obtaining a spatiotemporal correlation fusion feature tensor by weighted averaging the first attention weight map and the second attention weight map.
[0045] like Figure 4 As shown, the cross-modal attention fusion module uses the features of the temporal signal at a specific time point T (such as the peak time of epileptiform discharge) as a "trigger event." This event feature is used as a query vector to search for the most relevant anatomical region in the image's spatial feature map. Specifically, a cross-modal attention layer calculates the similarity between the temporal event features and all spatial location features of the image, generating an attention weight map. This weight map is overlaid on the original image, visually displaying "the possible origin or affected brain regions of the EEG event at time T." Conversely, features of a suspicious lesion region (such as a tumor) in the image can also be used as a query to retrieve features from all time points in the temporal signal, identifying specific signal patterns related to the lesion's activity. The final output is a spatiotemporally correlated fusion feature tensor. Figure 2 The diagram shows the steps involved in modal fusion using the cross-modal attention fusion module.
[0046] Risk assessment module: used to output diagnostic results based on the spatiotemporal correlation fusion feature tensor.
[0047] The risk assessment module dynamically outputs data based on the fused spatiotemporal features. This module not only outputs the disease lesion, but also predicts the future onset probability over time using multiple spatiotemporally correlated fused feature tensors, generating a risk time-series curve. Furthermore, it outputs an early warning signal when the risk time-series curve exceeds a preset threshold. This process generates risk time-series curves and event warning signals, achieving a leap from static diagnosis to dynamic monitoring.
[0048] In the aforementioned diagnostic assistance system based on a cross-modal attention mechanism, high-frequency physiological time-series data and image data are acquired, preprocessed, and timestamp aligned. Image spatial features are extracted from the image data, and physiological time-series information features are extracted from the physiological time-series data. Physiological time-series information features of abnormal time points, determined based on the sequence of physiological time-series information features, or image spatial features of abnormal regions, determined based on image spatial features, are acquired. Cross-modal attention calculation is performed using the physiological time-series information features of abnormal time points as query vectors and image spatial features as key vectors to generate a first attention weight map indicating the correlation between the time-series information features of abnormal time points and each region of image spatial features. Cross-modal attention calculation is then performed using the image spatial features of abnormal regions as query vectors and physiological time-series data as key vectors to generate a second attention weight map indicating the correlation between the image spatial features of abnormal regions and each time period of physiological time-series data. A spatiotemporal correlation fusion feature tensor is obtained by weighted averaging the first or second attention weight map. The diagnostic result is output based on the spatiotemporal correlation fusion feature tensor. Therefore, through the cross-modal attention fusion module of this application, physiological time series data at a specific time point and imaging data of a specific region can be aligned to establish a causal mapping between signal events and anatomical locations, thereby achieving a good alignment relationship and improving the accuracy of subsequent diagnosis.
[0049] In one embodiment, the system further includes: an interpretive output module; the interpretive output module includes:
[0050] Spatiotemporal correlation view unit: used to dynamically mark the attention weight distribution corresponding to the physiological time sequence information features on the image data according to the spatiotemporal correlation fusion feature tensor, and generate change animation;
[0051] Fusion Reporting Unit: Used to generate a natural language report based on the temporal information features of the abnormal time points and the image of the temporal information feature region of the abnormal time points.
[0052] In this embodiment, the interpretive output module connects to the risk assessment and is used to generate multi-dimensional visual explanations. It includes:
[0053] (1) Spatiotemporal correlation view: A piece of animation is dynamically played on the medical image to show the changes in brain regions activated by different EEG events over time.
[0054] (2) Fusion report: Generate a report, such as: "The system detected a spike-and-wave discharge at time point T, the spatial source of which is highly consistent with the right hippocampal sclerosis region shown in the MRI image (attention weight > 0.9), strongly suggesting that this is an epileptogenic focus."
[0055] In one embodiment, the cross-modal attention fusion module generates an attention weight map through the following steps:
[0056] In one embodiment, taking the localization of epileptogenic foci based on brain MRI and scalp EEG signals as an example, such as... Figure 3 As shown, the specific functions of each module in the diagnostic assistance system based on a cross-modal attention mechanism proposed in this application are as follows:
[0057] Multimodal data acquisition module: Collects high-resolution structural MRI data of the patient's brain. Simultaneously acquires several hours of scalp EEG data at a sampling rate of 1000Hz. Bandpass filtering and power frequency notch preprocessing are applied to the EEG data, and a synchronous marker signal is used to ensure indirect temporal alignment between the EEG and MRI.
[0058] Spatiotemporal Feature Extraction Module: The image spatial feature extraction unit uses a 3D ResNet model to extract spatial feature maps of the whole brain from MRI. The temporal signal feature extraction unit uses 1D-CNN + Transformer Encoder to process the raw EEG signal and output a feature vector for each time point (or short time window), which encodes the EEG activity pattern at that moment.
[0059] Anomaly capture module: Acquires epileptiform discharge events (such as spikes) in EEG signals marked by external input doctors, or epileptiform discharge events in EEG signals automatically determined by the anomaly recognition module, and extracts the feature vector of their peak time T.
[0060] Cross-modal attention fusion module: Using the EEG feature vector at time T as the query and the entire MRI spatial feature map as the key and value, it performs cross-modal attention computation. It generates an attention heatmap that highlights the brain regions most relevant to this spike event on the MRI.
[0061] The risk assessment module performs statistical analysis on multiple such events, outputs an epileptogenic focus probability map, and can calculate the frequency and spread trend of epileptic activity.
[0062] The interpretive output module generates the final localization report and dynamic visualization animation, which intuitively demonstrates the correlation between specific EEG events and brain anatomy.
[0063] Compared with the prior art, the present invention has the following significant advantages:
[0064] 1. Achieved true dynamic pathological capture: By fusing high-frequency time-series signals, the system is able to capture instantaneous physiological events and their dynamic processes that cannot be displayed by static images.
[0065] 2. Solved the problem of alignment and fusion of high-resolution multimodal data: The proposed time-triggered cross-modal attention mechanism establishes a precise bridge between millisecond-level signals and millimeter-level images, achieving an unprecedented level of fine fusion granularity.
[0066] 3. Diagnostic dimensions have expanded from static to dynamic: the output is no longer a single label, but a dynamic assessment that includes risk trends and event alerts, enabling real-time clinical monitoring.
[0067] 4. The pathophysiological mechanisms are highly interpretable: They can directly visualize the correspondence between physiological events and anatomical structures, providing doctors with a powerful tool for understanding disease mechanisms.
[0068] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0069] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A diagnostic assistance system based on cross-modal attention mechanism, characterized in that, The system comprises: a multi-modal data acquisition module: for acquiring image data and physiological time series data of the same position of a patient; and pre-processing the image data and the physiological time series data, and performing timestamp alignment; a spatio-temporal feature extraction module: for extracting image spatial features from the image data, and extracting physiological time series information features from the physiological time series data; an anomaly capturing module: for acquiring physiological time series information features of an abnormal time point determined according to the physiological time series information feature sequence, or image spatial features of an abnormal region determined according to the image spatial features; a cross-modal attention fusion module: for performing cross-modal attention calculation by taking the physiological time series information features of the abnormal time point as a query vector and taking the image spatial features as a key vector, to generate a first attention weight map indicating the relevance of the time series information features of the abnormal time point to each region of the image spatial features; and performing cross-modal attention calculation by taking the image spatial features of the abnormal region as a query vector and taking the physiological time series data as a key vector, to generate a second attention weight map indicating the relevance of the image spatial features of the abnormal region to each time period of the physiological time series data; and performing weighted averaging on the first attention weight map and the second attention weight map to obtain a spatio-temporal correlation fusion feature tensor; a risk assessment module: for outputting a diagnosis result based on the spatio-temporal correlation fusion feature tensor. 2.The diagnostic assistance system based on cross-modal attention mechanism of claim 1, wherein, The system further comprises an explanatory output module, which comprises: a spatio-temporal correlation view unit: for dynamically marking the attention weight distribution corresponding to the physiological time series information features on the image data according to the spatio-temporal correlation fusion feature tensor, and generating a change animation; a fusion report unit: for generating a natural language report according to the time series information features of the abnormal time point and the image of the region of the abnormal time point. 3.The diagnostic assistance system based on cross-modal attention mechanism of claim 1, wherein, The risk assessment module is further configured to: perform prediction on a plurality of obtained spatio-temporal correlation fusion feature tensors to obtain a risk time series curve of the probability of future attack changing over time, and output a warning signal when the risk time series curve exceeds a preset threshold. 4.The diagnostic assistance system based on cross-modal attention mechanism of claim 1, wherein, The pre-processing of the image data and the physiological time series data comprises: band-pass filtering and power frequency notch filtering of the physiological time series data. 5.The diagnostic assistance system based on cross-modal attention mechanism of claim 1, wherein, The spatio-temporal feature extraction module is specifically configured to extract image spatial features from the image data by using a three-dimensional convolutional neural network, and extract physiological time series information features containing timestamps from the physiological time series data by using a one-dimensional convolutional neural network. 6.The diagnostic assistance system based on cross-modal attention mechanism of claim 1, wherein, The system further comprises an anomaly identification module, which is configured to determine the physiological time series information features of the abnormal time point according to the physiological time series information feature sequence, or determine the image spatial features of the abnormal region according to the image spatial features.
7. The diagnostic assistance system based on cross-modal attention mechanism according to claim 6, characterized in that, The anomaly capturing module is configured to acquire the physiological time series information features of the abnormal time point or the image spatial features of the abnormal region from the anomaly identification module or external input instructions.
8. The diagnostic assistance system based on cross-modal attention mechanism according to claim 1, wherein, The cross-modal attention fusion module generates the attention weight map by the following steps: The query vector and the key vector are calculated for similarity to obtain an original similarity score; Normalization is performed on the original similarity score to generate an attention weight map.