A multimodal medical data intelligent fusion retrieval method and system

Abstract

The application relates to the field of information technology, in particular to a multi-modal medical data intelligent fusion retrieval method and system. The method comprises the following steps: acquiring multi-modal medical data, wherein the multi-modal medical data at least comprises at least two of medical image data, electronic medical record text data and physiological signal data; pre-processing and standardizing the multi-modal medical data, and extracting high-dimensional semantic features of the multi-modal medical data in a preset high-dimensional semantic space; constructing a semantic association between modes through a cross-modal alignment mechanism, dynamically fusing the high-dimensional semantic features according to the semantic association, and generating a fused joint representation vector; in response to a natural language query request input by a user, mapping the query request to the same semantic space as the joint representation vector, and calculating the similarity between the query request and the joint representation vector; sorting the multi-modal medical data according to the similarity, and returning the sorting result as a retrieval result output.

Description

Technical Field

[0001] This application relates to the field of information technology, specifically to a method and system for intelligent fusion retrieval of multimodal medical data. Background Technology

[0002] Against the backdrop of rapid development in medical informatization, medical institutions have accumulated massive amounts of multimodal data, including medical images (such as CT and MRI), electronic medical record texts (such as diagnostic records and laboratory reports), and physiological signals (such as electrocardiograms and electroencephalograms). These data each carry different dimensions of patient health information and are highly complementary. However, due to significant differences in structure, semantic expression, and sampling methods among different modalities, traditional retrieval systems typically only support single-modal queries and matching, making it difficult to achieve unified understanding and efficient retrieval at the cross-modal semantic level. Existing technologies have attempted to fuse multimodal information through feature concatenation or simple weighted fusion, but lack the ability to model deep semantic relationships between modalities, resulting in insufficient discriminative and generalization capabilities of the fused representations. Therefore, there is an urgent need for an intelligent retrieval technology capable of achieving deep semantic alignment, dynamic fusion, and natural language-driven retrieval of multimodal medical data to improve the efficiency of medical information utilization and assist in clinical decision-making and scientific research analysis. Summary of the Invention

[0003] This specification describes a multimodal medical data intelligent fusion retrieval method and system through several embodiments.

[0004] Firstly, embodiments of this specification provide a method for intelligent fusion retrieval of multimodal medical data, comprising the following steps:

[0005] Acquire multimodal medical data from different data sources, wherein the multimodal medical data includes at least two of the following: medical imaging data, electronic medical record text data, and physiological signal data;

[0006] The multimodal medical data is preprocessed and standardized, and high-dimensional semantic features of the multimodal medical data in a preset high-dimensional semantic space are extracted.

[0007] Semantic associations between modalities are constructed through a cross-modal alignment mechanism, and high-dimensional semantic features are dynamically fused based on these semantic associations to generate a fused joint representation vector.

[0008] In response to a user's input natural language query request, the query request is mapped to the same semantic space as the joint representation vector, and the similarity between the query request and the joint representation vector is calculated.

[0009] The multimodal medical data is sorted according to the similarity, and the sorting results are returned as the search results output.

[0010] Secondly, embodiments of this specification provide a multimodal medical data intelligent fusion retrieval system, including:

[0011] The acquisition module acquires multimodal medical data from different data sources, wherein the multimodal medical data includes at least two of the following: medical image data, electronic medical record text data, and physiological signal data.

[0012] The extraction module preprocesses and standardizes the multimodal medical data, and extracts the high-dimensional semantic features of the multimodal medical data in a preset high-dimensional semantic space.

[0013] The fusion module constructs semantic associations between modalities through a cross-modal alignment mechanism, and dynamically fuses high-dimensional semantic features based on the semantic associations to generate a fused joint representation vector.

[0014] The query module, in response to a natural language query request input by the user, maps the query request to the same semantic space as the joint representation vector, and calculates the similarity between the query request and the joint representation vector.

[0015] The output module sorts the multimodal medical data according to the similarity and returns the sorting results as the search results.

[0016] Thirdly, embodiments of this specification provide an electronic device, including a processor and a memory;

[0017] The processor is connected to the memory;

[0018] The memory is used to store executable program code;

[0019] The processor runs a program corresponding to the executable program code stored in the memory to perform the method described in any of the above aspects.

[0020] Fourthly, embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above aspects.

[0021] Fifthly, embodiments of this specification provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above aspects.

[0022] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0023] This specification provides a multimodal medical data intelligent fusion retrieval method and system in several embodiments. By constructing a unified high-dimensional semantic space and introducing a cross-modal alignment mechanism, it effectively bridges the semantic differences between heterogeneous data such as medical images, electronic medical record text, and physiological signals, achieving semantic association between different modalities. Through a dynamic weighted fusion mechanism, complementary information is adaptively integrated based on the semantic relevance between modalities to generate a more discriminative joint representation vector, thereby enhancing the relevance of retrieval results. It supports end-to-end mapping and matching of natural language query requests, lowering the barrier to entry for doctors and researchers. This helps improve the utilization efficiency of medical big data, providing support for precision diagnosis and treatment, case review, and medical research.

[0024] Other features and advantages of various embodiments of this specification will be further revealed in the following detailed description and accompanying drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram illustrating the intelligent fusion retrieval of multimodal medical data provided in this manual.

[0027] Figure 2 This is a schematic diagram of the multimodal medical data intelligent fusion retrieval method provided in this manual.

[0028] Figure 3 This is a schematic diagram illustrating the semantic relationships between the construction modalities provided in this specification.

[0029] Figure 4 This is a schematic diagram of the multimodal medical data intelligent fusion retrieval system provided in this manual.

[0030] Figure 5 A schematic diagram of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0031] The technical solutions of the embodiments of this specification will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of this specification and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of this specification.

[0032] The terms "first," "second," "third," etc., in the description, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0033] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to facilitate the description of the embodiments and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this specification.

[0034] All data involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0035] Before introducing the technical solutions described in this manual, the application scenarios and related technologies of the technical solutions will be introduced.

[0036] Multimodal medical data refers to the collection of various types of medical information generated from different sources and collection methods during clinical diagnosis and treatment, health monitoring, and medical research. This data typically includes medical images (such as CT, MRI, X-rays, and ultrasound), electronic medical record texts (such as chief complaints, present medical history, diagnostic records, and laboratory reports), and physiological signals (such as time-series data like electrocardiograms, electroencephalograms, blood oxygen saturation, and respiratory waveforms). Each modality characterizes a patient's health status from different dimensions: medical images provide intuitive visual evidence of anatomical structure or functional imaging; electronic medical records carry the professional judgment and semantic descriptions of clinicians; and physiological signals reflect the dynamic changes and real-time responses of the body over time. These data are highly complementary in content and have potential semantic connections.

[0037] The core value of multimodal medical data lies in its ability to construct more comprehensive and accurate patient representations. Information from a single modality often has limitations; for example, images alone may not be sufficient to determine the onset time of symptoms, textual descriptions alone may lack objective quantitative evidence, and isolated physiological signals lack contextual clinical background. By organically combining multiple modalities, not only can mutual verification be achieved and misjudgments reduced, but implicit patterns across modalities can also be revealed. For instance, certain imaging manifestations are often accompanied by certain textual keywords or specific ECG abnormality patterns, thus providing more reliable evidence for disease classification, prognostic assessment, and treatment response prediction. In medical research, researchers can use large-scale aligned multimodal databases to explore disease mechanisms, mine biomarkers, or train AI models. In telemedicine and tiered healthcare systems, multimodal data uploaded by primary healthcare institutions can be efficiently retrieved and interpreted by higher-level hospitals, improving the accessibility of high-quality resources. In the field of health management, combining continuously collected physiological signals from wearable devices with personal electronic health records can enable early warning and personalized intervention for chronic disease risks. However, research on the fusion of multimodal data and its application in retrieving multimodal medical data is currently lacking.

[0038] Therefore, this specification provides a method and system for intelligent fusion retrieval of multimodal medical data. Please refer to the appendix. Figure 1 First, multimodal medical data from different data sources are obtained from heterogeneous medical information systems. These data include at least two or more of the following: medical image data 11, electronic medical record text data 12, and physiological signal data 13. Then, the obtained multimodal data are preprocessed and standardized in a targeted manner. Based on this, a deep learning model is used to extract high-dimensional semantic features 14 of each in a unified pre-defined high-dimensional semantic space. By introducing a cross-modal alignment mechanism, the association between different modalities is constructed at the semantic level, generating a joint representation vector 15 that can comprehensively reflect the complementary information from multiple sources. When a user inputs a query request in natural language (such as "finding cases with pulmonary nodules and accompanying cough symptoms"), the query statement is mapped to the same high-dimensional semantic space as the joint representation vector 15 through a semantic encoder. The semantic similarity between the query statement and all joint representation vectors 15 in the database is calculated. Based on this similarity, the relevant multimodal medical data are sorted, and the most relevant cases or data fragments are returned as search results 21 according to their relevance.

[0039] This manual first provides a method for intelligent fusion retrieval of multimodal medical data. Please refer to the appendix. Figure 2 This includes the following steps:

[0040] Step S1) Obtain multimodal medical data from different data sources, wherein the multimodal medical data includes at least two of the following: medical image data 11, electronic medical record text data 12, and physiological signal data 13.

[0041] Medical data is typically stored across multiple independent information systems. For example, Picture Archiving and Communication Systems (PACS) store image data such as CT scans, MRI scans, and X-rays; Hospital Information Systems (HIS) or Electronic Medical Records (EMR) record structured or unstructured text content such as chief complaints, present medical history, diagnoses, and medication records; while Intensive Care Unit (ICU) information systems or wearable devices continuously collect dynamic physiological signals such as electrocardiograms (ECGs), blood oxygen saturation, respiratory rate, and blood pressure. Integrating these heterogeneous but semantically related data sets allows for the construction of a multimodal dataset tailored to the same patient. For example, in a typical cardiovascular disease diagnosis and treatment scenario, the system can simultaneously acquire a patient's coronary CTA images (used to observe the degree of vascular stenosis), the chief complaint text such as "chest pain, shortness of breath after exertion" recorded in the outpatient electronic medical record, and the 12-lead electrocardiogram time-series signal continuously monitored during hospitalization. Similarly, in neurology, for suspected epilepsy patients, the system can jointly retrieve their electroencephalogram (EEG) signals, cranial MRI images, and descriptive text about seizure frequency and triggers from the medical record. By aggregating data from at least two modalities, not only are conditions provided for subsequent semantic alignment and fusion, but the search results21 can also more comprehensively reflect the patient's overall condition.

[0042] Step S2) Preprocess and standardize the multimodal medical data, and extract the high-dimensional semantic features 14 of the multimodal medical data in the preset high-dimensional semantic space.

[0043] Methods for preprocessing and standardizing the multimodal medical data include:

[0044] Denoising, normalization, and region of interest extraction were performed on the medical image data 11.

[0045] The electronic medical record text data 12 was segmented, stop words were removed, entities were identified, and terminology was standardized.

[0046] The physiological signal data 13 was filtered, resampled, and time-aligned.

[0047] Preprocessing and standardization can eliminate the heterogeneity of different modal data in terms of acquisition equipment, format, scale and semantic expression, making them comparable and fusionable, and providing conditions for subsequent cross-modal semantic alignment.

[0048] For medical image data 11, the system first performs denoising operations to suppress artifacts or random noise introduced during the imaging process (e.g., using non-local means filtering or deep learning denoising models for low-dose CT images), then performs pixel value normalization (such as uniformly mapping Hounsfield units to the [0,1] interval), and automatically extracts regions of interest in combination with lesion detection models or doctor annotation information. For example, in lung CT, the nodule region is located, and in brain MRI, the hippocampal structure is segmented, thus focusing on key regions with clinical significance; for electronic medical record text data 12, Chinese word segmentation is performed in sequence (such as using medical terms like "chest pain" and "hypertension" as segmentation units), common stop words are removed (such as虚词 without substantial semantic meaning like "de" and "le"), named entity recognition is performed to extract key clinical entities such as diseases, symptoms, examination items, and drugs, and further mapping them to a unified medical ontology system through term standardization (such as standardizing "myocardial infarction" and "acute myocardial infarction" to the ICD-10 code I21.9), ensuring that different expressions refer to the same clinical concept; for physiological signal data 13, band-pass filtering is required to remove power frequency interference or baseline drift (such as using a 0.5–40Hz filter for ECG signals), then resampling is performed according to the target sampling rate to unify the time resolution, and time alignment processing is implemented between multi-channel or multi-period signals. For example, electrocardiogram, respiration, and blood oxygen signals are synchronously aligned according to event markers (such as the onset time of arrhythmia) to ensure the consistency of temporal semantics.

[0049] The method for extracting high-dimensional semantic features 14 of the multi-modal medical data in a preset high-dimensional semantic space includes:

[0050] Using a convolutional neural network to extract visual semantic features from medical image data 11;

[0051] Using a pre-trained language model to extract context-aware text semantic features from electronic medical record text data 12;

[0052] Using a temporal modeling network to extract dynamic temporal semantic features from physiological signal data 13;

[0053] Respectively inputting the visual semantic features, text semantic features, and dynamic temporal semantic features into corresponding pre-configured embedding models 31 to map them to the high-dimensional semantic space.

[0054] For medical image data 11, the system employs a deep convolutional neural network (such as ResNet-50, DenseNet, or Vision Transformer) as a visual feature extractor to automatically learn discriminative visual semantic features from preprocessed images or regions of interest. For example, in lung CT images, the network can capture subtle morphological features such as the edge sharpness, density distribution, and surrounding vascular traction of nodules, encoding them into a high-dimensional vector. This vector not only represents the image content but also implicitly contains semantic information related to clinical diagnosis. For electronic medical record text data 12, a pre-trained language model for the medical field (such as ClinicalBERT, BioBERT, or the Chinese medical version of ERNIE-Med) is used for context-aware semantic encoding. These models, pre-trained on large-scale medical text, can understand the potential association between "persistent chest pain with sweating" and "acute coronary syndrome," thus mapping descriptions with superficial lexical differences but similar semantics to neighboring semantic space locations. For example, when inputting an outpatient record containing "recurring dizziness and large fluctuations in blood pressure," the model can extract deep semantic features reflecting autonomic dysfunction or the risk of hypertensive emergencies. For physiological signal data13, the system employs a specially designed temporal modeling network (such as Temporal Convolutional Network, TCN; LSTM variant; or Transformer-based TimeSformer) to capture its dynamic evolution and key event patterns. For instance, in a 24-hour ECG monitoring signal, the model can not only identify the instantaneous waveform abnormalities of ventricular premature beats, but also model their frequency, diurnal rhythm variations, and coupling relationships with other physiological parameters, such as heart rate variability, thereby generating dynamic temporal semantic features rich in temporal context. After obtaining the original semantic features of the three modalities mentioned above, they are respectively input into the pre-configured embedding models 31. These embedding models 31 are usually composed of several fully connected layers and are jointly trained and optimized. Their function is to uniformly map the feature vectors of different modalities to the same preset high-dimensional semantic space (e.g., a 512-dimensional or 768-dimensional shared vector space). This space is not arbitrarily constructed, but is configured by the embedding models 31.

[0055] Specifically, the methods for configuring the embedded model 31 include:

[0056] Configure an image embedding model 31 based on a convolutional neural network or a visual Transformer for medical image data 11;

[0057] Configure a text embedding model based on BERT, BioBERT, or ClinicalBERT for electronic medical record text data 12;

[0058] Configure a signal embedding model 31 based on a one-dimensional convolutional network, a recurrent neural network, or a time series Transformer for physiological signal data 13;

[0059] Furthermore, each embedding model 31 is jointly fine-tuned under a preset reference semantic target, so that the output vectors are located in the same high-dimensional semantic space and are comparable.

[0060] Image embedding models 31 based on convolutional neural networks (such as ResNet, EfficientNet) or visual Transformers (ViT) are configured for medical image data 11; text embedding models 31 based on pre-trained language models for the medical field, such as BERT, BioBERT, or ClinicalBERT, are configured for electronic medical record text data 12; signal embedding models 31 based on one-dimensional convolutional neural networks (1D-CNN), recurrent neural networks, or time series Transformers (such as TimeSformer, Informer) are configured for physiological signal data 13; and each embedding model 31 is jointly fine-tuned under a preset reference semantic target so that their output vectors are located in the same high-dimensional semantic space and have cross-modal semantic comparability.

[0061] Image embedding models 31 are typically based on a pre-trained visual backbone network, with a projection head attached to its end to map globally pooled features to a semantic space of uniform dimension. For example, in the lung nodule screening task, the system can use ViT-Base pre-trained on ImageNet as the backbone, and then fine-tune it to make it more sensitive to CT image semantics such as "ground-glass opacity" and "lobulation sign", and output a 768-dimensional image embedding vector.

[0062] The text embedding model 31 utilizes a language model pre-trained on a large-scale biomedical corpus, such as ClinicalBERT. This model has been optimized on real clinical texts such as MIMIC-III and can accurately understand the implicit association between "hemoptysis with weight loss" and "possible malignant lung tumor". After inputting a standardized medical record text, the model takes the hidden state corresponding to the [CLS] label and generates a text semantic vector consistent with the image embedding dimension through a lightweight MLP projection layer.

[0063] For physiological signals, the system selects an appropriate time-series modeling architecture based on the signal characteristics. For example, for high-frequency, short-cycle signals such as electrocardiograms, stacked one-dimensional convolutional layers can be used to extract local waveform patterns, followed by LSTM to capture long-range dependencies. For multi-channel, long-duration intensive care signals (such as hourly sequences containing heart rate, respiration, and blood pressure simultaneously), a time-series Transformer is more suitable, using a self-attention mechanism to model complex interactions across channels and time periods. These signal embedding models 31 also have projection layers at the end, outputting dynamic semantic vectors of a unified dimension.

[0064] Step S3) Construct semantic associations between modalities through a cross-modal alignment mechanism, and dynamically fuse high-dimensional semantic features 14 according to the semantic associations to generate a fused joint representation vector 15.

[0065] Please see the appendix Figure 3 Methods for constructing semantic associations between modalities through cross-modal alignment mechanisms include:

[0066] Construct multimodal sample pairs consisting of medical image data 11, electronic medical record text data 12, or physiological signal data 13, wherein each sample pair corresponds to the same patient and has semantic consistency;

[0067] The multimodal data are input into the corresponding embedding model 31 to obtain their respective feature vectors in the high-dimensional semantic space;

[0068] Based on the contrastive learning strategy, a contrastive loss function is constructed by using cross-modal features of the same patient as positive sample pairs and cross-modal features of different patients or random combinations of cross-modal features as negative sample pairs.

[0069] By minimizing the contrast loss function, the parameters of the embedding model 31 are optimized, which brings positive sample pairs closer in the semantic space and pushes negative sample pairs further apart, thereby establishing a semantic alignment relationship between modalities.

[0070] Based on the similarity of feature vectors of semantically aligned multimodal data that is higher than a preset reference value, semantic associations of multimodal data are generated.

[0071] Multimodal sample pairs are constructed, consisting of medical imaging data 11, electronic medical record text data 12, or physiological signal data 13. Each sample pair originates from the same patient and is consistent in clinical semantics. For example, for a patient diagnosed with acute ischemic stroke, the corresponding multimodal sample pair may include high signal intensity in the left basal ganglia region shown by diffusion-weighted imaging (DWI) of the head, the chief complaint of "sudden onset of right limb weakness for 2 hours" recorded in the electronic medical record, and physiological signal fragments of sudden rise in blood pressure and fluctuations in heart rate collected by the monitor during the onset of the stroke. Although these data are different in form, they all point to the same pathological event, constituting a natural positive sample pair.

[0072] The data from each modality are input into their respective embedding models 31, such as ViT for MRI images, ClinicalBERT for medical record text, and TCN for physiological signals, to obtain their respective feature vectors in a unified high-dimensional semantic space (e.g., 768 dimensions). Based on this, a contrastive learning strategy is used to construct a contrastive loss function: cross-modal feature combinations originating from the same patient (e.g., MRI features and corresponding text features) are considered positive sample pairs, while cross-modal combinations between different patients (e.g., MRI from patient A and text from patient B) or random modal combinations within the same patient without clinical relevance (e.g., ECG from an unrelated follow-up and current stroke text) are considered negative sample pairs. By optimizing the contrastive loss function and iteratively updating the parameters of each embedding model 31, the cosine similarity or Euclidean distance of positive sample pairs in the semantic space is significantly reduced, while negative sample pairs are effectively pushed apart. After sufficient training, the originally heterogeneous modal features are aligned in the shared semantic space: for example, the text embedding of "left limb weakness" and the MRI embedding showing left motor cortex infarction are close to each other in space, while the embeddings related to "abdominal pain" or "pneumonia" are far apart.

[0073] Based on whether the similarity between the feature vectors of each modality after semantic alignment is higher than a preset reference threshold (such as cosine similarity > 0.7), it is determined whether there is a strong semantic association, and a reliable cross-modal semantic association map is generated based on this.

[0074] The method for dynamically fusing high-dimensional semantic features 14 based on the semantic association to generate a fused joint representation vector 15 includes:

[0075] The correlation between high-dimensional semantic features 14 with semantic association is calculated as a weight, and the high-dimensional semantic features 14 of different modalities are weighted and aggregated according to the weight to generate a fused joint representation vector 15.

[0076] The joint representation vector 15 can reflect the complementary semantic information of multimodal medical data.

[0077] The correlation between semantically related features across modalities (e.g., through attention mechanisms or similarity weighting) is calculated as a fusion weight, thereby weighted aggregation of features from different modalities. For example, in the aforementioned stroke case, if the similarity between MRI features and text features is as high as 0.85, while the similarity between physiological signals (such as blood pressure fluctuations) and text is only 0.65, the system assigns higher weights to images and text, and relatively lower weights to the signal modality, thus generating a joint representation that emphasizes structural abnormalities and clinical descriptions. The dynamic weighting mechanism adaptively highlights modal information that is more discriminative in specific cases, avoiding the dilution of key semantics caused by simple averaging. The generated joint representation vector15 not only preserves the original semantics of each modality but also integrates complementary cross-modal information, such as anatomical localization provided by images, symptom temporality described in text, and physiological stress states reflected by signals, thus forming a comprehensive, compact, and semantically rich patient-level representation, providing a high-quality indexing foundation for subsequent precise retrieval based on natural language queries.

[0078] Step S4) In response to a natural language query request input by the user, the query request is mapped to the same semantic space as the joint representation vector 15, and the similarity between the query request and the joint representation vector 15 is calculated.

[0079] When a user enters a free text query into the system interface, such as "finding middle-aged non-smoking patients with ground-glass opacities, accompanied by a persistent dry cough for more than two weeks and no fever," the system first performs the same preprocessing procedure as electronic medical record text on the query, including medical terminology recognition, stop word filtering, and standardization (e.g., standardizing "ground-glass opacity" to "ground-glass shadow"). The query is then input into the text embedding model 31 (e.g., ClinicalBERT), which has been configured and fine-tuned in step S2. Based on its contextual understanding capabilities learned from large-scale medical corpora and contrastive learning tasks, this model can accurately capture key clinical elements in the query: including imaging features (ground-glass opacity), symptom characteristics (dry cough, duration), exclusion criteria (no fever), and population attributes (middle-aged, non-smoker), and outputs a query semantic vector with dimensions consistent with the joint representation vector 15 (e.g., 768 dimensions). Since the text embedding model 31 has been aligned with the image and signal embedding model 31 in the same high-dimensional semantic space through cross-modal contrastive learning, the query vector is naturally in the same semantic coordinate system as the multimodal joint representation.

[0080] Then, the fused joint representation vectors of all patients in the database are traversed, and the semantic similarity between each vector and the query vector is calculated, typically using cosine similarity as the metric. For example, a patient's joint representation is formed by dynamically fusing the pure ground-glass nodules shown on CT scans, the record of "dry cough for 3 weeks, denial of fever and smoking history" in the medical record, and stable cardiopulmonary physiological signals. Its joint vector is highly similar to the query vector in the semantic space, with a cosine similarity of 0.88. Another patient, although having pulmonary nodules, also has high fever and purulent sputum; its joint representation conflicts with exclusion criteria such as "no fever," resulting in a significantly lower similarity (0.42). Through this semantic matching mechanism, multimodal cases that best match clinical intent can be intelligently identified without relying on precise keyword matching or manual rules.

[0081] Step S5) Sort the multimodal medical data according to the similarity and return the sorting results as the retrieval result 21.

[0082] After calculating the semantic similarity between the natural language query vector and the fused joint representation vector of each patient in the database (15), all candidate cases are sorted from high to low according to the similarity score, generating a list of search results (21) highly relevant to the user's query intent. For example, when a radiologist enters the query "Looking for young female patients with bilateral adrenal hyperplasia, hypokalemia, and hypertension," the system returns a list of cases sorted in descending order of semantic matching. The first case might be a complete multimodal record of a 28-year-old female patient: her abdominal CT clearly shows diffuse enlargement of both adrenal glands, her electronic medical record explicitly states "serum potassium 2.9 mmol / L, blood pressure 160 / 100 mmHg," and she has been diagnosed with primary aldosteronism by the endocrinology department; the physiological signal section includes blood pressure fluctuation curves and electrolyte test time-series data continuously monitored during hospitalization. This case, due to its high degree of similarity to the query across multiple dimensions such as imaging manifestations, text descriptions, and laboratory indicators, receives the highest similarity score (e.g., 0.91) and is prioritized for presentation. Following cases may have partial matches in images or text (e.g., only unilateral hyperplasia or no recorded serum potassium levels), with slightly lower similarity (e.g., 0.78), but are still of reference value. Cases that are clearly irrelevant (e.g., adrenal tumors or those without endocrine abnormalities) are ranked at the bottom or filtered out because their similarity is below the threshold.

[0083] On the other hand, this specification provides a multimodal medical data intelligent fusion retrieval system; please refer to the appendix. Figure 4 ,include:

[0084] The acquisition module 100 acquires multimodal medical data from different data sources, wherein the multimodal medical data includes at least two of the following: medical image data 11, electronic medical record text data 12, and physiological signal data 13.

[0085] Extraction module 200 preprocesses and standardizes the multimodal medical data, and extracts high-dimensional semantic features 14 of the multimodal medical data in a preset high-dimensional semantic space;

[0086] The fusion module 300 constructs semantic associations between modalities through a cross-modal alignment mechanism, and dynamically fuses the high-dimensional semantic features 14 according to the semantic associations to generate a fused joint representation vector 15.

[0087] The query module 400, in response to a natural language query request input by a user, maps the query request to the same semantic space as the joint representation vector 15, and calculates the similarity between the query request and the joint representation vector 15.

[0088] The output module 500 sorts the multimodal medical data according to the similarity and returns the sorting result as the retrieval result 21.

[0089] Please see Figure 5 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this specification.

[0090] like Figure 5 As shown, the electronic device 1100 may include: at least one processor 1101, at least one network interface 1104, a user interface 1103, a memory 1105, and at least one communication bus 1102. The communication bus 1102 can be used to connect and communicate with the various components mentioned above. The user interface 1103 may include buttons, and optionally may include standard wired or wireless interfaces. The network interface 1104 may include, but is not limited to, a Bluetooth module, an NFC module, or a Wi-Fi module. The processor 1101 may include one or more processing cores. The processor 1101 connects to various parts within the electronic device 1100 using various interfaces and lines, and performs various functions of the routing device and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1105, and by calling data stored in the memory 1105. Optionally, the processor 1101 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 1101 may integrate one or more combinations of CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content that the display screen needs to show; and the modem is used for wireless communication.

[0091] It is understandable that the aforementioned modem may not be integrated into the processor 1101, but may be implemented using a separate chip.

[0092] The memory 1105 may include RAM or ROM. Optionally, the memory 1105 may include a non-transitory computer-readable medium. The memory 1105 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1105 may also be at least one storage device located remotely from the aforementioned processor 1101. As a computer storage medium, the memory 1105 may include an operating system, a network communication module, a user interface module, and application programs. The processor 1101 may be used to call the application programs stored in the memory 1105 and execute the methods in the above-described embodiments.

[0093] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform multiple steps as described in the above embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0094] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the multiple steps described in the above embodiments.

[0095] Where there is no conflict, the technical features in this embodiment and implementation scheme can be combined arbitrarily.

[0096] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes multiple computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating multiple available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0097] When implemented through hardware or firmware, the aforementioned method flow is programmed into the hardware circuit to obtain the corresponding hardware circuit structure and achieve the corresponding function. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit, whose logic function is determined by the user programming the device. Designers can program a digital system onto a PLD themselves, eliminating the need for chip manufacturers to design and fabricate dedicated integrated circuit chips. Furthermore, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, similar to the software compiler used in program development. The original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There is not just one HDL, but many. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of the aforementioned hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logic method flow can be easily obtained.

[0098] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims of this specification.

Claims

1. A method for intelligent fusion retrieval of multimodal medical data, characterized in that, Includes the following steps: Acquire multimodal medical data from different data sources, wherein the multimodal medical data includes at least two of the following: medical imaging data, electronic medical record text data, and physiological signal data; The multimodal medical data is preprocessed and standardized, and high-dimensional semantic features of the multimodal medical data in a preset high-dimensional semantic space are extracted. Semantic associations between modalities are constructed through a cross-modal alignment mechanism, and high-dimensional semantic features are dynamically fused based on these semantic associations to generate a fused joint representation vector. In response to a user's input natural language query request, the query request is mapped to the same semantic space as the joint representation vector, and the similarity between the query request and the joint representation vector is calculated. The multimodal medical data is sorted according to the similarity, and the sorting results are returned as the search results output. The method for extracting high-dimensional semantic features of the multimodal medical data in a preset high-dimensional semantic space includes: Using convolutional neural networks to extract visual semantic features from medical image data; Context-aware semantic features of electronic medical record text data are extracted using a pre-trained language model. Dynamic temporal semantic features are extracted from physiological signal data using a temporal modeling network. Visual semantic features, textual semantic features, and dynamic temporal semantic features are input into their respective pre-configured embedding models and mapped to a high-dimensional semantic space. Methods for configuring embedded models include: Configure image embedding models based on convolutional neural networks or visual Transformers for medical image data; Configure a text embedding model based on BERT, BioBERT, or ClinicalBERT for electronic medical record text data; Configure signal embedding models for physiological signal data based on one-dimensional convolutional networks, recurrent neural networks, or time-series Transformers; Furthermore, each embedding model is jointly fine-tuned under a preset reference semantic objective, so that the output vectors are located in the same high-dimensional semantic space and are comparable; Methods for constructing semantic associations between modalities through cross-modal alignment mechanisms include: Construct multimodal sample pairs consisting of medical imaging data, electronic medical record text data, or physiological signal data, where each sample pair corresponds to the same patient and has semantic consistency; Multimodal data are input into the corresponding embedding models to obtain their respective feature vectors in the high-dimensional semantic space; Based on the contrastive learning strategy, a contrastive loss function is constructed by using cross-modal features of the same patient as positive sample pairs and cross-modal features of different patients or random combinations of cross-modal features as negative sample pairs. By minimizing the contrastive loss function and optimizing the embedding model parameters, the distance between positive sample pairs in the semantic space is shortened, while the distance between negative sample pairs is widened, thereby establishing a semantic alignment relationship between modalities. Based on the similarity of feature vectors of semantically aligned multimodal data that is higher than a preset reference value, semantic associations of multimodal data are generated.

2. The intelligent fusion retrieval method for multimodal medical data according to claim 1, characterized in that, Methods for preprocessing and standardizing the multimodal medical data include: Denoising, normalization, and region of interest extraction are performed on medical image data; The electronic medical record text data is segmented, stop words are removed, entities are identified, and terminology is standardized. Physiological signal data is filtered, resampled, and time-aligned.

3. The intelligent fusion retrieval method for multimodal medical data according to claim 1, characterized in that, The method for dynamically fusing high-dimensional semantic features based on the semantic association to generate a fused joint representation vector includes: The correlation between semantically related high-dimensional semantic features is calculated as a weight, and the high-dimensional semantic features of different modalities are weighted and aggregated according to the weight to generate a fused joint representation vector. The joint representation vector can reflect the complementary semantic information of multimodal medical data.

4. A multimodal medical data intelligent fusion retrieval system, characterized in that, include: The acquisition module acquires multimodal medical data from different data sources, wherein the multimodal medical data includes at least two of the following: medical image data, electronic medical record text data, and physiological signal data. The extraction module preprocesses and standardizes the multimodal medical data, and extracts the high-dimensional semantic features of the multimodal medical data in a preset high-dimensional semantic space. The fusion module constructs semantic associations between modalities through a cross-modal alignment mechanism, and dynamically fuses high-dimensional semantic features based on the semantic associations to generate a fused joint representation vector. The query module, in response to a natural language query request input by the user, maps the query request to the same semantic space as the joint representation vector, and calculates the similarity between the query request and the joint representation vector. The output module sorts the multimodal medical data according to the similarity and returns the sorting results as the search results. The method for extracting high-dimensional semantic features of the multimodal medical data in a preset high-dimensional semantic space includes: Using convolutional neural networks to extract visual semantic features from medical image data; Context-aware semantic features of electronic medical record text data are extracted using a pre-trained language model. Dynamic temporal semantic features are extracted from physiological signal data using a temporal modeling network. Visual semantic features, textual semantic features, and dynamic temporal semantic features are input into their respective pre-configured embedding models and mapped to a high-dimensional semantic space. Methods for configuring embedded models include: Configure image embedding models based on convolutional neural networks or visual Transformers for medical image data; Configure a text embedding model based on BERT, BioBERT, or ClinicalBERT for electronic medical record text data; Configure signal embedding models for physiological signal data based on one-dimensional convolutional networks, recurrent neural networks, or time-series Transformers; Furthermore, each embedding model is jointly fine-tuned under a preset reference semantic objective, so that the output vectors are located in the same high-dimensional semantic space and are comparable; Methods for constructing semantic associations between modalities through cross-modal alignment mechanisms include: Construct multimodal sample pairs consisting of medical imaging data, electronic medical record text data, or physiological signal data, where each sample pair corresponds to the same patient and has semantic consistency; Multimodal data are input into the corresponding embedding models to obtain their respective feature vectors in the high-dimensional semantic space; Based on the contrastive learning strategy, a contrastive loss function is constructed by using cross-modal features of the same patient as positive sample pairs and cross-modal features of different patients or random combinations of cross-modal features as negative sample pairs. By minimizing the contrastive loss function and optimizing the embedding model parameters, the distance between positive sample pairs in the semantic space is shortened, while the distance between negative sample pairs is widened, thereby establishing a semantic alignment relationship between modalities. Based on the similarity of feature vectors of semantically aligned multimodal data that is higher than a preset reference value, semantic associations of multimodal data are generated.

5. An electronic device, characterized in that, Including the processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-3.

Images