Cross-modal semantic-visual joint retrieval method based on multi-modal data lake warehouse

Abstract

The application provides a cross-modal semantic-visual joint retrieval method based on a multi-modal data lake warehouse, initiates a mode metadata index + semantic-visual content index double-layer structure, realizes unified index coding of medical multi-modal data, and solves the mode fragmentation problem. Differentiated metadata index rules are designed for text and medical images, combined with a medical pre-training model to generate a joint content vector, generate a mode adaptive index, and adapt to the characteristics of medical data. Based on the unified semantic-visual space, text ↔ image bidirectional retrieval is realized, metadata filtering and vector similarity are fused, cross-modal bidirectional precise matching is realized, and the matching precision of the medical scene is improved. Based on the reverse source integration mechanism, through the PID bidirectional association, the cross-modal complete data is automatically traced and integrated, the single mode retrieval result fragmentation problem is solved. Adapt to medical terminology, DICOM standard, privacy compliance, clinical high-precision demand, support large-scale data lake warehouse real-time retrieval.

Description

Technical Field

[0001] This invention relates to the fields of medical data processing, multimodal information retrieval, data lake warehouse and artificial intelligence technology, and in particular to a cross-modal semantic-visual joint retrieval platform based on a multimodal data lake warehouse, a cross-modal semantic-visual joint retrieval method based on a multimodal data lake warehouse, an electronic device and a computer-readable storage medium. Background Technology

[0002] With the development of smart healthcare and medical big data, medical data is exhibiting characteristics of multimodality, heterogeneity, and massive volume, mainly including: Text modality: Structured / unstructured text such as electronic medical records, laboratory reports, medical orders, medical literature, and diagnostic conclusions; Visual modalities: medical images such as CT, MRI, X-ray, ultrasound, pathological sections, and endoscopic images; Other modalities include: time-series physiological signals, gene sequences, and medical IoT sensor data.

[0003] Current medical data storage and retrieval systems generally adopt a hybrid architecture of data lake + data warehouse (data lake warehouse), but the following mainstream technical paths exist at the multimodal retrieval level: 1. Single-modal independent retrieval: Text retrieval is based on keywords / inverted indexes and vector retrieval; image retrieval is based on visual features (SIFT, CNN) and hash encoding; each modality database is isolated from each other and cross-modal related queries are not possible (such as "text description search for similar images" and "image search related to text report").

[0004] 2. Weakly related cross-modal retrieval: This method relies on manually annotated metadata (such as an image ID → report ID mapping table) and achieves modal switching through foreign key associations in the metadata. Essentially, it is still single-modal retrieval plus post-association, lacking a unified semantic space. Retrieval accuracy depends on annotation quality and semantic-level matching cannot be achieved.

[0005] 3. General Cross-Modal Model: A unified feature vector is generated using general image-text pre-trained models such as CLIP and BLIP, and similarity retrieval is performed in a vector library. However, the medical field faces challenges such as technical terminology barriers, the strong specificity of medical images, and complex modal semantic alignment. Consequently, the general model exhibits low accuracy and poor generalization in medical scenarios.

[0006] Therefore, in summary, the shortcomings of existing multimodal retrieval technologies include: 1. Modal fragmentation and lack of a unified indexing system: Text, images, and other modal data are stored and indexed independently, lacking a cross-modal semantic-visual joint indexing structure, making it impossible to achieve bidirectional semantic retrieval between "text → image" and "image → text"; 2. Lack of index generation mechanism: The system does not have a two-layer indexing system of modality adaptive metadata index + data content index designed for medical multimodal data. It only relies on shallow metadata association and cannot deeply explore the semantic associations within and between modalities. 3. Low retrieval matching accuracy: General models cannot adapt to professional semantics such as medical terminology, pathological features, and imaging signs, resulting in large semantic alignment errors; during retrieval, only coarse vector matching is performed, without a reverse tracing and result integration mechanism, which easily leads to mismatches and missed matches; 4. Poor adaptability to medical scenarios: It fails to meet the requirements of medical data privacy compliance, high accuracy of clinical retrieval, and low response latency. The index building and retrieval efficiency is insufficient, making it difficult to support real-time retrieval of large-scale medical data lake warehouses. Summary of the Invention

[0007] To address the technical problems existing in the prior art, the present invention provides the following technical solution: On the one hand, a cross-modal semantic-visual joint retrieval platform based on a multimodal data lake warehouse is provided, the platform comprising: The data access layer is used to uniformly access structured and unstructured multimodal medical data from the medical data lake warehouse platform; The data processing layer, connected to the data access layer, is used to standardize and preprocess the accessed multimodal medical data and generate metadata, assign globally unique identifiers to the processed data, and use medical models to convert text and image data into semantic feature vectors and visual feature vectors, respectively. An index building layer, connected to the data processing layer, is used to build a first index library based on the metadata and a second vector index library based on the semantic feature vector and the visual feature vector, wherein the stored vectors are associated with the globally unique identifier; The retrieval engine layer, connected to the index building layer, receives and parses user queries, performs parallel retrieval and matching based on the first index library and the second vector index library, and obtains cross-modal retrieval results; and The application interface layer is used to provide users with a search interface and return the cross-modal search results.

[0008] On the other hand, a cross-modal semantic-visual joint retrieval method based on a multimodal data lake warehouse is provided, implemented on the aforementioned platform. The method includes the following steps: Data Acquisition and Preprocessing: Collect multimodal medical data from the medical data lake warehouse and perform standardized processing, assigning a unified, globally unique identifier to the processed data; Modal metadata index generation: Based on the metadata of preprocessed data, construct a modality-adaptive first-level index for data of different modalities; Data content vector index generation: Through a coding model finely tuned in the medical field, preprocessed text and image data are converted into feature vectors, and a second-layer vector index is constructed; User retrieval information parsing: Receive and parse user retrieval requests, and generate query vectors and metadata filtering conditions; Cross-modal retrieval matching: Based on the metadata filtering conditions, filtering is performed in the first-level index, and based on the query vector, approximate nearest neighbor retrieval is performed in the second-level vector index. The results of the two are merged and sorted to obtain a candidate result set. Reverse source tracing of modal source data: Based on the globally unique identifier in the candidate result set, obtain the complete metadata of the corresponding data and other associated modal data information from the first-level index; Results Integration and Return: The multimodal data obtained from the source tracing is integrated and sorted according to the matching degree, and the structured results are returned to the user.

[0009] Preferably, the data acquisition and preprocessing specifically includes: Structured patient data and unstructured data, including at least electronic medical record text and medical images, are collected from the medical data lake warehouse; the text data is processed by medical terminology segmentation and standardization, and the image data is processed by format conversion and normalization; and each processed data is assigned a unique patient master data identifier as the globally unique identifier, while its modality type is labeled.

[0010] Preferably, the generation of the modal metadata index specifically includes: Build a text metadata index for text modal data based on an inverted index of medical terminology and an index of structured fields; Construct a multidimensional spatial index for visual modal data based on image type, location, and lesion type; and Construct a cross-modal association index that records the relationships between different modalities of data under the same patient or treatment event.

[0011] Preferably, the step of converting the preprocessed text and image data into feature vectors specifically includes: The preprocessed text data is input into the fine-tuned medical text encoder to generate a semantic feature vector with a unified dimension. The preprocessed image data is input into a finely tuned medical vision encoder to generate visual feature vectors of uniform dimensions; and For image-text pairs that have a corresponding relationship, the semantic feature vector and visual feature vector are mapped to a shared semantic space through a cross-modal alignment module, and feature fusion is performed to generate a joint semantic-visual enhanced feature vector.

[0012] Preferably, the cross-modal alignment module maps the semantic feature vector and visual feature vector to the same shared semantic space through a trainable projection layer, and optimizes it using a contrastive learning loss function to maximize the similarity of feature vectors of the same image-text pair in the shared space. Preferably, the construction of the second-layer vector index employs an approximate nearest neighbor indexing technique; specifically, it includes unifying and normalizing the dimensions of the semantic feature vector, visual feature vector, and joint semantic-visual enhanced feature vector, and constructing an index based on a hierarchical navigation small-world graph algorithm to establish a mapping relationship between the vector and the globally unique identifier. Preferably, the cross-modal retrieval matching supports at least one of the following scenarios: retrieving image visual feature vectors using text query vectors, or retrieving text semantic feature vectors using image query vectors. Preferably, the fusion ranking employs a weighted fusion strategy, assigning a first weight to the metadata filtering results and a second weight to the vector similarity results. Preferably, obtaining the associated other modal data information means obtaining the identifiers of other modal data associated with the globally unique identifier of the current candidate data by querying the cross-modal association index, thereby integrating them to form a complete diagnosis and treatment data unit including images, text reports, and electronic medical records.

[0013] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0014] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above method.

[0015] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. Improved retrieval accuracy: Medical terms and image features are deeply aligned, and the cross-modal matching accuracy is 25%~40% higher than that of general models, while the false negative rate and false positive rate are significantly reduced.

[0016] 2. Optimized retrieval efficiency: Parallel retrieval with dual-level indexes reduces response time by more than 50%, supporting millisecond-level retrieval of millions of medical data points.

[0017] 3. Enhanced cross-modal capabilities: Enables full functionality including "text-to-image search, image-to-text search, hybrid retrieval, and related tracing," covering retrieval needs across all clinical and research scenarios.

[0018] 4. Data governance value: A unified indexing system standardizes the management of multimodal medical data and improves the availability, searchability, and governability of data in the data lake warehouse.

[0019] 5. Clinical application value: It helps doctors quickly retrieve similar cases, image comparisons, and complete diagnostic and treatment data, thereby improving diagnostic efficiency and accuracy. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a platform architecture diagram provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a method provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0023] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0024] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0025] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0026] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0027] This invention discloses a cross-modal semantic-visual joint retrieval method for a medical multimodal data lake warehouse. Its core innovation lies in a complete technical system encompassing "multimodal data acquisition → modal metadata index generation → data content index generation → cross-modal retrieval and matching → reverse tracing and integration → result return." The core of this system is the construction of an adaptive two-layer index for medical multimodal data and cross-modal semantic-visual joint matching. It aims to solve the following technical problems: 1. Construct a semantic-visual dual-layer joint indexing system for a medical multimodal data lake warehouse to achieve unified indexing and encoding of modal metadata and data content; 2. Design a modality-adaptive index generation algorithm to automatically generate standardized index structures for different modalities such as text and medical images; 3. Achieve accurate cross-modal semantic-visual bidirectional matching, support text-to-image retrieval and image-to-text retrieval, and integrate complete results through reverse tracing; 4. Improve the accuracy, efficiency, and compliance of searches in medical scenarios to meet the search needs of multiple scenarios such as clinical diagnosis, medical research, and data governance.

[0028] The technical application principles and effects of this solution are described in detail below, in conjunction with the platform architecture and methodology.

[0029] I. Search Platform Architecture like Figure 1 As shown, the platform architecture includes: The data access layer is used to uniformly access structured data and unstructured multimodal medical data in the medical data lake warehouse platform; The data processing layer, connected to the data access layer, is used to perform standardized preprocessing on the accessed data, extract metadata, assign a globally unique identifier (PID) to the data, and convert the text data and image data into semantic feature vectors and visual feature vectors respectively through fine-tuned medical text models and medical image models. An index building layer, connected to the data processing layer, includes a modal metadata index library and a modal data content index vector library. The modal metadata index library is used to build a multi-dimensional index based on the PID, modal type, and business attributes generated after preprocessing. The modal data content index vector library is used to store the feature vectors generated by the data processing layer and associate them with the PID based on the nearest neighbor indexing technology. The retrieval engine layer, connected to the index building layer, receives user query requests, performs metadata retrieval and multimodal content vector retrieval in parallel based on the two-layer index, and performs cross-modal matching and result fusion; and The application interface layer provides a visual interface and API for receiving user search commands and returning search results.

[0030] The architecture is described below: 1. Data Access Layer (Bottom Layer: Data Sources and Entry Points) The platform first connects to the hospital's HIS, LIS, PACS and other data systems (data lake warehouse), automatically aggregating structured patient information, examination results and unstructured medical record text, medical images and other multimodal heterogeneous data. Through cleaning and labeling of this data (such as adding patient PID and modality type tags), it achieves unified encoding and association of cross-system and cross-format medical data within a logical framework, addressing the "data silo" problem at its source and establishing a unified data foundation for subsequent intelligent processing.

[0031] Core function: Connecting to a medical data lake repository platform (responsible for collecting patient visit data, including HIS, LIS, PACS, electronic medical records, etc.). This system uniformly connects to the lake repository platform to obtain the structured and unstructured data used by the system.

[0032] Structured data: such as patient basic information, diagnostic records, laboratory test results, medical orders, etc., is standardized data that can be stored in tables / fields.

[0033] Unstructured data: such as CT / MRI images, electronic medical record reports, electrocardiogram signals, and other heterogeneous data that cannot be directly stored in tables.

[0034] 2. Data Processing Layer (+ Model Layer: Data Cleaning and Feature Extraction) Its core function is to standardize the raw data and generate indexes and vector features that can be used by search engines, making it the system's "data processing center".

[0035] Metadata extraction: Extracting key attributes from structured / unstructured data, such as patient ID, data type, collection time, department, and other descriptive information.

[0036] Generate patient master data PID: Assign a unique ID (patient unique identifier) ​​to each patient, link the data of the same patient in different systems, and solve the problem of data silos.

[0037] PID Index Association: Bind the extracted metadata to the PID to establish a patient-dimensional index association, which facilitates quick location of the full patient data in the future.

[0038] Modal data vectorization: Text-based data (medical records, reports): Converted into semantic vectors using a medical NLP model; Image data (CT / MRI): Convert medical image models into visual feature vectors; transform unstructured data into machine-understandable vectors, providing a foundation for cross-modal retrieval.

[0039] Model layer (medical big model): As the core engine of vectorization, it is responsible for completing text semantic understanding and image feature extraction, and is the "brain" of cross-modal retrieval.

[0040] 3. Index Building Layer (Data Storage and Index Management) Core function: Stores processed data indexes and vectors, providing fast query capabilities for upper-layer search engines; it is the system's "data warehouse + index library".

[0041] Modal metadata index: Stores patient metadata and PID association information, and supports fast retrieval by patient ID, data type, time, and other conditions.

[0042] Modal data content index vector library: Stores vector features of text / image data, supporting semantic / content-based similarity retrieval (e.g., "find cases similar to this CT image").

[0043] The goal of this layer is to achieve efficient retrieval through a combination of metadata indexing and vector indexing, which combines structured condition filtering with unstructured content matching.

[0044] This platform utilizes a two-tiered composite index to deconstruct the "skeleton" and "soul" of the data. The platform innovatively establishes two sets of indexes simultaneously for the data, which is key to achieving accurate and efficient retrieval. The first-level index (“skeleton” index): Modal metadata indexes are built for both text and images to quickly identify “what it is.” For example, for a CT image, the index records structured tags such as patient information, examination site (e.g., cranium), and image type (e.g., MRI-T1 sequence). This index acts like a clearly categorized “library catalog,” allowing users to perform efficient and broad-based initial screening using structured criteria (e.g., “45-year-old female with thyroid TI-RADS 2 nodules”).

[0045] The second-layer index (the "soul" index): Utilizing medically optimized models (such as fine-tuned BioBERT and MedViT), the semantics of medical records and the lesion features of medical images are transformed into comparable semantic and visual vectors, respectively. For an image and its corresponding report text, the system can also use cross-modal alignment technology to achieve deep understanding and association between their vectors in the same "semantic-visual" space, generating a joint vector that simultaneously contains image lesion information and textual diagnostic descriptions. This layer of index digitizes the "connotation" of unstructured data, supporting content-based deep semantic matching.

[0046] 4. Search Engine Layer (Core Service Layer: Handles user queries) Core function: It receives user search requests, completes cross-modal matching and result integration, and is the "service hub" of the system.

[0047] Query parsing module: Understands the user's query intent, such as "find cases of 30-year-old males with lung cancer and accompanying CT images", and breaks down natural language / conditional queries into executable search instructions.

[0048] Multimodal retrieval module: Based on the parsed instructions, it simultaneously queries the vector library (image / text similarity) and the metadata index library (structured conditions) and returns the matching results.

[0049] Metadata retrieval module: Handles queries based on structured conditions (such as "laboratory data of all diabetic patients in 2023") separately, and quickly filters patient data that meets the conditions.

[0050] Results integration module: merges, deduplicates, and sorts the results of multimodal retrieval and metadata retrieval, organizing them into the format required by the user according to relevance / time order.

[0051] The search engine achieves accurate retrieval through cross-modal bidirectional fusion matching. When a doctor initiates a search (for example, entering a description such as "brain MRI showing cerebral infarction"), the platform's search engine will activate intelligent matching using a "two-layer parallel-fusion decision" process: First-line parallel processing: In the "skeleton index" (first-level index), relevant patients or imaging groups are quickly filtered out based on the structured conditions clearly defined in the query (such as "brain" or "MRI").

[0052] The second parallel approach: In the "Soul Index" (second-layer vector library), the user's query content (descriptive text or uploaded image) is also transformed into feature vectors, and intelligent algorithms are used to find candidate vectors that are most similar to it semantically or visually.

[0053] Fusion Decision: Finally, the engine uses preset fusion weights (e.g., vector similarity weight 0.7, metadata matching weight 0.3) to weight, synthesize, and rank the results from the two paths, selecting the candidate result that best matches the user's search intent. This fusion mechanism ensures precision while fully utilizing the deep semantics of the content.

[0054] 5. Application Interface Layer (User Interaction Layer: External Service Outlet) Core function: Provides access points for different users / systems, serving as the system's "external window".

[0055] Search API: This API allows third-party systems (such as hospital research platforms and clinical decision support systems) to call and integrate data retrieval capabilities.

[0056] Search Visual Interface: A visual operation interface provided for clinicians, researchers, and administrators, supporting interactive operations such as conditional filtering, result viewing, and case comparison.

[0057] Reverse tracing and outcome integration deliver a clinically friendly, complete view. After obtaining a high-quality list of matching PIDs, the platform doesn't simply return a bunch of file numbers. It leverages the cross-modal relationships established in the first-level index to automatically trace back to the source, intelligently linking a matching result to related medical records, lab reports, and data from all other modalities. Ultimately, the system presents this information to doctors clearly as a complete diagnostic unit (e.g., "Patient PID xxx: Thyroid nodule imaging + ultrasound report + thyroid function test + medical record summary"), rather than fragmented information, greatly facilitating clinical review and analysis.

[0058] Through the implementation of the above principles, this platform can achieve significant results in the practical application of medical multimodal retrieval, mainly reflected in three dimensions: accuracy, efficiency, and usability. 1. Significantly improves the accuracy and depth of search results. It enables bidirectional semantic-level search, allowing for both text-based image retrieval and image-based text retrieval. Doctors can directly use a symptom description to find cases with similar imaging features, or quickly find complete reports of similar historical cases using a single image to be examined, thus overcoming the shortcomings of traditional single-modal retrieval.

[0059] 2. Significantly optimizes search efficiency Thanks to the parallel retrieval design of the dual-level index, the system can perform lightning-fast high-speed filtering using metadata and quickly find the most similar "candidates" in a vector library of tens of millions of entries, achieving both speed and accuracy.

[0060] 3. Enhance the availability and manageability of clinical data Convenience for physicians: Through the integration and visualization of results, physicians can obtain a complete package of diagnostic and treatment information, including images and text, rather than an isolated file, which effectively assists in clinical diagnosis and scientific research comparison.

[0061] System-wide governance: The platform establishes a unified, standardized, and searchable digital "twin" for the dispersed, heterogeneous, and multimodal data within a hospital, greatly improving the management level and reuse value of data assets, and laying a solid data foundation for subsequent clinical decision support, scientific research data mining, and smart hospital construction.

[0062] The application principles of this platform will be further explained below in conjunction with the methodology and process.

[0063] II. Cross-modal semantic-visual joint retrieval method like Figure 2 The flowchart shown illustrates that this method mainly consists of the following steps: Step 1: (Medical Multimodal) Data Acquisition and Preprocessing 1. Data Access: Relying on a multimodal data lakehouse, structured data is extracted from the lakehouse database to retrieve patient visit-related data, while unstructured data is obtained from the object server by obtaining the object storage path through the lakehouse metadata information. Specifically: Text: Structured (patient ID, ICD-10 diagnostic code, laboratory indicators), unstructured (medical record text, imaging report, medical orders); Visual: DICOM medical images (CT / MRI / X-ray), pathological images, and endoscopic images; parsing DICOM tags and pixel data; 2. Pretreatment: Text: Medical word segmentation (adapted to UMLS, SNOMED-CT medical terminology set), stop word removal, standardization, vectorization; Imaging: DICOM to standard format conversion, noise reduction, normalization, lesion area annotation (automatic + semi-automatic), visual feature extraction; Uniformly allocate globally unique master index data PIDs and establish modal type labels (text / image / signal), data source labels, and patient ID association labels.

[0064] Step 2: Modal metadata index generation (first-level index) Modality-adaptive metadata indexes are generated for different modalities, forming a metadata index library: 1. Text Modal Metadata Index Metadata fields: PID, Modality, Patient ID, Document Type, Department, Creation Time, Medical Terminology Set, ICD-10 Code, Keywords, Text Length; Index structure: Inverted index (term → PID mapping) + structured index (field → PID mapping), supporting fast retrieval of terms and fields.

[0065] 2. Visual modality metadata index Metadata fields: PID, modality type, patient ID, image type (CT / MRI), location (brain / chest), sequence (T1 / T2), device model, lesion type, image size, acquisition time; Index structure: Multidimensional spatial index (R-tree index) + hash index, supporting fast filtering by dimensions such as location, type, and lesion.

[0066] 3. Cross-modal association metadata index Establish an association index for multimodal data under the same patient and the same diagnosis and treatment event (e.g., image PID → corresponding report PID, report PID → list of associated image PIDs). Metadata is uniformly stored in the metadata center of the data lake warehouse, supporting distributed querying and transaction consistency.

[0067] Step 3: Generate data content vector index (second-level index) Based on a pre-trained medical model, semantic-visual joint content index vectors are generated, and a unified vector index library is constructed. 1. Medical multimodal pre-trained model 1.1 Text Encoder Based on fine-tuning of a large medical model (such as MedBERT or BioBERT), the output is a medical semantic feature vector (dimensions: 512 / 768 / 1024). Specifically, the text encoder based on fine-tuning of the large medical model includes: a pre-trained medical language model, a text preprocessing module, a model fine-tuning module, and a feature vector output module. The pre-trained medical language model is MedBERT or BioBERT, whose input is a pre-processed sequence of medical text, and whose output is a contextual semantic feature representation containing multiple layers of Transformer encoders. The text preprocessing module is used to perform cleaning, standardization, word segmentation, and sequence length normalization operations on the medical text data to generate a token sequence, attention mask, and segmentation markers adapted to the model input format. The model fine-tuning module uses medical text-related tasks to fine-tune the pre-trained model, optimizing the model's semantic understanding of medical terms, clinical descriptions, and medical entities. The feature vector output module generates and outputs medical semantic feature vectors with dimensions of 512, 768, or 1024 based on the hidden layer output of the fine-tuned model for cross-modal retrieval.

[0068] 1.2 Visual Encoder The visual encoder, fine-tuned based on a pre-trained medical image model (such as MedViT or ResNet-Med), outputs medical visual feature vectors (of the same dimension). Specifically, the visual encoder fine-tuned based on the pre-trained medical image model includes: a medical image preprocessing module, a pre-trained medical image model, a model fine-tuning module, and a visual feature vector output module. The pre-trained medical image model is MedViT or ResNet-Med, used to extract basic visual features from medical images. The medical image preprocessing module performs standardization, enhancement, and format normalization operations on the original medical image data to generate image tensors adapted to the model's input format. The model fine-tuning module uses medical image-related tasks to fine-tune the pre-trained model to enhance its ability to represent lesion features, tissue structures, and pathological patterns. The visual feature vector output module generates and outputs medical visual feature vectors of the same dimension (512 / 768 / 1024) as the text encoder, based on the feature representation of the fine-tuned model, for use in cross-modal retrieval.

[0069] 2. Content Index Vector Generation Text data: Input preprocessed text → Text encoder → Semantic vector (content index); Image data: Input preprocessed image → visual encoder → visual vector (content index); Image-text data (e.g., images + reports): Joint input → Cross-modal alignment module → Joint semantic-visual vector (enhanced content indexing).

[0070] The specific implementation steps are as follows: Step (1): Dual-modal independent feature extraction Feature extraction was performed on both the image data and the text report data in the image-text pair: Image side: The preprocessed image data is input into the visual encoder, and the original visual feature vector is output. ; Text side: The preprocessed text report is input into the text encoder, which outputs the original semantic feature vector. ; in, For visual feature dimensions, These are text feature dimensions, and they can be the same or different.

[0071] Step (2): Cross-modal feature space alignment A shared semantic space is constructed through a cross-modal alignment module to achieve spatial mapping and dimensional unification of visual features and text features, including the following process: 1) Constructing the visual projection layer With text projection layer ,in For shared spatial dimensions; 2) Input the original visual feature vector V into the visual projection layer to obtain the aligned visual feature vector. ; 3) Input the original semantic feature vector T into the text projection layer to obtain the aligned text feature vector. ; 4). The contrastive learning loss function is used to optimize the training of the projection layer parameters, so that the same image-text pair... and By maximizing similarity in the shared space and minimizing the feature similarity of different image-text pairs, semantic alignment between modalities is achieved.

[0072] Step (3): Joint semantic-visual vector generation The aligned bimodal features are fused to generate an enhanced index vector containing multimodal complementary information, including the following process: 1) Employing a feature fusion strategy for and The following methods can be used for fusion: splicing and fusion: and By stitching along the dimensions, we obtain... ; Weighted summation fusion: Introducing learnable weights ,get ; Cross-attention fusion: Through a cross-modal attention layer, image features focus on key semantic information of the text, and text features focus on key visual regions of the image, outputting a fused feature vector; 2) The fused feature vector is optionally normalized or dimensionality reduced to obtain the joint semantic-visual enhanced index vector of the image-text pair data. This vector contains both image visual information and text semantic information, which can support cross-modal retrieval, image-text mutual search and multimodal association analysis.

[0073] 3) Vector index storage A vector library is constructed using the Approximate Nearest Neighbor (ANN) index (HNSW); The vector library and the metadata index library are bound together by PIDs to form a two-level index association system.

[0074] A multimodal vector library is constructed using a hierarchical navigation small-world (HNSW) approximate nearest neighbor (ANN) index. The specific implementation steps are as follows: First, the text semantic index vector, visual index vector, and joint semantic-visual enhancement index vector are preprocessed by unifying dimensions, L2 normalizing, and floating-point precision standardization to ensure the consistency of the vector space. Next, configure the HNSW core index parameters, including the number of nearest neighbors M, the neighbor exploration amount efConstruction during the graph construction phase, and the neighbor exploration amount efSearch during the retrieval phase; Subsequently, based on the exponential decay probability distribution, the vector is assigned a hierarchy, and a greedy nearest neighbor search is performed from the top to the bottom to build a hierarchical navigation graph structure layer by layer, thus completing the index topology establishment; The preprocessed vectors are batch inserted into the HNSW index, and each vector is assigned an internal unique identifier to establish a one-to-one mapping relationship between the vector's internal ID and the data unique identifier (PID). Finally, the HNSW graph structure, vector data, and mapping relationships are persistently stored to form an efficient searchable vector library that supports incremental data insertion, dynamic updates, and millisecond-level approximate nearest neighbor retrieval.

[0075] Step 4: Parsing User Retrieval Information 1. Receive user search requests, including the following types: Text retrieval: natural language (e.g., "images of cerebral infarction shown on cranial MRI"), keywords, medical terms, structured queries; Visual retrieval: Upload medical images and select lesion areas; Hybrid retrieval: combined text and visual queries.

[0076] 2. Query preprocessing: Text query: medical word segmentation, terminology standardization, vectorization (same as text encoder); Visual query: image preprocessing, feature extraction, vectorization (same as visual encoder); Generate query vectors and query metadata filtering conditions.

[0077] Step 5: Cross-modal retrieval and matching (core) 1. Parallel retrieval using a two-level index Metadata index retrieval (first-level index): Quickly filter candidate PID sets based on query metadata filtering conditions (part, type, department); Content vector retrieval (second-level index): Based on the query vector, perform ANN retrieval in the joint vector library to obtain the Top-K similar vectors and their corresponding PIDs.

[0078] 2. Cross-modal matching fusion Text → Image Matching: Text query vector → retrieve image visual vector → sort by cosine similarity; Image → Text Matching: Image query vector → retrieve text semantic vector → sort by similarity; Fusion strategy: Metadata filtering results (default weight 0.3) + vector similarity (default weight 0.7), weighted and sorted to obtain the candidate result set.

[0079] 3. Accuracy Optimization Enhanced weighting of medical terminology: Increased weighting of matching accuracy between medical terminology, ICD codes, and lesion terminology; Deduplication and noise reduction: Remove low similarity and irrelevant PIDs, and retain high confidence results.

[0080] Step 6: Reverse matching of modal source data 1. Based on the candidate result PID, perform a reverse query of the metadata index to obtain the data storage path and association information of each modality source. 2. Cross-modal source tracing: For each result PID, query the cross-modal association index to obtain its associated data in other modalities (e.g., image PID → corresponding report, medical record, test data). 3. Data Validation: Validate data integrity, privacy permissions, and timeliness, and remove invalid data.

[0081] Step 7: Results Information Integration and Return 1. Results Integration: Sort by similarity, relevance, and time; Integrate multimodal data to form complete diagnostic and treatment data units (e.g., images + reports + medical records + laboratory tests). The labeling and matching criteria include semantic matching points, visual similarity features, and association relationships.

[0082] 2. Results returned: The paginated results return structured results, including data for each modality, metadata, matching degree, and related data links; Supports visualization (image preview, text summary, association map).

[0083] The technical effects of this application are described in detail below with reference to the embodiments.

[0084] Example 1: Thyroid nodule screening and retrieval 1. Experimental configuration parameters The experimental environment and core parameters were standardized (Table 1 below), and all thyroid nodule screening and retrieval tests were performed based on this configuration. Table 1 2. Application Scenarios and Search Requirements This embodiment is applied to the thyroid nodule screening scenario in the endocrinology and ultrasound departments of a tertiary hospital. The precise clinical search criteria are: women aged 40–45, small cystic-solid nodules in the right lobe of the thyroid gland, TI-RADS2 category, and benign nodules with normal thyroid function. It is used to quickly screen for similar benign nodule cases, assist in clinical grading and case tracing.

[0085] 3. Core desensitization experimental sample data This example uses four clinical samples of thyroid nodules for desensitization, covering different ages, genders, nodule locations, nodule types, thyroid function indicators, and ultrasound grades, forming a complete test dataset. The core sample information is summarized in Table 2 below: Table 2 4. Core Logic of Data Preprocessing and Indexing Standardized processing of multimodal thyroid nodule data was implemented: ultrasound image data underwent noise reduction, size normalization, and nodule lesion region labeling, and a 768-dimensional visual feature vector was generated using a medical vision model; medical record text, thyroid function test indicators, and ultrasound grading reports were standardized with medical terminology and entity extraction, and a 768-dimensional semantic feature vector was generated using a medical text model; cross-modal attention mechanisms were used to align and fuse image and text features, generating a unified dimensional fusion vector. A first-layer metadata index (department, age, gender, nodule grade, thyroid function status) and a second-layer HNSW vector index were constructed sequentially, and multimodal data association and binding were completed.

[0086] 5. Retrieve quantification calculation results data Based on the dual-layer weighted fusion algorithm of this invention (metadata weight 0.3, vector weight 0.7), a full calculation was performed on the target retrieval requirements, and the candidate case matching scores and screening results are shown in Table 3 below: Table 3 The calculation results show that the present invention can accurately match TI-RADS2 benign cystic-solid nodules in the right lobe of the thyroid gland in women aged 40–45 years, and automatically removes interfering samples with incorrect gender, incorrect nodule location, abnormal grading, and abnormal thyroid function. The retrieval and matching accuracy is high and the anti-interference ability is strong.

[0087] 6. Overall performance comparison and improvement effect In a unified thyroid nodule screening dataset and hardware environment, the proposed solution is compared with traditional single-modal retrieval and general CLIP model retrieval in multiple dimensions. The technological advantages are significant, as shown in Table 4 below: Table 4 Experimental conclusion: In the case analysis, the platform was able to accurately locate the two most matching (PID00201, comprehensive score 0.9446) and the second most matching (PID00204, comprehensive score 0.9247) cases from the four interfering cases, which met all the search criteria, and automatically removed cases that did not match in terms of gender, lesion location, or grade, thus demonstrating good anti-interference capabilities.

[0088] Data from this model shows that its retrieval accuracy reached 95.1%, an improvement of nearly 25 percentage points compared to the general cross-modal model CLIP (70.5%); the false negative rate decreased to 2.4%, a significant reduction of nearly 10 percentage points compared to CLIP (12.3%). This indicates that the system's understanding of medical terminology and lesion characteristics far surpasses that of the general model. Thanks to its parallel retrieval design with a dual-level index, the system can perform lightning-fast filtering using metadata and quickly find the most similar "candidates" in a vector database of tens of millions of entries, achieving both speed and accuracy. In this case study, the average response time was only 32 milliseconds, far exceeding traditional single-modal retrieval (116 milliseconds) and general models (75 milliseconds), meaning it can basically support real-time clinical retrieval needs.

[0089] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0090] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0091] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0092] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0093] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0095] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0098] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-modal semantic-visual joint retrieval platform based on a multimodal data lake warehouse, characterized in that, The platform includes: The data access layer is used to uniformly access structured and unstructured multimodal medical data from the medical data lake warehouse platform; The data processing layer, connected to the data access layer, is used to standardize and preprocess the accessed multimodal medical data and generate metadata, assign globally unique identifiers to the processed data, and use medical models to convert text and image data into semantic feature vectors and visual feature vectors, respectively. An index building layer, connected to the data processing layer, is used to build a first index library based on the metadata and a second vector index library based on the semantic feature vector and the visual feature vector, wherein the stored vectors are associated with the globally unique identifier; The retrieval engine layer, connected to the index building layer, receives and parses user queries, performs parallel retrieval and matching based on the first index library and the second vector index library, and obtains cross-modal retrieval results; and The application interface layer is used to provide users with a search interface and return the cross-modal search results.

2. A cross-modal semantic-visual joint retrieval method based on a multimodal data lake warehouse, implemented on the platform described in claim 1, characterized in that, The method includes the following steps: Data Acquisition and Preprocessing: Collect multimodal medical data from the medical data lake warehouse and perform standardized processing, assigning a unified, globally unique identifier to the processed data; Modal metadata index generation: Based on the metadata of preprocessed data, construct a modality-adaptive first-level index for data of different modalities; Data content vector index generation: Through a coding model finely tuned in the medical field, preprocessed text and image data are converted into feature vectors, and a second-layer vector index is constructed; User retrieval information parsing: Receive and parse user retrieval requests, and generate query vectors and metadata filtering conditions; Cross-modal retrieval matching: Based on the metadata filtering conditions, filtering is performed in the first-level index, and based on the query vector, approximate nearest neighbor retrieval is performed in the second-level vector index. The results of the two are merged and sorted to obtain a candidate result set. Reverse source tracing of modal source data: Based on the globally unique identifier in the candidate result set, obtain the complete metadata of the corresponding data and other associated modal data information from the first-level index; Results Integration and Return: The multimodal data obtained from the source tracing is integrated and sorted according to the matching degree, and the structured results are returned to the user.

3. The method according to claim 2, characterized in that, The data acquisition and preprocessing specifically include: Structured patient data and unstructured data, including at least electronic medical record text and medical images, are collected from the medical data lake warehouse; the text data is processed by medical terminology segmentation and standardization, and the image data is processed by format conversion and normalization; and each processed data is assigned a unique patient master data identifier as the globally unique identifier, while its modality type is labeled.

4. The method according to claim 2, characterized in that, The generation of the modal metadata index specifically includes: Build a text metadata index for text modal data based on an inverted index of medical terminology and an index of structured fields; Construct a multidimensional spatial index for visual modal data based on image type, location, and lesion type; and Construct a cross-modal association index that records the relationships between different modalities of data under the same patient or treatment event.

5. The method according to claim 2, characterized in that, The process of converting preprocessed text and image data into feature vectors specifically includes: The preprocessed text data is input into the fine-tuned medical text encoder to generate a semantic feature vector with a unified dimension. The preprocessed image data is input into a finely tuned medical vision encoder to generate visual feature vectors of uniform dimensions; and For image-text pairs that have a corresponding relationship, the semantic feature vector and visual feature vector are mapped to a shared semantic space through a cross-modal alignment module, and feature fusion is performed to generate a joint semantic-visual enhanced feature vector.

6. The method according to claim 5, characterized in that, The cross-modal alignment module maps the semantic feature vector and visual feature vector to the same shared semantic space through a trainable projection layer, and optimizes it using a contrastive learning loss function to maximize the similarity of feature vectors of the same image-text pair in the shared space.

7. The method according to claim 2, characterized in that, The construction of the second-layer vector index adopts the approximate nearest neighbor indexing technique; specifically, it includes unifying and normalizing the dimensions of the semantic feature vector, visual feature vector and joint semantic-visual enhancement feature vector, and constructing an index based on the hierarchical navigation small world graph algorithm to establish a mapping relationship between the vector and the globally unique identifier.

8. The method according to claim 2, characterized in that, The cross-modal retrieval matching supports at least one of the following scenarios: retrieving image visual feature vectors using text query vectors, or retrieving text semantic feature vectors using image query vectors.

9. The method according to claim 2 or 8, characterized in that, The fusion ranking adopts a weighted fusion strategy, assigning a first weight to the metadata filtering results and a second weight to the vector similarity results.

10. The method according to claim 2, characterized in that, The acquisition of other modal data information refers to obtaining the identifiers of other modal data associated with the global unique identifier of the current candidate data by querying the cross-modal association index, thereby integrating them to form a complete diagnosis and treatment data unit including images, text reports, and electronic medical records.

Images