Diagnostic information driven multi-mode medical record similarity retrieval system

Abstract

The invention discloses a diagnostic information-driven multi-modal medical record similarity retrieval system, which comprises a data acquisition module, a data processing module and a similarity retrieval module, the feature extraction module is used for performing feature extraction on the electronic medical record data; the multi-modal semantic generation module is used for generating multi-modal semantic representation; the diagnosis code processing module is used for constructing a tree structure based on diagnosis codes in medical records, performing low-dimensional representation learning on the diagnosis codes by utilizing a Poincare embedding method, and realizing similarity measurement among diagnosis sets in combination with self-supervised contrast learning; the migration module is used for migrating the similarity knowledge of the diagnosis set to the learning process of the multi-modal semantic representation, and optimizing the multi-modal semantic representation to form medical record similarity representation driven by diagnosis information; and the retrieval module is used for performing similarity matching among the medical records in the shared semantic space and completing a retrieval task of the multi-modal medical records. According to the method, the problems of multi-mode isomerism and annotation data scarcity of the electronic medical record are effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical artificial intelligence and natural language processing, and in particular to a multimodal medical record similarity retrieval system driven by diagnostic information. Background Art

[0002] With the widespread adoption of electronic health records (EHRs) in modern healthcare, rapidly retrieving medical records with similar clinical characteristics from massive amounts of heterogeneous medical record data has become a key challenge in improving the efficiency and quality of diagnosis and treatment. Currently, medical record data typically contains both structured tabular information (such as examination indicators and medication records) and unstructured text (such as doctor's notes and medical history descriptions). Its multimodal nature makes it difficult for traditional single-modal analysis methods to fully capture the semantic information of medical records.

[0003] Furthermore, the high cost and subjectivity of acquiring high-quality annotated samples limit the performance of supervised learning models in modeling medical record similarity. Existing methods generally employ simple multimodal concatenation or ignore the semantic structure of diagnostic information, resulting in poor cross-modal alignment and a lack of interpretability and generalizability in medical record matching results.

[0004] Therefore, there is an urgent need for a medical record similarity modeling method that can fully utilize multimodal information, combine diagnostic structure semantics, and reduce dependence on labeled data to achieve more accurate and efficient similar medical record retrieval. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies and shortcomings of the existing technology and provide a multimodal medical record similarity retrieval system driven by diagnostic information. By fusing structured tables and unstructured text information and combining the hierarchical semantic relationship of diagnostic codes to model diagnostic sets, a more discriminative medical record similarity representation is constructed, thereby improving the accuracy and robustness of medical record retrieval, and achieving efficient and accurate medical record matching and retrieval, which is suitable for assisting clinical diagnosis and treatment decision-making and the construction of intelligent medical systems.

[0006] To achieve the above objectives, the present invention provides a technical solution: a multimodal medical record similarity retrieval system driven by diagnostic information, comprising:

[0007] A data acquisition module is used to acquire electronic medical record data, wherein the electronic medical record data includes structured table information, unstructured text information and diagnosis codes in the medical record;

[0008] The diagnostic code processing module constructs a diagnosis set based on the diagnostic codes in the medical records, then builds a tree structure based on the diagnosis set. It uses the Poincare embedding method to learn a low-dimensional representation of the diagnostic codes, and combines self-supervised contrastive learning to achieve similarity measurement between diagnosis sets.

[0009] The transfer module is used to transfer the similarity knowledge of the diagnosis set to the learning process of the multimodal semantic representation, construct the positive and negative sample medical record combinations required for training, and use self-supervised learning to optimize the multimodal semantic representation to form a medical record similarity representation driven by diagnosis information;

[0010] A feature extraction module is used to extract structured features from the structured table information to obtain a structured feature representation; and to extract text features from the unstructured text information to obtain a text feature representation;

[0011] The multimodal semantic generation module builds a dual-channel cross-attention mechanism, which establishes a cross-modal association between the structured feature representation and the text feature representation. It captures complementary features through a bidirectional cross-attention layer calculation, and uses a contrastive learning strategy based on the above-mentioned positive and negative sample medical records to align and fuse multimodal features, and generates a multimodal semantic representation through a gating mechanism.

[0012] The retrieval module is used to perform similarity matching between medical records in a shared semantic space and complete the retrieval task of multimodal medical records.

[0013] Furthermore, the structured feature extraction of the structured table information is performed as follows:

[0014] The TabNet model is used to perform multi-step serial attention processing on structured table information. In each step, key features are dynamically selected through the Attentive Transformer, and high-order interaction features are generated through the Feature Transformer. Finally, the initial table embedding tab_emb containing low-order features and high-order interaction features is output as the feature representation of the structured table.

[0015] Furthermore, the text feature extraction of the unstructured text information includes the following steps:

[0016] 1) The LLaMA model pre-trained by the autoregressive language modeling task is used to capture the semantic associations between medical terms in the unstructured text of medical records. Its mathematical formula is defined as:

[0017]

[0018] Where, represents the loss function of the language modeling task; t k represents the kth word in the medical record text sequence, that is, the language unit that needs to be predicted; t1, t2, ..., t k-1represents the medical record text sequence composed of the first k-1 tokens, which constitutes the historical context of the current prediction task; θ represents the set of learnable parameters of the LLaMA model, i.e., the model parameters; P(t k |t1,t2,...,t k-1 ; θ) represents the time from a given historical context t1 to t k-1 Under the conditions of model parameters θ, the LLaMA model is used to predict the kth word t k Estimated probability of occurrence;

[0019] The entire mathematical formula represents minimizing the negative log-likelihood loss by maximizing the conditional probability of the target word, thereby training the LLaMA model to better capture the semantic relationships and contextual dependencies in the medical record text;

[0020] 2) Using the decoder of the pre-trained LLaMA model to perform semantic modeling on the unstructured text information in the medical records;

[0021] 3) Extract the token at the end of the decoder and embed it into text_emb as the initial text embedding to represent the text feature. Its mathematical formula is defined as:

[0022] text_emb=Decoder(t′1,t′2,…,t′ k )[-1])

[0023] Where Decoder represents the decoder structure in the LLaMA model; t′1, t′2, ..., t′ k Represents the medical record text sequence after word segmentation, where t′ k Represents the kth word in the medical record text sequence.

[0024] Furthermore, the dual-channel cross-attention mechanism performs the following operations:

[0025] Construct a table-to-text pathway, using table features as query vectors and text features as key and value vectors, and calculate cross-modal attention weights to obtain text-enhanced table embedding h. b As table enhancement features, a text-to-table path is constructed at the same time, with text features as query vectors and table features as key and value vectors, and the attention weights are reversely calculated to obtain the table-enhanced text embedding h t As a text enhancement feature, the outputs of the text path and the table path are weighted fused through a dynamic gating mechanism to generate a fine-grained table-text fusion feature e.

[0026] Furthermore, the contrastive learning aligns and fuses the multimodal features of the table and text, as follows:

[0027] The InfoNCE loss function is used to narrow the similarity between the table-enhanced features and text-enhanced features from the same medical record, while simultaneously increasing the feature distance between different medical records. The cross-entropy loss of the main diagnostic classification task is combined to jointly optimize the discriminative ability of the multimodal fusion features. The parameters of the cross-attention layer and the gating mechanism are updated through a multi-task joint training strategy. The loss function calculation formula involved is as follows:

[0028]

[0029] Where, The loss function representing the alignment of table and text features; denote the text enhancement features and table enhancement features from medical record i respectively; h j represents the text enhancement feature or table enhancement feature from medical record j; τ1 represents the temperature coefficient, which is used to adjust the smoothness of the similarity distribution; N represents the number of samples in a batch of data; exp(·) and log(·) represent the exponential function and logarithmic function, respectively; sim(·) represents the similarity function, which is expressed as the inner product of two vectors.

[0030] Furthermore, a tree structure of the diagnosis set is constructed and similarity measurement is performed as follows:

[0031] 1) All diagnostic codes are modeled as a tree based on a hierarchical relationship. Each node in the tree represents a disease category or subcategory. Each node in the tree is a diagnosis node, and the semantic relationship between parent and child nodes is represented by edges.

[0032] 2) For each medical record’s diagnosis code set, map the diagnosis code to the leaf node in the tree, extract all leaf nodes and their ancestral paths, and connect them through the root node to form a minimum spanning supertree t containing all diagnosis nodes as the diagnosis set supertree of the medical record. Each diagnosis node has a corresponding weight w i and its own eigenvector x i ,The feature vector comes from the text encoding of the diagnosis code description corresponding to the diagnosis node, and is encoded using the BioBERT model;

[0033] 3) Given a diagnostic set hypertree Introducing a learnable mapping function f θ (·) The characteristic vector x of each diagnostic node in Euclidean space i Mapped into the Poincare sphere model in the hyperbolic space, the low-dimensional representation z of the diagnostic node in the hyperbolic space is obtained i , the corresponding mathematical formula is as follows:

[0034]

[0035] Where, Represents a projection operation for mapping a vector in Euclidean space to a d-dimensional Poincare sphere model Medium; low-dimensional representation z i Represents the input vector x i Through the function f θ (·) The embedded vector obtained after mapping is located in the Poincare sphere model middle;

[0036] 4) Based on the embedding vector z of all diagnostic nodes i Compute the weighted hyperbolic centroid of the entire tree And based on the hyperbolic distance from each diagnostic node to the centroid and its weight w i Generate attention weight α i , the mathematical formula involved is as follows;

[0037]

[0038] Where exp0(·) represents the mapping from hyperbolic space to tangent space; log0(z i ) represents the mapping from the tangent space back to the Poincare sphere model; cosh -1 (·) represents the hyperbolic inverse cosine function, which is used to calculate the distance in hyperbolic space; ‖·‖ represents the Euclidean norm, that is, the two-norm of the vector; weighted hyperbolic centroid Represents the embedding vector of the center of mass of tree T in the Poincare sphere model, which also satisfies Represents vector z i With vector The Euclidean distance between them; λ represents the balance factor, which controls the relative influence between the distance term and the weight term; ∈ represents a small positive constant, which is used to avoid the zero value problem in the logarithmic function; τ2 represents the temperature coefficient, which is used to adjust the smoothness of the attention score distribution;

[0039] 5) Using attention weight α i Vector x for diagnostic nodes i Perform weighted averaging on Riemannian geometry and aggregate into a preliminary tree embedding representation z T , its mathematical formula is as follows:

[0040]

[0041] 6) Hyperbolic variance and maximum hyperbolic distance are introduced as scatter metrics, which are combined with the preliminary tree embedding to form a diagnostic tree representation vector with discriminative ability. The mathematical formula involved is as follows:

[0042]

[0043] Where, Var TRepresentation of nodes in the Poincare sphere model The hyperbolic variance in is used to measure the vector z of all diagnostic nodes i Relative to vector The degree of dispersion; n represents the number of diagnostic nodes in the tree T; R T represents the maximum hyperbolic distance between all diagnostic nodes and the center of mass, which is used to measure the structural tension; h T Represents the final vector representation of the entire tree; z T ‖Var T ‖R T It means concatenating three vectors to generate a new vector; Indicates that the dimension of the final vector is d+2, where d represents the embedding dimension, plus the variance Var T and the maximum distance R T two scalars;

[0044] 7) By randomly deleting, replacing, and adding the diagnostic tree T corresponding to the anchor samples a Subtree of the construct similar diagnostic tree as the positive sample T p , select other diagnostic trees as negative samples T n , get the triplet sample, and use the triplet loss function to jointly optimize the mapping function f θ The (·) parameter makes similar diagnostic trees close together in the shared semantic space and different diagnostic trees far apart, thus achieving hierarchical perception and discriminative representation learning of the diagnostic set. The mathematical formula of the triplet loss function is as follows:

[0045]

[0046] Where, Represents the triplet loss function, which is used to measure the relative distance between anchor samples, positive samples and negative samples, and optimize the model so that the distance between similar samples is closer and the distance between dissimilar samples is farther; represents the hyperbolic distance defined in the Poincare sphere model, which is used to measure the geometric difference between two embedding vectors; Respectively represent the diagnostic tree T corresponding to the anchor sample a , positive sample T p And negative samples T n Tree representation in the Poincare sphere model; γ represents a marginal hyperparameter that controls the minimum acceptable distance difference between positive and negative samples. This parameter encourages negative samples to be at least γ farther away than positive samples.

[0047] Furthermore, the learning process of transferring the similarity knowledge of the diagnosis set to the multimodal semantic representation includes the following steps:

[0048] 1) The diagnostic tree corresponding to the diagnostic set is mapped through the mapping function fθ (·) is mapped to the Poincare sphere model in hyperbolic space, and the hyperbolic distance between the diagnosis trees is calculated as the similarity measure. Based on the similarity measure results between the diagnosis trees, the positive sample medical records E are screened out. + With negative sample medical records E - Combination, construct the sample combination required for training;

[0049] 2) The learning process of multimodal semantic representation of medical records is guided by diagnostic information, and the contrastive learning strategy is used to optimize model parameters. The mathematical formula of the loss function for medical record similarity learning is as follows:

[0050]

[0051] Where, represents the loss function for medical record similarity learning; e q 、e + 、e - Represents query medical records and positive sample medical records E respectively + And negative sample medical records E - The corresponding multimodal fusion features; τ3 represents the temperature coefficient, which is used to adjust the smoothness of the similarity distribution; N - Indicates the number of negative samples;

[0052] 3) Enhance the ability of multimodal fusion features to understand the semantics of medical records through the knowledge transfer mechanism, and generate more discriminative medical record similarity representations.

[0053] Furthermore, similarity matching between medical records in the shared semantic space is performed as follows:

[0054] The multimodal semantic representation and the medical record similarity representation are jointly encoded and mapped to a unified shared semantic space. In this space, the semantic similarity between medical records is calculated using the cosine similarity distance measurement method. The list of the most similar medical records is returned according to the set threshold or sorting strategy, completing the efficient retrieval task of multimodal medical records.

[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0056] 1. This paper establishes an explicit connection between structured tabular information and unstructured text information by constructing a dual-channel cross-attention mechanism, capturing cross-modal complementary features. Compared to traditional methods that use simple splicing or unidirectional attention, this method achieves more fine-grained and semantically aligned modalities, significantly improving the model's understanding of medical record content.

[0057] 2. The present invention adopts a contrastive learning strategy to align and fuse structured features and text features, making information of different modalities more compact in a shared semantic space, enhancing the model's discriminative ability, and avoiding the problem of information loss caused by the excessive semantic gap between modalities in traditional fusion methods.

[0058] 3. The present invention constructs a minimum spanning hypertree structure of the medical record diagnosis set based on the hierarchical relationship of diagnosis codes, fully retaining the classification logic and semantic associations between diseases, providing a more compact and semantically rich representation for subsequent similarity calculations, and breaking through the limitation of traditional methods that only rely on label overlap to measure similarity.

[0059] 4. This paper maps diagnostic nodes onto a Poincare sphere model in hyperbolic space, combining a learnable mapping function with an attention mechanism to generate discriminative low-dimensional representations. Compared to traditional Euclidean space embedding methods, this method better preserves the hierarchical structure of diagnostic codes, improving the discriminability and generalization of diagnostic sets.

[0060] 5. The present invention constructs positive and negative sample triplets and adopts the triplet loss function to jointly optimize the embedding parameters, so that diagnostic trees with similar structures are close to each other in the shared semantic space, and diagnostic trees with different structures are far away from each other, which effectively solves the problem of scarce labeled data and improves the robustness and migration ability of the model.

[0061] 6. This invention introduces a dynamic gating mechanism into the dual-channel cross-attention mechanism. Fusion weights are dynamically generated based on the combined representation of table and text features, ensuring flexible adjustment of the fusion strategy based on modality importance in different cases. This mechanism enhances the model's sensitivity to key information and improves the flexibility and accuracy of multimodal fusion.

[0062] 7. This invention, for the first time, uses similarity knowledge across diagnostic sets as a guiding signal to drive the learning process of multimodal semantic representations. By constructing a combination of positive and negative medical records based on diagnostic information and optimizing model parameters using self-supervised comparative learning, it achieves end-to-end knowledge transfer from structure to semantics, significantly improving the accuracy of medical record similarity modeling.

[0063] 8. The present invention jointly encodes multimodal semantic representation and medical record similarity representation, maps them to a unified shared semantic space, and performs similarity matching between medical records in this space. This not only improves retrieval efficiency, but also enhances the interpretability of the results. It is suitable for fast query and intelligent recommendation of large-scale electronic medical record databases.

[0064] 9. This invention employs self-supervised learning strategies, including contrastive learning and triplet learning, which significantly reduces reliance on manually labeled samples and lowers model training costs. Furthermore, the model possesses excellent generalization capabilities and is suitable for modeling medical record similarity across a variety of medical scenarios.

[0065] 10. The present invention provides doctors with treatment plans, medication records and prognosis information of similar historical cases by accurately matching similar medical records, which helps to formulate personalized diagnosis and treatment plans and improve the quality and efficiency of medical services.

[0066] 11. The multimodal semantic generation module and diagnostic coding processing module proposed in this invention can be widely applied to other medical text analysis tasks, such as disease prediction, treatment pathway recommendation, patient clustering, etc., and have strong versatility and engineering implementation value.

[0067] 12. The present invention retains some adjustable parameters during the model training process, supports incremental learning and online updates when new medical record data is accumulated, and ensures that the model adapts to the evolution trend of medical data in the long term. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a diagram showing the structure of tabular data and text data in medical records; the PPD test in the tabular part of the figure represents the tuberculin test.

[0069] Figure 2 This is a flowchart of the multi-step attention mechanism of the TabNet model; the ReLU in the figure is a linear rectification unit.

[0070] Figure 3 This is a flow chart of the multimodal semantic generation module. In the figure, Text2Tab Cross-Attention represents the path from table to text, Tab2Text Cross-Attention represents the path from text to table, MLP represents the gating mechanism, and Query represents the embedding of the input path as the query in the attention calculation.

[0071] Figure 4 The hierarchical division of the diagnostic coding system.

[0072] Figure 5 is a schematic diagram of the construction of a triplet sample; in the figure, x n Indicates inserting a new node. DETAILED DESCRIPTION

[0073] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0074] This embodiment discloses a multimodal medical record similarity retrieval system driven by diagnostic information. It can effectively integrate structured table information and unstructured text information in electronic medical records, and combine the hierarchical semantic relationship of diagnostic codes to model diagnostic sets, ultimately achieving efficient and accurate medical record matching and retrieval in a shared semantic space. The system includes:

[0075] A data acquisition module is used to acquire electronic medical record data, wherein the electronic medical record data includes structured table information, unstructured text information and diagnosis codes in the medical record;

[0076] The diagnostic code processing module constructs a diagnosis set based on the diagnostic codes in the medical records, then builds a tree structure based on the diagnosis set. It uses the Poincare embedding method to learn a low-dimensional representation of the diagnostic codes, and combines self-supervised contrastive learning to achieve similarity measurement between diagnosis sets.

[0077] The transfer module is used to transfer the similarity knowledge of the diagnosis set to the learning process of the multimodal semantic representation, construct the positive and negative sample medical record combinations required for training, and use self-supervised learning to optimize the multimodal semantic representation to form a medical record similarity representation driven by diagnosis information;

[0078] A feature extraction module is used to extract structured features from the structured table information to obtain a structured feature representation; and to extract text features from the unstructured text information to obtain a text feature representation;

[0079] The multimodal semantic generation module builds a dual-channel cross-attention mechanism, which establishes a cross-modal association between the structured feature representation and the text feature representation. It captures complementary features through a bidirectional cross-attention layer calculation, and uses a contrastive learning strategy based on the above-mentioned positive and negative sample medical records to align and fuse multimodal features, and generates a multimodal semantic representation through a gating mechanism.

[0080] The retrieval module is used to perform similarity matching between medical records in a shared semantic space and complete the retrieval task of multimodal medical records.

[0081] The following is a detailed description of each module of the present invention.

[0082] ①Data acquisition module:

[0083] First, electronic medical record data is obtained from hospital information systems or public medical datasets (such as MIMIC-III). This electronic medical record data includes structured table information and unstructured text information. The structured table information includes basic patient demographic information (such as age and gender) and laboratory test indicators (such as blood sugar and white blood cell count); the unstructured text information includes doctor's notes, medical records, and descriptions of chief complaints.

[0084] like Figure 1 As shown in the figure, the structure of tabular data and text data in a typical medical record is demonstrated, reflecting the complementarity of the two in expressing the patient's health status.

[0085] ②Feature extraction module:

[0086] For the tabular information in medical records, numerical features are first normalized, while categorical features are converted to numerical form through label encoding. After all features are processed, they are input into the TabNet model to obtain a structured representation. To enhance the model's ability to model the importance of different features, TabNet is pre-trained. The pre-training task is set to predict the primary diagnosis label of the medical record based on the tabular data, thereby guiding the model to learn key features closely related to the primary diagnosis.

[0087] The TabNet model performs multi-step sequential attention processing on structured tabular information. In each step, the Attentive Transformer dynamically selects key features, and the Feature Transformer generates high-order interaction features. The final output is the initial table embedding tab_emb, which contains both low-order and high-order interaction features, as a structured feature representation.

[0088] like Figure 2 As shown in the figure, the multi-step attention mechanism process of the TabNet model is demonstrated, including feature selection and feature transformation modules, as well as the feedback mechanism between each step, which fully demonstrates the model's ability to accurately extract task-related core information from complex case tables.

[0089] For unstructured text information, the decoder part of the LLaMA model is used for semantic modeling. This model is pre-trained based on the autoregressive language modeling task, and its goal is to maximize the conditional probability of the target word. Its mathematical formula is defined as:

[0090]

[0091] Where, represents the loss function of the language modeling task; t k represents the kth word in the medical record text sequence, that is, the language unit that needs to be predicted; t1, t2, ..., t k-1 represents the medical record text sequence composed of the first k-1 tokens, which constitutes the historical context of the current prediction task; θ represents the set of learnable parameters of the LLaMA model, i.e., the model parameters; P(t k |t1,t2,...,t k-1 ; θ) represents the time from a given historical context t1 to t k-1 Under the conditions of model parameters θ, the LLaMA model is used to predict the kth word t k Finally, extract the token embedding text_emb at the end of the trained LLaMA model decoder as the initial text embedding to represent the text features.

[0092] LLaMA-3.2-1B was selected as the encoding model for text data in medical records, mainly based on its balance between adaptability and efficiency in medical scenarios. The text in medical records is often very long, highly professional, and loosely structured (including chief complaints, medical history, etc.). The 1B-scale LLaMA model supports long text input and can better learn key medical terms and semantic relationships in medical records. At the same time, the pre-training corpus of the model contains a large amount of scientific text, which enables it to have a preliminary semantic understanding of medical terms (such as "glomerular filtration rate" and "myocardial enzyme spectrum"). Compared with larger models, this model has lower computational costs and stronger and more accurate representation capabilities than smaller models.

[0093] The autoregressive decoder of the LLaMA model uses a unidirectional attention mechanism, which aggregates contextual information from left to right. Under this mechanism, the final token accumulates semantic information of the entire sequence through multiple layers of attention weights. Therefore, the final token embedding output by the LLaMA model decoder is used as the initial representation of the medical record text. Its mathematical formula is defined as:

[0094] text_emb=Decoder(t1′,t2′,...,t k ′)[-1])

[0095] Where Decoder represents the decoder structure in the LLaMA model; t1′, t2′, ..., t k ′ represents the medical record text sequence after word segmentation processing, where t k ′ represents the kth word in the medical record text sequence.

[0096] This process enables the model to capture the semantic associations between medical terms in medical record text, and is particularly suitable for long and highly professional clinical record scenarios.

[0097] ③Multimodal semantic generation module:

[0098] A dual-channel cross-attention mechanism is constructed to achieve bidirectional interaction between structured table information and unstructured text information. Specifically:

[0099] 1. Table-to-text pathway: Using table features as query vectors and text features as key and value vectors, we calculate cross-modal attention weights to obtain table-enhanced text features.

[0100] 2. Text-to-table pathway: Using text features as query vectors and table features as key and value vectors, the attention weights are calculated in reverse order to obtain text-enhanced table features.

[0101] 3. Dynamic gating mechanism: Dynamically generates fusion weights based on the joint representation of table and text features to achieve fine-grained multimodal fusion feature output.

[0102] Among them, 1 and 2 constitute a dual-channel cross attention layer, such as Figure 3 As shown in the figure, this figure intuitively shows how table features and text features guide each other, allowing the model to focus on the parts that are connected between the table and the text, such as establishing a semantic connection between "white blood cell count" and "description of infection symptoms", thereby enhancing cross-modal understanding capabilities.

[0103] To further improve the quality of multimodal fusion, a contrastive learning strategy is introduced to narrow the similarity between tabular features and text features from the same medical record, while increasing the distance between features from different medical records. The parameters of the cross-attention layer and the gating mechanism are updated. The loss function calculation formula involved is as follows:

[0104]

[0105] Where, The loss function representing the alignment of table and text features; denote the text enhancement features and table enhancement features from medical record i respectively; h j represents the text enhancement feature or table enhancement feature from medical record j; τ1 represents the temperature coefficient, which is used to adjust the smoothness of the similarity distribution; N represents the number of samples in a batch of data; exp(·) and log(·) represent the exponential function and logarithmic function, respectively; sim(·) represents the similarity function, which is expressed as the inner product of two vectors.

[0106] ④ Diagnostic coding processing module:

[0107] like Figure 4 As shown in the figure, the hierarchical division of the ICD coding system includes four levels: chapter, section, class and subclass, reflecting the classification logic and semantic attribution relationship between diseases.

[0108] All diagnostic codes can be modeled as a tree according to the hierarchical relationship. Each node in the tree represents a disease category or subcategory. Each node in the tree is a diagnostic node, and the semantic relationship between parent and child nodes is represented by edges. For each diagnostic code set of a medical record, the diagnostic code is mapped to the leaf node in the tree. All leaf nodes and their ancestral paths are extracted and connected through the root node to form a minimum spanning supertree t containing all diagnostic nodes as the diagnostic set supertree of the medical record. Each diagnostic node has a corresponding weight w. i and its own eigenvector x i ,The feature vector comes from the text encoding of the diagnosis code description corresponding to the diagnosis node, and is encoded using the BioBERT model;

[0109] Given a diagnosis set hypertree Introducing a learnable mapping function f θ(·) The characteristic vector x of each diagnostic node in Euclidean space i Mapped into the Poincare sphere model in the hyperbolic space, the low-dimensional representation z of the diagnostic node in the hyperbolic space is obtained i , the corresponding mathematical formula is as follows:

[0110]

[0111] Where, Represents a projection operation for mapping a vector in Euclidean space to a d-dimensional Poincare sphere model Medium; low-dimensional representation z i Represents the input vector x i Through the function f θ (·) The embedded vector obtained after mapping is located in the Poincare sphere model middle;

[0112] Then, based on the embedding vector z of all diagnosis nodes i Compute the weighted hyperbolic centroid of the entire tree And based on the hyperbolic distance from each diagnostic node to the centroid and its weight w i Generate attention weight α i , the mathematical formula involved is as follows;

[0113]

[0114] Where exp0(·) represents the mapping from hyperbolic space to tangent space; log0(z i ) represents the mapping from the tangent space back to the Poincare sphere model; cosh -1 (·) represents the hyperbolic inverse cosine function, which is used to calculate the distance in hyperbolic space; ‖·‖ represents the Euclidean norm, that is, the two-norm of the vector; weighted hyperbolic centroid Represents the embedding vector of the center of mass of tree T in the Poincare sphere model, which also satisfies Represents vector z i With vector The Euclidean distance between them; λ represents the balance factor, which controls the relative influence between the distance term and the weight term; ∈ represents a small positive constant, which is used to avoid the zero value problem in the logarithmic function; τ2 represents the temperature coefficient, which is used to adjust the smoothness of the attention score distribution;

[0115] Use attention weight α i Vector x for diagnostic nodes i Perform weighted averaging on Riemannian geometry and aggregate into a preliminary tree embedding representation z T , its mathematical formula is as follows:

[0116]

[0117] Hyperbolic variance and maximum hyperbolic distance are introduced as scatter metrics, which are concatenated with the preliminary tree embedding to form a diagnostic tree representation vector with discriminative ability, which is used as the final diagnostic set representation. The mathematical formula involved is as follows:

[0118]

[0119] Where, Var T Representation of nodes in the Poincare sphere model The hyperbolic variance in is used to measure the vector z of all diagnostic nodes i Relative to vector The degree of dispersion; n represents the number of diagnostic nodes in the tree T; R T represents the maximum hyperbolic distance between all diagnostic nodes and the center of mass, which is used to measure the structural tension; h T Represents the final vector representation of the entire tree; z T ‖Var T ‖R T It means concatenating three vectors to generate a new vector; Indicates that the dimension of the final vector is d+2, where d represents the embedding dimension, plus the variance Var T and the maximum distance R T two scalars;

[0120] In order to improve the discriminative ability of diagnostic set representation, a self-supervised triple learning framework is designed. By randomly deleting, replacing, and adding the diagnostic tree T corresponding to the anchor sample a Subtree of the construct similar diagnostic tree as the positive sample T p , select other diagnostic trees as negative samples T n , get the triplet sample, and use the triplet loss function to jointly optimize the mapping function f θ The (·) parameter makes similar diagnostic trees close together in the shared semantic space and different diagnostic trees far apart, thus achieving hierarchical perception and discriminative representation learning of the diagnostic set. The mathematical formula of the triplet loss function is as follows:

[0121]

[0122] Where, Represents the triplet loss function, which is used to measure the relative distance between anchor samples, positive samples and negative samples, and optimize the model so that the distance between similar samples is closer and the distance between dissimilar samples is farther; represents the hyperbolic distance defined in the Poincare sphere model, which is used to measure the geometric difference between two embedding vectors; Respectively represent the diagnostic tree T corresponding to the anchor samplea , positive sample T p And negative samples T n Tree representation in the Poincare sphere model; γ represents a marginal hyperparameter that controls the minimum acceptable distance difference between positive and negative samples. This parameter encourages negative samples to be at least γ farther away than positive samples.

[0123] like Figure 5 As shown in the figure, the construction process of triple samples is demonstrated, including the selection strategies of anchor samples, positive samples and negative samples, as well as their distribution trends in the hyperbolic space, which reflects the model's ability to learn structured semantics under unsupervised conditions.

[0124] ⑤Migration module:

[0125] The diagnostic trees corresponding to the diagnosis set are mapped to hyperbolic space using a mapping function, and the hyperbolic distance between the diagnostic trees is calculated as a similarity measure. Based on the similarity between the diagnostic trees, positive and negative sample medical record combinations are screened to construct the sample pairs required for training. Based on the constructed sample pairs, a contrastive learning strategy is used to optimize the model parameters. The knowledge transfer mechanism is used to enhance the multimodal fusion module's understanding of medical record semantics, generating a more discriminative medical record similarity representation. The mathematical formula for the loss function of medical record similarity learning is as follows:

[0126]

[0127] Where, represents the loss function for medical record similarity learning; e q 、e + 、e - Represents query medical records and positive sample medical records E respectively + And negative sample medical records E - The corresponding multimodal fusion features; τ3 represents the temperature coefficient, which is used to adjust the smoothness of the similarity distribution; N - Indicates the number of negative samples;

[0128] ⑥Search module:

[0129] Multimodal semantic representations and medical record similarity representations are jointly encoded and mapped into a unified shared semantic space. Within this space, semantic similarity between medical records is calculated using distance metrics such as cosine similarity. Based on a predefined threshold or sorting strategy, a list of the most similar medical records is returned, enabling efficient retrieval of multimodal medical records.

[0130] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A multimodal medical record similarity retrieval system driven by diagnostic information, characterized by: include: A data acquisition module is used to acquire electronic medical record data, wherein the electronic medical record data includes structured table information, unstructured text information and diagnosis codes in the medical record; The diagnostic code processing module constructs a diagnosis set based on the diagnostic codes in the medical records, then builds a tree structure based on the diagnosis set. It uses the Poincare embedding method to learn a low-dimensional representation of the diagnostic codes, and combines self-supervised contrastive learning to achieve similarity measurement between diagnosis sets. The transfer module is used to transfer the similarity knowledge of the diagnosis set to the learning process of the multimodal semantic representation, construct the positive and negative sample medical record combinations required for training, and use self-supervised learning to optimize the multimodal semantic representation to form a medical record similarity representation driven by diagnosis information; A feature extraction module, configured to extract structured features from the structured table information to obtain a structured feature representation; Performing text feature extraction on the unstructured text information to obtain text feature representation; The multimodal semantic generation module builds a dual-channel cross-attention mechanism, which establishes a cross-modal association between the structured feature representation and the text feature representation. It captures complementary features through a bidirectional cross-attention layer calculation, and uses a contrastive learning strategy based on the above-mentioned positive and negative sample medical records to align and fuse multimodal features, and generates a multimodal semantic representation through a gating mechanism. The retrieval module is used to perform similarity matching between medical records in a shared semantic space and complete the retrieval task of multimodal medical records.

2. The multimodal medical record similarity retrieval system driven by diagnostic information according to claim 1, characterized in that: The structured feature extraction of the structured table information is performed as follows: The TabNet model is used to perform multi-step serial attention processing on structured table information. In each step, key features are dynamically selected through the Attentive Transformer, and high-order interaction features are generated through the Feature Transformer. Finally, the initial table embedding tab_emb containing low-order features and high-order interaction features is output as the feature representation of the structured table.

3. The multimodal medical record similarity retrieval system driven by diagnostic information according to claim 1, characterized in that: The text feature extraction of unstructured text information includes the following steps: 1) The LLaMA model pre-trained by the autoregressive language modeling task is used to capture the semantic associations between medical terms in the unstructured text of medical records. Its mathematical formula is defined as: Where, represents the loss function of the language modeling task; t k represents the kth word in the medical record text sequence, that is, the language unit that needs to be predicted; t1, t2, ..., t k-1 represents the medical record text sequence composed of the first k-1 tokens, which constitutes the historical context of the current prediction task; θ represents the set of learnable parameters of the LLaMA model, i.e., the model parameters; P(t k |t1,t2,...,t k-1 ; θ) represents the time from a given historical context t1 to t k-1 Under the conditions of model parameters θ, the LLaMA model is used to predict the kth word t k Estimated probability of occurrence; The entire mathematical formula represents minimizing the negative log-likelihood loss by maximizing the conditional probability of the target word, thereby training the LLaMA model to better capture the semantic relationships and contextual dependencies in the medical record text; 2) Using the decoder of the pre-trained LLaMA model to perform semantic modeling on the unstructured text information in the medical records; 3) Extract the token at the end of the decoder and embed it into text_emb as the initial text embedding to represent the text feature. Its mathematical formula is defined as: text_emb=Decoder(t1′,t2′,...,t k ′)[-1]) Where Decoder represents the decoder structure in the LLaMA model; t1′, t2′, ..., t k ′ represents the medical record text sequence after word segmentation processing, where t k ′ represents the kth word in the medical record text sequence.

4. The diagnostic information-driven multimodal medical record similarity retrieval system according to claim 1, characterized in that: The dual-channel crisscross attention mechanism performs the following operations: Construct a table-to-text pathway, using table features as query vectors and text features as key and value vectors, and calculate cross-modal attention weights to obtain text-enhanced table embedding h. b As table enhancement features, a text-to-table path is constructed at the same time, with text features as query vectors and table features as key and value vectors, and the attention weights are reversely calculated to obtain the table-enhanced text embedding h t As a text enhancement feature, the outputs of the text path and the table path are weighted fused through a dynamic gating mechanism to generate a fine-grained table-text fusion feature e.

5. The diagnostic information driven multimodal medical record similarity retrieval system according to claim 1, characterized in that: The contrastive learning aligns and fuses the multimodal features of the table and text as follows: The InfoNCE loss function is used to narrow the similarity between the table-enhanced features and text-enhanced features from the same medical record, while simultaneously increasing the feature distance between different medical records. The cross-entropy loss of the main diagnostic classification task is combined to jointly optimize the discriminative ability of the multimodal fusion features. The parameters of the cross-attention layer and the gating mechanism are updated through a multi-task joint training strategy. The loss function calculation formula involved is as follows: Where, The loss function representing the alignment of table and text features; denote the text enhancement features and table enhancement features from medical record i respectively; h j represents the text enhancement feature or table enhancement feature from medical record j; τ1 represents the temperature coefficient, which is used to adjust the smoothness of the similarity distribution; N represents the number of samples in a batch of data; exp(·) and log(·) represent the exponential function and logarithmic function, respectively; sim(·) represents the similarity function, which is expressed as the inner product of two vectors.

6. The diagnostic information driven multimodal medical record similarity retrieval system according to claim 1, characterized in that: Construct a tree structure of the diagnosis set and perform similarity measurement as follows: 1) All diagnostic codes are modeled as a tree based on a hierarchical relationship. Each node in the tree represents a disease category or subcategory. Each node in the tree is a diagnosis node, and the semantic relationship between parent and child nodes is represented by edges. 2) For each medical record’s diagnosis code set, map the diagnosis code to the leaf node in the tree, extract all leaf nodes and their ancestral paths, and connect them through the root node to form a minimum spanning supertree t containing all diagnosis nodes as the diagnosis set supertree of the medical record. Each diagnosis node has a corresponding weight w i and its own eigenvector x i ,The feature vector comes from the text encoding of the diagnosis code description corresponding to the diagnosis node, and is encoded using the BioBERT model; 3) Given a diagnostic set hypertree Introducing a learnable mapping function f θ (·) The characteristic vector x of each diagnostic node in Euclidean space i Mapped into the Poincare sphere model in the hyperbolic space, the low-dimensional representation z of the diagnostic node in the hyperbolic space is obtained i , the corresponding mathematical formula is as follows: Where, Represents a projection operation for mapping a vector in Euclidean space to a d-dimensional Poincare sphere model Medium; low-dimensional representation z i Represents the input vector x i Through the function f θ (·) The embedded vector obtained after mapping is located in the Poincare sphere model middle; 4) Based on the embedding vector z of all diagnosis nodes i Compute the weighted hyperbolic centroid of the entire tree And based on the hyperbolic distance from each diagnostic node to the centroid and its weight w i Generate attention weight α i , the mathematical formula involved is as follows; Where exp0(·) represents the mapping from hyperbolic space to tangent space; log0(z i ) represents the mapping from the tangent space back to the Poincare sphere model; cosh -1 (·) represents the hyperbolic inverse cosine function, which is used to calculate the distance in hyperbolic space; ‖·‖ represents the Euclidean norm, that is, the two-norm of the vector; weighted hyperbolic centroid Represents the embedding vector of the center of mass of tree T in the Poincare sphere model, which also satisfies Represents vector z i With vector The Euclidean distance between them; λ represents the balance factor, which controls the relative influence between the distance term and the weight term; ∈ represents a small positive constant, which is used to avoid the zero value problem in the logarithmic function; τ2 represents the temperature coefficient, which is used to adjust the smoothness of the attention score distribution; 5) Using attention weight α i Vector x for diagnostic nodes i Perform weighted averaging on Riemannian geometry and aggregate into a preliminary tree embedding representation z T , its mathematical formula is as follows: 6) Hyperbolic variance and maximum hyperbolic distance are introduced as scatter metrics, which are combined with the preliminary tree embedding to form a diagnostic tree representation vector with discriminative ability. The mathematical formula involved is as follows: Where, Var T Representation of nodes in the Poincare sphere model The hyperbolic variance in is used to measure the vector z of all diagnostic nodes i Relative to vector The degree of dispersion; n represents the number of diagnostic nodes in the tree T; R T represents the maximum hyperbolic distance between all diagnostic nodes and the center of mass, which is used to measure the structural tension; h T Represents the final vector representation of the entire tree; z T ‖Var T ‖R T It means concatenating three vectors to generate a new vector; Indicates that the dimension of the final vector is d+2, where d represents the embedding dimension, plus the variance Var T and the maximum distance R T two scalars; 7) By randomly deleting, replacing, and adding the diagnostic tree T corresponding to the anchor samples a Subtree of the construct similar diagnostic tree as the positive sample T p , select other diagnostic trees as negative samples T n , get the triplet sample, and use the triplet loss function to jointly optimize the mapping function f θ The (·) parameter makes similar diagnostic trees close together in the shared semantic space and different diagnostic trees far apart, thus achieving hierarchical perception and discriminative representation learning of the diagnostic set. The mathematical formula of the triplet loss function is as follows: Where, Represents the triplet loss function, which is used to measure the relative distance between anchor samples, positive samples and negative samples, and optimize the model so that the distance between similar samples is closer and the distance between dissimilar samples is farther; represents the hyperbolic distance defined in the Poincare sphere model, which is used to measure the geometric difference between two embedding vectors; Respectively represent the diagnostic tree T corresponding to the anchor sample a , positive sample T p And negative samples T n Tree representation in the Poincare sphere model; γ represents a marginal hyperparameter that controls the minimum acceptable distance difference between positive and negative samples. This parameter encourages negative samples to be at least γ farther away than positive samples.

7. The diagnostic information driven multimodal medical record similarity retrieval system according to claim 1, characterized in that: The learning process of transferring the similarity knowledge of the diagnosis set to the multimodal semantic representation includes the following steps: 1) The diagnostic tree corresponding to the diagnostic set is mapped through the mapping function f θ (·) is mapped to the Poincare sphere model in hyperbolic space, and the hyperbolic distance between the diagnosis trees is calculated as the similarity measure. Based on the similarity measure results between the diagnosis trees, the positive sample medical records E are screened out. + With negative sample medical records E - Combination, construct the sample combination required for training; 2) The learning process of multimodal semantic representation of medical records is guided by diagnostic information, and the contrastive learning strategy is used to optimize model parameters. The mathematical formula of the loss function for medical record similarity learning is as follows: Where, represents the loss function for medical record similarity learning; e q 、e + 、e - Represents query medical records and positive sample medical records E respectively + And negative sample medical records E - The corresponding multimodal fusion features; τ3 represents the temperature coefficient, which is used to adjust the smoothness of the similarity distribution; N - Indicates the number of negative samples; 3) Enhance the ability of multimodal fusion features to understand the semantics of medical records through the knowledge transfer mechanism, and generate more discriminative medical record similarity representations.

8. The diagnostic information driven multimodal medical record similarity retrieval system according to claim 1, characterized in that: The similarity matching between medical records in the shared semantic space is specifically as follows: The multimodal semantic representation and the medical record similarity representation are jointly encoded and mapped to a unified shared semantic space. In this space, the semantic similarity between medical records is calculated using the cosine similarity distance measurement method. The list of the most similar medical records is returned according to the set threshold or sorting strategy, completing the efficient retrieval task of multimodal medical records.

Images