Medical diagnosis method and system based on multi-modal retrieval enhancement and guide guidance

Abstract

The invention relates to a medical diagnosis method and system based on multi-modal retrieval enhancement and guide guidance. The method comprises the steps that text information including reports and / or electronic health records and medical image information are obtained; encoding the medical image information and the text information by using an image encoder and a text encoder respectively to obtain visual features and text features; respectively utilizing a guide branch decoder and a label branch decoder, taking diagnosis guide features and disease type labels of samples in the training stage as queries of a Transform structure, taking splicing features obtained by splicing text features and visual features as keys and values, and decoding to obtain first prediction probability distribution and second prediction probability distribution of disease types, so as to obtain first prediction probability distribution and second prediction probability distribution of the disease types; and a final disease prediction result is obtained. According to the method, disease specificity knowledge is dynamically retrieved based on a multi-source medical knowledge base, redundancy and noise are removed through a large language model, a standardized and structured diagnosis guide is generated, and explicit guidance of the knowledge is achieved.

Description

Technical Field

[0001] The present invention relates to the fields of multimodal data processing and computer technology, and in particular to a medical diagnosis method and system based on multimodal retrieval enhancement and guideline guidance. Background Art

[0002] In recent years, multimodal learning technologies (such as image-text joint analysis) have made significant progress in the field of medical diagnosis. They improve diagnostic accuracy by integrating multi-source data such as radiological images, pathology reports, and electronic health records (EHRs). However, existing technologies are often unable to efficiently use medical knowledge to guide models for diagnosis. Mainstream methods often rely on pre-trained text encoders to inject broad medical knowledge into them (such as training based on pure text or using medical knowledge graphs), and it is difficult to dynamically introduce fine-grained disease guideline knowledge in specific downstream tasks, resulting in the model being difficult to adapt to the differentiated needs of different clinical scenarios. In addition, existing models lack explicit constraints in the feature extraction and fusion process, and fail to incorporate knowledge from the corresponding guidelines.

[0003] This also leads to poor model interpretability. Current black-box models make it difficult to verify whether their decisions adhere to medical guidelines. In particular, the lack of quantitative evaluation mechanisms for key indicators in text and the accuracy of visual lesion localization limits clinical credibility. Some existing solutions include:

[0004] The paper "Knowledge-enhanced visual-language pre-training on chestradiology images" (KAD) is a medical pre-training method based on knowledge graph enhancement. Its core approach is to train a text encoder using a structured medical entity graph (covering diseases, symptoms, medications, etc.), embedding entity relationships in the knowledge graph into text representations. Although KAD improves the semantic understanding capabilities of the text encoder through pre-training, its knowledge integration remains limited to the pre-training stage and cannot be dynamically adjusted for downstream diagnostic tasks. Furthermore, it does not cover the fine-grained diagnostic criteria in imaging modalities and clinical guidelines, resulting in a deviation between the basis for cross-modal decision-making and the actual clinical pathway.

[0005] The paper "Healnet: Multimodal fusion for heterogeneous biomedical data" (Healnet) discloses a hybrid early fusion architecture for heterogeneous medical data, mainly dealing with image and structured genetic data. The method designs an early fusion decoder architecture, builds a shared latent space to capture cross-modal interaction relationships, and uses a gated attention mechanism to dynamically adjust feature weights. However, the feature fusion process of Healnet is completely data-driven, without introducing clinical guidelines as prior knowledge constraints. The model is easily disturbed by non-specific features (such as general descriptive words in reports), and cannot focus on the key diagnostic indicators recommended by the guidelines through explicit knowledge guidance, resulting in low accuracy of lesion positioning and low degree of conformity with clinical standards.

[0006] In summary, there is currently a lack of a medical diagnosis method for a computer to solve or partially solve the aforementioned problems. SUMMARY

[0007] The purpose of the present application is to provide a medical diagnosis method and system based on multimodal retrieval enhancement and guideline guidance to solve or partially solve the problem of limited knowledge injection method, in order to overcome the defects of the prior art.

[0008] The purpose of the present application can be achieved by the following technical solutions:

[0009] In one aspect of the present application, a medical diagnosis method based on multimodal retrieval enhancement and guideline guidance is provided, applied to an electronic device, the method comprising the following steps:

[0010] Obtaining text information including reports and / or electronic health records, and medical image information;

[0011] Encoding the medical image information using an image encoder to obtain visual features;

[0012] Using a text encoder to encode the text information respectively to obtain text features;

[0013] Using a guideline branch decoder, the diagnosis guideline features of the sample in the training stage are used as the query of the Transformer structure, and the spliced features after splicing the text features and the visual features are used as the key and value of the Transformer structure, to decode to obtain the first prediction probability distribution of the disease type;

[0014] Using a label branch decoder, the disease type label of the sample in the training stage is used as the query, and the spliced features are used as the key and value of the Transformer structure, to decode to obtain the second prediction probability distribution of the disease type;

[0015] Based on the first prediction probability distribution and the second prediction probability distribution, a final disease prediction result is obtained.

[0016] As a preferred technical solution, the training process of the image encoder, text encoder, guide branch decoder and label branch decoder includes the following steps:

[0017] Obtain a training set including multiple samples, where each sample includes text information, medical image information, and a disease type label;

[0018] Based on the disease type labels of the samples, a retrieval model is used to retrieve the most relevant documents from the medical knowledge base. The large language model is used to refine the corresponding diagnostic guidelines for each sample, and a text encoder is used to form a diagnostic guideline feature set.

[0019] Performing negative sample balancing on the diagnostic guideline feature set, calculating the guideline enhancement contrast loss by calculating the similarity between the sample multimodal features and the diagnostic guideline features;

[0020] Calculate the cross entropy loss between the guide branch and the label branch;

[0021] Training is achieved based on the guide enhanced contrast loss and the cross entropy loss.

[0022] As a preferred technical solution, the guide enhancement contrast loss is calculated using the following formula:

[0023]

[0024] in, is the guide enhancement contrast loss, N is the number of training set samples, α is a parameter, ||[·] is the indicator function, the value is 1 when the [·] condition is met, otherwise it is 0, P i represents the positive guide feature set corresponding to sample i, Represents the modal feature I of the i-th sample i With Guide Features S i The loss between modal features I i Can be a text feature T i , visual features V i ,y i,j represents the jth label in the i-th sample, φ(I i ,S i,j ) represents the modal feature I of sample i i Features & Guide Features i , log(·) is the logarithmic function, P is the positive label subset of the original guide feature set, Q is the set after sampling the negative label subset of the original guide feature set, and S is the guide feature set after negative sample balancing.

[0025] As a preferred technical solution, the cross-entropy loss of the guideline branch and the label branch is calculated by the following formula:

[0026]

[0027] Wherein, is the cross-entropy loss of the guideline branch and the label branch, N is the number of training set samples, represents the cross-entropy loss operator, respectively, the first prediction probability distribution and the second prediction probability distribution of the disease type decoded by the guideline branch decoder and the label branch decoder of the i-th sample, y i is the true disease type label of the i-th sample.

[0028] As a preferred technical solution, the medical knowledge base includes a knowledge base, a scientific research literature base, a clinical guideline base and a medical textbook base, wherein the knowledge base includes medical entries of Wikipedia and standardized disease descriptions of authoritative medical knowledge, the scientific research literature base includes research literature related to medicine and biology based on PubMed database, the clinical guideline base includes clinical practice guidelines, and the medical textbook base includes documents of medical electronic textbooks.

[0029] As a preferred technical solution, the retrieval model is a Med-CPT model.

[0030] As a preferred technical solution, the following evaluation process is further included:

[0031] A set of key indicators is extracted from the diagnostic guidelines, the key indicators are matched in the text information, the average attention weight of the model at the corresponding indicator position is calculated, and the proportion of the average value exceeding a fixed threshold is calculated as the guideline recall rate;

[0032] The attention heat map of the model on the medical image information is obtained, the intersection over union of the area with the highest attention value and the lesion area labeled by experts is calculated as the positioning accuracy, and the ability of the model to locate the abnormal lesion in the picture is evaluated;

[0033] Based on the guideline recall rate and the positioning accuracy, the model evaluation is realized.

[0034] As a preferred technical solution, the process of obtaining the final disease prediction result includes the following steps:

[0035] The first prediction probability distribution and the second prediction probability distribution are weighted and averaged, and binarization processing is performed based on a preset threshold to obtain a binarized disease prediction result for each disease type.

[0036] In another aspect of the present application, a medical diagnosis system based on multi-modal retrieval enhancement and guideline guidance is provided for implementing the aforementioned medical diagnosis method based on multi-modal retrieval enhancement and guideline guidance, and the medical diagnosis system comprises:

[0037] a text and image data collection module configured to acquire text information including reports and / or electronic health records, and medical image information;

[0038] a diagnosis guideline storage module configured to, in a training stage, retrieve a plurality of documents with the highest relevance from a medical knowledge base based on disease type labels of samples, refine diagnosis guideline features corresponding to each sample by a large language model, and save the diagnosis guideline features and the disease type labels of the samples in the training stage;

[0039] an encoding module configured to encode the medical image information by an image encoder to obtain visual features, and encode the text information by a text encoder to obtain text features;

[0040] a double-branch decoding module configured to decode, by a guideline branch decoder, the diagnosis guideline features of the samples in the training stage as queries of a Transformer structure, and decode, by a label branch decoder, the disease type labels of the samples in the training stage as queries, and decode the concatenated features obtained by concatenating the text features and the visual features as keys and values of the Transformer structure to obtain a first prediction probability distribution of the disease type and a second prediction probability distribution of the disease type;

[0041] a medical diagnosis module configured to obtain a final disease prediction result based on the first prediction probability distribution and the second prediction probability distribution.

[0042] In another aspect of the present application, a computer-readable storage medium is provided, comprising one or more programs for execution by one or more processors of an electronic device, and the one or more programs comprise instructions for executing the aforementioned medical diagnosis method based on multi-modal retrieval enhancement and guideline guidance.

[0043] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0044] (1) The knowledge injection mode combined with the diagnosis guide is used to realize medical diagnosis: the text information and medical image information are first encoded, and then the diagnosis guide features and disease type labels of the samples in the training stage are used as the queries of the Transformer structure, and the spliced features after the splicing of the text features and the visual features are used as the keys and values of the Transformer structure, so that two prediction probability distributions of the disease types are obtained by decoding, thereby realizing the knowledge injection of the diagnosis guide. The diagnosis guide breaks through the limitation of traditional static pre-training by dynamically retrieving a multi-source medical knowledge base and refining the retrieved knowledge, can adapt to disease-specific guidelines in real time, and significantly improves the diagnosis ability of the model and the generalization ability of the model for different diseases.

[0045] (2) Strong explainability and strong evaluation: the model performance is evaluated by calculating the guide recall rate and the lesion positioning accuracy, thereby constructing a double-axis explainability evaluation system, synchronously evaluating the explainability of the model in the text and visual modalities, realizing the visualization and standardization verification of the diagnosis basis, and improving the credibility of the clinical deployment of the model.

[0046] (3) Realize the retrieval and refinement of knowledge: based on the disease type labels of the samples, the retrieval model is used to retrieve the most relevant documents from the medical knowledge base, the large language model is used for refinement to obtain the diagnosis guide corresponding to each sample, the text encoder is used to form a diagnosis guide feature set, the disease-specific knowledge is dynamically retrieved based on the multi-source medical knowledge base (guidelines, literature, textbooks, etc.), and the large language model (LLM) is used to remove redundancy and noise, to generate standardized and structured diagnosis guides, and realize the explicit guidance of knowledge.

[0047] (4) Ensure the consistency of the diagnosis result and the standard: the guide supervised contrast learning loss is designed, the distance between the multi-modal features and the hidden space of the guide knowledge is explicitly constrained, the alignment of the image and the text features to the guide is strengthened, and the consistency of the diagnosis result and the authoritative standard is improved. The double-path decoding architecture is adopted, and the guide knowledge and the disease name are used to guide the fusion of the visual and text features, and the double paths are complementary to realize strong performance. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The flowchart of the multi-modal retrieval enhanced and guide-based medical diagnosis method in the embodiment;

[0049] Figure 2 The schematic diagram of the retrieval enhanced guide multi-modal diagnosis method framework in the embodiment;

[0050] Figure 3 The schematic diagram of the multi-modal retrieval enhanced and guide-based medical diagnosis system in the embodiment;

[0051] Figure 4 A schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work should fall within the protection scope of the present application.

[0053] The terms “first”, “second” are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first”, “second” can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of “multiple” is two or more, unless otherwise specifically limited.

[0054] Embodiment 1

[0055] The present embodiment provides a medical diagnosis method based on multi-modal retrieval enhancement and guideline guidance, mainly including the following points:

[0056] (1) In view of the problem that the static knowledge injection of the prior art depends on the static knowledge integration in the pre-training stage, it is difficult to dynamically adapt to the demand of medical guideline update or various downstream tasks, and it is unable to cope with the adaptation scene of new diseases, a mechanism of dynamically retrieving knowledge according to specific tasks is proposed, and through real-time calling of a multi-source medical knowledge base, flexible adaptation of disease-specific knowledge is realized, breaking through the limitations of static pre-training.

[0057] (2) In view of the problem that the traditional method does not explicitly introduce medical knowledge to guide model training on the downstream task, resulting in disconnection between model decision-making process and diagnosis standard, and the output result lacks traceability and clinical consistency, the present embodiment designs a guideline enhancement contrast loss and a double-axis diagnosis decoder, directly guides the model feature learning and multi-modal fusion two processes with the retrieved guideline knowledge, and ensures that the model feature extraction and decision-making are strictly aligned with the guideline.

[0058] (3) In view of the problem that the black box characteristics of the neural network model make the existing method unable to provide the correlation between its decision and the medical guideline, and lack a systematic quantitative evaluation system, resulting in insufficient trust of the clinical doctors on the AI decision, the present embodiment constructs a double-modal explainability evaluation system, verifies the matching degree of the model in the image and text two modalities with the clinical guideline through quantitative indexes, and provides a verifiable evidence chain for credible diagnosis.

[0059] Reference Figure 1 and Figure 2, the method comprises the following steps:

[0060] Step S1: Acquire text information including reports and / or electronic health records, and medical imaging information.

[0061] Medical imaging information x can be CT, MRI, and other imaging information, and text information t can be the corresponding report and electronic health record (EHR). For example, imaging information includes fundus, brain, dermatoscope, and chest X-ray.

[0062] Step S2: Encode the medical image information using an image encoder to obtain visual features.

[0063] Using the image encoder Φ img (·) Process the image information x and output visual features V.

[0064] Step S3: Encode the text information using a text encoder to obtain text features.

[0065] Using the text encoder Φ text (·) Process report and EHR data t and output text features T.

[0066] It should be noted that the order of step S2 and step S3 can be interchanged.

[0067] In step S4, a guide branch decoder is used to take the diagnostic guide features of the sample in the training phase as the query of the Transformer structure, and the concatenated features of the text features and the visual features as the key and value of the Transformer structure, and decode to obtain the first predicted probability distribution of the disease type.

[0068] The label decoder and the guide decoder constitute a dual-path diagnostic decoder, which further transforms the obtained features into the final predicted probability distribution, which includes a guide branch decoder and a label branch decoder. Use the guide features of the samples extracted during the training phase as the query in the Transformer structure, and use the concatenated text features and visual features to obtain multimodal features. As Key and Value, the obtained predicted probability distribution is used as the first predicted probability distribution.

[0069] In step S5, a label branch decoder is used to decode the disease type label of the sample in the training phase as a query and the concatenated features as the key and value of the Transformer structure to obtain a second predicted probability distribution of the disease type.

[0070] Another label branch decoder The disease name (i.e., disease type label) embedding of the training stage sample is used as a query, and the concatenated multi-modal feature is also used as a key and a value, and the obtained prediction probability distribution is used as a second prediction probability distribution.

[0071] It should be noted that the order of step S4 and step S5 can be interchanged.

[0072] Step S6, based on the first prediction probability distribution and the second prediction probability distribution, obtains the final disease prediction result.

[0073] In this embodiment, the prediction probability distributions of the two branches of step S4 and step S5 are weighted and averaged, and then binarized according to a fixed threshold. The probability corresponding to each disease is less than the threshold, which is 0 (healthy), and greater than the threshold, which is 1 (sick).

[0074] It should be noted that the diagnosis (classification) task of step S6 can be replaced by other multi-modal tasks, such as segmentation.

[0075] Before formally implementing steps S1-S6 of the method, the following training phase is also included, which aims to train the text encoder, image encoder, guideline branch decoder, and label branch decoder:

[0076] Step S01, obtaining a training set containing N samples Where x i represents a medical image, t i is the corresponding report and electronic health record (EHR), y i is a one-hot multi-label vector (as the disease type label of the sample in step S5), corresponding to m diseases. The goal of training is to enable the model to predict the disease label y i based on multi-modal data.

[0077] Step S02, construct a multi-source medical knowledge base, including but not limited to: a knowledge base, integrating medical articles from Wikipedia and standardized disease descriptions from authoritative medical encyclopedias. A research literature database based on PubMed database to obtain medical and biological research literature in the past five years. A clinical guideline database containing more than 40,000 clinical practice guidelines from multiple authoritative agencies. A medical textbook database covering more than 18,000 medical electronic textbooks.

[0078] Step S03, based on the given sample set, input the disease label to the double-tower medical retrieval model Med-CPT to retrieve the corresponding knowledge, and return the top-k most relevant documents from the above 4 sources (usually set k = 10). Next, use a large language model (LLM) to refine the retrieved documents to compress the content and reduce redundancy:

[0079] g i= LLM{(Prompt, c i,1 ,…, c i,k )}, (1)

[0080] where c i is the k documents retrieved in the last step. Finally, we get the standardized diagnosis guideline g i for each disease, where Qwen2.5-72B is chosen as the LLM.

[0081] It should be noted that the knowledge retrieval model Med-CPT can be replaced by any cross-modal retrieval model of the same type, and the large language model Qwen2.5-72B can also be replaced by other similar large language models.

[0082] Step S04, using the image encoder Φ img (·) to process the radiological image x i , output the visual feature V i , the text encoder Φ text (·) to process the report and EHR data t i , output the text feature T i , and the same text encoder to process the guideline g, output the guideline feature G (as the diagnosis guideline feature of the sample in step S4), the process is as follows:

[0083]

[0084] where hw is the height and width of the image respectively, d is the standard dimension of the model, l is the maximum input length of the text encoder, and m is the number of disease categories.

[0085] After pooling the features, the designed guideline enhanced contrastive loss is calculated. Considering the negative impact of excessive negative samples on the training effect in contrastive learning, G is divided into a positive label subset P and a negative label subset N according to the value of the sample label y i , and a subset Q is sampled from N, and the guideline set S = P∪Q used for calculating the loss is finally obtained, and the loss function is:

[0086]

[0087] where log(·) is the logarithmic function, φ(·) is the similarity between the sample feature and the guideline feature, and ||[·] is the indicator function.

[0088] It should be noted that the diagnosis guideline feature of the sample in step S4 is the full output guideline feature G rather than the guideline set S after negative sample balancing, to avoid data leakage.

[0089] Step S05, the two-way diagnostic decoder further converts the obtained features into the final predicted probability distribution, which includes the guideline branch decoder and the label branch decoder. The guideline branch decoder Using the extracted guideline features as Query in the Transformer structure, using the concatenated multi-modal features As Key and Value, the obtained predicted probability distribution is:

[0090]

[0091] Another branch of the label branch decoder Using the disease name embedding as Query, using the concatenated multi-modal features as Key and Value, the obtained predicted probability distribution is:

[0092]

[0093] The final training total loss function contains the cross-entropy loss of the two branches (calculated with the predicted probability distribution and the real label) and the guideline enhanced contrastive loss:

[0094]

[0095] Wherein represents the cross-entropy loss, and β is the weight coefficient between the two losses.

[0096] Step S06, training effect evaluation.

[0097] This step provides a multi-modal explainability evaluation system, including the text index - guideline recall and the visual index - Intersection over Union (IoU).

[0098] The guideline recall quantifies the degree of compliance with the diagnostic guideline, and the calculation process is as follows: a set of key indicators is extracted from the guideline, these key indicators are matched in the text, the average attention weight of the model at the corresponding position is calculated, the proportion of the average value exceeding a fixed threshold θ is calculated, and the guideline recall is obtained.

[0099] The Intersection over Union (IoU) first obtains the model attention heat map on the image data, calculates the region with the highest attention value with the lesion region annotated by experts, and evaluates the ability of the model to locate the abnormal lesion in the picture.

[0100] To verify the effectiveness of the method, experiments were conducted on four multi-modal disease diagnosis data sets, covering four different parts of the human body, including the chest, brain, eye and skin. The proposed method outperforms a variety of multi-modal fusion methods and medical knowledge injection methods in terms of performance, significantly surpassing existing methods in various indicators. In addition, from the perspective of explainability, the trained model can focus on important indicators in the text data and lesion locations in the image data according to the diagnostic guidelines, demonstrating stronger explainability and credibility.

[0101] Embodiment 2

[0102] Based on embodiment 1, referring to Figure 3 The embodiment provides a medical diagnosis system based on multi-modal retrieval enhancement and guideline guidance, which is used to implement the medical diagnosis method based on multi-modal retrieval enhancement and guideline guidance of embodiment 1. The medical diagnosis system comprises:

[0103] (1) A text and image data acquisition module is used to acquire text information including reports and / or electronic health records, and medical image information.

[0104] (2) A diagnostic guideline storage module is used to retrieve one or more documents with the highest relevance from a medical knowledge base based on the disease type label of the sample during the training phase, refine the diagnostic guideline features corresponding to each sample through a large language model, and save the diagnostic guideline features and disease type labels of the samples in the training phase.

[0105] (3) An encoding module is used to encode the medical image information using an image encoder to obtain visual features, and encode the text information using a text encoder to obtain text features.

[0106] (4) A double-branch decoding module is used to use the diagnostic guideline features of the sample in the training phase as the query of the Transformer structure, and use the spliced features after splicing the text features and the visual features as the key and value of the Transformer structure, to decode to obtain the first prediction probability distribution of the disease type, and use the label branch decoder to use the disease type label of the sample in the training phase as the query, and use the spliced features as the key and value of the Transformer structure, to decode to obtain the second prediction probability distribution of the disease type.

[0107] (5) A medical diagnosis module is used to obtain the final disease prediction result based on the first prediction probability distribution and the second prediction probability distribution.

[0108] The present application has the following advantages:

[0109] (1)Knowledge retrieval and refinement. Based on dynamic retrieval of disease-specific knowledge from multi-source medical knowledge bases (guidelines, literature, textbooks, etc.), and through a large language model (LLM) to remove redundancy and noise, a standardized and structured diagnostic guideline is generated, realizing the explicit guidance of knowledge. By dynamically retrieving multi-source medical knowledge bases and refining the retrieved knowledge, the limitations of traditional static pre-training are overcome, enabling real-time adaptation of disease-specific guidelines and significantly improving the model's diagnostic ability and generalization ability for different diseases. The invention designs a comprehensive framework for using external medical knowledge to guide multi-modal disease diagnosis, fully guiding feature learning and modality fusion of the model, and the framework can be flexibly integrated into other systems.

[0110] (2)Guideline-enhanced contrast loss. A guideline-guided supervised contrast learning loss is designed to explicitly constrain the distance between multi-modal features and the hidden space of guideline knowledge, strengthen the alignment of image and text features to the guideline, and improve the consistency of the diagnostic result with the authoritative standard.

[0111] (3)Dual-axis diagnostic decoder. A dual-path decoding architecture is used to guide the fusion of visual and text features with guideline knowledge and disease names, respectively, and the dual paths complement each other to achieve strong performance. By constructing a dual-axis explainability evaluation system, the explainability of the model in both text and visual modalities is evaluated simultaneously, enabling visualization and standardized verification of diagnostic basis and improving the credibility of the model's clinical deployment.

[0112] Embodiment 3

[0113] The embodiment provides an electronic device, comprising: one or more processors and a memory, the memory having one or more programs stored therein, the one or more programs comprising instructions for performing the multi-modal retrieval enhanced and guideline guided medical diagnosis as described in embodiment 1.

[0114] As Figure 4 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course, other hardware required for business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the above Figure 1 described method. Of course, in addition to the software implementation, the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0115] The memory can include non-persistent memory in a computer readable medium, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer readable medium.

[0116] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0117] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A medical diagnosis method based on multimodal retrieval enhancement and guideline guidance, characterized in that: Applied to electronic equipment, the method comprises the following steps: Access to text information including reports and / or electronic health records, as well as medical imaging information; Encoding the medical image information using an image encoder to obtain visual features; Using a text encoder, respectively encode the text information to obtain text features; Using a guideline branch decoder, the diagnostic guideline features of the sample in the training phase are used as queries of the Transformer structure, and the concatenated features of the text features and the visual features are used as keys and values ​​of the Transformer structure to decode and obtain a first predicted probability distribution of the disease type; Using a label branch decoder, the disease type label of the sample in the training phase is used as a query, and the spliced ​​features are used as the key and value of the Transformer structure to decode and obtain a second predicted probability distribution of the disease type; Based on the first prediction probability distribution and the second prediction probability distribution, a final disease prediction result is obtained.

2. A medical diagnosis method based on multimodal retrieval enhancement and guideline guidance according to claim 1, characterized in that: The training process of the image encoder, text encoder, guide branch decoder and label branch decoder includes the following steps: Obtain a training set including multiple samples, where each sample includes text information, medical image information, and a disease type label; Based on the disease type labels of the samples, a retrieval model is used to retrieve the most relevant documents from the medical knowledge base. The large language model is used to refine the corresponding diagnostic guidelines for each sample, and a text encoder is used to form a diagnostic guideline feature set. Performing negative sample balancing on the diagnostic guideline feature set, calculating the guideline enhancement contrast loss by calculating the similarity between the sample multimodal features and the diagnostic guideline features; Calculate the cross entropy loss between the guide branch and the label branch; Training is achieved based on the guide enhanced contrast loss and the cross entropy loss.

3. A medical diagnosis method based on multimodal retrieval enhancement and guideline guidance according to claim 2, characterized in that: The guideline enhancement contrast loss is calculated using the following formula: in, is the guide enhancement contrast loss, N is the number of training set samples, α is a parameter, ||[·] is the indicator function, the value is 1 when the [·] condition is met, otherwise it is 0, P i represents the positive guide feature set corresponding to sample i, Represents the modal feature I of the i-th sample i With Guide Features S i The loss between modal features I i Can be a text feature T i , visual features V i ,y i,j represents the jth label in the i-th sample, φ(I i ,S i,j ) represents the modal feature I of sample i i Features & Guide Features i , log(·) is the logarithmic function, P is the positive label subset of the original guide feature set, Q is the set after sampling the negative label subset of the original guide feature set, and S is the guide feature set after negative sample balancing.

4. A medical diagnosis method based on multimodal retrieval enhancement and guideline guidance according to claim 2, characterized in that: The cross entropy loss of the guide branch and the label branch is calculated using the following formula: in, is the cross entropy loss between the guide branch and the label branch, N is the number of training set samples, represents the cross entropy loss operator, The guide branch decoder and label branch decoder of the i-th sample respectively decode the first predicted probability distribution and the second predicted probability distribution of the disease type, y i is the true disease type label of the i-th sample.

5. The medical diagnosis method based on multimodal retrieval enhancement and guideline guidance according to claim 2, characterized in that: The medical knowledge base includes an encyclopedia knowledge base, a scientific research literature base, a clinical guideline base and a medical textbook base, wherein the encyclopedia knowledge base includes medical entries from Wikipedia and standardized disease descriptions from authoritative medical encyclopedias, the scientific research literature base includes medical and biology-related research literature obtained based on the PubMed database, the clinical guideline base includes clinical practice guidelines, and the medical textbook base includes documents of medical electronic textbooks.

6. The medical diagnosis method based on multimodal retrieval enhancement and guideline guidance according to claim 2, characterized in that: The retrieval model is the Med-CPT model.

7. The medical diagnosis method based on multimodal retrieval enhancement and guideline guidance according to claim 1, characterized in that: The assessment process also includes the following: Extract a set of key indicators from the diagnostic guideline, match the key indicators in the text information, calculate the average of the model's attention weights at the corresponding indicator positions, and calculate the proportion of the average exceeding a fixed threshold as the guideline recall rate; Obtain a heatmap of the model's attention on the medical image information and calculate the intersection-over-union ratio of the area with the highest attention value and the lesion area annotated by the expert as the positioning accuracy to evaluate the model's ability to locate abnormal lesions in the image. Model evaluation is performed based on the guide recall rate and the positioning accuracy.

8. The medical diagnosis method based on multimodal retrieval enhancement and guideline guidance according to claim 1, characterized in that: The process of obtaining the final disease prediction result includes the following steps: A weighted average is performed on the first prediction probability distribution and the second prediction probability distribution, and binarization is performed based on a preset threshold to obtain a binary disease prediction result for each disease type.

9. A medical diagnosis system based on multimodal retrieval enhancement and guideline guidance, characterized in that: For implementing the medical diagnosis method based on multimodal retrieval enhancement and guideline guidance as described in any one of claims 1 to 8, the medical diagnosis system comprises: a text and image data acquisition module for acquiring text information including reports and / or electronic health records, as well as medical imaging information; The diagnostic guideline storage module is used to retrieve the most relevant documents from the medical knowledge base based on the disease type labels of the samples during the training phase using a retrieval model. The module then uses a large language model to refine the diagnostic guideline features corresponding to each sample and saves the diagnostic guideline features and disease type labels of the samples during the training phase. an encoding module, configured to encode the medical image information using an image encoder to obtain visual features, and to encode the text information using a text encoder to obtain text features; a dual-branch decoding module, configured to utilize a guide branch decoder to use the diagnostic guideline features of the samples in the training phase as queries for the Transformer structure, and to use a concatenated feature obtained by concatenating the text features and the visual features as keys and values ​​of the Transformer structure, to decode and obtain a first predicted probability distribution of the disease type; and utilize a label branch decoder to use the disease type labels of the samples in the training phase as queries, and to use the concatenated features as keys and values ​​of the Transformer structure, to decode and obtain a second predicted probability distribution of the disease type; The medical diagnosis module is used to obtain a final disease prediction result based on the first prediction probability distribution and the second prediction probability distribution.

10. A computer-readable storage medium, characterized in that The invention comprises one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the medical diagnosis method based on multimodal retrieval enhancement and guideline guidance as described in any one of claims 1-8.

Images