Multi-mode credible dialogue type retrieval enhancement generation system for medical diagnosis

Abstract

The invention discloses a medical diagnosis-oriented multi-modal credible dialogue type retrieval enhancement generation system, and relates to the field of artificial intelligence, in the system, a multi-modal image analysis module generates a structured image feature data packet based on a multi-modal large model according to an original medical image; the knowledge database construction module is used for constructing a multi-center collaborative medical knowledge database; the image knowledge retrieval module is used for obtaining image feature associated knowledge; the clinical knowledge retrieval module is used for obtaining a candidate knowledge set; a multi-dimensional weighting reordering module obtains a knowledge list; the medical inquiry generation module is used for generating structured preliminary diagnosis based on the user questions, the image abstract and the knowledge list; the credibility verification module is used for verifying the structured preliminary diagnosis to obtain a final diagnosis report; according to the method, the medical knowledge retrieval precision can be improved, the image information fusion capability is enhanced, and high-credibility and traceable intelligent diagnosis and treatment auxiliary service oriented to doctor-patient scenes is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a multimodal, trustworthy, conversational retrieval enhancement generation system for medical diagnosis. Background Technology

[0002] With the widespread application of Large Language Models (LLM) in the field of artificial intelligence, they have been successfully implemented in various scenarios such as finance and education. Retrieval Augmentation (RAG) systems have become a typical solution for leveraging the core capabilities of LLM models. RAG systems, by cleverly combining a retrieval module, a generative model, and a fusion mechanism, significantly improve the richness and accuracy of information responses, demonstrating strong practical potential. RAG systems utilize the retrieval module to quickly locate relevant information from massive amounts of data, then leverage the semantic integration and logical reconstruction of the generative model, and finally integrate this information into a logical and targeted response through a fusion mechanism, providing answers to users.

[0003] In recent years, in medical settings, the RAG system, by using a retrieval module, generative model, and fusion mechanism in synergy, enables LLM models to generate richer and more accurate answers in conjunction with external knowledge bases. This has become an important technical approach to improving the knowledge utilization rate of LLM models. The RAG system quickly locates external medical literature or guidelines through its retrieval module, performs semantic reconstruction using its generative model, and then integrates them through its fusion mechanism to create logical and targeted answers, providing users with high-quality information services.

[0004] However, general-purpose LLM models are not designed for medical scenarios and have the following inherent shortcomings: (1) Medical terminology is abstract and complex, and the domain language system is highly specialized, making it difficult for general-purpose models to understand accurately; (2) Medical knowledge is huge and has a short update cycle, making it difficult for the limited context window of LLM models to fully contain it; (3) The generation process of LLM models is "black box" and is prone to hallucinatory content, making it impossible for users to verify the credibility of their diagnosis and advice sources; (4) LLM models mainly process text and do not adequately support multimodal information such as images, test reports, and structured scales that play an important role in clinical practice. In particular, their generalization ability is weak in the interpretation of medical images, making it difficult to reliably analyze image features.

[0005] Therefore, even though there are already modified models for the medical field, such as Sunsimiao, QiZhenGPT, BianQue, and PULSE, these models are mostly based on general LLM models that have been retrained. Although they have a stronger grasp of medical knowledge, they still cannot solve the core challenges such as lack of image understanding, uncontrollable reasoning chains, and lack of verifiable evidence.

[0006] In practical clinical applications, these shortcomings pose significant risks, especially in disease scenarios requiring comprehensive judgment based on imaging, laboratory indicators, past medical history, and lifestyle data. Take functional hypothalamic amenorrhea (FHA) as an example. This disease is caused by a decrease or deficiency of hypothalamic gonadotropin-releasing hormone (GnRH), leading to decreased FSH and LH levels, and is one of the most common causes of secondary amenorrhea in young women. Its etiology is complex (weight loss, excessive exercise, psychological stress, etc.), requiring not only written medical history but also a high degree of reliance on multimodal information such as ultrasound imaging, biochemical indicators, and bone mineral density reports for comprehensive analysis. Furthermore, even after removing the triggering factors, most patients still require a long time for menstrual recovery, making adherence management and medication education crucial. However, existing consultation-based LLM models struggle to provide comprehensive judgment based on imaging, laboratory, and behavioral data, and cannot reliably verify the source of the generated content, limiting their application value in the management of diseases such as FHA.

[0007] In summary, existing medical LLM models and RAG systems still have significant shortcomings in medical image understanding, knowledge traceability, multimodal fusion, and security verification, making it difficult to meet the stringent requirements of accuracy, interpretability, and verifiability in medical scenarios. Therefore, there is an urgent need to construct a novel RAG system that integrates multimodal medical image interpretation with a multi-level trusted verification mechanism to improve the accuracy of medical knowledge retrieval, enhance the integration of image information, and achieve highly reliable, traceable, and intelligent diagnostic and treatment assistance services for doctor-patient scenarios. Summary of the Invention

[0008] The purpose of this application is to provide a multimodal, trustworthy, conversational retrieval enhancement and generation system for medical diagnosis, which can improve the accuracy of medical knowledge retrieval, enhance the ability to integrate image information, and realize highly trustworthy, traceable intelligent diagnostic and treatment assistance services for doctor-patient scenarios.

[0009] To achieve the above objectives, this application provides the following solution: This application provides a multimodal trusted conversational retrieval enhancement generation system for medical diagnosis, the multimodal trusted conversational retrieval enhancement generation system for medical diagnosis includes: The multimodal image parsing module is used to generate structured image feature data packages based on a multimodal large model from the original medical images; the structured image feature data packages include: organ morphology, lesion location, echo characteristics, boundary characteristics, and size; A knowledge database construction module is used to build a multi-center collaborative medical knowledge database; the multi-center collaborative medical knowledge database includes: an image feature knowledge base, a clinical guideline knowledge base, and a similar case diagnosis database. The image knowledge retrieval module is used to perform vector encoding on the structured image feature data package and retrieve image feature association knowledge from the image feature knowledge base based on a hybrid retrieval strategy; the hybrid retrieval strategy is a combination of dense vector index retrieval and sparse vector index retrieval. The clinical knowledge retrieval module is used to generate image summaries based on image feature association knowledge and original medical images using a multimodal large model; based on user questions and image summaries, it performs a dual-database search in the clinical guideline knowledge base and the similar case diagnosis database to obtain a candidate knowledge set; the candidate knowledge set includes: standardized diagnosis and treatment criteria and case data; The multidimensional weighted sorting module is used to weight and sort the candidate knowledge set according to image feature matching weight, guideline authority weight, and case similarity weight to obtain a knowledge list; The medical consultation generation module is used to generate a structured preliminary diagnosis based on the user's question, the image summary, and the knowledge list; the structured preliminary diagnosis includes: diagnosis results, diagnostic basis, recommended examinations, preliminary treatment plan, and risk warnings; The credibility verification module is used to perform source tracing and annotation, credibility scoring and logical link decomposition on the structured preliminary diagnosis to obtain the final diagnosis report; the final diagnosis report includes: diagnosis conclusion, source of evidence, credibility score and reasoning chain.

[0010] Optionally, the execution process of the multimodal image analysis module specifically includes: Based on the examination site, imaging type, and image table structure, the original medical images are classified using a preset logical routing decision tree; Determine the corresponding image description prompt template based on the image classification results; The original medical images are analyzed based on the image description prompt template and the multimodal large model to obtain a structured image feature data package.

[0011] Optionally, the construction process of the knowledge database construction module specifically includes: Acquire unstructured case data from multiple medical centers; Data cleaning was performed on unstructured case data from multiple medical centers; Based on the cleaned unstructured case data, a named entity recognition model was used to identify medical entities; and the medical entities were standardized and coded. The standardized data is segmented and sliced ​​according to semantics, and then converted into embedded vectors and stored in the vector retrieval engine.

[0012] Optionally, the execution process of the image knowledge retrieval module specifically includes: Based on generalized linear model embedding and medical corpus of hypothalamic amenorrhea, a medical word embedding model is obtained; The structured image feature data packet is vector-encoded using a medical term embedding model; Based on the vector-encoded data packets, image feature association knowledge is retrieved from the image feature knowledge base using a hybrid retrieval strategy.

[0013] Optionally, the dense vector index is dense_clinical_semantic_index; the sparse vector index is sparse_medical_lexical_index.

[0014] Optionally, the clinical knowledge retrieval module may also include the following before execution: The query is rewritten and decomposed based on the user's query to obtain multiple sub-queries covering different etiological chains.

[0015] This application provides a multimodal, trustworthy, conversational retrieval enhancement and generation system for medical diagnosis. By constructing a multi-center collaborative medical knowledge database, it achieves standardized integration and efficient reuse of multi-center case data, significantly improving data usability and semantic consistency. Through vector encoding of the structured image feature data package and retrieval in the image feature knowledge base based on a hybrid retrieval strategy, it obtains image feature-related knowledge, achieving standardized, multi-level structured interpretation of medical images and improving the accuracy and standardization of image information extraction. Based on image feature-related knowledge and original medical images, it generates image summaries using a multimodal large model. Based on user questions and image summaries, it performs a dual-database search in the clinical guideline knowledge base and the similar case diagnosis database to obtain a candidate knowledge set, achieving a "secondary enhancement" of image description, making the image information summary more complete, standardized, and closer to clinical diagnostic requirements. By weighting and ranking the candidate knowledge set according to image feature matching weights, guideline authority weights, and case similarity weights, it significantly improves the accuracy and relevance of medical knowledge retrieval. Finally, it enhances the interpretability and security of diagnostic results through "authoritative knowledge constraints + trustworthiness scoring + logical chain decomposition." Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1This is a schematic diagram of the structure of a multimodal trusted conversational retrieval enhancement generation system for medical diagnosis, as described in one embodiment of this application. Figure 2 A schematic diagram of the standardized coding process for medical entities. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] In one exemplary embodiment, such as Figure 1 As shown, a multimodal, trustworthy, conversational retrieval enhancement generation system for medical diagnosis is provided. It is suitable for medical scenarios such as hypothalamic amenorrhea (FHA) that require joint image-text judgment. It achieves highly reliable medical diagnostic assistance through multi-level feature extraction, hybrid retrieval, generative reasoning, and trust verification. The system includes: a multimodal image parsing module 100, a knowledge database construction module, an image knowledge retrieval module 200, a clinical knowledge retrieval module 300, a multidimensional weighted ranking module 400, a medical consultation generation module 500, and a trust verification module 600. The multimodal image parsing module 100 is used to generate a structured image feature data package based on a multimodal large model from the original medical images; the structured image feature data package includes, but is not limited to: organ morphology, lesion location, echo features, boundary features, and size; The execution process of the multimodal image analysis module 100 specifically includes: S11, Based on the examination site, imaging type (such as B-ultrasound, MRI) and image table structure, the original medical images are classified using a preset logical routing decision tree; S12, determine the corresponding image description prompt template based on the image classification results to avoid the problem that a single template cannot be adapted to different examination types; S13, based on the image description prompt template and the multimodal large model, the original medical image is parsed to obtain a structured image feature data packet T_img.

[0021] The knowledge database construction module is used to construct a multi-center collaborative medical knowledge database; the multi-center collaborative medical knowledge database includes: an image feature knowledge base 101, a clinical guideline knowledge base 102, and a similar case diagnosis database 103; The construction process of the knowledge database construction module specifically includes: S21, acquire unstructured case data from multiple medical centers; The unstructured medical record data includes laboratory reports from laboratory examination reports; outpatient and inpatient medical records, progress notes; medical images and reports such as CT / MRI / ultrasound; follow-up / re-examination records of reasons for adjusting patient medication, efficacy evaluation, description of adverse reactions, reasons for discontinuing medication, etc. S22, Data cleaning of unstructured case data from multiple medical centers; Data cleaning includes: format correction, missing data completion, and bias correction; S23, Based on the cleaned unstructured case data, a named entity recognition model is used to identify medical entities; and the medical entities are standardized and coded, as follows: Figure 2 As shown, the database is uniformly coded according to the "Technical Specifications for Real-World Research in Traditional Chinese Medicine Database Construction and Data Preprocessing" (2021); medical entities include TCM and Western medicine terms, symptoms, diseases, and drug information. Specifically, named entity recognition models include: medical diagnostic standardization models, traditional Chinese medicine diagnostic standardization models, and drug information standardization models. For example, in an electronic medical record stating "Aspirin enteric-coated tablets 100 mg qd orally, Bayer aspirin has been discontinued. Metformin extended-release tablets 0.5 g bid orally are added," the drug information standardization model identifies three drug entities and then normalizes them to pharmacopoeia standard entries (standard generic name in the pharmacopoeia + pharmacopoeia code). S24. The standardized data is segmented and sliced ​​according to semantics and converted into embedded vectors for storage in the vector retrieval engine.

[0022] The image knowledge retrieval module 200 is used to perform vector encoding on the structured image feature data package and retrieve image feature association knowledge from the image feature knowledge base 101 based on a hybrid retrieval strategy. The hybrid retrieval strategy combines dense vector index retrieval and sparse vector index retrieval, which ensures both concept expansion capability and semantic retrieval accuracy. The dense vector index is called dense_clinical_semantic_index; the sparse vector index is called sparse_medical_lexical_index.

[0023] The execution process of the image knowledge retrieval module 200 specifically includes: S31. Based on the generalized linear model embedding (GLM Embedding 3) and the medical corpus of hypothalamic amenorrhea, a medical word embedding model is obtained. S32, The structured image feature data packet is vector-encoded using a medical term embedding model; S33, based on the vector-encoded data packet, retrieve the image feature association knowledge K_img from the image feature knowledge base using a hybrid retrieval strategy.

[0024] The clinical knowledge retrieval module 300 is used to generate image summaries based on image feature association knowledge and original medical images using a multimodal large model; based on the user question Q_user and the image summary, it performs a dual-database search in the clinical guideline knowledge base 102 and the similar case diagnosis database 103 to obtain a candidate knowledge set; the candidate knowledge set includes: standardized diagnosis and treatment criteria and case data; This process is a secondary image parsing based on image feature knowledge. The original medical image and K_img are input into the multimodal LLM. Secondary reasoning is performed through the "image content - feature knowledge" dual-anchored prompt template to generate an accurate image summary A_img for subsequent clinical knowledge retrieval. Before the clinical knowledge retrieval module 300 is executed, it also includes: The query Q_user is rewritten and decomposed to obtain multiple sub-queries covering different etiological chains. For example, the etiologies of FHA include: malnutrition, excessive exercise, and mental stress. The execution process of the clinical knowledge retrieval module 300 specifically includes: Search for standardized diagnostic and treatment guidelines (G_clinical) in the Clinical Guidelines Knowledge Base 102. Retrieve case data C_case from the similar case diagnosis database 103; The candidate knowledge set C_all = {G_clinical, C_case} is obtained; The multidimensional weighted sorting module 400 is used to perform weighted sorting on the candidate knowledge set according to the image feature matching weight W_img, the guide authority weight W_guide, and the case similarity weight W_case to obtain a knowledge list; The process for determining the above weights is as follows: First, a reasonable range should be determined. Clinical guidelines are the most authoritative in medical Q&A and therefore have the highest basic weight. At the same time, the proportion of similar case diagnosis is greater than that of imaging feature knowledge to prevent the system from overriding the conclusions of guidelines in cases with uncertain images or noisy images.

[0025] Secondly, a validation set was prepared to fine-tune the three sets of weights to obtain the optimal values. Objectives such as nDCG@5 / nDCG@10 (ranking quality), Recall@K (whether the key evidence can enter the Top-K) and MRR (how quickly the first key evidence appears) were set to test the RRF re-ranking effect of each validation set. Finally, select the weighted combination that best combines the overall indicators.

[0026] Weighted score Reorder the knowledge list to obtain the knowledge list R_sorted; The medical consultation generation module 500 is used to generate a structured preliminary diagnosis based on the user's question, the image summary, and the knowledge list; the structured preliminary diagnosis includes: diagnosis results, diagnostic basis, recommended examinations, preliminary treatment plan, and risk warnings; Specifically, based on the user question Q_user, the image summary A_img, and the knowledge list R_sorted, a structured preliminary diagnosis D_raw is generated based on the preset consultation prompt template in the consultation LLM; The credibility verification module 600 is used to perform source tracing and annotation, credibility scoring and logical link decomposition on the structured preliminary diagnosis to obtain the final diagnosis report D_final; the final diagnosis report includes: diagnosis conclusion, source of evidence, credibility score and reasoning chain.

[0027] Source tracing and annotation involves labeling each sentence in the diagnostic conclusion with the retrieved knowledge fragment from which it originates. The credibility scoring model calculates a credibility score S_conf based on the knowledge source type, retrieval matching degree, and semantic consistency. Logical Link Decomposition (CoT) outputs a decision link consisting of "image features → clinical evidence → diagnostic reasoning," improving transparency.

[0028] This application achieves multi-level performance improvements that are difficult to achieve in existing medical question-answering systems by integrating multimodal image analysis, structured medical knowledge construction, hybrid retrieval enhancement, and a trusted verification mechanism. These improvements are mainly reflected in the following five aspects: (1) Achieve standardized integration and efficient reuse of multi-center case data, significantly improving data availability and semantic consistency: By performing format correction, content validation, and missing data completion on unstructured case data from different medical centers, this application first addresses the "data unavailability" problem that affects retrieval and model training.

[0029] After named entity recognition and structuring, texts that originally had terminological differences were uniformly mapped to the standard medical vocabulary system, improving the semantic consistency of cross-center data.

[0030] Compared with existing technologies, the multi-center collaborative medical knowledge database established in this application can achieve unified management and reusability of case data across departments and institutions, and significantly improve the completeness and accuracy of knowledge retrieval.

[0031] (2) To achieve standardized, multi-level structured interpretation of medical images and improve the accuracy and standardization of image information extraction: This application uses a logical routing mechanism and multi-level image prompt templates to ensure that different types of medical images can be matched with the most suitable prompt structure.

[0032] By utilizing feature-based template extraction and location-based cue templates, multimodal LLM can be guided to focus on lesion areas and key imaging features, thereby improving the accuracy and structuring of image descriptions.

[0033] Compared with traditional single-template image analysis schemes, this application significantly reduces the risk of image interpretation bias and improves the consistency and reusability of image structured output.

[0034] (3) Implement a secondary image parsing mechanism that aligns the "knowledge semantics - image pixels" dual channels to improve the integrity of image diagnostic information: This application inputs the image feature knowledge retrieved by the RAG system and the original image into a multimodal model, and compensates for potential misjudgments of the image model through knowledge constraints: it can correct omissions or deviations in the initial structured extraction, and further explore potential lesion features based on authoritative knowledge prompts.

[0035] This mechanism achieves "secondary enhancement" of image description, making image information summaries more complete, standardized, and closer to clinical diagnostic requirements.

[0036] Compared with existing technologies that can only generate descriptions based on a single feature extraction, this application realizes the verification and error correction of image information, which greatly improves the quality of image information summarization.

[0037] (4) By query rewriting, query decomposition, and multidimensional weighted ranking, the accuracy and relevance of medical knowledge retrieval are significantly improved: This application breaks down complex image summaries into independently searchable multidimensional subqueries, avoiding the omission of diagnostic criteria caused by the "single query dimension" in traditional retrieval.

[0038] At the same time, a weighted ranking strategy is used to sort clinical guidelines and similar cases, effectively filtering out redundant knowledge and low-relevance information.

[0039] Compared with traditional RAG systems based on Boolean logic or single-vector retrieval, the hybrid retrieval strategy of this application significantly improves the recall and precision of highly relevant clinical evidence.

[0040] (5) Improve the interpretability and security of diagnostic results through "authoritative knowledge constraints + credibility scoring + logical chain decomposition": This application employs a dual risk control mechanism: all diagnostic conclusions are derived from a structured retrieval knowledge base, significantly reducing the probability of hallucination content appearing; the reasoning chain, source of evidence, and credibility score can be displayed, making the diagnostic process transparent and verifiable; Compared to traditional black-box generation models, the diagnostic output of this application is traceable, verifiable, and interpretable, meeting the requirements of safety and prudence in medical scenarios and helping to reduce the risk of misdiagnosis.

[0041] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0042] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0043] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0044] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal, trustworthy, conversational retrieval enhancement generation system for medical diagnosis, characterized in that: The aforementioned multimodal trusted conversational retrieval enhancement generation system for medical diagnosis includes: The multimodal image parsing module is used to generate structured image feature data packages based on a multimodal large model from the original medical images; the structured image feature data packages include: organ morphology, lesion location, echo characteristics, boundary characteristics, and size; A knowledge database construction module is used to build a multi-center collaborative medical knowledge database; the multi-center collaborative medical knowledge database includes: an image feature knowledge base, a clinical guideline knowledge base, and a similar case diagnosis database. The image knowledge retrieval module is used to perform vector encoding on the structured image feature data package and retrieve image feature association knowledge from the image feature knowledge base based on a hybrid retrieval strategy; the hybrid retrieval strategy is a combination of dense vector index retrieval and sparse vector index retrieval. The clinical knowledge retrieval module is used to generate image summaries based on image feature association knowledge and original medical images using a multimodal large model; based on user questions and image summaries, it performs a dual-database search in the clinical guideline knowledge base and the similar case diagnosis database to obtain a candidate knowledge set; the candidate knowledge set includes: standardized diagnosis and treatment criteria and case data; The multidimensional weighted sorting module is used to weight and sort the candidate knowledge set according to image feature matching weight, guideline authority weight, and case similarity weight to obtain a knowledge list; The medical consultation generation module is used to generate a structured preliminary diagnosis based on the user's question, the image summary, and the knowledge list; the structured preliminary diagnosis includes: diagnosis results, diagnostic basis, recommended examinations, preliminary treatment plan, and risk warnings; The credibility verification module is used to perform source tracing and annotation, credibility scoring and logical link decomposition on the structured preliminary diagnosis to obtain the final diagnosis report; the final diagnosis report includes: diagnosis conclusion, source of evidence, credibility score and reasoning chain.

2. The multimodal trusted conversational retrieval enhancement generation system for medical diagnosis according to claim 1, characterized in that, The execution process of the multimodal image analysis module specifically includes: Based on the examination site, imaging type, and image table structure, the original medical images are classified using a preset logical routing decision tree; Determine the corresponding image description prompt template based on the image classification results; The original medical images are analyzed based on the image description prompt template and the multimodal large model to obtain a structured image feature data package.

3. The multimodal trusted conversational retrieval enhancement generation system for medical diagnosis according to claim 1, characterized in that, The construction process of the knowledge database construction module specifically includes: Acquire unstructured case data from multiple medical centers; Data cleaning was performed on unstructured case data from multiple medical centers; Based on the cleaned unstructured case data, a named entity recognition model was used to identify medical entities; and the medical entities were standardized and coded. The standardized data is segmented and sliced ​​according to semantics, and then converted into embedded vectors and stored in the vector retrieval engine.

4. The multimodal trusted conversational retrieval enhancement generation system for medical diagnosis according to claim 1, characterized in that, The execution process of the image knowledge retrieval module specifically includes: Based on generalized linear model embedding and medical corpus of hypothalamic amenorrhea, a medical word embedding model is obtained; The structured image feature data packet is vector-encoded using a medical term embedding model; Based on the vector-encoded data packets, image feature association knowledge is retrieved from the image feature knowledge base using a hybrid retrieval strategy.

5. The multimodal trusted conversational retrieval enhancement generation system for medical diagnosis according to claim 1, characterized in that, The dense vector index is called dense_clinical_semantic_index; the sparse vector index is called sparse_medical_lexical_index.

6. The multimodal trusted conversational retrieval enhancement generation system for medical diagnosis according to claim 1, characterized in that, The clinical knowledge retrieval module also includes the following steps before execution: The query is rewritten and decomposed based on the user's query to obtain multiple sub-queries covering different etiological chains.

Images