Dual-domain rag-driven multi-omics fusion pathological analysis system

Abstract

The application discloses a dual-domain RAG-driven multi-omics fusion pathological analysis system, and belongs to the technical field of artificial intelligence of medical data. Pathological image feature data and structured multi-omics data of a patient are fused at a semantic layer by a multi-modal fusion module to obtain a semantic layer fusion result and generate a comprehensive representation vector of the patient. A dual-domain retrieval module obtains internal reference evidence corresponding to an in-hospital case knowledge base and external reference evidence corresponding to an external medical literature knowledge base. A consistency gating fusion module analyzes the consistency of the internal reference evidence and the external reference evidence, fuses the internal reference evidence and the external reference evidence to obtain fused credible evidence, and a report generation module generates a medical auxiliary report with an evidence chain based on a large language model according to the semantic layer fusion result and the credible evidence. The embodiment of the application can enhance the explainability and credibility of a diagnosis conclusion.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology for medical data, and in particular to a dual-domain RAG-driven multi-omics fusion pathological analysis system. Background Technology

[0002] Large Language Models (LLMs) are a type of artificial intelligence technology based on deep learning. Their core principle is to train a model using large-scale datasets, enabling it to generate natural language text or understand the meaning of spoken text. The main differences between large and small language models lie in model size, training data, and computational resources. Large models are characterized by a large number of parameters, abundant training data, and ample computational resources, and possess the ability to solve general tasks, follow human instructions, and perform complex reasoning.

[0003] In recent years, large-scale visual language models have shown great potential in the medical field. These models can combine image and text information to describe or answer questions using natural language. However, when applied in medicine, these large-scale models suffer from insufficient factual accuracy. Without external knowledge support, they may generate conclusions that do not conform to medical facts, thus failing to provide comprehensive and accurate medical auxiliary reports. Summary of the Invention

[0004] This application provides a dual-domain RAG-driven multi-omics fusion pathological analysis system, which achieves the technical effect of combining pathological slide images and multi-omics data, along with empirical and cutting-edge knowledge, to provide more comprehensive and accurate medical auxiliary reports.

[0005] This application provides a dual-domain RAG-driven multi-omics fusion pathological analysis system, the system comprising:

[0006] The data preprocessing module is used to perform format conversion and standardized encoding on the patient's multi-omics data and pathological slide image data to obtain the patient's structured multi-omics data and structured pathological slide image data.

[0007] The image analysis module is used to extract key pathological features from the patient's structured pathological slide image data based on a pre-trained visual language large model, and generate pathological image feature data.

[0008] The multimodal fusion module is used to fuse the pathological image feature data and structured multi-omics data at the semantic layer to obtain the semantic layer fusion result, and generate the comprehensive representation vector of the patient based on the semantic layer fusion result;

[0009] The dual-domain retrieval module is used to enhance the retrieval of the hospital case knowledge base and the external medical literature knowledge base based on the comprehensive representation vector and the semantic layer fusion result, respectively, to obtain the internal reference evidence corresponding to the hospital case knowledge base and the external reference evidence corresponding to the external medical literature knowledge base.

[0010] The consistency-gated fusion module is used to analyze the consistency between the internal reference evidence and the external reference evidence. When the internal reference evidence and the external reference evidence are consistent, the internal reference evidence and the external reference evidence are fused to obtain fused credible evidence.

[0011] The report generation module is used to generate a medical auxiliary report with a chain of evidence based on the semantic layer fusion results and credible evidence, using a pre-trained large language model.

[0012] In some implementations, the consistency-gated fusion module is further configured to:

[0013] When the internal reference evidence and the external reference evidence are inconsistent, the credibility priority of the relevant evidence is determined according to preset rules, and some of the relevant evidence is screened or retained as credible evidence.

[0014] In some implementations, the consistency gating fusion module is further used to analyze the consistency between the internal reference evidence and the external reference evidence based on a pre-set consistency criterion or by training a small discriminant model. When the internal reference evidence and the external reference evidence are consistent, the credible evidence is given a higher confidence level, and the relevant process data of the gating is retained.

[0015] In some implementations, the dual-domain retrieval module includes:

[0016] The in-hospital case knowledge base retrieval submodule is used to connect to the hospital's local in-hospital case knowledge base, retrieve data from the in-hospital case knowledge base, and obtain internal reference evidence;

[0017] The external medical literature knowledge base retrieval submodule is used to connect to external medical literature knowledge bases and obtain external reference evidence related to the current patient's multi-omics data and pathological image feature data;

[0018] The hospital's case knowledge base stores historical case records in structured vector form. Each historical case record includes corresponding scoring feedback, process data, multi-omics data, and pathological slide data.

[0019] In some implementations, the process of fusing the pathological image feature data and structured multi-omics data at the semantic layer to obtain a semantic layer fusion result, and generating a comprehensive representation vector of the patient based on the semantic layer fusion result, includes:

[0020] Based on the Transformers fusion network, self-attention encoding is performed on pathological image feature data and structured multi-omics data to obtain semantic layer representations of both.

[0021] Through the cross-attention mechanism, the semantic layer representation of relevant pathological image feature data is linked with the semantic layer representation of structured multi-omics data;

[0022] The semantic layer representations of the two are fused to obtain a semantic layer fusion result. The semantic layer fusion result is then subjected to a nonlinear transformation to obtain a comprehensive representation vector of the patient with a uniform length.

[0023] Accordingly, the retrieval of data from the hospital's case knowledge base includes:

[0024] In the hospital's case knowledge base, a vector similar to the comprehensive representation vector is retrieved from the historical case records in vector form to obtain the hospital's historical cases similar to the patient as internal reference evidence.

[0025] Accordingly, the acquisition of external reference evidence related to the current patient's multi-omics data and pathological image feature data includes:

[0026] Based on the semantic layer fusion results of multi-omics data and pathological image feature data, search terms are generated, and relevant external reference evidence is obtained by searching through the search terms.

[0027] In some implementations, it also includes:

[0028] The user interaction and feedback module is used to receive user ratings and feedback on the medical auxiliary report.

[0029] The update and optimization module is used to structure the scoring feedback, process data, desensitized multi-omics data of the patients, and pathological slide image data, and store them in the hospital's case knowledge base, and to train the visual language model and the large language model.

[0030] In some implementations, the image analysis module is also used to: extract key pathological features from the patient's structured pathological slide image data based on a pre-trained visual language large model, and generate pathological image feature data in vector form as a numerical representation of the image data.

[0031] In some embodiments, the pathological features include at least one of the following: cell morphology, histological pattern, and atypical areas;

[0032] The multi-omics data includes at least one of the following: electronic medical records, laboratory test data, gene sequencing data, and metabolomics data;

[0033] The medical auxiliary report includes at least one of the following: a comprehensive analysis of the patient's pathological slide data and multi-omics data, a list of possible conclusions, and a list of credible evidence corresponding to each possible conclusion.

[0034] In some implementations, training the visual language big model and the big language model includes performing reinforcement learning or low-rank adaptive fine-tuning on the visual language big model and the big language model.

[0035] In some implementations, the update and optimization module is also used to: update the version number of the in-hospital case knowledge base after it has been entered into the database, and update the index of historical cases in the in-hospital case knowledge base.

[0036] This embodiment of a dual-domain RAG-driven multi-omics fusion pathological analysis system addresses the problem of insufficient factual accuracy in conclusions obtained when using large visual language models in the medical field. This is achieved through a data preprocessing module, an image analysis module, a multimodal fusion module, a dual-domain retrieval module, a consistency-gated fusion module, and a report generation module. The system solves this problem by comprehensively considering external knowledge and in-hospital case data to provide conclusions with a sufficient chain of evidence, thereby improving the interpretability of the system's conclusions and increasing physician trust. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the structure of a dual-domain RAG-driven multi-omics fusion pathological analysis system provided in Embodiment 1 of this application;

[0039] Figure 2 This is a schematic diagram of the structure of a dual-domain RAG-driven multi-omics fusion pathological analysis system provided in Embodiment 2 of this application. Detailed Implementation

[0040] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] Example 1

[0042] Pathological analysis is the gold standard for the clinical diagnosis of many diseases, especially tumors. Traditional pathological analysis mainly relies on pathologists observing and analyzing pathological slides under a microscope, combined with the patient's clinical information to make a judgment. This method is highly dependent on the physician's experience, the diagnostic process is time-consuming, and has a certain degree of subjectivity. With the development of digital pathology technology, pathological slides are scanned into high-resolution whole-slide images (WSI), making computer-aided analysis possible. Patient multi-omics data (such as genome sequencing, transcriptomics, proteomics, metabolomics, etc.), electronic medical records (EMR), and laboratory test results play an important role in diagnosis. For example, the diagnosis and subtyping of certain cancers require the combination of specific gene mutation status, immunohistochemical results, etc., for accurate completion. The dual-domain RAG-driven multi-omics fusion pathological analysis method described in this application can be applied to the auxiliary analysis of pathological diagnosis of malignant tumors. A dual-domain RAG-driven multi-omics fusion pathology analysis system was installed and deployed on the hospital's server. The hospital's past tumor cases were collected and organized in advance to form an internal case knowledge base. The internal case knowledge base stores structured vectors and text index information of tumor cases. Commonly used tumor literature databases were also organized as an external medical literature knowledge base.

[0043] Figure 1 This is a schematic diagram of the structure of a dual-domain RAG-driven multi-omics fusion pathological analysis system provided in Embodiment 1 of this application. The system described in this embodiment can be used to automatically analyze the patient's condition. The system includes a data preprocessing module 110, an image analysis module 120, a multimodal fusion module 130, a dual-domain retrieval module 140, a consistency-gated fusion module 150, and a report generation module 160.

[0044] The data preprocessing module 110 is used to perform format conversion and standardized encoding on the patient's multi-omics data and pathological slide image data to obtain the patient's structured multi-omics data and structured pathological slide image data.

[0045] Multi-omics data can include at least one of the following: electronic medical records, laboratory test data, gene sequencing data, and metabolomics data. Laboratory test data can include complete blood counts and biochemical indicators, while gene sequencing data can be whole exome sequencing (WES) data or whole genome re-sequencing (WGS) data. The data preprocessing module 110 can convert laboratory test data into symbolic representations of abnormal indicators and extract a list of key mutated genes from gene sequencing data. The data preprocessing module 110 performs format conversion and standardization encoding on data from different sources for subsequent processing.

[0046] For example, a hospital receives a suspected lung cancer patient, scans the patient's pathological slide images to obtain pathological slide image data, and uploads it to the data preprocessing module 110. The data preprocessing module 110 inputs the patient's multi-omics data (related clinical data), including: electronic medical records (patient's basic information, symptoms), laboratory test data (blood test results), and gene sequencing data (genetic testing showing EGFR gene mutation), etc. The data preprocessing module 110 analyzes and extracts structured key-value pairs from the electronic medical records and laboratory test data, such as "Smoking history: Yes," "White blood cell count: Normal," "EGFR L858R: Mutation positive," etc. All textual information is uniformly encoded into 8-bit (UTF-8) format, and numerical indicators are standardized to multiples or categories relative to normal values ​​(high / low / normal). The data preprocessing module 110 adjusts the resolution format of the pathological slide image data to obtain structured pathological slide image data for use by the subsequent image analysis module 120.

[0047] The image analysis module 120 is used to extract key pathological features from the patient's structured pathological slide image data based on a pre-trained visual language large model, and generate pathological image feature data.

[0048] The pre-trained visual language model can be a visual language model that has already been deployed within the hospital. This visual language model is pre-trained using millions of pathological slide image data and their reports, and has the ability to describe structured pathological slide image data.

[0049] The image analysis module 120 is also used to: extract key pathological features from the patient's structured pathological slide image data based on a pre-trained visual language large model, and generate pathological image feature data in vector form as a numerical representation of the image data.

[0050] This module uses a pre-trained visual language model to extract features and generate preliminary pathological descriptions from structured pathological slide image data. This module extracts key pathological features from the patient's structured pathological slide image data, including but not limited to cell morphology, histological patterns, and atypical regions, and generates corresponding feature vector representations of pathological image feature data for subsequent processing. Here, atypical regions can refer to suspicious lesion areas.

[0051] For example, the image analysis module 120 calls the visual language big model deployed in the hospital to process the input structured pathological slide image data of lung tissue, extract features such as "tumor cells arranged like adenocarcinoma" and "signet ring cells appear in some areas", and generates a preliminary description "histologically considered lung adenocarcinoma, moderately differentiated, with some signet ring cell carcinoma components". Based on the preliminary description, a feature vector (e.g., 1024-dimensional) of the pathological slide image is generated as pathological image feature data.

[0052] The multimodal fusion module 130 is used to fuse the pathological image feature data and structured multi-omics data at the semantic layer to obtain the semantic layer fusion result, and generate the comprehensive representation vector of the patient based on the semantic layer fusion result.

[0053] The pathological image feature data generated by the image analysis module 120 and the patient's structured multi-omics data are used as input. In the multimodal fusion module 130, they are fused using an attention mechanism or other fusion network to obtain a semantic layer fusion result. This semantic layer fusion result represents the patient's fusion feature summary information. Then, based on this semantic layer fusion result, a comprehensive representation vector of the patient can be further generated. This comprehensive representation vector condenses the patient's multi-omics data features and pathological slide image data features, providing a basis for subsequent retrieval. The advantage of this setup is that the multi-omics fusion pathological analysis system described in this embodiment simultaneously utilizes pathological slide images and other important patient data, achieving the fusion of multi-omics data without requiring doctors to manually integrate multimodal information, reducing the cognitive burden on doctors, and fully utilizing information to improve diagnostic accuracy.

[0054] Optionally, pathological image feature data and structured multi-omics data are fused at the semantic layer to obtain a semantic layer fusion result. A comprehensive representation vector of the patient is then generated based on this semantic layer fusion result. This process includes: using a Transformer fusion network, performing self-attention encoding on the pathological image feature data and structured multi-omics data to obtain their semantic layer representations; using a cross-attention mechanism to link the semantic layer representations of related pathological image feature data and structured multi-omics data; performing feature fusion (concatenate) on the semantic layer representations of the two to obtain the semantic layer fusion result; and performing a nonlinear transformation on the semantic layer fusion result to obtain a comprehensive representation vector of the patient of uniform length. Here, Transformer is a deep learning model based on a self-attention mechanism. Its core is to replace traditional recurrent neural networks with a self-attention mechanism, significantly improving the performance of processing tasks.

[0055] For example, the multimodal fusion module 130 obtains a comprehensive representation vector E_patient of a uniform length. The comprehensive representation vector E_patient summarizes the patient's key pathological features (e.g., signs of lung adenocarcinoma) and structured key information (e.g., EGFR mutation positivity).

[0056] The dual-domain retrieval module 140 is used to enhance the retrieval of the hospital case knowledge base and the external medical literature knowledge base based on the comprehensive representation vector and the semantic layer fusion result, respectively, to obtain the internal reference evidence corresponding to the hospital case knowledge base and the external reference evidence corresponding to the external medical literature knowledge base.

[0057] The "dual-domain" approach described in this application refers to performing Retrieval-augmented Generation (RAG) operations on two knowledge domains. RAG technology improves the accuracy and reliability of reports generated from large models by incorporating references from external databases during the report generation process. The advantage of this approach in this application is that it overcomes the limitations of searching from a single-source database. It fully considers the specificities of local patients through the hospital's case knowledge base, while also taking into account the latest advancements in medical research through an external medical literature knowledge base.

[0058] Optionally, the dual-domain retrieval module 140 includes:

[0059] The in-hospital case knowledge base retrieval submodule is used to connect to the hospital's local in-hospital case knowledge base, retrieve data from the in-hospital case knowledge base, and obtain internal reference evidence;

[0060] The external medical literature knowledge base retrieval submodule is used to connect to external medical literature knowledge bases and obtain external reference evidence related to the current patient's multi-omics data and pathological image feature data;

[0061] The hospital's case knowledge base stores historical case records in structured vector form. Each historical case record includes corresponding scoring feedback, process data, multi-omics data, and pathological slide data.

[0062] Optionally, data in the hospital's case knowledge base is retrieved, including: retrieving vectors similar to the composite representation vector from historical case records in vector form within the hospital's case knowledge base, and obtaining similar historical cases within the hospital as internal reference evidence. Using vector retrieval technology, the similarity between the patient's composite representation vector and the structured vectors of historical cases in the hospital's case knowledge base is calculated to identify one or more historical cases similar to the patient in category or features as reference evidence.

[0063] Optionally, if the hospital's case knowledge base supports keyword retrieval, keyword matching can be used to retrieve one or more historical cases similar to the patient as reference evidence. Then, the diagnostic conclusions and relevant characteristics of similar historical cases can be extracted for doctors' reference.

[0064] Optionally, external reference evidence related to the current patient's multi-omics data and pathological image feature data can be obtained, including: generating search terms based on the semantic layer fusion results of the multi-omics data and pathological image feature data, and obtaining relevant external reference evidence through the search terms.

[0065] External medical literature knowledge bases can be pre-built medical literature corpora, internet medical literature search databases, clinical guidelines, etc. These knowledge bases provide access to the latest knowledge relevant to the current patient's condition. The external medical literature knowledge base retrieval submodule retrieves highly relevant literature passages or key points of clinical guidelines based on the patient's key characteristics. These key characteristics can be represented by search terms, such as the patient's initial diagnostic direction or abnormal biomarkers. For example, when multi-omics data and pathological slide image data suggest that the patient may have a rare tumor subtype, the external medical literature knowledge base retrieval submodule can retrieve the latest research papers or authoritative clinical guidelines describing and defining the subtype as external reference evidence.

[0066] For example, the in-hospital case knowledge base retrieval submodule receives the patient's comprehensive representation vector E_patient and performs a similar vector search within the in-hospital case knowledge base. For instance, it finds three cases with high similarity: Case A is "lung adenocarcinoma with signet ring cell component, EGFR mutation positive, diagnosed as stage IV lung cancer"; Case B is "lung adenocarcinoma, highly differentiated, EGFR wild-type"; and Case C is "lung squamous cell carcinoma, EGFR wild-type". The in-hospital case knowledge base retrieval submodule selects Case A with the highest similarity and Case B with the second highest similarity as internal reference evidence and provides the diagnostic records for Cases A and B. For example, the diagnostic record for Case A is "lung adenocarcinoma with special type component, diagnosed by the chief physician, with good prognosis and follow-up results". Simultaneously, the in-hospital case knowledge base retrieval submodule can also use keywords for supplementary searching. For example, it can search the in-hospital case knowledge base for keywords "EGFR mutation, lung adenocarcinoma" to ensure that historical cases with similar vectors but highly related text are not missed.

[0067] The external medical literature knowledge base retrieval submodule searches external medical literature knowledge bases, generating search terms based on the semantic layer fusion results of multi-omics data and pathological image feature data. For example, search terms could include "EGFR-mutant lung adenocarcinoma signet ring cell literature" or "lung adenocarcinoma molecular subtyping guidelines." The search results might include a recently published tumor pathology research paper mentioning that "lung adenocarcinoma with signet ring cell components often indicates a poor prognosis and needs to be differentiated from metastatic gastric cancer," or treatment recommendations for EGFR mutations in the latest national lung cancer diagnosis and treatment guidelines. The external medical literature knowledge base retrieval submodule extracts these literature paragraphs as external reference evidence, including the source (paper title, journal, or guideline name).

[0068] The consistency-gated fusion module 150 is used to analyze the consistency between the internal reference evidence and the external reference evidence. When the internal reference evidence and the external reference evidence are consistent, the internal reference evidence and the external reference evidence are fused to obtain fused credible evidence.

[0069] Optionally, the consistency gating fusion module 150 is further configured to determine the credibility priority of relevant evidence according to preset rules when the internal reference evidence and external reference evidence are inconsistent, and to screen or retain some relevant evidence as credible evidence.

[0070] The consistency-gated fusion module 150 can compare, filter, and fuse internal and external reference evidence to ensure that the evidence ultimately provided to the report generation module 160 is context-consistent and reliable. Understandably, after filtering or retaining some relevant evidence as credible, inconsistencies can be highlighted for subsequent reference by physicians.

[0071] Optionally, the consistency gating fusion module 150 is further configured to analyze the consistency between the internal reference evidence and the external reference evidence based on a pre-set consistency criterion or by training a small discriminant model. When the internal reference evidence and the external reference evidence are consistent, the credible evidence is given a higher confidence level, and the relevant process data of the gating is retained.

[0072] The consistency-gated fusion module 150 first analyzes the consistency of diagnostic conclusions between internal and external reference evidence. If the two corroborate each other, for example, if the diagnostic conclusions of historical cases in the internal reference evidence are consistent with the diagnostic conclusions recommended by clinical guidelines in the external reference evidence, then the consistent evidence is retained and given a higher confidence level. If the two are inconsistent, for example, if the diagnostic conclusions of historical cases in the internal reference evidence are for disease A, while the diagnostic conclusions recommended by clinical guidelines in the external reference evidence are for disease B, then the evidence with higher confidence is retained through a gating mechanism. For example, the clinical guidelines in the external reference evidence are chosen to be more trusted, or the internal reference evidence is judged to be more reliable based on the number of historical cases in the internal reference evidence.

[0073] Optionally, the consistency-gated fusion module 150 can also retain all inconsistent evidence, while marking the inconsistent situations, and output the inconsistent internal reference evidence, external reference evidence, and inconsistent situations to the report generation module 160, so that doctors can select them as needed later.

[0074] Optionally, the consistency-gated fusion module 150 can also filter out portions of internal and external reference evidence that are clearly inconsistent with the patient's situation, retaining evidence with higher reference value. By setting the consistency-gated fusion module 150, obviously irrelevant or contradictory evidence is filtered out, noise interference is reduced, and the report generation module 160 is ensured to use more credible evidence.

[0075] For example, the consistency-gated fusion module 150 assesses the consistency between internal reference evidence (Case A) and external reference evidence. Through natural language processing analysis, the consistency-gated fusion module 150 determines that the diagnostic conclusion of Case A is generally consistent with the external reference evidence, both pointing to the diagnostic conclusion of "lung adenocarcinoma with specific histological components." However, regarding the prognosis, Case A shows a good prognosis, while the external reference evidence suggests a poor prognosis; this is a slight inconsistency. For such slight inconsistencies, the consistency-gated fusion module 150 can avoid deleting either side's evidence. Instead, while obtaining credible evidence, it can add a label indicating the existence of differing viewpoints. This allows the report generation module 160 to prompt doctors to identify the existing uncertainties when generating a medical auxiliary report. The consistency-gated fusion module 150 assesses the consistency between internal reference evidence Case B and external reference evidence. Since Case B (EGFR wild-type) and the patient (EGFR mutation positive) have significantly different gene mutation states, even when internal reference evidence Case B and external reference evidence are consistent, the consistency-gated fusion module 150 retains only the standard diagnostic points for lung adenocarcinoma from internal reference evidence Case B, filtering out the diagnostic conclusion of Case B. In summary, the fused credible evidence includes: the standard diagnostic points for lung adenocarcinoma from internal reference evidence Case A and internal reference evidence Case B, and the external reference evidence.

[0076] The report generation module 160 is used to generate a medical auxiliary report with a chain of evidence based on the semantic layer fusion results and credible evidence using a pre-trained large language model.

[0077] The report generation module 160 generates a text-based medical auxiliary report based on the semantic layer fusion result obtained by the multimodal fusion module 130 by fusing the pathological image feature data and structured multi-omics data at the semantic layer, and the credible evidence output by the consistency-gated fusion module 150.

[0078] Optionally, the medical support report includes at least one of the following: a comprehensive analysis of the patient's pathological slide data and multi-omics data, a list of possible conclusions, and a list of credible evidence for each possible conclusion. For example, the medical support report could state: "Based on the pathological morphological characteristics (XXX) combined with the gene mutation (YYY), a preliminary diagnosis of type A is considered. [Literature: mentions that patients with YYY mutations are mostly type A]; at the same time, we found a similar case in our hospital (Case ID123) that was diagnosed with type A and responded well to SSS therapy [Hospital Case Knowledge Base]. Therefore, this patient is most likely to have type A disease." Through the above medical support report, doctors can not only see the conclusions provided by the dual-domain RAG-driven multi-omics fusion pathological analysis system, but also see the reasoning process and the source of evidence, improving the transparency and credibility of the conclusions.

[0079] For example, the large language model can be a generative pre-trained transformer (GPT) architecture model specifically fine-tuned in the medical field. The report generation module 160 calls the large language model to generate a medical assistance report with a chain of evidence. The medical assistance report should be clearly structured, providing possible conclusions and their evidence, and may also list excluded conclusions and their reasons, and provide treatment suggestions for reference. Credible evidence can be marked with "【】" for doctors to consult, or links can be set in the user interface to view the specific content of the credible evidence.

[0080] As an example, suppose the semantic layer fusion result generated by the multimodal fusion module 130 represents a comprehensive feature summary containing "middle-aged male, lung adenocarcinoma histological features, EGFR L858R mutation," combined with the credible evidence output by the consistency-gated fusion module 150, namely labeled internal and external reference evidence representing internal / external sources and brief citations. Inputting this information into a pre-trained large model, an exemplary medical assistance report is generated as follows:

[0081] "Based on a comprehensive analysis of the morphological characteristics and molecular testing results of the patient's lung tumor, the primary diagnosis is lung adenocarcinoma (with signet ring cell component). Supporting evidence includes: a similar patient (Case A) diagnosed with lung adenocarcinoma with signet ring cell component in the hospital's case database, whose EGFR gene also showed an L858R mutation [Hospital Case Database]. Furthermore, literature reports that the signet ring cell subtype of lung adenocarcinoma may have a worse prognosis than general adenocarcinoma, requiring differentiation from gastric metastatic carcinoma [XXX Journal 2024]." [Zhang San et al.]. This patient requires further clinical examination to rule out the possibility of a primary gastrointestinal tumor and metastasis. For EGFR mutation-positive lung adenocarcinoma, the latest "XXX Treatment Guidelines" recommend first-line treatment with EGFR-targeted tyrosine kinase inhibitors [2025 Edition XXX Treatment Guidelines]. Differential diagnosis: 1) Lung squamous cell carcinoma: The histology in this case is more consistent with adenocarcinoma, not squamous cell carcinoma. 2) Metastatic gastric cancer: Endoscopy and other examinations are needed to rule out the possibility of gastric tumor metastasis to the lung. Conclusion: Suspected primary lung adenocarcinoma (moderately differentiated, with signet ring cell components), EGFR mutation-targeted therapy is recommended.

[0082] This embodiment of a dual-domain RAG-driven multi-omics fusion pathological analysis system, through a data preprocessing module 110, an image analysis module 120, a multimodal fusion module 130, a dual-domain retrieval module 140, a consistency-gated fusion module 150, and a report generation module 160, addresses the problem that when large visual language models are used in the medical field, the conclusions obtained are not factually accurate, and without external knowledge support, the large model may generate conclusions that are inconsistent with medical facts. It achieves the effect of comprehensively considering external knowledge and in-hospital cases, providing conclusions with a sufficient chain of evidence, improving the interpretability of the system's conclusions, and increasing doctors' trust.

[0083] The technical solution described in this application, through multimodal fusion, integrates pathological slide images with multi-omics data such as gene sequencing data for analysis. This overcomes the limitations of focusing solely on pathological slide images or partial clinical data, providing a more comprehensive and accurate medical auxiliary report. By searching both the hospital's case knowledge base and external medical literature knowledge base, it fully utilizes the valuable case experience accumulated over many years within the hospital while promptly referencing the latest external medical literature and authoritative clinical guidelines, achieving a combination of experiential knowledge and cutting-edge knowledge, avoiding the bias that may result from a single-source database. By providing medical auxiliary reports with a clear chain of evidence, it enhances the interpretability and credibility of the conclusions, achieving the effect of allowing doctors to trace the evidence. The dual-domain RAG-driven multi-omics fusion pathology analysis system described in this embodiment, through intelligent retrieval and automatic generation of medical auxiliary reports, significantly reduces the time doctors spend manually searching for information and writing reports, improving work efficiency and quality. It can also serve as an auxiliary teaching tool for young doctors, providing diagnostic thinking references.

[0084] Example 2

[0085] Figure 2 The figure shows a schematic diagram of a dual-domain RAG-driven multi-omics fusion pathological analysis system provided in Embodiment 2 of this application. Compared with Embodiment 1, the dual-domain RAG-driven multi-omics fusion pathological analysis system in this embodiment adds a user interaction and feedback module 170 and an update and optimization module 180.

[0086] The user interaction and feedback module 170 receives user ratings and feedback on the medical auxiliary report. The medical auxiliary report generated by the report generation module 160 is presented to the doctor for review via the user interface, allowing the doctor to make a final diagnosis based on its content. Doctors can rate the usefulness and accuracy of the medical auxiliary report generated by the dual-domain RAG-driven multi-omics fusion pathology analysis system, for example, using a 1-5 star rating system, where 5 stars represents complete approval and 1 star represents complete disapproval. Doctors can also input brief comments or suggestions for correction. The user interaction and feedback module 170 records the doctor's ratings and feedback.

[0087] The update and optimization module 180 is used to structure scoring feedback, process data, desensitized multi-omics data of patients, and pathological slide image data, and store them in the hospital's case knowledge base to train the visual language model and the large language model.

[0088] After the dual-domain RAG-driven multi-omics fusion pathology analysis system completes an analysis of a patient, the update and optimization module 180 supports one-click structuring of various patient information and storing it in the hospital's case knowledge base. The stored content can include: the patient's multi-omics data, clinical data, pathological slide image data, medical auxiliary reports, scoring feedback, and the doctor's final conclusions. During storage, the update and optimization module 180 performs data anonymization processing, such as removing personally identifiable information like the patient's name and ID card number, retaining only anonymized data related to the patient's condition.

[0089] The update and optimization module 180 is also used to update the version number of the in-hospital case knowledge base after it has been added to the database, and to update the index of historical cases in the in-hospital case knowledge base. Each time new information is structured and added to the in-hospital case knowledge base, the update and optimization module 180 can assign a new version number to the newly obtained in-hospital case knowledge base to facilitate subsequent tracking and analysis of the update and derivation process of historical cases. After new cases are added to the in-hospital case knowledge base, an index update mechanism is automatically triggered to rebuild or incrementally update the retrieval index of the in-hospital case knowledge base. For example, it calculates the embedding vector of the newly added cases and adds it to the vector index library, updates the keyword index library to cover newly emerging terms, etc., thereby ensuring that the dual-domain retrieval module 140 can utilize the latest cases during subsequent searches.

[0090] For example, after reading the medical assistance report on the user interface, the doctor believes that the content and conclusions recorded in the report are basically consistent with their experience, meaning that the medical assistance report has reference value for diagnosis. The doctor's final conclusion is "Lung adenocarcinoma (with signet ring cell component), stage IIIA, EGFR mutation positive, targeted therapy recommended according to guidelines, excluding primary gastrointestinal carcinoma." The doctor gives the medical assistance report a 5-star rating and adds a brief comment, "The report is detailed and well-founded, and provides helpful suggestions for differential diagnosis," as feedback. The user interaction and feedback module 170 is responsible for receiving the doctor's rating feedback.

[0091] The update and optimization module 180 is responsible for the structured storage of scoring feedback, process data, anonymized multi-omics data of patients, and pathological slide image data. For example, after receiving feedback from a doctor clicking "One-click storage," all relevant data of that patient can be summarized and stored as a new entry in the hospital's case knowledge base. The stored content may include: basic anonymized patient information (such as gender, age group, etc.), pathological slide image data (such as pathological image feature data, etc.), multi-omics data (such as EGFR L858R mutation positivity, etc.), medical auxiliary reports, the doctor's final conclusion, and scoring feedback. The update and optimization module 180 assigns a unique ID and timestamp to this new entry in the hospital's case knowledge base and records that the current version of the hospital's case knowledge base has been updated to v1.xx. The update and optimization module 180 triggers an index update mechanism, calculates the embedding vector of the new entry and adds it to the vector index library, parses new terms appearing in the new entry, and updates the keyword index library. The update and optimization module 180 caches relevant literature citations for external medical literature knowledge bases.

[0092] The update and optimization module 180 is used to train the visual language large model and the large language model. The update and optimization module 180 can periodically train the visual language large model and the large language model. For example, every time N new cases are added to the hospital's case knowledge base or at fixed intervals, the update and optimization module 180 can adjust the training of the visual language large model and the large language model based on the doctors' rating feedback.

[0093] Optionally, training the visual language model and the large language model includes reinforcement learning or low-rank adaptive fine-tuning. The update and optimization module 180 can employ reinforcement learning or fine-tuning to achieve the following effects: for cases with high ratings, strengthen the tendency of the visual language model and the large language model to generate similar conclusions; for cases with low ratings, analyze the reasons and reduce the probability of the visual language model and the large language model repeating similar errors. Reinforcement learning refers to using doctors' rating feedback as a reward signal, optimizing the model through reinforcement learning, such as policy gradient or human feedback reinforcement learning (RLHF), to make it tend to produce medical auxiliary reports with high ratings. Low-rank adaptive fine-tuning (LoRA) refers to using small-scale incremental update techniques to adjust some parameters of the model, thereby absorbing new knowledge, not forgetting existing capabilities, and avoiding overfitting. For example, the visual language model can be fine-tuned by combining pathological slide image data of new cases with relevant conclusions to continuously improve the recognition accuracy of special and rare pathological slide image data. By setting up update and optimization module 180, the visual language big model and the big language model can be retrained periodically, so that the dual-domain RAG-driven multi-omics fusion pathology analysis system will perform better and better in searching the hospital's case knowledge base over time.

[0094] For example, over a three-month period, this dual-domain RAG-driven multi-omics fusion pathology analysis system generated medical auxiliary reports for 100 patients. Of these, 80 received 4-5 star ratings (high-scoring feedback), 15 received 3-star ratings (medium-scoring feedback), and 5 received 1-2 star ratings (low-scoring feedback). The update and optimization module 180 used cases corresponding to high-scoring feedback as positive samples and cases corresponding to low-scoring feedback as negative samples, retraining some parameters of the model using RLHF to make it more prone to generating high-scoring feedback in similar situations. For the LoRA technique, since only some weights of the model were adjusted, training was fast and avoided excessive forgetting of existing knowledge. After training the visual-language large model and the large language model, the update and optimization module 180 updated the model to the new version and recorded the version number and effective date.

[0095] This embodiment, by setting up a user interaction and feedback module 170 and an update and optimization module 180, enables the dual-domain RAG-driven multi-omics fusion pathological analysis system to continuously update based on the doctor's scoring feedback, thereby improving the system's practicality and reliability. This allows the system to provide real diagnostic conclusions, enabling the visual language model and the large language model to gradually improve their performance through continuous learning.

[0096] This embodiment constructs a learning loop involving doctors. Doctors' ratings and feedback on each medical assistance report are directly used for model training and optimization, while each new case further supplements and improves the hospital's case knowledge base. This collaboration between the model, knowledge base, and doctors makes the system increasingly intelligent with use, realizing the evolution of AI-assisted diagnosis from a static tool to a dynamic assistant.

[0097] The designs of the various embodiments in this application possess strong versatility, scalability, and portability, and can be extended to other fields of pathology and medical imaging diagnosis. For example, in addition to histopathology, cytological images and gene testing can be integrated for auxiliary diagnosis; in radiological imaging diagnosis, image scan results can be combined with laboratory data and biomarkers to aid interpretation. The dual-domain RAG framework can also be extended to knowledge bases in different fields, such as a hospital historical image diagnosis report database within the hospital, and a medical imaging-related literature database outside the hospital. Similarly, the physician feedback closed-loop mechanism can be used to continuously improve the performance of radiological diagnostic artificial intelligence (AI). Furthermore, in a comprehensive tumor diagnosis and treatment decision support system, the system of this application can serve as a module to provide pathological diagnostic opinions, and then collaborate with AI decision-making modules in other departments (radiology, internal medicine) to form a multidisciplinary consultation AI assistant.

[0098] The technical solutions described in the various embodiments of this application can assist in identifying complex and rare cases, improving diagnostic accuracy, and reducing missed diagnoses and misdiagnoses. Especially in primary hospitals, inexperienced doctors can better complete diagnostic work with the support provided by the system. Secondly, as the knowledge base continues to expand, the system will gradually accumulate large amounts of case data from the hospital and even multiple centers, which can be used for medical research, teaching and training, becoming a valuable experience base. The system described in this application is expected to interface with Hospital Information Systems (HIS), Laboratory Information Systems (LIS), and Picture Archiving and Communication Systems (PACS) as part of a comprehensive smart hospital solution, achieving the integration of clinical information and promoting the intelligent upgrading of medical processes. The continuous learning capability of the dual-domain RAG-driven multi-omics fusion pathology analysis system gives it long-term service value, continuously improving performance and adapting to new medical discoveries and clinical guidelines.

[0099] It is worth noting that the various units and modules included in the above embodiments are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0100] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A dual-domain RAG-driven multi-omic fusion pathology analysis system, characterized in that, The system comprises: a data preprocessing module for format conversion and standardized coding of the patient's multi-omics data and pathological section image data to obtain structured multi-omics data and structured pathological section image data of the patient; an image analysis module for extracting key pathological features from the structured pathological section image data of the patient based on a pre-trained visual language large model to generate pathological image feature data; a multi-modal fusion module for fusing the pathological image feature data and structured multi-omics data at a semantic layer to obtain a semantic layer fusion result and generating a comprehensive representation vector of the patient according to the semantic layer fusion result; a dual-domain retrieval module for retrieving and enhancing the in-hospital case knowledge base and external medical literature knowledge base based on the comprehensive representation vector and semantic layer fusion result to obtain internal reference evidence corresponding to the in-hospital case knowledge base and external reference evidence corresponding to the external medical literature knowledge base; a consistency gating fusion module for analyzing the consistency of the internal reference evidence and external reference evidence, and fusing the internal reference evidence and external reference evidence to obtain fused credible evidence when the internal reference evidence and external reference evidence are consistent; a report generation module for generating a medical auxiliary report with evidence chain based on a pre-trained large language model according to the semantic layer fusion result and credible evidence.

2. The dual-domain RAG-driven multi-omics fusion pathology analysis system of claim 1, wherein, The consistency gating fusion module is further configured to: determine the credible priority of relevant evidence according to a preset rule and screen or retain part of the relevant evidence as credible evidence when the internal reference evidence and external reference evidence are inconsistent.

3. The dual-domain RAG-driven multi-omics fusion pathology analysis system of any one of claims 1-2, wherein, The consistency gating fusion module is further configured to analyze the consistency of the internal reference evidence and external reference evidence based on a pre-set consistency criterion or by training a small discriminant model, and give the credible evidence a higher confidence when the internal reference evidence and external reference evidence are consistent, and retain the gated relevant process data.

4. The dual-domain RAG-driven multi-omics fusion pathology analysis system of claim 3, wherein, The dual-domain retrieval module comprises: an in-hospital case knowledge base retrieval submodule for connecting the in-hospital case knowledge base of the hospital to retrieve data in the in-hospital case knowledge base and obtain internal reference evidence; an external medical literature knowledge base retrieval submodule for connecting the external medical literature knowledge base to obtain external reference evidence related to the multi-omics data and pathological image feature data of the current patient; wherein the in-hospital case knowledge base stores historical case records in the form of structured vectors, and each historical case record includes corresponding score feedback, process data, multi-omics data and pathological section data.

5. The dual-domain RAG-driven multi-omics fusion pathological analysis system of claim 4, wherein, The fusion of the pathological image feature data and structured multi-omics data at the semantic layer to obtain a semantic layer fusion result and generating a comprehensive representation vector of the patient according to the semantic layer fusion result comprises: based on a Transformers fusion network, self-attention coding of the pathological image feature data and structured multi-omics data to obtain semantic layer representations of both; through a cross-attention mechanism, the semantic layer representations of the relevant pathological image feature data and the semantic layer representations of the structured multi-omics data are linked to each other; The semantic layer representations of the two are fused to obtain a semantic layer fusion result, and a unified length comprehensive representation vector of the patient is obtained by performing nonlinear transformation on the semantic layer fusion result; Correspondingly, the data in the in-hospital case knowledge base is retrieved, including: In the in-hospital case knowledge base, vectors similar to the comprehensive representation vector are searched in the historical case records in vector form, and in-hospital historical cases similar to the patient are obtained as internal reference evidence; Correspondingly, the external reference evidence related to the multi-omics data and the pathological image feature data of the current patient is obtained, including: According to the semantic layer fusion result of the multi-omics data and the pathological image feature data, a search term group is generated, and the related external reference evidence is searched by the search term group.

6. The dual-domain RAG-driven multi-omics fusion pathological analysis system of claim 1, wherein, Also includes: A user interaction and feedback module for receiving user feedback on the medical auxiliary report; An update and optimization module for structuring the feedback, process data, desensitized multi-omics data and pathological section image data of the patient, and storing them in the in-hospital case knowledge base, and training the visual language large model and the large language model.

7. The dual-domain RAG-driven multi-omics fusion pathological analysis system of claim 1, wherein, The image analysis module is also used to: based on the pre-trained visual language large model, extract key pathological features from the structured pathological section image data of the patient, generate vector form pathological image feature data as numerical representation of image data.

8. The dual-domain RAG-driven multi-omics fusion pathology analysis system of claim 7, wherein, The pathological features include at least one of the following: cell morphology, histological pattern and atypical area; The multi-omics data includes at least one of the following: electronic medical record, laboratory examination data, gene sequencing data and metabolomics data; The medical auxiliary report includes at least one of the following: comprehensive analysis of the patient's pathological section data and multi-omics data, list of possible conclusions, and list of credible evidence corresponding to each possible conclusion.

9. The dual-domain RAG-driven multi-omics fusion pathological analysis system of claim 6, wherein, Training the visual language large model and the large language model includes reinforcement learning or low-rank adaptive fine-tuning of the visual language large model and the large language model.

10. The dual-domain RAG-driven multi-omics fusion pathological analysis system of claim 6, wherein, The update and optimization module is also used to update the version number of the in-hospital case knowledge base after storage, and update the index of the historical cases in the in-hospital case knowledge base.

Images