Long-range medical record diagnosis and treatment question-answering system and method based on dynamic timing semantic graph

Abstract

The application relates to a long-range medical record diagnosis and treatment question-answering system and method based on a dynamic time sequence semantic graph, and belongs to the technical fields of medical artificial intelligence and information processing, and solves the problems, such as insufficient reasoning ability, of the prior art in processing long-range medical records. The system comprises: a preprocessing module, which preprocesses the long-range medical record information of a patient to generate standardized data; a medical entity recognition module, which recognizes event class entities and descriptive entities in the standardized data; associates time attributes with the event class entities to generate time sequence entity tuples; outputs a medical entity recognition result; a dynamic semantic graph construction module, which constructs a specific dynamic semantic graph of the corresponding patient; a text-graph dual-mode fusion query engine, which is used for analyzing a medical question raised by a user and performing text-graph dual-mode fusion query based on the specific dynamic semantic graph; and a diagnosis and treatment reply module, which is used for generating a diagnosis and treatment reply result according to the text-graph dual-mode fusion query result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical artificial intelligence and information processing technology, and in particular to a long-range medical record diagnosis and treatment question-answering system and method based on a dynamic time sequence semantic graph. BACKGROUND

[0002] With the development of modern medicine and the deepening of informationization construction, the accumulation of clinical data presents an explosive growth. Especially for patients with chronic diseases such as tumors, cardiovascular diseases, and diabetes, their complete electronic medical records (EMR) often span several years or even decades, containing hundreds of independent medical documents, such as admission records, medical records, examination and test reports, imaging reports, surgical records, nursing records, and discharge summaries, with a total text volume easily exceeding hundreds of thousands of words. These medical data spanning the patient's life cycle are the cornerstone of precision medicine, clinical research, and individualized health management, and contain great value.

[0003] However, how to efficiently and accurately extract and understand information from these vast amounts of text with different formats and high levels of unstructuredness is a core bottleneck faced by current medical informatics. Traditional keyword search or simple text mining techniques cannot meet the clinical needs.

[0004] In recent years, large language models (LLM) represented by retrieval augmented generation (RAG) have been introduced into the medical field. The core idea is to cut long text into semantically related chunks, use vector retrieval technology to find the most relevant text chunks to the user's question, and then provide these text chunks as context to the LLM to generate answers. Although RAG has improved the performance of question-answering systems to some extent, its inherent technical flaws are exposed when dealing with ultra-long medical records:

[0005] (1) Fundamental long-distance dependency relationship rupture problem: The core value of ultra-long medical records lies in their internal, time-crossing causal, logical, and evolutionary relationships. For example, a patient used oxaliplatin chemotherapy after gastric cancer surgery in 2015, and in the medical record in 2018, there were persistent symptoms of neurotendinous inflammation. The chunking mechanism of RAG is almost impossible to recall and place these two events recorded in documents three years apart in the limited context window of the LLM, so it cannot answer the question "What previous treatment may be related to the patient's neurotendinous inflammation symptoms?"

[0006] (2) "Peeking at the tiger in the tube" type of incomplete context: the retrieved isolated text blocks, like the fragments of a puzzle, lack the overall picture of the patient's health status and disease evolution. LLMs answer based on these fragmented information, which is prone to bias and even contradicts the patient's overall medical history, resulting in factual errors.

[0007] (3) "Incomplete recall" of information aggregation and statistical problems: for problems that require a global perspective, such as "summarize the patient's lowest white blood cell count after each chemotherapy and the corresponding date", RAG needs to theoretically recall each chemotherapy record and all subsequent blood routine reports. The retrieval mechanism based on semantic similarity performs poorly in such tasks, often missing key information points, resulting in inaccurate statistical results.

[0008] As a technology that can effectively organize and express complex relationships between entities, knowledge graph is considered a potential solution to the above problems. However, existing general medical knowledge graphs (such as UMLS, SNOMED-CT) are static and universal knowledge bases. They can tell us that "oxaliplatin" may cause "neurotrophic inflammation", but they cannot answer whether and when this event occurred in this particular patient. Therefore, there is a lack of an innovative method that can dynamically construct a personal "medical life trajectory" graph for each patient and seamlessly integrate the structured reasoning ability of the graph with the rich details of the original text. SUMMARY

[0009] In view of the above analysis, the embodiments of the present application aim to provide a long-range medical record diagnosis and treatment question and answer system and method based on dynamic time sequence semantic graph, to solve the core technical problems of existing technologies in handling long-range, massive medical records, such as broken long-range dependency relationships, incomplete context information, difficulty in global information aggregation, and insufficient reasoning ability.

[0010] In one aspect, the present application provides a long-range medical record diagnosis and treatment question and answer system based on dynamic time sequence semantic graph, which comprises:

[0011] A preprocessing module for preprocessing the long-range medical record information of a patient to generate standardized data;

[0012] A medical entity recognition module for identifying event class entities and descriptive entities in the standardized data; associating time attributes with event class entities to generate time sequence entity tuples; aggregating time sequence entity tuples and descriptive entities to output medical entity recognition results;

[0013] A dynamic semantic graph construction module is configured to construct a specific dynamic semantic graph of a patient according to the medical entity recognition result, by taking each time sequence entity tuple as a graph node and taking information extraction result of a descriptive entity related to the time sequence entity tuple as attribute information of the time sequence entity tuple.

[0014] A text-graph dual-mode fusion query engine is configured to analyze a medical question raised by a user and perform text-graph dual-mode fusion query based on the specific dynamic semantic graph by using a cooperative working mode based on a graph path retrieval mode and a context-aware text retrieval mode.

[0015] A diagnosis and treatment reply module is configured to generate a diagnosis and treatment reply result according to the text-graph dual-mode fusion query result.

[0016] Based on the above scheme, the present application further makes the following improvements:

[0017] Further, the medical entity recognition module is configured to perform:

[0018] The standardized data is subjected to medical entity recognition by using a sequence labeling model based on BERT-CRF to obtain medical entities of various types.

[0019] It is determined whether each medical entity recognized is an event type entity.

[0020] The time attribute of the event type entity is associated to generate a time sequence entity tuple.

[0021] Text and medical type information of the descriptive entity is extracted.

[0022] Further, the dynamic semantic graph construction module comprises a cross-text complex relationship inference and linking unit and a graph dynamic aggregation and storage unit; wherein,

[0023] The cross-text complex relationship inference and linking unit is configured to construct a relationship classification model based on a graph neural network according to the medical entity recognition result.

[0024] The graph dynamic aggregation and storage unit is configured to construct a specific dynamic semantic graph of a patient according to the relationship classification model based on the graph neural network.

[0025] Further, the cross-text complex relationship inference and linking unit is configured to perform:

[0026] The information extraction result of the descriptive entity related to the time sequence entity tuple is taken as attribute information of the time sequence entity tuple, the relationship between two time sequence entity tuples is analyzed, and a candidate relationship pair is determined.

[0027] The relationship classification model based on the graph neural network is constructed according to all candidate relationship pairs.

[0028] Further, the relationship classification model based on the graph neural network is constructed, and the following is performed:

[0029] Each time sequence entity tuple is taken as a graph node, and the corresponding candidate relationship pair is taken as a candidate edge between the corresponding graph nodes.

[0030] The feature vector of each graph node is composed of entity text embedding, type embedding and time embedding, and the feature vector is propagated and aggregated by graph convolution or graph attention network.

[0031] For each candidate edge, an edge classifier is used to predict the existence probability and the relationship type of the corresponding candidate edge.

[0032] Further, the graph atlas dynamic aggregation and storage unit performs the following:

[0033] The graph nodes in the relationship classification model based on the graph neural network are taken as graph atlas nodes, the candidate edges with an existence probability greater than a probability threshold are taken as directed edges between the corresponding graph atlas nodes, and the relationship type of the candidate edge is taken as the relationship type of the corresponding directed edge, thereby constructing a specific dynamic semantic graph atlas of the corresponding patient.

[0034] Further, the relationship between each pair of time sequence entity tuples is analyzed to determine the candidate relationship pair, and the following is performed:

[0035] For any two time sequence entity tuples A and B, if they are mentioned in the same case, or the time stamps are separated within a preset threshold, or the combination of medical entity types has a known association in a medical knowledge base, they are included in the candidate relationship pair.

[0036] Further, the text-atlas dual-mode fusion query engine performs the following:

[0037] The medical problems proposed by the user are subjected to intent recognition and query conversion, and the corresponding graph query language is generated.

[0038] A coarse-grained text retrieval method based on problem vector representation is adopted to retrieve the patient's super-long medical record information and determine a preliminary text segment set in the patient's super-long medical record information.

[0039] The generated graph query language is input into the specific dynamic semantic graph atlas to obtain a graph path retrieval result.

[0040] According to each path in the graph path retrieval result, a corresponding text retrieval query statement is generated; and the text retrieval query statement is used to filter and reorder in the preliminary text segment set to obtain an accurate medical record segment in the patient's super-long medical record information.

[0041] Further, the diagnosis and treatment reply module comprises a bidirectional alignment and cross-validation unit and a traceable answer generation unit; wherein,

[0042] A bidirectional alignment and cross verification unit is configured to bidirectionally align and cross verify the path returned by the graph path retrieval and the accurate medical record fragment returned by the context-aware text retrieval to obtain a verified and corrected fusion evidence set;

[0043] A traceable answer generation unit is configured to organize the medical question and the fusion evidence set according to the structured prompt word, and generate a diagnosis and treatment answer result.

[0044] In another aspect, the application also provides a long-range medical record diagnosis and treatment question and answer method based on a dynamic time sequence semantic graph, the method comprising:

[0045] Pretreating the long-range medical record information of a patient to generate standardized data;

[0046] Identifying event class entities and descriptive entities in the standardized data, associating time attributes with the event class entities to generate time sequence entity tuples, and summarizing the time sequence entity tuples and the descriptive entities to output a medical entity recognition result;

[0047] According to the medical entity recognition result, each time sequence entity tuple is taken as a graph node, and the information extraction result of the descriptive entity related to the time sequence entity tuple is taken as attribute information of the time sequence entity tuple, to construct a specific dynamic semantic graph of the corresponding patient;

[0048] Analyzing a medical question raised by a user, and based on the specific dynamic semantic graph, using a collaborative working mode based on a graph path retrieval mode and a context-aware text retrieval mode to perform text-graph dual-mode fusion query;

[0049] Generating a diagnosis and treatment answer result according to the text-graph dual-mode fusion query result.

[0050] Compared with the prior art, the application can at least achieve one of the following beneficial effects:

[0051] The application provides a long-range medical record diagnosis and treatment question and answer system and method based on a dynamic time sequence semantic graph, relates to an intelligent information acquisition method for a mass, long-range, unstructured electronic medical record, dynamically constructs a personal time sequence semantic graph for each patient, and combines a graph-text dual-mode fusion retrieval technology, can deeply and globally understand and reason a patient's decades of medical history, and answers complex clinical questions requiring long-range tracing with high accuracy. The application can completely solve the core technical problems of the prior art in processing long-range, mass medical records, such as breaking of long-distance dependency relationship, incompleteness of context information, difficulty in global information summarization, and insufficient reasoning ability, and realizes a diagnosis and treatment question and answer system with high precision, high timeliness, and strong logical reasoning ability.

[0052] The technical solutions in the present application can be combined with each other to realize more preferred combination solutions. Other features and advantages of the present application will be described in the following description, and some advantages will become apparent from the description, or will be learned by practice of the present application. The objects and other advantages of the present application can be realized and obtained by the content particularly pointed out in the description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the present application, and together with the description serve to explain the principles of the present application, and should not be considered limiting of the present application in scope, as numerous embodiments of the present application can be made and only specific ones are illustrated in the drawings as typical embodiments.

[0054] Figure 1 A structure schematic diagram of a super-long-range medical record diagnosis and treatment question-answering system based on a dynamic timing semantic graph is provided for Embodiment 1 of the present application.

[0055] Figure 2 A flowchart of a super-long-range medical record diagnosis and treatment question-answering method based on a dynamic timing semantic graph is provided for Embodiment 2 of the present application. DETAILED DESCRIPTION

[0056] Preferred embodiments of the present application will be described in detail below with reference to the drawings, wherein the drawings constitute a part of this specification and are used to explain the principles of the embodiments of the present application together with the embodiments of the present application, and should not be considered limiting the scope of the present application, as numerous embodiments of the present application can be made and only specific ones are illustrated in the drawings as typical embodiments.

[0057] In Embodiment 1 of the present application, a super-long-range medical record diagnosis and treatment question-answering system based on a dynamic timing semantic graph is disclosed, and a structure schematic diagram of the question-answering system is shown in FIG. 1. Figure 1 As shown in FIG. 1, the question-answering system includes a pre-processing module, a data standardization module, a data analysis module, a question generation module, a question answering module, a result output module, and a result feedback module.

[0058] (1) Pre-processing module

[0059] The pre-processing module is used to pre-process the super-long-range medical record information of a patient to generate standardized data.

[0060] It should be noted that in the present embodiment, the super-long-range medical record of a patient covers all medical information up to the present time. Therefore, in the data source access process, the super-long-range medical record of the patient needs to be received and integrated to ensure coverage of the overall picture of the patient's medical history.

[0061] The medical record types of the super-long-range medical record include structured data, unstructured data, and semi-structured data.

[0062] The structured data includes test data presented in table form, such as blood routine, biochemical test, tumor marker, etc. These data have clear table headers and corresponding test result values, and are usually organized in the form of rows and columns, facilitating data retrieval and analysis.

[0063] Unstructured data includes various types of recorded texts in medical processes, such as admission records, progress records, operation records, discharge summaries, and nursing records.

[0064] Semi-structured data includes medical examination reports and image data, such as electrocardiogram reports, pathology reports, and medical image reports. Specifically, the textual description part in electrocardiogram reports, pathology reports, and medical image reports is usually unstructured text information, as they are recorded in natural language form without fixed fields and formats. However, these reports and image files may also contain some structured information, such as examination date, patient basic information, examination equipment parameters, etc., which belong to structured data.

[0065] For medical data of different medical record types in the ultra-long medical record of a patient, the following methods are used for preprocessing, respectively.

[0066] For structured data, the following method is used for preprocessing: mapping each table header in the structured data to a standardized test item name, and accurately associating each test result with its corresponding test date and time, and converting it to a standardized text format to form formatted data. Specifically, the structured data is cleaned, different table headers (such as "white blood cell count" and "WBC") are mapped to standardized test item names, and each test result is accurately associated with its corresponding test date and time. The processed structured data is converted to a standardized text format or JSON format, which specifically includes the following fields: {test item standard name, test value, unit, reference range, test timestamp, source document ID}. This standardized formatted text representation not only preserves the accuracy of structured data, but also facilitates subsequent unified processing and graph construction. For example, a record "WBC: 4.5 (10^9 / L), 2020-06-20 08:30" in the original table will be standardized to {"test item": "white blood cell count", "test value": 4.5, "unit": "10^9 / L", "reference range": "3.5-9.5", "test timestamp": "2020-06-20 08:30:00", "source document ID": "test report_20200620"}.

[0067] For unstructured data, the following method is used for preprocessing: text cleaning, document deduplication, and content segmentation are performed on the unstructured data to obtain standardized text. Exemplarily, text cleaning involves removing irrelevant characters, format markers, and other steps. In the content segmentation process, preliminary division can be made according to natural paragraphs, titles, or record dates.

[0068] It should be noted that in the implementation process, if the structured data or unstructured data is in the form of a scanned copy, optical character recognition (OCR) technology is first used to convert the scanned paper medical records or picture format reports into editable text, and then the corresponding preprocessing operation is carried out.

[0069] For semi-structured data, preprocessing is performed as follows: structured data and unstructured data in semi-structured data are extracted respectively, the extracted structured data and unstructured data are processed according to the corresponding preprocessing method respectively, and the processed formatted data and standardized text are associated with the reports / images in the semi-structured data.

[0070] In addition, for various types of medical data, a unified method is used for time information normalization to form a unified standardized time format. Specifically, ambiguous time expressions such as "3 days after surgery", "today morning", and "spring 2018" are analyzed in combination with the context and converted to the standard YYYY-MM-DD HH:MM:SS format.

[0071] (2) Medical entity recognition module

[0072] The medical entity recognition module is used to identify event class entities and descriptive entities in the standardized data, associate time attributes with the event class entities, generate time sequence entity tuples, and output medical entity recognition results by summarizing the time sequence entity tuples and the descriptive entities.

[0073] First of all, it needs to be clear that the current medical entity recognition technology (such as the BERT-based named entity recognition model) is mainly aimed at identifying and classifying text fragments of medical concepts from text (for example, identifying "aspirin" as a drug entity). However, these methods have the following limitations: the identified entities are static and isolated, lacking a time dimension; they cannot distinguish between multiple occurrences of the same entity at different time points (such as a patient using the same drug multiple times at different times); and they are difficult to directly support time sequence reasoning and causal analysis. Although adding time attributes to entities is a technically feasible approach in theory, in existing technologies in the field, the extraction of time information is usually treated as an independent time expression recognition task, separate from entity recognition. Even if there are studies that attempt to associate entities with time, they are limited to simple co-occurrence association and lack a strict event-time binding mechanism. There is a lack of specialized solutions for medical field characteristics such as complex time expressions and multiple time associations.

[0074] To solve the above problems, the embodiment innovatively proposes the concept of "time entity tuple", which expands the traditional entity recognition to "eventized entity recognition". Specifically, in the embodiment, not only the medical entity itself is recognized, but more importantly, the occurrence time of the medical event corresponding to the entity is recognized, and one or more time attributes are forcibly bound to each recognized event class entity to form a "time entity" tuple. This technical breakthrough enables the subsequent graph construction to naturally support complex queries such as time reasoning and causal tracing. Through a specially designed joint training model and strict time constraint mechanism, the embodiment ensures that each event class entity can accurately associate its occurrence time.

[0075] In the embodiment, the event class entity refers to a medical event with explicit time attribute occurring or recorded at a specific time point, i.e., a clinical activity or state change observed in the diagnosis and treatment process of a patient with a time start and end point. The event class entity specifically includes but is not limited to the following events:

[0076] Diagnosis events: [diagnosis] (referring to a clear diagnosis event, such as "2020-05-10 diagnosed as lung cancer"), [suspected diagnosis] (referring to a preliminary diagnosis result), [diagnosis revision] (referring to a change to the original diagnosis);

[0077] Treatment events: [drugs] (a drug use event, including starting medication, stopping medication, and dose adjustment), [surgery] (a surgical operation), [radiotherapy] (a radiotherapy course), [chemotherapy] (a chemotherapy cycle);

[0078] Examination events: [examination items] (such as an imaging examination, including CT and MRI), [laboratory tests] (such as a laboratory test, e.g., blood routine), [pathological examination] (such as a histopathological examination);

[0079] Symptom and sign events: [symptoms] (appearance, aggravation, and relief of symptoms, such as "2020-06-01 cough appeared"), [signs] (abnormal signs found in physical examination);

[0080] Clinical decision events: [treatment plan adjustment], [consultation], [transfer];

[0081] Nursing record events: [vital sign measurement], [nursing operation];

[0082] Patient report events: [adverse reaction report], [efficacy evaluation].

[0083] In contrast to event-type entities, descriptive entities (i.e. non-event-type entities) include anatomical sites (e.g. "left upper lobe of lung"), disease names (as general concepts rather than specific diagnostic events), medical facility names, and physician names, etc. These entities do not have time occurrence attributes by themselves.

[0084] In particular, the medical entity recognition module of the present embodiment identifies event-type entities and descriptive entities in the standardized data by the following steps:

[0085] 1) A sequence labeling model based on BERT-CRF is used to perform medical entity recognition on the standardized data, to obtain various types of medical entities.

[0086] Using the sequence labeling model, all types of medical entities in the standardized data can be identified, including diagnoses, drugs, surgeries, symptoms, anatomical sites, and test items, etc.

[0087] 2) It is determined whether each identified medical entity is an event-type entity.

[0088] By way of example, according to the identified medical entity and its context semantics in the standardized data, it is determined whether the corresponding medical entity belongs to an "event-type entity". For example, in the text "a mass is visible in the left upper lobe of the patient's lung" in the standardized data, "left upper lobe of lung" is identified as an anatomical site (descriptive entity), while "mass" is identified as an imaging finding (event-type entity, corresponding to the result of a certain examination).

[0089] 3) Time attribute association is performed on the event-type entities to generate time-series entity tuples.

[0090] The medical entity recognition module in the present embodiment can identify all medical entities in the standardized data, but only performs mandatory time association on event-type entities, which is an extension and deepening based on traditional entity recognition. Specifically, in the present embodiment, the association of time attributes is achieved by the following manner: according to the time description of the identified event-type entity in the context of the standardized data, the time attribute associated therewith is determined. Specifically, an event-type entity can be associated with a single timestamp (e.g. an examination on a certain specific date), or a time period (e.g. a chemotherapy cycle lasting for several weeks, including a start time and an end time). In the specific implementation process, a joint learning manner can be used to simultaneously train the entity recognition and time relation extraction models. For each identified event-type entity, the model searches for a time expression within a text window around the entity, and determines the relationship type (such as "occurrence time", "start time", "end time") between the time expression and the entity through a relationship classifier. If the time is not explicitly given in the text, the system will backtrack to the metadata timestamp (such as the recording date) of the paragraph or document as the default time.

[0091] For the event class entity, one or more time attributes are forced to be associated, and the corresponding time sequence entity tuple is constructed. That is, the time sequence entity tuple in the embodiment finally generates structured information in the form of a tuple, and the format is defined as: (entity content, entity type, start timestamp, end timestamp, source document and position). For example, ("osimertinib", "drug", "2020-03-05 09:00:00", "2021-01-10 17:00:00", "long-term medical order sheet ID123").

[0092] The medical entity recognition module also extracts text and medical type information of descriptive entities, and does not bind time attributes. The information extraction result of the descriptive entity is used as the attribute or modification information of the event class entity in the subsequent graph construction stage, and is used to enrich the semantic description of the graph node. For example, the anatomical site can be used as the spatial positioning attribute of the diagnosis or surgery event, and the disease name can be used for type labeling of the diagnosis event, thereby enhancing the semantic integrity of the graph and the accuracy of the query.

[0093] (3) Dynamic semantic graph construction module

[0094] The dynamic semantic graph construction module is used to construct a specific dynamic semantic graph for each patient according to the medical entity recognition result, by taking each time sequence entity tuple as a graph node and taking the information extraction result of the descriptive entity related to the time sequence entity tuple as the attribute information of the time sequence entity tuple.

[0095] The specific dynamic semantic graph constructed in the embodiment has the function of converting isolated and disordered medical records into a structured and computable personal health knowledge network with the patient as the center and time as the core dimension.

[0096] In the embodiment, the dynamic semantic graph construction module includes a cross-text complex relationship inference and linking unit and a graph dynamic aggregation and storage unit, which are specifically described as follows.

[0097] 1) Cross-text complex relationship inference and linking unit

[0098] The cross-text complex relationship inference and linking unit is used to construct a relationship classification model based on a graph neural network according to the medical entity recognition result.

[0099] Preferably, the embodiment adopts a global relationship candidate generation mechanism to process the medical entity recognition result and construct a global candidate relationship graph. It should be noted that in the embodiment, the time sequence entity tuple is a structured data unit formed by binding an event class entity and a time attribute, and its format is: entity content, entity type, start timestamp, end timestamp, source document, and position, which are basic constituent elements of the graph nodes in the embodiment, and the time sequence entity refers to the core medical event entity contained in the time sequence entity tuple. The embodiment processes complete time sequence entity tuples, rather than individual event class entities, because the time attribute and source information contained in the time sequence entity tuple are needed for relationship inference, and the specific implementation manner is described as follows.

[0100] First, the relationship between two time sequence entity tuples is analyzed to determine the candidate relationship pair. Specifically, for any two time sequence entity tuples A and B, if they are mentioned in the same case, or their timestamps are separated by a preset threshold, or the medical entity type combination has a known association in the medical knowledge base, they are included in the candidate relationship pair. In the specific implementation process, the information extraction result of the descriptive entity related to the time sequence entity tuple can also be used as attribute information of the time sequence entity tuple, on the basis of which the relationship between two time sequence entity tuples is analyzed to determine the candidate relationship pair. In addition, the embodiment also introduces a time sequence consistency constraint and a medical knowledge enhancement mechanism. In the relationship inference process, the time sequence logic constraint and the external medical knowledge graph are introduced to filter the candidate relationships that do not meet the time sequence logic or are obviously in conflict with the medical knowledge, so as to determine the final candidate relationship pair. It should be noted that the relationship inference basis includes entity type combination, time logic, semantic context, and global co-occurrence frequency, and these bases are used for preliminary screening and filtering in the candidate relationship pair determination stage to ensure that only meaningful entity pairs are subjected to deep relationship classification in the subsequent stage.

[0101] According to all candidate relationship pairs, a relationship classification model based on a graph neural network is constructed. Specifically, each time sequence entity tuple is taken as a graph node, and the corresponding candidate relationship pair is taken as a candidate edge between the corresponding graph nodes. The feature vector of each graph node is composed of entity text embedding, type embedding, and time embedding, and the feature vector is propagated and aggregated by graph convolution or graph attention network. For each candidate edge, an edge classifier is used to predict the existence probability and relationship type of the corresponding candidate edge.

[0102] In this embodiment, the edge classifier can be trained in the following manner: the training data includes medical expert annotated relationship pairs, remote supervision data based on external knowledge base and negative samples, and the relationship classification accuracy is improved by introducing semantic context and global co-occurrence frequency and other features. The relationship types predicted by the edge classifier include but are not limited to: [treatment], [induction], [improvement], [worsening], [accompanying], [diagnostic basis], [preceding] and the like, which are consistent with the relationship types defined below.

[0103] It should be noted that the traditional method mainly extracts relationships within a single sentence or a single document, and cannot handle long-distance relationships across documents; the existing knowledge graph construction method mainly depends on pre-defined templates or rules, and lacks the ability to mine complex and implicit relationships; most methods do not fully utilize time information as a constraint for relationship inference. The cross-text complex relationship inference and linking unit provided in the embodiment is a breakthrough in the core technology of graph construction, which embodies the following three key innovations:

[0104] Innovation point one: long-range, cross-document long-distance relationship inference capability

[0105] This unit can perform long-distance relationship inference across several years and hundreds of documents on the set of all "time sequence entity tuples". Traditional relationship extraction methods can only handle local and short-distance entity relationships and cannot establish causal or time sequence relationships between events separated by several years and distributed in different documents. The present application can discover potential causal relationships between "chemotherapy medication in 2015" and "neurological damage symptoms appearing in 2018" through global perspective relationship inference.

[0106] Innovation point two: intelligent relationship judgment integrating multi-dimensional information

[0107] The relationship inference comprehensively considers information in multiple dimensions such as entity type, time logic, semantic context, medical knowledge priori and global statistical features for intelligent judgment.

[0108] Innovation point three: dynamic and adaptive relationship network construction

[0109] The relationship network constructed by this unit is dynamically generated for a specific patient based on his individual medical record. The system can update the relationship network in real time according to the newly added medical record data, so that the graph always reflects the latest health status of the patient.

[0110] 2) Graph dynamic aggregation and storage unit

[0111] The graph dynamic aggregation and storage unit is used to construct a specific dynamic semantic graph for a corresponding patient based on a graph neural network-based relationship classification model.

[0112] In the implementation process, the dynamic graph aggregation and storage unit constructs a specific dynamic semantic graph of the patient, taking the graph nodes in the relationship classification model based on the graph neural network as the graph nodes, taking the candidate edges with the existence probability greater than the probability threshold as the directed edges between the corresponding graph nodes, and taking the relationship type of the candidate edge as the relationship type of the corresponding directed edge.

[0113] That is, the dynamic graph aggregation and storage unit in the embodiment takes the generated "time entity tuple" as a graph node and takes the "inferred relationship" as a directed edge. The "time entity tuple" herein corresponds to the time entity tuple in the medical entity recognition result output by the medical entity recognition module, and the "inferred relationship" corresponds to the relationship type and the corresponding entity pair whose confidence meets the threshold requirement predicted by the edge classifier of the cross-text complex relationship inference and linking unit, thereby constructing a specific dynamic semantic graph applicable only to the patient. The graph is stored in a high-performance graph database to support subsequent complex path queries and dynamic incremental updates.

[0114] In addition, the embodiment also sets a dynamic incremental update mechanism: when the patient generates new medical record data, the system only performs preprocessing and entity recognition on the newly added documents to generate new time entity tuples; the new entities are inserted into the specific dynamic semantic graph after disambiguation and deduplication, and relationship inference is performed with the existing nodes within a defined time window, and finally the incremental writing of graph nodes and directed edges is completed through the graph database transaction mechanism to realize the dynamic incremental update of the specific dynamic semantic graph.

[0115] (4) Text-graph dual-mode fusion query engine

[0116] The text-graph dual-mode fusion query engine is used to analyze the medical problems raised by the user and perform text-graph dual-mode fusion query based on the specific dynamic semantic graph and using the cooperative working mode of the graph path retrieval mode and the context-aware text retrieval mode.

[0117] In the embodiment, the text-graph dual-mode fusion query engine is the key to realize high-precision question answering. It receives the medical problems raised by the user in the natural language form, cooperatively works based on the graph path retrieval mode and the context-aware text retrieval mode, and realizes text-graph dual-mode fusion query.

[0118] The medical problems raised by the user are subjected to intent recognition and query conversion to generate the corresponding graph query language.

[0119] In practice, the intent recognition and slot filling model transforms user-generated medical questions in natural language into graph query languages ​​(such as Cypher or GQL). The training data for the intent recognition and slot filling model consists of manually labeled data, automatically generated templates, and real user query annotations. This model can translate questions such as "What chemotherapy drugs did the patient use after lung cancer surgery?" into corresponding Cypher queries.

[0120] For example, regarding the question "Did the medication B used by the patient after surgery A cause abnormal liver function?",

[0121] Intent recognition and slot filling model generation:

[0122] MATCH p = (s:surgery{name:'A'})-[:after]->(m:drug{name:'B'})-[:inducing]->(l:test items{name:'liver function', result:'abnormal'}) WHERE s.starttimestamp < m.starttimestamp AND m.starttimestamp < l.starttimestamp RETURN p

[0123] 1) Graph Path Retrieval Mode

[0124] The generated graph query language is input into the specific dynamic semantic graph to obtain the graph path retrieval results. The graph path retrieval results are presented in the form of a set of paths, each path consisting of several temporal entity tuple nodes connected by directed relation edges, reflecting the causal, temporal, or accompanying relationships between the medical events involved in the user's question.

[0125] Specifically, the generated graph query language is input into the specific dynamic semantic graph. The following path search and verification operations are used to obtain graph path retrieval results: First, the generated graph query language is input into the specific dynamic semantic graph. During graph traversal, time-constrained pruning, bidirectional breadth-first search, and heuristic search strategies based on graph statistics are introduced to find and verify paths that meet the query conditions. Next, the found paths are scored for confidence, and finally, the paths with the highest confidence are returned. These paths constitute the output of the graph path retrieval mode, i.e., the "graph path retrieval return results".

[0126] 2) Context-aware text retrieval mode

[0127] In practice, when a user submits a question, the system initiates graph path retrieval and coarse-grained text retrieval based on question vector representation in parallel.

[0128] The super-long medical record information of the patient is retrieved by using a coarse-grained text retrieval method based on a problem vector representation to determine a preliminary text segment set in the super-long medical record information of the patient.

[0129] According to each path in the atlas path retrieval result, a corresponding text retrieval query sentence is generated; and the text retrieval query sentence is used to filter and reorder in the preliminary text segment set to obtain accurate medical record segments in the super-long medical record information of the patient.

[0130] That is, after the atlas path retrieval returns a result, the system extracts key entities, time windows, and source document IDs from the path to construct an enhanced text retrieval query sentence: adopt entity keyword filtering, time range filtering, and document source priority strategy to perform fine filtering and reordering on the coarse recall result. The coarse recall result referred to here is the preliminary text segment set returned by the coarse-grained text retrieval based on the problem vector representation, rather than the several paths returned by the atlas path retrieval.

[0131] In the specific implementation process, the text retrieval can be implemented through a search engine (such as Elasticsearch). First, coarse-grained retrieval is performed based on the vector representation of the original question to obtain a preliminary recall result; then, according to the key entities, time windows, and source document IDs extracted from the atlas path, an enhanced text retrieval query sentence is constructed, and on the basis of vector similarity retrieval, a Boolean query and time interval filtering are superimposed to perform fine filtering and reordering on the coarse recall result, and finally highly relevant text segments are returned.

[0132] Therefore, the text-atlas dual-mode fusion query result in the embodiment includes the atlas path retrieval result and the accurate medical record segments in the super-long medical record information.

[0133] (5) Diagnosis and treatment reply module

[0134] The diagnosis and treatment reply module, also known as the multi-source evidence fusion and traceable answer generation module, is used to generate a diagnosis and treatment reply result according to the text-atlas dual-mode fusion query result.

[0135] The diagnosis and treatment reply module includes a bidirectional alignment and cross-validation unit and a traceable answer generation unit, and the functions thereof are specifically described as follows.

[0136] 1) Bidirectional alignment and cross-validation unit

[0137] The bidirectional alignment and cross-validation unit is used to perform bidirectional alignment and cross-validation on the (structured) path returned by the atlas path retrieval and the accurate medical record segments (i.e., original narrative segments) returned by the context-aware text retrieval to obtain a verified and corrected fusion evidence set.

[0138] Verification from graph to text: using the source document and location information of the node, verify whether the corresponding event or relationship is explicitly or implicitly described in the original text;

[0139] Verification from text to graph: perform quick entity and relationship recognition on the text segment, check whether it has corresponding nodes or edges in the graph path.

[0140] By scoring and ranking the "path text representation" and "original text segment" pairs through the cross-encoder model, the most reliable evidence combination is selected.

[0141] 2) Traceable answer generation unit

[0142] The traceable answer generation unit is used to organize medical questions and fuse evidence sets according to structured prompt words, and generate diagnosis and treatment reply results.

[0143] Specifically, in this embodiment, the structured prompt words can be constructed in the following manner: the question, evidence list and reference rules are organized into structured prompt words, and the ID, type, content and source of each evidence are listed. The question in the structured prompt word is determined based on the medical question, and the evidence list and reference rules are determined based on the fused evidence set.

[0144] The process of traceable answer can be implemented based on a large language model which is instructed to generate answers strictly according to evidence, and attach evidence ID after each key argument. For example:

[0145] The patient started taking osimertinib on March 5, 2020 [G1]. On June 18, 2020, the medical record recorded that he had a dry cough [T1]. The subsequent CT examination (2020-06-20) showed "multiple patchy ground glass density shadows in both lungs" [T2], based on which the clinic suspected drug-induced interstitial pneumonia on June 21, 2020 [T3], and took measures such as stopping osimertinib and hormone treatment [T3]. The first suspicion of this side effect was the imaging findings of multiple patchy ground glass density shadows in both lungs [T2].

[0146] In the specific implementation process, the system backend analyzes the evidence ID, and the graph subgraph visualization and original medical record highlight display can be realized in the front-end interface, so as to realize a completely traceable answer.

[0147] Specific embodiment 2 of the present application discloses a long-range medical record diagnosis and treatment question and answer method based on dynamic timing semantic graph, and the flow chart of the method is as shown in Figure 2 The method comprises the following steps:

[0148] Step S1: pre-process the long-range medical record information of the patient to generate standardized data;

[0149] Step S2: identify event class entities and descriptive entities in the standardized data; associate time attributes with the event class entities to generate time sequence entity tuples; aggregate the time sequence entity tuples and the descriptive entities to output a medical entity recognition result;

[0150] Step S3: according to the medical entity recognition result, taking each time sequence entity tuple as a graph node respectively, and taking the information extraction result of the descriptive entity related to the time sequence entity tuple as the attribute information of the time sequence entity tuple, a specific dynamic semantic graph of the corresponding patient is constructed;

[0151] Step S4: analyze the medical question raised by the user, and based on the specific dynamic semantic graph, adopt a collaborative working mode based on graph path retrieval mode and context-aware text retrieval mode to perform text-graph dual-mode fusion query;

[0152] Step S5: generate a diagnosis and treatment reply result according to the text-graph dual-mode fusion query result.

[0153] The method embodiment and the device embodiment described above are based on the same principle, can be mutually borrowed, and can achieve the same technical effect.

[0154] Specific embodiment 3 of the present application details the implementation process of the method of the present application through a specific clinical scenario.

[0155] Scenario setting: a 65-year-old lung cancer patient has an electronic medical record of up to 8 years, with a text volume of about 350,000 words. A clinician raises a complex question: "After the patient started receiving 'osimertinib' targeted therapy in 2020, how was the risk of 'interstitial pneumonia' monitored and handled? What was the imaging manifestation when the side effect was first suspected?"

[0156] Step S1: load / build patient-specific graph.

[0157] The system first checks whether the dynamic time sequence semantic graph of the patient exists. If not, the graph construction module is started to process the 8-year medical record in full, generate the graph and store it.

[0158] Step S2: start the dual-mode fusion query engine.

[0159] After the engine receives the question, the collaborative retrieval process is started.

[0160] Step S3: perform graph path retrieval.

[0161] The question is parsed as a graph query: find a path from the [drug: osimertinib, medication time > 2020-01-01] node that can connect to the [diagnosis: interstitial pneumonia] or [symptom: shortness of breath / cough] related nodes.

[0162] The graph search result can be a path: (osimertinib, 2020-03-05) -> (patient) -> (symptom: dry cough, 2020-06-18) -> (imaging: CT examination: bilateral ground-glass opacities, 2020-06-20) -> (diagnosis: suspected drug-induced interstitial pneumonia, 2020-06-21)

[0163] This path clearly reveals the logic and time sequence of events.

[0164] Step S4: Perform context-aware text search.

[0165] Extract key entities and time from the graph path in step S3: osimertinib, interstitial pneumonia, dry cough, CT, ground-glass opacities, 2020-06-18 to 2020-06-21.

[0166] This information is used to construct a highly focused query, guiding the vector search model to accurately find text passages containing these keywords and with a timestamp around June 2020 in the 350,000-word original medical record.

[0167] The search results may include: outpatient medical record on June 18, 2020, CT report on June 20, 2020, and medical record on June 21, 2020 (which records the measures such as stopping osimertinib and giving hormone shock treatment).

[0168] Step S5: Perform multi-source evidence fusion and answer generation.

[0169] After receiving the graph path and related text passages, verify that the graph logic chain is detailed in the original text, and perform high-confidence scoring on evidence combination. Finally, these evidences are integrated into a large language model to generate answers with evidence identification.

[0170] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

[0171] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily thought of by those skilled in the art, and should be covered within the protection scope of the present application.

Claims

1. A dynamic timing semantic graph-based ultra-long-range medical record diagnosis and treatment question and answer system, characterized in that, The system comprises: a preprocessing module configured to preprocess the ultra-long medical record information of the patient to generate standardized data; a medical entity recognition module configured to recognize event class entities and descriptive entities in the standardized data, associate time attributes with the event class entities to generate time sequence entity tuples, and aggregate the time sequence entity tuples and the descriptive entities to output medical entity recognition results; a dynamic semantic graph construction module configured to construct a specific dynamic semantic graph of the patient according to the medical entity recognition results, by taking each time sequence entity tuple as a graph node and taking information extraction results of the descriptive entities related to the time sequence entity tuple as attribute information of the time sequence entity tuple; a text-graph dual-mode fusion query engine configured to analyze a medical question raised by a user and perform text-graph dual-mode fusion query based on the specific dynamic semantic graph by using a collaborative working mode based on a graph path retrieval mode and a context-aware text retrieval mode; a diagnosis and treatment reply module configured to generate diagnosis and treatment reply results according to the text-graph dual-mode fusion query results; the text-graph dual-mode fusion query engine is configured to: perform intent recognition and query conversion on the medical question raised by the user to generate corresponding graph query language; perform retrieval on the ultra-long medical record information of the patient by using a coarse-grained text retrieval mode based on question vector representation to determine a preliminary text segment set in the ultra-long medical record information of the patient; input the generated graph query language into the specific dynamic semantic graph to obtain graph path retrieval results; generate corresponding text retrieval query sentences according to each path in the graph path retrieval results, and filter and reorder the text retrieval query sentences in the preliminary text segment set to obtain accurate medical record segments in the ultra-long medical record information of the patient; the diagnosis and treatment reply module comprises a bidirectional alignment and cross-validation unit and a traceable answer generation unit; wherein the bidirectional alignment and cross-validation unit is configured to perform bidirectional alignment and cross-validation on the paths returned by the graph path retrieval and the accurate medical record segments returned by the context-aware text retrieval to obtain a verified and corrected fusion evidence set; the traceable answer generation unit is configured to organize the medical question and the fusion evidence set according to structured prompt words to generate diagnosis and treatment reply results.

2. The dynamic timing semantic graph based ultra-long-range medical record diagnosis and treatment question-answering system according to claim 1, characterized in that, the medical entity recognition module is configured to: perform medical entity recognition on the standardized data by using a sequence labeling model based on BERT-CRF to obtain medical entities of various types; determine whether each recognized medical entity is an event class entity; associate time attributes with the event class entities to generate time sequence entity tuples; perform text and medical type information extraction on the descriptive entities.

3. The dynamic timing semantic graph based ultra-long-range medical record diagnosis and treatment question-answering system according to claim 2, characterized in that, the dynamic semantic graph construction module comprises a cross-text complex relationship inference and linking unit and a graph dynamic aggregation and storage unit; wherein the cross-text complex relationship inference and linking unit is configured to construct a relationship classification model based on a graph neural network according to the medical entity recognition results; the graph dynamic aggregation and storage unit is configured to construct the specific dynamic semantic graph of the patient according to the relationship classification model based on the graph neural network.

4. The dynamic timing semantic graph-based ultra-long-range medical record diagnosis and treatment question-answering system according to claim 3, characterized in that, the cross-text complex relationship inference and linking unit is configured to: Information of a descriptive entity related to the time sequence entity tuple is extracted as attribute information of the time sequence entity tuple, a relationship between two time sequence entity tuples is analyzed, and a candidate relationship pair is determined; A relationship classification model based on a graph neural network is constructed according to all candidate relationship pairs.

5. The dynamic timing semantic graph based ultra-long-range medical record diagnosis and treatment question-answering system according to claim 4, characterized in that, The relationship classification model based on the graph neural network is constructed, and the following is performed: Each time sequence entity tuple is taken as a graph node, and the corresponding candidate relationship pair is taken as a candidate edge between the corresponding graph nodes; A feature vector of each graph node is composed of an entity text embedding, a type embedding and a time embedding, and the feature vector is propagated and aggregated by a graph convolution or a graph attention network; For each candidate edge, an edge classifier is used to predict an existence probability and a relationship type of the corresponding candidate edge.

6. The dynamic timing semantic graph based ultra-long-range medical record diagnosis and treatment question-answering system according to claim 5, characterized in that, The graph atlas dynamic aggregation and storage unit performs the following: The graph nodes in the relationship classification model based on the graph neural network are taken as graph atlas nodes, the candidate edges with an existence probability greater than a probability threshold are taken as directed edges between the corresponding graph atlas nodes, and the relationship types of the candidate edges are taken as relationship types of the corresponding directed edges, to construct a specific dynamic semantic graph atlas of the corresponding patient.

7. The dynamic timing semantic graph-based ultra-long-range medical record diagnosis and treatment question-answering system according to claim 6, characterized in that, The relationship between two time sequence entity tuples is analyzed, and the following is performed to determine the candidate relationship pair: For any two time sequence entity tuples A and B, if they are mentioned in the same case, or the time stamps are separated within a preset threshold, or the combination of medical entity types has a known association in a medical knowledge base, the candidate relationship pair is included.

8. A dynamic timing semantic graph-based ultra-long-range medical record diagnosis and treatment question and answer method, characterized in that, The method is implemented in the long-range medical record diagnosis and treatment question-answering system based on the dynamic time sequence semantic graph atlas according to claim 1, and the method comprises: Pretreating the long-range medical record information of the patient to generate standardized data; Identifying event class entities and descriptive entities in the standardized data, associating time attributes with the event class entities to generate time sequence entity tuples, and outputting medical entity recognition results by aggregating the time sequence entity tuples and the descriptive entities; According to the medical entity recognition results, each time sequence entity tuple is taken as a graph node, and information of a descriptive entity related to the time sequence entity tuple is extracted as attribute information of the time sequence entity tuple, to construct a specific dynamic semantic graph atlas of the corresponding patient; Analyzing a medical question raised by a user, and based on the specific dynamic semantic graph atlas, adopting a collaborative working mode based on a graph atlas path retrieval mode and a context-aware text retrieval mode to perform text-atlas dual-mode fusion query; Generating a diagnosis and treatment reply result according to the text-atlas dual-mode fusion query result.

Images