Image-text generation system based on medical knowledge and patient data

Abstract

An image-text generation system based on medical knowledge and patient data comprises an acquisition processing unit used for accessing an external authoritative medical knowledge source and executing structured and standardized processing to obtain first authoritative data; the access processing unit is used for collecting target clinical data and extracting key features to obtain second clinical data; the analysis processing unit is used for constructing the first authoritative data into an authoritative knowledge graph, and performing semantic matching and bidirectional reasoning on the first authoritative data and the second clinical data to generate fused medical decision knowledge data; the image-text generation unit is used for generating an image-text report based on the medical decision knowledge data; and the display unit is used for displaying the image-text report. Through collaborative innovation of knowledge graph optimization, data standardization processing, hierarchical visualization and automatic verification, the problem of fusion of medical knowledge and clinical data is effectively solved, and the intelligent level of diagnosis and treatment decision and the clinical practicability of an image-text report are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of medical data processing technology, and more specifically, to a graphic and text generation system based on medical knowledge and patient data. Background Technology

[0002] In the current medical field, the fragmentation of medical knowledge, the heterogeneity of clinical data from multiple sources, and the inefficiency of generating graphic and textual reports have become key issues restricting intelligent diagnosis and treatment decisions.

[0003] For example: External authoritative medical knowledge sources, such as guidelines, literature, and pharmacopoeias, are mostly unstructured or semi-structured data with inconsistent terminology and circular dependencies, which directly leads to logical conflicts during knowledge graph construction and affects the accuracy of reasoning.

[0004] The target clinical data includes both structured and unstructured data. Existing systems lack standardized feature extraction processes, which can easily lead to data redundancy or omission of key information, resulting in unreliable semantic matching results.

[0005] Existing reports are mainly generated using a single output format, which cannot distinguish information levels according to clinical importance, and lacks an intuitive visual mapping of medical semantic relationships. This causes doctors to spend a lot of time screening key information, prolonging the diagnosis time and increasing the risk of misdiagnosis.

[0006] Therefore, the existing technology has problems and needs further improvement and development. Summary of the Invention

[0007] (I) Purpose of the invention: In order to solve the problems existing in the prior art, the purpose of the present invention is to provide an intelligent system that can efficiently integrate medical knowledge and clinical data, optimize the knowledge graph structure, and generate hierarchical and visualized graphic reports.

[0008] (II) Technical Solution: To solve the above-mentioned technical problems, this technical solution provides a graphic and text generation system based on medical knowledge and patient data, including: The acquisition and processing unit is used to access external authoritative medical knowledge sources and perform structured and standardized processing to obtain the first authoritative data; The access processing unit is used to collect target clinical data and extract key features to obtain second clinical data. The analysis and processing unit is used to construct an authoritative knowledge graph from the first authoritative data and perform semantic matching and bidirectional reasoning with the second clinical data to generate fused medical decision-making knowledge data. The graphic and text generation unit is used to generate graphic and text reports from medical decision-making knowledge data. The display unit is used to display the graphic report.

[0009] The aforementioned image and text generation system based on medical knowledge and patient data includes an image and text generation unit comprising a layered rendering module. This layered rendering module performs multi-dimensional layered rendering of medical decision-making knowledge data, specifically including the following steps. The information in the medical decision-making knowledge data is divided into a core layer, a supporting layer, and an interpretive layer according to clinical importance; High-contrast visual parameters are assigned to information in the core layer, medium-contrast parameters are assigned to information in the support layer, and low-contrast parameters are assigned to information in the interpretation layer. Linear transition processing is performed at the boundaries between different levels of information.

[0010] The aforementioned image and text generation system based on medical knowledge and patient data includes an image and text generation unit comprising a feature mapping module. The feature mapping module establishes mapping rules between medical semantic relationships and visual elements, and invokes the mapping rules based on semantic matching results when generating an image and text report.

[0011] The aforementioned image and text generation system based on medical knowledge and patient data includes a feature mapping module that establishes mapping rules between medical semantic relationships and visual elements, specifically comprising the following steps: The feature mapping module maps entity relationships in the authoritative knowledge graph to visual connections; Map entity attributes to color codes.

[0012] In the aforementioned image and text generation system based on medical knowledge and patient data, when the analysis and processing unit constructs an authoritative knowledge graph from the first authoritative data, if there are circular dependencies of medical terms or diagnostic criteria in the authoritative knowledge graph, it converts them into a unidirectional chain structure. Hierarchical abstraction abstracts the inner levels in a circular dependency, generating virtual nodes; The outer layer is adjusted to generate an intermediate graph without circular dependencies based on the virtual nodes and the remaining nodes; The original layer replacement restores the virtual nodes to their original inner layer structure, forming a unidirectional reasoning logic chain.

[0013] The aforementioned image and text generation system based on medical knowledge and patient data requires that the virtual nodes include original hierarchical identifiers, dependency summaries, and virtual type tags. The original level identifier records the original node identifier of the abstracted inner level; the dependency summary describes the core dependency relationship of the inner level; and the virtual type label distinguishes virtual nodes from ordinary nodes.

[0014] The aforementioned image and text generation system based on medical knowledge and patient data includes a processing unit that performs structured and standardized processing on the accessed authoritative medical knowledge, including: Perform data cleaning and format conversion on structured knowledge; Extract core entities and entity relationships from unstructured knowledge to generate structured knowledge entries.

[0015] The aforementioned image and text generation system based on medical knowledge and patient data, wherein the access processing unit extracts key features from the target clinical data, including: When the target clinical data is structured clinical data, perform standardized mapping and time series alignment on the target clinical data; When the target clinical data is unstructured, patient feature entities are extracted from the target clinical data and associated with structured data to obtain the second clinical data.

[0016] The aforementioned image and text generation system based on medical knowledge and patient data includes an analysis and processing unit that constructs an authoritative knowledge graph from first authoritative data. This authoritative knowledge graph is a multi-level medical knowledge graph. The top layer of the authoritative knowledge graph is the medical domain ontology, the middle layer is entity attributes, and the bottom layer is entity instances. The analysis and processing unit obtains the authoritative knowledge graph based on the semantic relationships between entities stored in the storage unit.

[0017] The aforementioned image and text generation system based on medical knowledge and patient data includes an analysis and processing unit that performs semantic matching between the authoritative knowledge graph and second clinical data, specifically including: The patient feature entities in the second clinical data are semantically similarly matched with entities in the authoritative knowledge graph to locate the associated authoritative knowledge nodes.

[0018] The aforementioned image and text generation system based on medical knowledge and patient data includes an analysis and processing unit that performs bidirectional reasoning between the authoritative knowledge graph and the second clinical data, based on semantic matching results. This includes reasoning from the authoritative knowledge graph to the second clinical data and reasoning from the second clinical data to the authoritative knowledge graph.

[0019] The aforementioned graphic and text generation system based on medical knowledge and patient data further includes a verification unit, which verifies the accuracy of the graphic and text report. Specifically, the verification unit verifies the accuracy of medical terminology, data consistency, and clinical logic of the generated graphic and text report.

[0020] (III) Beneficial Effects: This invention provides a graphic and text generation system based on medical knowledge and patient data, which optimizes the structure of the medical knowledge graph and improves the reliability of reasoning; standardizes the processing of clinical data and enhances data usability; generates hierarchical graphic and text reports, which improves clinical readability; and verifies the reports to ensure their accuracy. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the structure of a graphic and text generation system based on medical knowledge and patient data according to the present invention; Figure 2 This is a schematic diagram illustrating the steps of a layered rendering module in a graphic generation system based on medical knowledge and patient data, which performs multi-dimensional layered rendering of medical decision-making knowledge data. Figure 3 This is a schematic diagram illustrating the steps of the image generation system based on medical knowledge and patient data in this invention, which converts the circular dependencies of medical terms or diagnostic criteria in an authoritative knowledge graph into a unidirectional chain structure. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to preferred embodiments. More details are set forth in the following description in order to provide a full understanding of the present invention. However, the present invention can obviously be implemented in many other ways different from those described herein. Those skilled in the art can make similar extensions and derivations based on actual application situations without departing from the spirit of the present invention. Therefore, the scope of protection of the present invention should not be limited by the content of this specific embodiment.

[0023] The accompanying drawings are schematic diagrams of embodiments of the present invention. It should be noted that these drawings are for illustrative purposes only and are not drawn to scale, and should not be construed as limiting the actual scope of protection of the present invention.

[0024] A graphic and text generation system based on medical knowledge and patient data, such as Figure 1 As shown, it includes an acquisition processing unit, an access processing unit, an analysis processing unit, an image and text generation unit, a verification unit, and a display unit.

[0025] The acquisition and processing unit is used to access external authoritative medical knowledge sources and perform structured and standardized processing to obtain first authoritative data. The access and processing unit is used to collect target clinical data and extract key features to obtain second clinical data. The analysis and processing unit is used to construct an authoritative knowledge graph from the first authoritative data and perform semantic matching and bidirectional reasoning with the second clinical data to generate fused medical decision-making knowledge data. The graphic and text generation unit generates a graphic and text report based on the medical decision-making knowledge data. The verification unit verifies the graphic and text report and iterates according to clinical needs. The display unit is used to display the graphic and text report.

[0026] The acquisition and processing unit performs structured and standardized processing on the accessed authoritative medical knowledge, including data cleaning and format conversion of structured knowledge; extracting core entities and entity relationships from unstructured knowledge to generate structured knowledge entries.

[0027] The acquisition and processing unit includes a knowledge source access module and a knowledge preprocessing module. The knowledge source access module connects to at least one authoritative medical knowledge source via an application programming interface (API) or direct database connection. Authoritative medical knowledge sources include medical guideline databases, medical literature databases, medical terminology standard databases, and medical imaging knowledge bases. Specifically, medical guideline databases may include UpToDate (a clinical advisory service) and clinical treatment guideline databases. Medical literature databases may include the biomedical literature retrieval system PubMed and the CNKI medical database. Medical terminology standard databases may include the Unified Medical Language System (UMLS) and the Clinical Medical Terminology Standard System (SNOMED CT). Medical imaging knowledge bases may include the Radiopaedia (a radiology education system).

[0028] The knowledge preprocessing module categorizes and processes the accessed authoritative medical knowledge: Perform data cleaning and format conversion on structured knowledge; for example, remove duplicates and fill in missing values ​​for diagnostic criteria and medication recommendation tables in guidelines, and convert them into unified structured data; Unstructured knowledge is processed using a bidirectional pre-trained BERT language model to extract core entities and entity relationships, generating structured knowledge entries. For example, the BERT model extracts core entities from full-text documents and image descriptions: disease name, pathological mechanism, treatment plan, and entity relationships: disease-complications, drug-indications, generating structured knowledge entries. These structured knowledge entries include metadata, entities, relationships, and attributes.

[0029] The access processing unit extracts key features from the target clinical data, including: when the target clinical data is structured clinical data, performing standardized mapping and time series alignment on the target clinical data; when the target clinical data is unstructured clinical data, extracting patient feature entities from the target clinical data and associating them with structured data to obtain second clinical data.

[0030] The access processing unit includes a clinical data access module and a clinical data processing module. The clinical data is accessed through standard interfaces by the access module to the Hospital Information System (HIS), Electronic Medical Record System (EMR), Laboratory Information System (LIS), and Medical Image Archiving and Communication System (PACS) to obtain the clinical data of the target patient. The clinical data includes structured and unstructured data. Structured data includes laboratory indicators such as complete blood count and liver and kidney function tests, while unstructured data includes outpatient medical records, surgical records, and imaging reports. The clinical data processing module performs standardized mapping and time-series alignment on the structured clinical data. Standardization mapping associates the values ​​of laboratory indicators with reference ranges and marks outliers. Time-series alignment sorts the laboratory indicators by acquisition time, generating an indicator trend dataset. The clinical data processing module extracts patient characteristic entities from the unstructured clinical data using a sequence-labeled BiLSTM-CRF model, such as gender: male, chief complaint: cough for 3 days, CT findings: bilateral ground-glass opacities. These patient characteristic entities are then associated with the structured data to form a comprehensive patient feature set, i.e., the second clinical data.

[0031] The analysis and processing unit includes a map construction module, a feature matching module, and a fusion engine.

[0032] The graph construction module constructs an authoritative knowledge graph from the first authoritative data. This authoritative knowledge graph is a multi-level medical knowledge graph. The top layer of the authoritative knowledge graph is a medical domain ontology, the middle layer is entity attributes, and the bottom layer is entity instances. The analysis and processing unit obtains the authoritative knowledge graph based on the semantic relationships between entities stored in the storage unit. The top-level medical domain ontology can be diseases, symptoms, drugs, examinations, etc.; the middle-level entity attributes can be the etiology, clinical manifestations, and treatment principles of diseases, etc.; and the bottom-level entity instances can be type 2 diabetes, glycated hemoglobin, etc. The authoritative knowledge graph is obtained through the semantic relationships between entities stored in the storage unit, for example, type 2 diabetes - common complications - diabetic nephropathy.

[0033] The feature matching module performs semantic matching between the authoritative knowledge graph and the second clinical data. Specifically, this includes performing semantic similarity matching between patient feature entities in the second clinical data and entities in the authoritative knowledge graph to locate associated authoritative knowledge nodes. For example, the patient feature entities in the second clinical data, such as bilateral lung ground-glass opacities and white blood cell count of 15×10⁻⁶, are used to locate the associated authoritative knowledge nodes. 9 / L performs semantic similarity matching with entity instances in the authoritative knowledge graph to locate associated authoritative knowledge nodes. For example, bilateral lung ground-glass opacities are associated with the disease pneumonia. Semantic similarity matching can be based on cosine similarity calculated using Word2Vec word vectors, with a similarity threshold of 0.85.

[0034] The fusion engine performs bidirectional reasoning between the authoritative knowledge graph and the second clinical data to generate fused medical decision-making knowledge data. This bidirectional reasoning is performed based on semantic matching results and includes: reasoning from the authoritative knowledge graph to the second clinical data, and reasoning from the second clinical data to the authoritative knowledge graph. For example, based on the patient's characteristic entity "diabetes" in the second clinical data, the recommended path from "diabetes-treatment drugs-metformin" in the second clinical data is invoked. Combined with the patient's liver and kidney function indicator (creatinine 130 μmol / L), suitable drugs are screened, excluding nephrotoxic drugs. Reasoning from the second clinical data to the authoritative knowledge graph: based on the patient's abnormal indicator "glycated hemoglobin 8.5%", the entity relationship of "poor blood glucose control-possible causes-improper diet / poor medication adherence" is traced in the authoritative knowledge graph. After obtaining the analysis conclusion, the fused medical decision-making knowledge data is output. This medical decision-making knowledge data includes disease diagnosis criteria, individualized treatment recommendations, prognostic assessments, and mechanistic explanations.

[0035] The image and text generation unit generates image and text reports based on medical decision-making knowledge data, including a text generation module and an image generation module. The text generation module, based on the fused medical decision-making knowledge package, calls a pre-trained clinical text generation model to generate the target text. The pre-trained clinical text generation model can be a language model, fine-tuned using a medical domain corpus. The target text includes professional diagnostic reports, patient education texts, and research summary texts. Professional diagnostic reports include basic patient information, chief complaint, present medical history, diagnostic basis, and treatment plan. Patient education texts translate technical terms into plain language, for example, converting glycated hemoglobin into the average blood glucose level over the past three months. Research summary texts include comparative features of patient data with similar cases, generating text including case characteristics, literature comparison, and clinical implications.

[0036] The image generation module generates target images based on the target text content: clinical data visualization charts and mechanism diagrams. Specifically, clinical data visualization charts can be trend charts of test indicators generated by the Matplotlib engine, such as the glycated hemoglobin change curve over the past 6 months; or comparison charts of examination results generated by Plotly, such as comparisons of lesion volume on CT images before and after treatment. Mechanism diagrams can be generated using the Stable Diffusion medical fine-tuning model, combined with pathological mechanisms from authoritative knowledge graphs, such as the process of viral invasion of alveolar epithelial cells, to create clearly labeled diagrams of anatomical locations. The graphic report includes target text and target images, namely, professional diagnostic reports, educational texts, research and study texts, clinical data visualization charts, and case mechanism diagrams.

[0037] The verification unit verifies the image and text report and iterates according to clinical needs, including: The generated text and image reports are validated, including medical terminology accuracy verification, data consistency verification, and clinical logic verification. Medical terminology accuracy verification can be performed by calling the Application Programming Interface (API) of the Unified Medical Language System (UMLS) terminology library to check terminology standardization; for example, correcting "high blood sugar" to "hyperglycemia." Data consistency verification verifies the accuracy of phrases like "white blood cell count 15 × 10⁻⁶" in the text. 9 Does " / L" match the value in the chart? For example, verify that the white blood cell count in the text is 15 × 10⁻⁶. 9 Does / L match the value in the chart? Clinical logic verification uses the reasoning rule base of an authoritative knowledge graph to verify whether there are contraindications. For example, it checks whether there are contraindications for the use of glucocorticoids in diabetic patients. The system receives modification instructions from clinicians through an input unit, such as adding descriptions of drug side effects or adjusting the time range of trend graphs. It also collects feedback data to update the attention weights of the text generation model and the parameters of the image generation model. The feedback data includes the number of modifications and the type of modification. Updating the attention weights of the text generation model and the parameters of the image generation model can be achieved through the reinforcement learning algorithm PPO strategy, such as increasing the generation priority of drug side effects in diabetic patient reports and optimizing the color contrast of lesion annotations.

[0038] The display unit is used to display the graphic report, which includes a graphic-text linkage function. By clicking on the glycated hemoglobin in the target text through the input unit, one can jump to the corresponding trend chart. Clicking on the lesion area in the image can display the pathological mechanism explanation associated in the authoritative knowledge graph.

[0039] The image and text generation unit includes a layered rendering module, which performs multi-dimensional layered rendering of medical decision-making knowledge data, such as... Figure 2 As shown, the specific steps include: Step 101: Divide the information in the medical decision knowledge data into a core layer, a supporting layer, and an interpretive layer according to clinical importance; Step 102: Assign high-contrast visual parameters to the information in the core layer, medium-contrast visual parameters to the information in the support layer, and low-contrast visual parameters to the information in the interpretation layer. Step 103: Perform linear transition processing at the boundaries of different levels of information.

[0040] The core layer includes diagnostic conclusions and key test indicators, the supporting layer includes symptom descriptions and medication history, and the explanatory layer includes explanations of the disease mechanism.

[0041] High-contrast visual parameters are assigned to the information in the core layer, medium-contrast parameters are assigned to the information in the support layer, and low-contrast parameters are assigned to the information in the interpretation layer. Specifically, the high-contrast visual parameters assigned to the information in the core layer are bold red font and yellow highlighted background; the medium-contrast parameters assigned to the information in the support layer are regular black font and white background; and the low-contrast parameters assigned to the information in the interpretation layer are gray italic font.

[0042] Linear transitions are applied to the boundaries between different levels of information. For example, gradient color blocks and dashed borders are used to separate the core inspection indicators and the indicator trend analysis areas to avoid visual disjointness and improve the readability of the report.

[0043] In step 101, the hierarchical division criteria are based on the rules governing the importance of clinical information and its usage scenarios. The storage unit stores the hierarchical rules, and the hierarchical rendering module matches these rules with keywords in the medical decision-making knowledge data. The hierarchical rules are shown in the table below.

[0044] In step 102, the visual parameters of the core layer can be as follows:

[0045] The specific visual parameters of the support layer can be as follows:

[0046] The specific visual parameters of the background layer can be as follows.

[0047] In step 103, a linear transition process is performed at the intersection of different levels of information, which can be specifically as follows: The boundary between the core layer and the support layer: Insert a 1-line-high gradient divider at the end of the core paragraph. The color transitions linearly from the light yellow background of the core layer to the white background of the support layer. Set the divider height to 4px and the width to 100%. There should be no text content. Insert a rectangle using Word's shape function and fill it with the gradient color; or use Cascading Style Sheets (CSS) settings. Add a guiding subheading below the separator bar. The text content is "Analysis of Indicator Changes". Use the support layer font style, black regular font, 11pt, with a left indent of 2 characters, and align it with the support layer body text.

[0048] At the junction of the support layer and the background layer: Insert a dashed border and icon at the end of the supporting paragraph. Insert a 1px wide gray dashed border in Word's border function and set the style to dashed; or use CSS styles. Add knowledge expansion icons and text below the border. You can use Word's icon insertion function to select information icons and set the font color to gray, which is the same as the background text color. The first line of the background paragraph is indented by 2 characters and aligned with the text guiding the icon to create a visual connection.

[0049] Cross-level list item boundary: Use red bullet points for core list items; in Word, select solid circles for bullet points and set the color to red; or use Cascading Style Sheets (CSS) settings. Use lowercase black letters for the support layer list items, and select a, b, c as the Word bullet points with black color; or set the Cascading Style Sheets (CSS) settings. Insert a 0.5-line-high blank paragraph between two lists, or set the paragraph spacing in Word to 0.5 lines after the paragraph; or use Cascading Style Sheets (CSS) to avoid visual clutter.

[0050] The graphic generation unit, when generating visual charts, also includes texture enhancement for the second clinical data: adding clinical feature textures to the charts corresponding to the time-series data or multi-dimensional comparative data in the second clinical data. Time-series data in the second clinical data refers to data that changes over time, such as blood glucose levels or body temperature changes over the past three months. Multi-dimensional comparative data in the second clinical data refers to data that changes according to different treatment plans, such as the effects of different drug treatments.

[0051] Specifically, including, Abnormal data points are overlaid with a red grid texture, while normal data points are overlaid with a green smooth texture. Abnormal data points are values ​​whose exponential values ​​exceed the first health threshold, such as blood glucose levels > 11.1 mmol / L. Normal data points are values ​​that meet the first health threshold. In the trend curve, the upward trend segment is superimposed with an upward-sloping arrow texture, and the downward trend segment is superimposed with a downward-sloping arrow texture. Texture intensity is dynamically adjusted based on data fluctuations. When the fluctuation range is greater than 30%, the texture density is increased; when the fluctuation range is less than 10%, the texture density is decreased. This variation in texture intensity based on fluctuation range makes abnormal data and trends more intuitive to identify.

[0052] When generating visual charts, the image and text generation unit enhances the texture of the second clinical data using tools such as Excel and PowerPoint. The second clinical data was categorized into two types: time series data and multi-dimensional comparative data. Outlier marking was then applied to the categorized data. Specifically, the outlier marking could be achieved by setting a first health threshold using Excel's conditional formatting cell highlighting rules. When the blood glucose level was >11.1 mmol / L, the cell was marked as outlier. Normal data was marked as normal when the blood glucose level was between 3.9 and 7.0 mmol / L. For discrete data points in time series or multi-dimensional data, textures are simulated by filling cells with patterns, as shown in the table below.

[0053] For trend curves in time series data, directional textures are simulated by inserting arrow shapes: A basic trend curve is generated, and a line chart is inserted in Excel with the X-axis representing dates and the Y-axis representing blood glucose levels. The chart is copied and pasted into PowerPoint. Data labels are added to the curve, retaining outlier labels. An upward trend segment texture, such as a slanted upward arrow, is added. When three consecutive data points show an upward trend, it is determined to be an upward trend segment. A 45° slanted upward arrow is inserted, with the arrow line color set to orange and the line width 1.5pt. The arrow is then superimposed along the upward curve segment, covering the entire trend segment, with an opacity set to 50%. A downward trend segment texture, such as a slanted downward arrow, is added. When three consecutive data points show a downward trend, a -45° slanted downward arrow is inserted, with the line color set to blue, the line width 1.5pt, and the opacity 50%. This arrow is then superimposed along the downward curve segment. Adjust the texture intensity based on the data fluctuation range. Fluctuation range = |(maximum - minimum) / average| × 100%. For a fluctuation range > 30%, use a strong texture: change the pattern style of the red grid texture for outlier data points to a dense grid, darken the line color to dark red, increase the arrow width of the trend arrow to 2pt, and set the arrow head size to large (5mm width, 5mm height). For a fluctuation range of 10%-30%, use a medium texture: retain the fine grid texture for outlier data points, keep the line color unchanged, set the trend arrow line width to 1.5pt, and the arrow head size to medium (3mm width, 3mm height). For a fluctuation range < 10%, use a weak texture: change the grid texture of outlier data points to a dashed grid, lighten the line color to light red, set the trend arrow line width to 1pt, and the arrow head size to small (2mm width, 2mm height), and reduce the opacity to 30%.

[0054] By following the steps above, the normal / abnormal states and trends of data can be intuitively distinguished in the visualization charts, thereby improving the efficiency of clinical data interpretation.

[0055] The image and text generation unit includes a feature mapping module, which establishes mapping rules between medical semantic relationships and visual elements, and calls the mapping rules based on the semantic matching results when generating an image and text report.

[0056] The feature mapping module establishes mapping rules between medical semantic relationships and visual elements, specifically including the following steps: The feature mapping module maps entity relationships in the authoritative knowledge graph to visual connections, with solid lines representing direct associations and dashed lines representing indirect associations. Map entity attributes to color codes.

[0057] When generating graphic reports, the above mapping rules are automatically invoked based on semantic matching results. For example, next to the diagnosis of diabetes, retinopathy complications are connected by a red dotted line and the risk level is marked as: high risk, glycated hemoglobin > 9%, making the abstract semantic relationship visible.

[0058] The feature mapping module maps entity relationships in the authoritative knowledge graph to visual connections. This can be achieved through a preset relationship connection lookup table, as shown in the table below.

[0059] Mapping entity attributes to color codes includes mapping risk level, time, and strength of evidence. This can be achieved through pre-defined risk level mapping tables, time mapping tables, and strength of evidence mapping tables. The risk level mapping table includes risk level attribute values ​​and matching color codes, shape codes, and font styles. The time mapping table includes time attribute values ​​and matching visual elements and configuration rules. The strength of evidence mapping table includes strength of evidence attribute values ​​and matching visual elements and configuration rules.

[0060] The image and text generation unit also includes an overlay rendering module, which is used to construct a base layer, a feature enhancement layer, and a decision layer to perform layered overlay rendering of the second clinical data.

[0061] Specifically, including, Base layer construction: The structured data in the second clinical data is used as the bottom layer, such as test indicators and medication records. The raw values ​​are recorded in a table format, and a white background and black regular font are set as the inherent color information of the first texture. Feature enhancement layer overlay: Create a new analysis layer on the base layer, add ambient occlusion light simulation effect to abnormal data, such as gray shadow border, and add bump texture simulation effect to trend change data as a second texture for ambient occlusion light and bump information. For example, add up / down arrow texture to blood sugar fluctuations over the past 3 months. Decision layer display: A decision layer is overlaid on the analysis layer. The core conclusions of the medical decision knowledge data generated by the analysis and processing unit, such as high risk of diabetes, are displayed with a semi-transparent orange block. The text inside the block is in bold white font. The lighting and shadow effects of the third texture are enhanced to give the decision information the highest visual priority.

[0062] When extracting key features, the access processing unit also includes detecting circular dependencies in the second clinical data through hierarchical traversal: hierarchical traversal, starting from the root node of data collection, recording the upstream dependency list of each data item layer by layer; circular marking, if the upstream dependency list of a data item contains its downstream data item, it is marked as a circular dependency; conflict prompt, outputting the specific path of the circular dependency, pausing the feature extraction process, and prompting the user to intervene manually.

[0063] When the analysis and processing unit constructs an authoritative knowledge graph from the first authoritative data, if there are circular dependencies of medical terms or diagnostic criteria in the authoritative knowledge graph, it converts them into a unidirectional chain structure to avoid circular conflicts during semantic matching.

[0064] like Figure 3 As shown, the specific steps are as follows: Hierarchical abstraction abstracts the inner levels in a circular dependency, generating virtual nodes; The outer layer is adjusted to generate an intermediate graph without circular dependencies based on the virtual nodes and the remaining nodes; The original layer replacement restores the virtual nodes to their original inner layer structure, forming a unidirectional reasoning logic chain.

[0065] The virtual node must include an original level identifier, a dependency summary, and a virtual type label. The original level identifier records the original node identifier of the abstracted inner level; the dependency summary describes the core dependencies of the inner level; the dependency summary can describe the core dependencies of the inner level in text form; the virtual type label distinguishes the virtual node from ordinary nodes. Specifically, the virtual type label can be marked as a circular dependency abstract node.

[0066] In hierarchical abstraction, the inner levels of a circular dependency are abstracted to generate virtual nodes. The inner level of a circular dependency is determined by a preset dependency depth threshold. If the dependency depth of a node at a certain level in a circular dependency, that is, the path length from that node to the starting point of the ring, is less than or equal to 2, it is determined to be an inner level. For example, in a circular structure where A depends on B, B depends on C, and C depends on A, the path length of B depending on C is 1, which is less than the threshold 2, so it is determined to be an inner level. A virtual node is generated, and the original node identifier of the abstracted inner level is recorded as the original level identifier; the core dependency relationship of the inner level is described in text form as a dependency summary; the virtual node is marked as a circular dependency abstract node to distinguish it from ordinary nodes, thus obtaining the virtual node.

[0067] In the outer layer adjustment, an intermediate graph without circular dependencies is generated based on the virtual nodes and the remaining nodes. After the intermediate graph is generated, the graph is checked for acyclicity: starting from the virtual node, all dependency paths of the outer nodes are traversed. If there is a circular path of virtual node, outer node, and virtual node, the dependency order of the outer nodes is readjusted. Specifically, the nodes can be sorted in lexicographical order by name until all paths are unidirectional and acyclic.

[0068] In the original layer replacement, the virtual nodes are restored to their original inner layer structure, forming a unidirectional reasoning logic chain. After restoring the virtual nodes to their original inner layer structure, the following verifications are performed: Node number consistency verification: the number of inner layer nodes after restoration is completely consistent with that before abstraction; Dependency direction consistency verification: the dependency direction of the inner layer after restoration is completely consistent with the original structure; Global acyclicity verification: the entire knowledge graph is sorted, specifically using a topological sorting algorithm. If all nodes can enter the sorted sequence, then the circular dependency is determined to have been eliminated.

[0069] Sort using a topological sorting algorithm, specifically including the following steps. Entities in the knowledge graph are abstracted as vertices, each assigned a unique identifier. Directed edges are generated based on the dependencies between entities. Specifically, if entity A is a prerequisite for entity B, a directed edge A→B is generated, with the head pointing to the dependent entity. For example, if hyperglycemia S002 is a diagnostic criterion for diabetes D001, a directed edge S002→D001 is generated. If entity C is an exclusion condition for entity D, a directed edge C→¬D is generated. Directed edges with negation are stored separately in the exclusion relation table of the storage unit. For example, if hypoglycemia S003 excludes diabetes D001, a directed edge S003→¬D001 is generated.

[0070] The in-degree of each vertex is the number of directed edges pointing to that vertex, excluding negative edges. For example, if diabetes D001 depends on hyperglycemia S002 and elevated glycated hemoglobin S004, then the in-degree of D001 is 2. The in-degree is calculated by traversing all directed edges, counting the in-degree value of each vertex, and storing it in an in-degree table. This in-degree table is a static table and is only manually refreshed when the knowledge graph is updated. Negative edges in the exclusion relation table are not included in the in-degree calculation; they are only used for subsequent reasoning and verification. The initial queue rule is to add vertices with an in-degree of 0, i.e., basic entities without prior dependencies, to the queue as sorting starting points, such as gender S001 and age S005.

[0071] The queue is processed according to the following fixed rules to generate a topological sequence: the first vertex is taken from the queue, added to the topological sequence list, and marked as sorted; the in-degree of adjacent vertices is updated, all outgoing edges of the vertex are traversed, i.e., the dependent entities of the vertex as a precondition, the in-degree of each adjacent vertex is decremented by 1, and the in-degree list is updated; new vertices with an in-degree of 0 are added to the queue. If the in-degree of an adjacent vertex is updated to 0 and it has not been marked as sorted, it is added to the queue; the above steps are repeated until the queue is empty.

[0072] If the number of vertices in the topological sequence list equals the total number of vertices in the knowledge graph, the sorting is successful, generating an acyclic entity dependency sequence; if the number of vertices in the "topological sequence list" is less than the total number of vertices, there are unprocessed circular dependencies. Output a circular dependency residual report, which includes the identifiers of unqueued vertices and their dependencies, and trigger a manual intervention process.

[0073] A graphic and text generation system based on medical knowledge and patient data effectively solves the problem of integrating medical knowledge and clinical data through collaborative innovation of knowledge graph optimization, data standardization processing, hierarchical visualization, and automated verification. It significantly improves the intelligence level of diagnosis and treatment decision-making and the clinical practicality of graphic and text reports, and has important medical value and social significance.

[0074] Optimize the structure of medical knowledge graphs to improve reasoning reliability: By hierarchical abstraction, outer layer adjustment, and original layer replacement, the circular dependencies in the knowledge graph are transformed into a unidirectional chain structure, eliminating logical conflicts and making semantic matching and bidirectional reasoning processes free of cyclic redundancy, thus improving the accuracy of reasoning; the authoritative knowledge graph is divided into a three-layer structure, combined with standardized processing procedures, to achieve structured storage and efficient retrieval of knowledge, thereby improving the efficiency of knowledge integration.

[0075] Standardized clinical data processing enhances data usability: For structured and unstructured clinical data, standardized mapping, time series alignment and entity extraction, and association fusion strategies are adopted respectively to ensure that key features are extracted without omission, thereby improving data integrity. Through bidirectional reasoning from knowledge graph to clinical data and from clinical data to knowledge graph, deep integration of medical knowledge and patient data is achieved, which improves the decision support coverage of complex cases and reduces the one-sidedness caused by single reasoning.

[0076] Hierarchical graphic report generation enhances clinical readability: By dividing the report into core, support, and interpretation layers, and combining contrast parameters with linear transition processing, key information is visually highlighted, improving doctors' efficiency in obtaining information; entity relationships in the knowledge graph are mapped to visual connections and entity attributes are mapped to color codes, intuitively presenting the logical connections between diseases, symptoms, and treatments, reducing the difficulty of medical semantic understanding.

[0077] Full-process automated verification ensures report accuracy: The verification unit reduces the report error rate by verifying the accuracy of medical terminology, data consistency, and clinical logic, making it far lower than the error rate of traditional manual verification.

[0078] System scalability and universality: Each unit is independently packaged, allowing for flexible replacement of functional modules to adapt to the personalized needs of different departments and improve the system's reusability.

[0079] The above description illustrates preferred embodiments of the present invention and helps those skilled in the art to more fully understand the technical solution of the present invention. However, these embodiments are merely illustrative and should not be construed as limiting the specific implementation of the present invention to these embodiments. For those skilled in the art, several simple deductions and modifications can be made without departing from the inventive concept, and all such modifications should be considered within the protection scope of the present invention.

Claims

1. A graphic and text generation system based on medical knowledge and patient data, characterized in that, include, The acquisition and processing unit is used to access external authoritative medical knowledge sources and perform structured and standardized processing to obtain the first authoritative data; The access processing unit is used to collect target clinical data and extract key features to obtain second clinical data. The analysis and processing unit is used to construct an authoritative knowledge graph from the first authoritative data and perform semantic matching and bidirectional reasoning with the second clinical data to generate fused medical decision-making knowledge data. The graphic and text generation unit generates graphic and text reports based on medical decision-making knowledge data; The display unit is used to display the graphic report.

2. The image and text generation system based on medical knowledge and patient data according to claim 1, characterized in that, The image and text generation unit includes a layered rendering module, which performs multi-dimensional layered rendering of medical decision-making knowledge data, specifically including the following steps. The information in the medical decision-making knowledge data is divided into a core layer, a supporting layer, and an interpretive layer according to clinical importance; High-contrast visual parameters are assigned to information in the core layer, medium-contrast parameters are assigned to information in the support layer, and low-contrast parameters are assigned to information in the interpretation layer. Linear transition processing is performed at the boundaries between different levels of information.

3. The image and text generation system based on medical knowledge and patient data according to claim 1, characterized in that, The image and text generation unit includes a feature mapping module, which establishes mapping rules between medical semantic relationships and visual elements, and calls the mapping rules based on the semantic matching results when generating an image and text report.

4. The image and text generation system based on medical knowledge and patient data according to claim 3, characterized in that, The feature mapping module establishes mapping rules between medical semantic relationships and visual elements, specifically... Includes the following steps, The feature mapping module maps entity relationships in the authoritative knowledge graph to visual connections; Map entity attributes to color codes.

5. The image and text generation system based on medical knowledge and patient data according to claim 1, characterized in that, When the analysis and processing unit constructs an authoritative knowledge graph from the first authoritative data, if there are circular dependencies of medical terms or diagnostic criteria in the authoritative knowledge graph, it converts them into a unidirectional chain structure: Hierarchical abstraction abstracts the inner levels in a circular dependency, generating virtual nodes; The outer layer is adjusted to generate an intermediate graph without circular dependencies based on the virtual nodes and the remaining nodes; The original layer replacement restores the virtual nodes to their original inner layer structure, forming a unidirectional reasoning logic chain.

6. The image and text generation system based on medical knowledge and patient data according to claim 5, characterized in that, The virtual node must include the original level identifier, dependency summary, and virtual type label. The original level identifier records the original node identifier of the abstracted inner level; the dependency summary describes the core dependency relationship of the inner level; and the virtual type label distinguishes virtual nodes from ordinary nodes.

7. The image and text generation system based on medical knowledge and patient data according to claim 1, characterized in that, The acquisition and processing unit performs structured and standardized processing on the accessed authoritative medical knowledge, including: Perform data cleaning and format conversion on structured knowledge; Extract core entities and entity relationships from unstructured knowledge to generate structured knowledge entries.

8. The image and text generation system based on medical knowledge and patient data according to claim 1, characterized in that, The access processing unit extracts key features from the target clinical data, including: When the target clinical data is structured clinical data, perform standardized mapping and time series alignment on the target clinical data; When the target clinical data is unstructured, patient feature entities are extracted from the target clinical data and associated with structured data to obtain the second clinical data.

9. The image and text generation system based on medical knowledge and patient data according to claim 1, characterized in that, The analysis and processing unit constructs an authoritative knowledge graph from the first authoritative data. This authoritative knowledge graph is a multi-level medical knowledge graph. The top layer of the authoritative knowledge graph is the medical domain ontology, the middle layer is entity attributes, and the bottom layer is entity instances. The analysis and processing unit obtains the authoritative knowledge graph based on the semantic relationships between entities stored in the storage unit.

10. The image and text generation system based on medical knowledge and patient data according to claim 1, characterized in that, The analysis and processing unit performs semantic matching between the authoritative knowledge graph and the second clinical data, specifically including: The patient feature entities in the second clinical data are semantically similarly matched with entities in the authoritative knowledge graph to locate the associated authoritative knowledge nodes.

11. The image and text generation system based on medical knowledge and patient data according to claim 1, characterized in that, The analysis and processing unit performs bidirectional reasoning between the authoritative knowledge graph and the second clinical data, based on the semantic matching results, including: reasoning from the authoritative knowledge graph to the second clinical data, and reasoning from the second clinical data to the authoritative knowledge graph.

12. The image and text generation system based on medical knowledge and patient data according to claim 1, characterized in that, It also includes a verification unit, which verifies the accuracy of the graphic report, specifically including verifying the accuracy of medical terminology, data consistency, and clinical logic of the generated graphic report.

Images