Multi-modal medical text extraction method based on adaptive domain knowledge fusion

Abstract

The invention relates to a multi-modal medical text extraction method based on adaptive domain knowledge fusion, and belongs to the technical field of medical text mining, and the method comprises the following steps: text entity-relationship mining: extracting related entities and semantic relationships from clinical structured and unstructured texts, and generating a preliminary text disease network; professional knowledge linking: performing knowledge alignment by using an external medical knowledge graph; and extracting a medical spatio-temporal event chain: extracting an event sequence with a time sequence and a spatial position from a clinical text, constructing a dynamic text disease knowledge network, and extracting a multi-modal medical text.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical text mining technology and relates to a multimodal medical text extraction method based on adaptive domain knowledge fusion. Background Art

[0002] Medical text, as unstructured data, contains a wealth of medical information, such as patient symptoms, diseases, diagnoses, treatments, and demographic data. This information is of great value for research in clinical decision support, intelligent diagnosis and treatment, and disease course prediction. Due to its unstructured nature, rich technical terminology, large content, and non-standardized expression, medical text is difficult to directly and effectively utilize. In recent years, some research has attempted to use neural networks to extract knowledge from medical text. For example, Ma et al. proposed an event extraction model based on the BiGRU-CRF architecture, which can effectively identify trigger words and extract relevant entity results from electronic medical records. Recently, many scholars have also explored the use of graph neural networks. For example, Wu et al. developed a hybrid neural network called BioIE, which uses GCNs enhanced with multi-head attention to capture the complex relationships and contextual information in biomedical text. Another important research advance is the integration of medical knowledge into pre-trained language models. For example, Roy et al. explored techniques to enhance BERT for clinical relationship extraction, demonstrating that integrating medical knowledge significantly improves performance on domain-specific tasks. Furthermore, integrating different neural network architectures has also been a focus of recent research. Yang et al. proposed a hybrid neural network model combining CNN and BiLSTM to effectively capture local and global features of medical text. Knez et al. introduced a bimodal architecture for temporal relationship extraction that integrates information from medical text and external knowledge. Furthermore, the recent rise of large language models (LLMs) has further enhanced the ability to extract knowledge from text. For example, Cao et al. proposed an AutoRD system that leverages ontology-enhanced LLMs to automatically extract rare disease information and its relationships with medical concepts from medical text in an end-to-end manner. These studies lack the extraction of spatiotemporal features from clinical text and overlook the importance of medical dictionaries and knowledge graphs, which in turn affects the accuracy of disease-related knowledge extraction from medical text. Therefore, to overcome these difficulties, it is necessary to fully extract the temporal and spatial features of medical text and combine them with domain dictionaries and external knowledge graphs to supplement and correct them, thereby fully and accurately extracting knowledge from medical text. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a multimodal medical text extraction method based on adaptive domain knowledge fusion, which combines knowledge graphs, medical dictionaries and spatiotemporal features, and realizes the structured organization of medical free text by fine-grained mining of spatiotemporal event chains in medical texts and constructing them into text knowledge networks.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A multimodal medical text extraction method based on adaptive domain knowledge fusion includes the following steps:

[0006] Text entity-relationship mining: extract relevant entities and semantic relationships from clinical structured and unstructured texts to generate preliminary text disease networks;

[0007] Professional knowledge linking: using external medical knowledge graphs for knowledge alignment;

[0008] Medical spatiotemporal event chain extraction: Extract event sequences with temporal order and spatial location from clinical texts, build dynamic text disease knowledge networks, and extract multimodal medical texts.

[0009] Furthermore, the text entity-relationship mining includes the following steps:

[0010] S11: For the text sequence X=[w1,w2,…,w n ], generate context-related vector H through medical text pre-training, and then optimize the vector through the medical domain dictionary to generate H lex ;

[0011] S12: Define the legal relationships between entity types through medical system nomenclature and construct the corresponding adjacency matrix A ontology ∈{0,1} |S|×|S| , where S is the set of entity types and |S| is the number of types;

[0012] S13: Using graph convolutional neural network (GCN) to fuse neighborhood information constrained by ontology:

[0013]

[0014] Among them, G (0) =H lex is the initial input, is the learnable parameter of the lth layer, σ(·) is the activation function; the loss function uses the cross entropy of the weighted design of nodes and edges;

[0015] S14: Use conditional random field CRF and Softmax to perform text entity recognition and relationship classification, generate entity label sequence score function S(y) and relationship probability P(r|e i ,e j ), thereby extracting the corresponding entities and relationships in the medical text and generating a preliminary text disease network

[0016] Furthermore, in step S11, BioBERT-large is used, fine-tuned based on PubMed+PMC corpus, and the domain dictionary embedding is added to the input layer to generate the domain dictionary enhanced medical text sequence vector H lex .

[0017] Furthermore, the professional knowledge link specifically includes the following steps:

[0018] S21: Introducing external medical knowledge graphs The cross-graph bilinear attention mechanism is used to calculate the similarity between nodes, integrating KG and for

[0019] S22: Generated by knowledge graph representation method TransH The corresponding embedding g={g1,g2,…,g n};

[0020] S23: Aggregation using graph attention networks Through continuous training and network convergence, the entities with similarity close to 1 are merged, their terminology descriptions are unified, and the The number of entities is

[0021] Furthermore, the medical spatiotemporal event chain extraction specifically includes the following steps:

[0022] S31: Use regular expressions to extract time features from medical texts. The regular expression template is:

[0023] T abs =Regex(r′\d4-\d2-\d2′)

[0024] T rel =Regex(r′\d+(day|monther|year)(before|after)′)

[0025] T abs =T base +Δt·unit_coe

[0026] Generate a sequential event chain Each event i Representation z at time t i (t) Updated by the gated network GRU:

[0027]

[0028] in, represents the initial embedding of time, represents the time difference encoding function;

[0029] S32: Calculate event using Softmax function i At time t j Impact event event j The probability P(event i |event j );

[0030] S33: Use the medical anatomy dictionary to identify the anatomical location of the event and obtain the event i Anatomical location encoding Location(event i ), and then construct the adjacency matrix A based on this location ∈{0,1} N×N ;

[0031] S34: Construct a spatiotemporal text disease network, integrate spatial and temporal features, and use GCN to learn the embedding matrix of all events:

[0032]

[0033] Among them, ⊙ represents element-by-element multiplication, i.e., hadmard product, which integrates spatiotemporal constraints to generate a spatiotemporal event network G T-L ;

[0034] S35: After extracting the spatiotemporal event chain, the spatiotemporal graph convolutional network T-GCN is used to integrate the text knowledge network that integrates domain knowledge and the spatiotemporal event network G T-L Aggregate medical knowledge to obtain a dynamic text disease network

[0035] Furthermore, in step S33, the three-dimensional anatomical coordinate system x, y, and z axes are constructed based on the Terminologia Anatomica dictionary, which respectively represent the human body direction, anatomical level, and organ system, thereby obtaining the anatomical position code Location (event) i ).

[0036] An electronic device comprising a memory and a processor;

[0037] The memory is used to store computer programs;

[0038] The processor is configured to implement the method for constructing a medical visual disease network with enhanced spatiotemporal features as described in any one of the above items when executing the computer program.

[0039] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing a medical visual disease network with enhanced spatiotemporal features as described in any one of the above items.

[0040] The beneficial effects of this invention lie in its ability to accurately extract knowledge from medical texts and construct networks by combining external knowledge graphs with prior knowledge in relevant medical fields, mining multi-scale entities, relationships, and spatiotemporal events within the text. This technological breakthrough enables semantic mapping of specialized knowledge between medical texts and external knowledge graphs, resolving issues such as ambiguity. Furthermore, it synchronizes timestamps and spatial locations in the extraction of disease course events, enabling accurate association of spatiotemporal features and providing an effective and feasible solution for disease course management.

[0041] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0043] Figure 1 This is a flow chart of the multimodal medical text extraction method based on adaptive domain knowledge fusion. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0046] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0047] Example 1:

[0048] like Figure 1 As shown, the present invention provides a multimodal medical text extraction method based on adaptive domain knowledge fusion. Disease text data includes electronic medical records, physical examinations, surgical records, and omics data. These data contain many professional terms, and the various characteristics of diseases change over time and with changes in human organ structure. Therefore, this section proposes a domain knowledge-driven medical text knowledge extraction method, which mainly includes three aspects: text entity-relationship mining, professional knowledge linking, and medical spatiotemporal event chain extraction.

[0049] In order to effectively extract entities and relationships from clinical text data, the present invention first extracts relevant entities and semantic relationships from clinical structured and unstructured text. Medical terms in clinical data are ambiguous. For example, "LHV" can refer to left ventricular hypertrophy or lumbar spinal stenosis. There is also the problem of nested relationships. For example, left ventricular hypertrophy leads to diastolic dysfunction, which in turn causes heart failure. Therefore, for the text sequence X = [w1, w2, ..., w n ], generate context-related vector H through medical text pre-training, and then optimize the vector through the medical domain dictionary to generate H lex Then, the legal relationship between entity types is defined through the medical system nomenclature, and the corresponding adjacency matrix A is constructed. ontology ∈{0,1} |S|×|S| , where S is the set of entity types and |S| is the number of types. Graph convolutional neural network (GCN) is then used to fuse the neighborhood information of ontology constraints:

[0050]

[0051] Among them, G (0) =H lex is the initial input, is the learnable parameter of the first layer, and σ(·) is the activation function. Afterwards, the conditional random field (CRF) and Softmax are used to perform text entity recognition and relationship classification, generating the entity label sequence score function S(y) and the relationship probability P(r|e i ,e j ), thereby extracting the corresponding entities and relationships in the medical text and generating a preliminary text disease network

[0052] At the same time, clinical data exists in multi-source texts, such as medical records, health records, physical examinations, etc., which contain many heterogeneous terms, such as anaphora, coreference, etc., which require the introduction of external knowledge graphs. Eliminate the differences in term expression. Therefore, the present invention fuses KG and for Then generate it through the knowledge graph representation method TransH The corresponding embedding g={g1,g2,…,g n}, and then use the graph attention network to aggregate Through continuous training and network convergence, entities with similarity close to 1 can be merged, their terminology descriptions can be unified, and the The number of entities is

[0053] In addition, medical clinical texts contain temporal and spatial features, which often influence the development direction of the disease. Therefore, the present invention extracts event sequences with temporal order and spatial location from clinical texts to construct a disease dynamic evolution model. First, regular expressions are used to extract the temporal features in medical texts, and then a temporal event chain is generated. Each event i Representation z at time t i (t) Updated by Gated RU:

[0054]

[0055] in, represents the initial embedding of time, Represents the time difference encoding function. Then, the Softmax function is used to calculate the event event i At time t j Impact event event j The probability P(event i |event j After processing the temporal relationship, the medical anatomy dictionary is used to identify the anatomical location of the event and obtain the event iAnatomical location encoding Location(event i ), and then construct the adjacency matrix A based on this location ∈{0,1} N×N After that, spatial and temporal features are fused and GCN is used to learn the embedding matrix of all events:

[0056]

[0057] Among them, ⊙ represents element-by-element multiplication, i.e., hadmard product, which integrates spatiotemporal constraints to generate a spatiotemporal event network G T-L After extracting the spatiotemporal event chain, the text knowledge network that integrates domain knowledge and the spatiotemporal event network G T-L Aggregate medical knowledge and generate HHD text disease knowledge network

[0058] Example 2:

[0059] In this embodiment, the above technical solution includes the following steps:

[0060] Step 1: This example uses the MIMIC-III clinical text dataset. To address the difficulty of extracting knowledge from freely annotated medical text, this stage uses an entity relationship mining method enhanced by domain knowledge. First, BioBERT-large (fine-tuned based on PubMed+PMC corpus) is used while adding a domain dictionary embedding (SNOMED CT ontology dictionary) to the input layer, thereby generating a domain dictionary-enhanced medical text sequence vector H. lex Then, the legal relationship between entities in the text sequence is defined using the medical system nomenclature, such as anatomical structure-pathological changes, symptoms-complications, treatment-contraindications, etc., so as to construct the corresponding graph and generate the adjacency matrix A. ontology Secondly, a two-layer graph convolutional neural network is used to integrate ontological constraints. The loss function uses the cross entropy of weighted nodes and edges. After training the graph neural network, an initial text disease map can be generated.

[0061] Step 2: To address the issues of complicated professional terminology, coreference, and anaphora in medical texts, we use external medical knowledge graphs (such as Hetionet) for knowledge alignment, and embed the knowledge graph using the TransH model. Then, we use the cross-graph bilinear attention mechanism to calculate the similarity between nodes and generate Finally, yes A simplified verification was performed to ensure that there were no isolated nodes in the structure, and that the semantics met the requirements after manual verification (200 random nodes were selected to confirm that they had a high accuracy rate).

[0062] Step 3: To address the problem of implicit spatiotemporal logic in disease text data, use regular expression templates:

[0063] T abs =Regex(r′\d4-\d2-\d2′)

[0064] T rel =Regex(r′\d+(day|monther|year)(before|after)′)

[0065] T abs =T base +Δt·unit_coe

[0066] Extracting a chain of temporal events The event chain is encoded through the network; then, spatial position encoding is performed, and the three-dimensional anatomical coordinate system x, y, and z axes are constructed based on the Terminologia Anatomica dictionary, representing the human body direction, anatomical level, and organ system respectively. Thus, the anatomical position encoding Location (event i ); Secondly, construct a spatiotemporal text disease network, whose nodes are +Spatiotemporal event nodes, using the spatiotemporal graph convolutional network T-GCN to combine spatiotemporal features with The characteristics of the medical knowledge are aggregated, and finally a dynamic text disease network is obtained. Effectively manage heterogeneous medical texts and improve the efficiency of data management in the medical industry.

[0067] In the above embodiments, references to "this embodiment" in the specification indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily refer to the same embodiment.

[0068] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the present invention are intended to encompass all such alternatives, modifications, and variations that fall within the broad scope of the appended claims.

[0069] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0070] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0071] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0072] Regarding the computer-readable storage medium in this embodiment, those skilled in the art will appreciate that all or part of the steps in the aforementioned method embodiments can be implemented using hardware associated with the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps in the aforementioned method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0073] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used for communication, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes the various steps of the above method.

[0074] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0075] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0076] The present invention can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above.

[0077] The present invention may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal medical text extraction method based on adaptive domain knowledge fusion, characterized by: The following steps are involved: Text entity-relationship mining: extract relevant entities and semantic relationships from clinical structured and unstructured texts to generate preliminary text disease networks; Professional knowledge linking: using external medical knowledge graphs for knowledge alignment; Medical spatiotemporal event chain extraction: Extract event sequences with temporal order and spatial location from clinical texts, build dynamic text disease knowledge networks, and extract multimodal medical texts.

2. The multimodal medical text extraction method based on adaptive domain knowledge fusion according to claim 1 is characterized by: The text entity-relationship mining includes the following steps: S11: For the text sequence X=[w1,w2,…,w n ], generate context-related vector H through medical text pre-training, and then optimize the vector through the medical domain dictionary to generate H lex ; S12: Define the legal relationships between entity types through medical system nomenclature and construct the corresponding adjacency matrix A ontology ∈{0,1} |S|×|S| , where S is the set of entity types and |S| is the number of types; S13: Using graph convolutional neural network (GCN) to fuse neighborhood information constrained by ontology: Among them, G (0) =H lex is the initial input, is the learnable parameter of the lth layer, σ(·) is the activation function; the loss function uses the cross entropy of the weighted design of nodes and edges; S14: Use conditional random field CRF and Softmax to perform text entity recognition and relationship classification, generate entity label sequence score function S(y) and relationship probability P(r|e i ,e j ), thereby extracting the corresponding entities and relationships in the medical text and generating a preliminary text disease network 3. The multimodal medical text extraction method based on adaptive domain knowledge fusion according to claim 2 is characterized by: In step S11, BioBERT-large is used, fine-tuned based on PubMed+PMC corpus, and the domain dictionary embedding is added to the input layer to generate the domain dictionary enhanced medical text sequence vector H lex .

4. The multimodal medical text extraction method based on adaptive domain knowledge fusion according to claim 1 is characterized by: The professional knowledge link specifically includes the following steps: S21: Introducing external medical knowledge graphs The cross-graph bilinear attention mechanism is used to calculate the similarity between nodes, integrating KG and for S22: Generated by knowledge graph representation method TransH The corresponding embedding g={g1,g2,…,g n }; S23: Aggregation using graph attention networks Through continuous training and network convergence, the entities with similarity close to 1 are merged, their terminology descriptions are unified, and the The number of entities is 5. The multimodal medical text extraction method based on adaptive domain knowledge fusion according to claim 1 is characterized by: The medical spatiotemporal event chain extraction specifically includes the following steps: S31: Use regular expressions to extract time features from medical texts. The regular expression template is: T abs =Regex(r′ \ d4-\d2-\d2′) T rel =Regex(r ′\ d+(day|monther|year)(before|after)′) T abs =T base +Δt·unit_coe Generate a sequential event chain Each event i Representation z at time t i (t) Updated by the gated network GRU: in, represents the initial embedding of time, represents the time difference encoding function; S32: Calculate event using Softmax function i At time t j Impact event event j The probability P(event i |event j ); S33: Use the medical anatomy dictionary to identify the anatomical location of the event and obtain the event i Anatomical location encoding Location(event i ), and then construct the adjacency matrix A based on this location ∈{0,1} N×N ; S34: Construct a spatiotemporal text disease network, integrate spatial and temporal features, and use GCN to learn the embedding matrix of all events: Among them, ⊙ represents element-by-element multiplication, i.e., hadmard product, which integrates spatiotemporal constraints to generate a spatiotemporal event network G T-L ; S35: After extracting the spatiotemporal event chain, the spatiotemporal graph convolutional network T-GCN is used to integrate the text knowledge network that integrates domain knowledge and the spatiotemporal event network G T-L Aggregate medical knowledge to obtain a dynamic text disease network 6. The multimodal medical text extraction method based on adaptive domain knowledge fusion according to claim 5 is characterized by: In step S33, the three-dimensional anatomical coordinate system x, y, and z axes are constructed based on the Terminologia Anatomica dictionary, which respectively represent the human body direction, anatomical level, and organ system, thereby obtaining the anatomical position code Location (event) i ).

7. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the method for constructing a medical visual disease network with enhanced spatiotemporal features as described in any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method for constructing a medical visual disease network with enhanced spatiotemporal features as described in any one of claims 1 to 6 is implemented.

Images