Multi-modal medical knowledge graph construction method and device and computer equipment

Abstract

The invention provides a multi-modal medical knowledge graph construction method and device and computer equipment, and relates to the technical field of medical knowledge graph construction, and the method comprises the steps: obtaining multi-modal medical data; the multi-modal medical data comprises a multi-modal medical data subset of a plurality of medical keywords; preprocessing the multi-modal medical data to obtain multi-modal text data of each medical keyword; based on a preset medical knowledge extraction model, obtaining a structured medical data set of each medical keyword according to the multi-modal text data of each medical keyword; the medical knowledge extraction model is a model obtained by training based on a pre-training model; and constructing a multi-modal medical knowledge graph based on the structured medical data set. According to the method, the construction of the multi-modal medical knowledge graph can be realized in a non-labeled medical data scene.

Description

Technical Field

[0001] The present invention relates to the technical field of medical knowledge graph construction, and specifically to a multimodal medical knowledge graph construction method, device and computer equipment. Background Art

[0002] Medical data comes from a wide range of sources, including but not limited to clinical records, medical images, biological specimens, and genomic data. Therefore, medical data usually presents multimodal characteristics, including text, images, videos, and audio. It is very important to effectively integrate and fuse these multimodal medical data in a unified architecture to build a unified medical knowledge graph.

[0003] However, due to the different data types and formats of multimodal medical data, there are multiple forms such as structured and unstructured, text and multimedia, resulting in high diversity and complexity of data, making it difficult for existing knowledge graph structures to represent multimodal medical data. In addition, the acquisition and annotation of medical data usually requires manual processing by professional medical personnel, which is costly and time-consuming.

[0004] Based on this, how to realize the construction of multimodal medical knowledge graph in the scenario of unlabeled medical data has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, in order to solve the above-mentioned technical problems, the present invention provides a method, device and computer equipment for constructing a multimodal medical knowledge graph.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for constructing a multimodal medical knowledge graph, comprising:

[0008] Acquire multimodal medical data; the multimodal medical data includes a multimodal medical data subset of multiple medical keywords;

[0009] Preprocessing the multimodal medical data to obtain multimodal text data of each of the medical keywords;

[0010] Based on a preset medical knowledge extraction model, a structured medical data group of each medical keyword is obtained according to the multimodal text data of each medical keyword; the medical knowledge extraction model is a model trained based on a pre-trained model;

[0011] Based on the structured medical data set, a multimodal medical knowledge graph is constructed.

[0012] Optionally, the multimodal medical data subset includes Internet search results corresponding to the medical keywords, and the data form of the Internet search results includes pictures, videos and texts.

[0013] Optionally, the preprocessing of the multimodal medical data to obtain multimodal text data of each medical keyword specifically includes:

[0014] For each of the medical keywords, a five-tuple of data mapping relationships of the medical keywords is generated according to the Internet search results of the medical keywords; the five-tuple of data mapping relationships includes a video identifier, a picture identifier, a text identifier, the medical keyword and a data source; the data source is used to indicate the source of the Internet search results of the medical keywords;

[0015] Extracting audio text from the video corresponding to the video identifier to obtain the audio text of the video;

[0016] Processing the image corresponding to the image identifier based on OCR technology to obtain an image description of the image;

[0017] According to the data mapping relationship quintuple, the audio text and the picture description, multimodal text data of the medical keywords are generated, and the multimodal text data includes the medical keywords, the text identifier, the audio text and the picture description.

[0018] Optionally, the training process of the medical knowledge extraction model includes:

[0019] Acquire a model training sample set; the model training sample set includes a plurality of sample data, and the sample data is a multimodal text including a plurality of medical entities and the relationship between the medical entities;

[0020] Performing iterative model training using the model training sample set to obtain an initial medical knowledge extraction model;

[0021] Transfer learning is performed on the initial medical knowledge extraction model to obtain the medical knowledge extraction model.

[0022] Optionally, the using the model training sample set to perform model iterative training to obtain an initial medical knowledge extraction model specifically includes:

[0023] The sample data is used to perform the following model training process in sequence until the preset end training condition is reached:

[0024] Perform context representation on the sample data using a pre-trained model to obtain a context representation result;

[0025] According to the context representation result, a context representation tensor is obtained, and then an attention weight matrix is ​​obtained;

[0026] Embedding the attention weight matrix into an entity location;

[0027] Expanding the dimension of the context representation tensor to obtain an entity relationship matrix; the entity relationship matrix is ​​used to reflect the relationship between medical entities;

[0028] Multiplying the entity relationship matrix and the attention weight matrix to obtain an entity relationship score matrix;

[0029] Obtaining medical knowledge extraction results according to the entity relationship score matrix;

[0030] Determining a loss value corresponding to the medical knowledge extraction result according to a pre-constructed ontology layer;

[0031] Model parameters are adjusted based on the loss value.

[0032] Optionally, the ontology layer includes multiple types of medical entity categories and multiple types of relationship categories.

[0033] Optionally, constructing a multimodal medical knowledge graph based on the structured medical data set specifically includes:

[0034] For each structured medical data group, determining target multimodal source data and target data source corresponding to the structured medical data group; the target multimodal source data is used to reflect which data the structured medical data group is obtained by performing knowledge extraction on;

[0035] Integrating the structured medical data set, the target multimodal source data, and the target data source to obtain an integrated data set;

[0036] Using a preset graphic database to store the integrated data set;

[0037] A connection is established between the preset graph database and the InteractiveGraph framework.

[0038] Optionally, the structured medical data set includes a head medical entity, a head medical entity type, a relationship, a tail medical entity and a tail medical entity type.

[0039] In a second aspect, the present invention provides a multimodal medical knowledge graph construction device, comprising:

[0040] An acquisition module, configured to acquire multimodal medical data; the multimodal medical data includes a multimodal medical data subset of a plurality of medical keywords;

[0041] A preprocessing module, used for preprocessing the multimodal medical data to obtain multimodal text data of each medical keyword;

[0042] An extraction module, used to obtain a structured medical data group of each medical keyword based on the multimodal text data of each medical keyword based on a preset medical knowledge extraction model; the medical knowledge extraction model is a model trained based on a pre-trained model;

[0043] A construction module is used to construct a multimodal medical knowledge graph based on the structured medical data group.

[0044] In a third aspect, the present invention provides a computer device, comprising:

[0045] at least one processor; and,

[0046] a memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the multimodal medical knowledge graph construction method as described above.

[0048] The present invention adopts the above technical scheme, a method for constructing a multimodal medical knowledge graph, including: obtaining multimodal medical data; the multimodal medical data includes a multimodal medical data subset of multiple medical keywords; preprocessing the multimodal medical data to obtain multimodal text data of each medical keyword; based on a preset medical knowledge extraction model, obtaining a structured medical data group of each medical keyword according to the multimodal text data of each medical keyword; based on the structured medical data group, constructing a multimodal medical knowledge graph. Based on this, by taking the medical keywords as the hub, preprocessing the multimodal medical data to obtain the multimodal text data of each medical keyword, and establishing an initial data link relationship. Then, using pre-training technology, fully explore the potential correlation between medical data to obtain a structured medical data group of each medical keyword. Finally, based on the structured medical data group, construct a multimodal medical knowledge graph. In this way, the present invention can realize the construction of a multimodal medical knowledge graph in a scenario without labeled medical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 It is a flowchart of a method for constructing a multimodal medical knowledge graph provided by an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of a multimodal medical data processing flow provided by an embodiment of the present invention;

[0052] Figure 3 This is a general architecture diagram of the pre-trained model;

[0053] Figure 4 It is a structural schematic diagram of a multimodal medical knowledge graph construction device provided by an embodiment of the present invention;

[0054] Figure 5 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0056] Example

[0057] Figure 1 FIG. 1 is a flow chart of a method for constructing a multimodal medical knowledge graph provided by an embodiment of the present invention. Figure 1 As shown, the multimodal medical knowledge graph construction method includes:

[0058] Step 101: Acquire multimodal medical data; the multimodal medical data includes a multimodal medical data subset of multiple medical keywords.

[0059] Specifically, the multimodal medical data subset includes Internet search results corresponding to medical keywords and their associated metadata, wherein the data form of the Internet search results includes pictures, videos and texts, and the metadata includes the corresponding medical keywords and the corresponding data source, and the data source is used to indicate the source of the Internet search results corresponding to the medical keywords, for example, the data source may be a related website.

[0060] Here, when acquiring multimodal medical data, we first define a keyword set, and then crawl the search engine data according to the keywords, including multimodal data such as disease introduction diagrams, popular science videos, and related text descriptions. The specific steps are as follows:

[0061] (1) Obtain a medical keyword set; the medical keyword set is a keyword set constructed based on MedDRA (Medical Dictionary for Regulatory Activities) and UMLS (Unified Medical Language System).

[0062] (2) For each medical keyword in the medical keyword set, a search result list page corresponding to the medical keyword is obtained based on a web crawler tool and an Internet search engine. The search result list page contains picture information, video introduction information, and ordinary text information of the medical keyword.

[0063] (3) Using an HTML (HyperText Markup Language) parser to parse the search result list page, and extracting a preset number of Internet search results. The preset number of Internet search results include Internet search results in the form of images, Internet search results in the form of videos, and Internet search results in the form of text.

[0064] In a specific example, taking the search for a certain disease as an example, by clicking on the search result, you can see the disease introduction information containing multimodal data, including a plain text introduction to the disease, a picture introduction to the disease, and a popular science video introduction to the disease.

[0065] (4) Storing the extracted Internet search results of the first preset number of items and their associated metadata. Specifically, downloading the Internet search results in the form of images and the Internet search results in the form of videos to a local computer, and storing the Internet search results in the form of text to a local computer, and storing the corresponding metadata, thereby obtaining a multimodal medical data subset of the current medical keyword.

[0066] Step 102: Preprocess the multimodal medical data to obtain multimodal text data of each medical keyword.

[0067] Figure 2 It is a schematic diagram of a multimodal medical data processing flow provided by an embodiment of the present invention. Among them, HAS and CONTAINS respectively represent different meanings of inclusion, wherein HAS specifically refers to which modal data the Internet search results of medical keywords have, for example, the Internet search results of medical keywords include Internet search results in the form of pictures, Internet search results in the form of videos, and Internet search results in the form of texts. CONTAINS specifically refers to a certain entity or category containing another entity or category, for example, a video contains audio text.

[0068] IS-A, PART-OF and RELATED-TO represent relationships with different meanings. Taking A and B as an example, IS-A means that A is B, PART-OF means that A is a type of B, and RELATED-TO means that A and B are related.

[0069] refer to Figure 2 , preprocess the multimodal medical data to obtain multimodal text data of each medical keyword, which may specifically include:

[0070] (1) For each medical keyword, a five-tuple of data mapping relationships of the medical keyword is generated according to the Internet search results of the medical keyword; the five-tuple of data mapping relationships includes a video identifier, an image identifier, a text identifier, a medical keyword, and a data source; the data source is used to represent the source of the Internet search results of the medical keyword.

[0071] Here, the identifier may specifically be an ID (Identity document). Based on this, with the medical keyword as the pivot, the initial data mapping relationship quintuple <video ID, picture ID, text ID, medical keyword, data source> can be obtained.

[0072] (2) Extract the audio text of the video corresponding to the video identifier to obtain the audio text of the video.

[0073] Specifically, video data is multimodal information, including audio and image information, which needs to be extracted and converted into corresponding structured data. The inventors analyzed the medical introduction videos obtained and found that most of the key frame information of medical introduction videos is single, for example, the whole process is just a doctor's expert's description of the disease, and there is no medical image display in the video. Therefore, it is only necessary to extract the audio information of such videos.

[0074] When extracting audio text from a video corresponding to a video identifier, video-to-text conversion can be implemented based on the Video Description item to obtain an audio text, and further a triplet <video ID, audio ID, audio text> can be defined.

[0075] (3) Processing the image corresponding to the image identifier based on OCR (Optical Character Recognition) technology to obtain an image description of the image.

[0076] Specifically, the image data is processed, and the text description and entity information in the image are extracted based on the OCR technology to form a tuple <image ID, image description>.

[0077] (4) Generate multimodal text data of medical keywords based on the data mapping relationship quintuple, audio text, and picture description. The multimodal text data includes medical keywords, text identifiers, audio text, and picture description.

[0078] Specifically, the audio text is used to replace the corresponding video in the data mapping relationship quintuple, and the picture description is used to replace the corresponding picture in the data mapping relationship quintuple, so as to obtain multimodal text data <medical keywords, text ID, audio text, picture description> with medical keywords as the hub, which will be used as the data source for the subsequent medical knowledge extraction model until the medical entity and relationship structured data are finally extracted.

[0079] Step 103: Based on a preset medical knowledge extraction model, a structured medical data group of each medical keyword is obtained according to the multimodal text data of each medical keyword; the medical knowledge extraction model is a model trained based on a pre-trained model.

[0080] Specifically, the structured medical data set may include a head medical entity, a head medical entity type, a relationship, a tail medical entity, and a tail medical entity type.

[0081] It should be noted that a medical keyword corresponds to a set of multimodal text data (i.e. <medical keyword, text ID, audio text, image description>), and based on the preset medical knowledge extraction model, according to the multimodal text data of the medical keyword, at least one structured medical data group of the medical keyword can be obtained (i.e. <head medical entity, head medical entity type, relationship, tail medical entity, tail medical entity type>).

[0082] Step 104: Construct a multimodal medical knowledge graph based on the structured medical data set.

[0083] Here, based on the structured medical data set, a multimodal medical knowledge graph is constructed, which may specifically include:

[0084] (1) For each structured medical data group, determine the target multimodal source data and the target data source corresponding to the structured medical data group; the target multimodal source data is used to reflect the data from which the structured medical data group is obtained by knowledge extraction.

[0085] (2) Integrate the structured medical data set, the target multimodal source data, and the target data source to obtain an integrated data set.

[0086] Specifically, from the foregoing content, it can be seen that the structured medical data group can be <head medical entity, head medical entity type, relationship, tail medical entity, tail medical entity type>. Based on this, after determining the multimodal source data and data source corresponding to the structured medical data group, it can be integrated to obtain the integrated data group <head medical entity, head medical entity type, relationship, tail medical entity, tail medical entity type, multimodal source data, data source>.

[0087] Among them, the multimodal source data is used to reflect the data from which the structured medical data group is obtained by knowledge extraction. For example, this structured medical data group is obtained by knowledge extraction of the target audio text, and the multimodal source data is used to reflect the target audio text. Specifically, the multimodal source data can be the ID of the target audio text.

[0088] (3) Use a preset graphical database to store the integrated data set.

[0089] Specifically, in order to store the integrated data set in the form of files in the hard disk of the server and establish a multimodal data index associated with structured medical knowledge, the preset graph database can be a Neo4J graph database.

[0090] Specifically, the processed integrated data set is imported into Neo4J, and basic database functions such as adding, deleting, modifying and querying data can be realized by using Neo4J's unique query language CQL. For structured medical knowledge, this solution uses Neo4J as the knowledge graph storage engine, which can provide users with a more convenient and efficient knowledge management and application experience.

[0091] (4) Establish a connection between the preset graph database and the InteractiveGraph framework.

[0092] Specifically, InteractiveGraph is a powerful open source framework that aims to provide Web-based interactive operation tools for large graph data. This framework greatly simplifies the visualization of graph data and the interaction between users and graph data. InteractiveGraph provides GraphNavigator (graph navigator), GraphExplorer (graph browser) and RelationFinder (relationship finder) functions. After establishing a connection between the preset graph database and the InteractiveGraph framework, you can design pages that can realize user interaction functions to realize node retrieval, path retrieval and global browsing functions.

[0093] In an embodiment of the present invention, the training process of the medical knowledge extraction model may include:

[0094] (1) Obtain a model training sample set; the model training sample set includes multiple sample data, and the sample data is a multimodal text including multiple medical entities and relationships between medical entities.

[0095] In a specific example, a piece of sample data may be: diabetes requires insulin treatment, and patients with diabetes may also have a history of hypertension.

[0096] (2) Use the model training sample set to perform iterative model training to obtain the initial medical knowledge extraction model.

[0097] Specifically, the model training sample set is used to perform iterative model training to obtain an initial medical knowledge extraction model, which may include:

[0098] Use sample data to perform the following model training process in sequence until the preset end training condition is reached:

[0099] (2.1) Use the pre-trained model to perform contextual representation on the sample data and obtain the contextual representation result.

[0100] Specifically, in order to better extract information between multimodal text data, a pre-trained model is used to build a knowledge representation layer. Since the pre-trained model has learned common features on a large-scale data set, the model can capture more high-level and abstract semantic information than a general network model. Specifically, a pre-trained model and a linear mapping method can be used to convert a text sequence (i.e., sample data) into a contextual representation result of a preset dimension. Among them, in order to map the representation to the appropriate dimension, a linear layer (also called a fully connected layer or a dense layer) can be applied. This linear layer will receive the original representation generated by the pre-trained model as input and output a new representation with the desired dimension. This step is optional, but in some cases, it can help adjust the dimension of the representation to suit a specific downstream task.

[0101] Figure 3 This is a general architecture diagram of the pre-trained model. Figure 3 The pre-trained model uses character-level n-grams to represent a word, integrating the morphological features within the word. The core idea is to average the word and n-gram vectors of the entire text to get the text vector, and then use the text vector for softmax multi-classification.

[0102] In the present invention, the output of the last hidden layer of the pre-trained model is used as the context representation result of the multimodal medical text (i.e., the context representation result).

[0103] The pre-trained model can be an existing BERT or ERNIE, etc.

[0104] (2.2) Based on the context representation result, the context representation tensor X is obtained, and then the attention weight matrix Z is obtained.

[0105] Specifically, the attention weight matrix is ​​used to make the model pay more attention to the sequence that is effective for the result. The present invention realizes the mapping of the context representation tensor to the entity or relationship type space by constructing the attention weight matrix, and finally converts it into a matrix representing the medical entity relationship score. According to the context representation result, the context representation tensor X is obtained, and then the specific implementation method of obtaining the attention weight matrix Z is as follows:

[0106] (2.2.1) Prepare input data

[0107] Specifically, according to the context representation result generated by the pre-trained model in the early stage, a context representation tensor X is generated, where each row of X is a word embedding representation, that is, the context representation result, the sequence length of X is represented by n, and each word embedding dimension is represented by d.

[0108] (2.2.2) Linear transformation

[0109] Specifically, the context representation tensor X is transformed three times linearly to generate the query Q (Query), key K (Key) and value V (Value) matrices. Q , W K and W V is the weight parameter matrix. The linear transformation formula is as follows:......(1)

[0110] Q=XW Q ......(1)

[0111] K=XW K ......(2)

[0112] V=XW V ......(3)

[0113] Among them, the dimensions of Q, K and V are (n, d k ), where d k Usually equal to d.

[0114] (2.2.3) Calculate attention score

[0115] Specifically, the dot product between the query Q and the key K is calculated to obtain the attention score matrix A, that is:

[0116] A=QK T / sqrt(d k )......(4)

[0117] Then, the attention score matrix A is scaled to make the value of any element in the attention score matrix A within a preset range, so as to prevent the value of the element in the attention score matrix A from being too large.

[0118] (2.2.4) Apply Softmax

[0119] Specifically, the Softmax function is applied to each row of the attention score matrix A to obtain the matrix α, that is:

[0120] α=softmax(A)......(5)

[0121] Among them, the Softmax function is used to convert the score into a probability distribution.

[0122] (2.2.5) Calculate the weighted sum

[0123] Specifically, the matrix α is used to perform weighted summation on the value matrix V to obtain the attention weight matrix Z, that is:

[0124] Z=αV......(6)

[0125] In the present invention, the final matrix Z is defined as a constructed attention weight matrix, which is used to focus the model's attention on the knowledge extraction task in the input sequence, which helps the model better capture key contextual information, thereby improving the performance of entity relationship extraction.

[0126] (2.3) Embed the attention weight matrix into entity positioning.

[0127] Specifically, first construct the entity positioning matrix w emb , Sine position embedding is a method of representing position information, generated by sine and cosine functions. For each position γ and each dimension i of the text, the embedded value is calculated as follows:

[0128]

[0129] Where d represents the output dimension. The generated embedding matrix w emb Reflects the relative relationship between text positions.

[0130] Then, the embedding matrix w emb Used to adjust the attention weight matrix Z to obtain the adjusted attention weight matrix Z. The formula is as follows:

[0131] Z=Z⊙sin(w emb )+w q′ ⊙cos(w emb )......(8)

[0132] Among them, ⊙ represents element-by-element multiplication; the formula means stacking the odd and even columns in the attention weight matrix Z on the last dimension.

[0133] Finally, the attention weight matrix Z obtained after entity localization embedding adjustment enables the weight matrix to better capture the relative position information between elements in the input sequence, which helps to model the relationship between entities more accurately.

[0134] (2.4) The context representation tensor X is dimensionally expanded to obtain the entity relationship matrix R; the entity relationship matrix is ​​used to reflect the relationship between medical entities.

[0135] Specifically, according to the definition of the entity relationship matrix, the dimension of the multimodal text context representation tensor X obtained previously is expanded to construct the corresponding entity relationship matrix R, so that each row and column of R represents an entity, and the elements of R represent the relationship between entities.

[0136] In a concrete example, suppose we have the following entities and relationships:

[0137] Entities: E1 = "Diabetes", E2 = "Insulin", E3 = "Hypertension".

[0138] Relationship: R1 = "treats", R2 = "associates with".

[0139] The corresponding entity relationship matrix R can be intuitively displayed using the following Table 1.

[0140] Table 1

[0141] diabetes insulin hypertension diabetes 0 treat Relationship insulin treat 0 0 hypertension Relationship 0 0

[0142] (2.5) Multiply the entity relationship matrix and the attention weight matrix to obtain the entity relationship score matrix w score ,Right now:

[0143] w score =R·Z...... (9)

[0144] Among them, w score Each element (i, j, k, l) in represents the relationship score between the j-th entity type at the k-th position and the l-th position in the input sequence in the i-th multimodal text sample.

[0145] Furthermore, the score can be optimized by combining the attention mask mechanism and the lower triangle mask optimization mechanism to filter out invalid position information and enhance the attention of the current position model.

[0146] In addition, an entity relationship score matrix is ​​constructed. Each element value in the matrix represents the score of the corresponding entity relationship. By setting the score threshold, the joint extraction of entity relationships in multimodal text can be achieved.

[0147] (2.6) Based on the entity relationship score matrix, the medical knowledge extraction results are obtained.

[0148] Specifically, the medical knowledge extraction result can be obtained through normalization and decoding, and the decoder is the decoder corresponding to the pre-trained model.

[0149] Furthermore, when the structured medical data group output by the medical knowledge extraction model is in the form of <head medical entity, head medical entity type, relationship, tail medical entity, tail medical entity type>, the form of the medical knowledge extraction result is also <head medical entity, head medical entity type, relationship, tail medical entity, tail medical entity type>.

[0150] (2.7) Based on the pre-constructed ontology layer, determine the loss value corresponding to the medical knowledge extraction result.

[0151] Specifically, the ontology layer can be constructed using the OWL semantic standard based on the Protégé ontology construction tool.

[0152] The ontology layer contains multiple medical entity categories and multiple relationship categories. In a specific example, the ontology layer is expressed in the form of "concept set + relationship set", and 11 medical entity categories are constructed, namely: disease, symptoms, drugs, site, prognosis, sociology, surgical treatment, other treatments, examination, epidemiology, and others. In addition, 55 relationship categories are constructed, namely: complications, pathological classification, related (cause), related (transformation), related (symptoms), differential diagnosis, synonyms, clinical manifestations, symptoms after treatment, symptoms of invasion and metastasis to surrounding tissues, drug treatment, site of onset, metastasis site, invasion site, prognosis, prognosis survival rate, etiology, high-risk factors, risk assessment factors, medical history, genetic factors, pathogenesis, pathophysiology, surgical treatment, adjuvant therapy, chemotherapy, radiotherapy, laboratory tests, imaging examinations, auxiliary examinations, histological examinations, endoscopy, screening, common groups, incidence, age of onset, common areas, gender tendency of onset, mortality, transmission route, common season, prevention, stage, treatment department, and 11 types of synonym relationships.

[0153] (2.8) Adjust model parameters based on the loss value.

[0154] (3) Perform transfer learning on the initial medical knowledge extraction model to obtain a medical knowledge extraction model.

[0155] Specifically, in order to enable the medical knowledge extraction model to integrate more medical knowledge base information, the constructed medical knowledge extraction model will be trained in advance on more external medical knowledge bases, and then the trained model parameters will be migrated to the target domain, that is, the final knowledge extraction task will be completed in multimodal text (unlabeled medical video text scenario).

[0156] The specific implementation process of transferring learning the initial medical knowledge extraction model to obtain the medical knowledge extraction model includes:

[0157] (3.1) Obtain transfer learning sample data through external knowledge bases, which can include the medical CKBP (Commonsense Knowledge Base Population) dataset, CHIP (Chinese Health Information Processing Conference) dataset, and i2b2 dataset. The texts of these datasets come from unstructured medical text data such as medical textbooks, clinical practice, and electronic medical records.

[0158] The difference between the transfer learning sample data and the aforementioned model training sample set is that the data sources are different, but both contain multiple sample data, and the sample data is multimodal text containing multiple medical entities and the relationships between medical entities.

[0159] (3.2) Fine-tune the parameters of the initial medical knowledge extraction model on each sample data of the transfer learning sample data to ensure that the medical knowledge extraction model integrates the medical knowledge covered by different data sets and can perform better in the knowledge extraction task.

[0160] Finally, we obtained a medical knowledge extraction model that integrates multiple medical knowledge. The model parameters were migrated to the unlabeled medical video text scenario, which can complete the task of extracting the relationship between entities. It provides strong support for information extraction tasks in the medical field and lays the foundation for the subsequent construction of a medical knowledge graph.

[0161] It should be noted that the specific implementation process of transfer learning of the initial medical knowledge extraction model is the same or similar to the aforementioned model iteration training process. For the specific implementation method, please refer to the relevant description of the aforementioned model iteration process.

[0162] The present invention adopts the above-mentioned technical scheme. The medical knowledge extraction model is based on a pre-trained model as the context representation basis, and then defines an entity relationship score matrix, adds an entity positioning embedding mechanism, considers the positional relationship of the text, and calculates the relative position score, so that the medical knowledge extraction model can more accurately extract the entity and relationship concepts of the text, and the trained medical knowledge extraction model can perform knowledge extraction on unlabeled medical text data to obtain a structured medical data group used to construct a knowledge graph.

[0163] In the process of constructing a multimodal medical knowledge graph, by taking medical keywords as the hub, data of different modalities are converted to obtain multimodal text data of each medical keyword, and an initial data link relationship is established. Then, pre-training technology is used to fully explore the potential correlation between medical data and obtain a structured medical data group for each medical keyword. Finally, based on the structured medical data group, a multimodal medical knowledge graph is constructed, and visualization and complex retrieval functions are realized. In this way, the present invention can realize the construction of a multimodal medical knowledge graph in an unlabeled medical data scenario, and realize the visualization interaction and retrieval application of a multimodal medical knowledge graph, and then realize multimodal medical knowledge retrieval and reasoning, which is conducive to improving the accuracy and personalization of medical decision-making, and improving the quality and efficiency of medical diagnosis in complex multimodal data scenarios.

[0164] Based on a general inventive concept, the present invention also provides a multimodal medical knowledge graph construction device.

[0165] Figure 4 Schematic diagram of a multimodal medical knowledge graph construction device provided by an embodiment of the present invention. Figure 4 As shown, the multimodal medical knowledge graph construction device includes:

[0166] The acquisition module 41 is used to acquire multimodal medical data; the multimodal medical data includes a multimodal medical data subset of multiple medical keywords.

[0167] The preprocessing module 42 is used to preprocess the multimodal medical data to obtain multimodal text data of each medical keyword.

[0168] The extraction module 43 is used to obtain a structured medical data group of each medical keyword based on the multimodal text data of each medical keyword based on a preset medical knowledge extraction model; the medical knowledge extraction model is a model trained based on a pre-trained model.

[0169] The construction module 44 is used to construct a multimodal medical knowledge graph based on the structured medical data set.

[0170] Optionally, the multimodal medical data subset includes Internet search results corresponding to the medical keywords, and the data form of the Internet search results includes pictures, videos and texts.

[0171] Optionally, the preprocessing module 42 may be specifically used for:

[0172] (1) For each of the medical keywords, a five-tuple of data mapping relationships of the medical keywords is generated according to the Internet search results of the medical keywords; the five-tuple of data mapping relationships includes a video identifier, a picture identifier, a text identifier, the medical keyword, and a data source; the data source is used to represent the source of the Internet search results of the medical keywords.

[0173] (2) Extracting audio text from the video corresponding to the video identifier to obtain the audio text of the video.

[0174] (3) Processing the image corresponding to the image identifier based on OCR technology to obtain an image description of the image.

[0175] (4) Generate multimodal text data of the medical keywords based on the data mapping relationship quintuple, the audio text and the picture description, wherein the multimodal text data includes the medical keywords, the text identifier, the audio text and the picture description.

[0176] Optionally, the multimodal medical knowledge graph construction device further includes a model training module, which specifically includes:

[0177] The acquisition unit is used to acquire a model training sample set; the model training sample set includes multiple sample data, and the sample data is a multimodal text including multiple medical entities and the relationship between the medical entities.

[0178] A training unit is used to perform iterative model training using the model training sample set to obtain an initial medical knowledge extraction model.

[0179] A transfer learning unit is used to perform transfer learning on the initial medical knowledge extraction model to obtain the medical knowledge extraction model.

[0180] Optional, training unit, specifically used for:

[0181] (1) Using the sample data, the following model training process is performed in sequence until the preset training end condition is reached.

[0182] (2) Use the pre-trained model to perform contextual representation on the sample data to obtain a contextual representation result.

[0183] (3) Based on the context representation result, a context representation tensor is obtained, and then an attention weight matrix is ​​obtained.

[0184] (4) Embedding the attention weight matrix into entity positioning.

[0185] (5) Expanding the dimension of the context representation tensor to obtain an entity relationship matrix; the entity relationship matrix is ​​used to reflect the relationship between medical entities.

[0186] (6) Multiply the entity relationship matrix and the attention weight matrix to obtain an entity relationship score matrix.

[0187] (7) Obtain medical knowledge extraction results based on the entity relationship score matrix.

[0188] (8) Determine the loss value corresponding to the medical knowledge extraction result based on the pre-constructed ontology layer.

[0189] (9) Adjusting model parameters based on the loss value.

[0190] Optionally, the ontology layer includes multiple types of medical entity categories and multiple types of relationship categories.

[0191] Optionally, the building block 44 is specifically used for:

[0192] (1) For each structured medical data group, determine the target multimodal source data and the target data source corresponding to the structured medical data group; the target multimodal source data is used to reflect the data from which the structured medical data group is obtained by knowledge extraction.

[0193] (2) Integrate the structured medical data set, the target multimodal source data, and the target data source to obtain an integrated data set.

[0194] (3) Using a preset graphic database to store the integrated data set.

[0195] (4) Establish a connection between the preset graph database and the InteractiveGraph framework.

[0196] Optionally, the structured medical data set includes a head medical entity, a head medical entity type, a relationship, a tail medical entity and a tail medical entity type.

[0197] Based on a general inventive concept, the present invention also provides a computer device.

[0198] Figure 5 Schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 5 As shown, the computer device 500 includes:

[0199] at least one processor 510; and,

[0200] A memory 530 is communicatively connected to at least one processor 510; wherein,

[0201] The memory 530 stores instructions 520 that can be executed by at least one processor, and the instructions 520 are executed by at least one processor 510 so that the at least one processor 510 can implement the multimodal medical knowledge graph construction method described in the above embodiment.

[0202] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0203] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0204] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0205] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0206] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0207] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0208] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0209] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0210] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for constructing a multimodal medical knowledge graph, characterized in that: include: Acquiring multimodal medical data; The multimodal medical data includes a multimodal medical data subset of a plurality of medical keywords; Preprocessing the multimodal medical data to obtain multimodal text data of each of the medical keywords; Based on a preset medical knowledge extraction model, a structured medical data group of each medical keyword is obtained according to the multimodal text data of each medical keyword; the medical knowledge extraction model is a model trained based on a pre-trained model; Based on the structured medical data set, a multimodal medical knowledge graph is constructed.

2. The method for constructing a multimodal medical knowledge graph according to claim 1, characterized in that: The multimodal medical data subset includes Internet search results corresponding to the medical keywords, and the data forms of the Internet search results include pictures, videos and texts.

3. The method for constructing a multimodal medical knowledge graph according to claim 2, characterized in that: The preprocessing of the multimodal medical data to obtain multimodal text data of each medical keyword specifically includes: For each of the medical keywords, a five-tuple of data mapping relationships of the medical keywords is generated according to the Internet search results of the medical keywords; the five-tuple of data mapping relationships includes a video identifier, a picture identifier, a text identifier, the medical keyword and a data source; the data source is used to indicate the source of the Internet search results of the medical keywords; Extracting audio text from the video corresponding to the video identifier to obtain the audio text of the video; Processing the image corresponding to the image identifier based on OCR technology to obtain an image description of the image; According to the data mapping relationship quintuple, the audio text and the picture description, multimodal text data of the medical keywords are generated, and the multimodal text data includes the medical keywords, the text identifier, the audio text and the picture description.

4. The method for constructing a multimodal medical knowledge graph according to claim 1, characterized in that: The training process of the medical knowledge extraction model includes: Acquire a model training sample set; the model training sample set includes a plurality of sample data, and the sample data is a multimodal text including a plurality of medical entities and the relationship between the medical entities; Performing iterative model training using the model training sample set to obtain an initial medical knowledge extraction model; Transfer learning is performed on the initial medical knowledge extraction model to obtain the medical knowledge extraction model.

5. The method for constructing a multimodal medical knowledge graph according to claim 4, characterized in that: The method of using the model training sample set to perform model iterative training to obtain an initial medical knowledge extraction model specifically includes: The sample data is used to perform the following model training process in sequence until the preset end training condition is reached: Perform context representation on the sample data using a pre-trained model to obtain a context representation result; According to the context representation result, a context representation tensor is obtained, and then an attention weight matrix is ​​obtained; Embedding the attention weight matrix into an entity location; Expanding the dimension of the context representation tensor to obtain an entity relationship matrix; the entity relationship matrix is ​​used to reflect the relationship between medical entities; Multiplying the entity relationship matrix and the attention weight matrix to obtain an entity relationship score matrix; Obtaining medical knowledge extraction results according to the entity relationship score matrix; Determining a loss value corresponding to the medical knowledge extraction result according to a pre-constructed ontology layer; Model parameters are adjusted based on the loss value.

6. The method for constructing a multimodal medical knowledge graph according to claim 5, characterized in that: The ontology layer includes multiple medical entity categories and multiple relationship categories.

7. The method for constructing a multimodal medical knowledge graph according to claim 1, characterized in that: The step of constructing a multimodal medical knowledge graph based on the structured medical data set specifically includes: For each structured medical data group, determining target multimodal source data and target data source corresponding to the structured medical data group; the target multimodal source data is used to reflect which data the structured medical data group is obtained by performing knowledge extraction on; Integrating the structured medical data set, the target multimodal source data, and the target data source to obtain an integrated data set; Using a preset graphic database to store the integrated data set; A connection is established between the preset graph database and the InteractiveGraph framework.

8. The method for constructing a multimodal medical knowledge graph according to any one of claims 1 to 7, characterized in that: The structured medical data set includes head medical entities, head medical entity types, relationships, tail medical entities, and tail medical entity types.

9. A multimodal medical knowledge graph construction device, characterized in that: include: An acquisition module, used for acquiring multimodal medical data; The multimodal medical data includes a multimodal medical data subset of a plurality of medical keywords; A preprocessing module, used for preprocessing the multimodal medical data to obtain multimodal text data of each medical keyword; An extraction module, used to obtain a structured medical data group of each medical keyword based on the multimodal text data of each medical keyword based on a preset medical knowledge extraction model; the medical knowledge extraction model is a model trained based on a pre-trained model; A construction module is used to construct a multimodal medical knowledge graph based on the structured medical data group.

10. A computer device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the multimodal medical knowledge graph construction method as described in any one of claims 1 to 8.

Images