A method for constructing a multimodal knowledge graph for critical illnesses
By constructing a multimodal knowledge graph for critically ill patients, combining natural language understanding and multimodal data, the problem of insufficient personalization of the online consultation system is solved, and the accurate collection and personalization of the condition information of critically ill patients is achieved. It is recommended to improve the efficiency and accuracy of consultation.
Patent Information
- Application Number
- CN202411216407.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The existing online consultation system lacks personalization and flexibility when dealing with critically ill patients, making it difficult for doctors to obtain accurate condition information, affecting the accuracy and efficiency of diagnosis and treatment plans.
Using a multimodal knowledge graph construction method for critical and critical illness, we use the acquisition and preprocessing of medical data, extract medical entities and relationships, combine image and text data to build a multimodal knowledge graph, and extract and transform it through neural network models. Generate low-dimensional vector representations to achieve accurate collection and personalized suggestions for patient condition information.
It improves the efficiency and accuracy of the consultation process, can better adapt to complex and diverse patient consultation scenarios, provide more accurate and personalized medical advice, and reduce misdiagnosis and missed diagnosis.
Smart Images

Figure CN118761464B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of digital medical technology, and in particular to a method for constructing a multimodal knowledge graph for critical and severe diseases. Background Art
[0002] Traditional outpatient medical treatment methods have problems such as long waiting times and insufficient doctor resources, which limit patients' ability to obtain timely medical services. With the rapid development and popularization of Internet technology, people's demand for convenient and efficient medical services is growing. Online consultation, as an emerging medical treatment method, provides patients with the convenience of communicating and consulting with doctors anytime and anywhere through online platforms.
[0003] Although online consultations have solved some of the problems of traditional medical treatment, there are still some shortcomings. When dealing with a large number of consultations from online patients, doctors need to provide diagnosis and treatment recommendations efficiently, so some platforms use pre-designed question and answer templates to improve efficiency. However, these templates often lack personalization and specificity, and only provide responses based on common questions and symptoms. Since patients' conditions and needs vary, templates cannot fully adapt to the unique circumstances of each patient, making it difficult for doctors to obtain accurate consultation information, which in turn affects the accuracy and personalization of diagnosis and treatment plans.
[0004] Existing technologies have obvious deficiencies in personalized medical services. At present, there is a lack of effective means of identifying and classifying emergency calls, making it difficult to timely and reasonably allocate limited emergency dispatchers and emergency resources to those who need emergency treatment. In order to improve the quality and efficiency of online consultations, it is necessary to develop a more intelligent and personalized system that can provide accurate and comprehensive medical advice based on the patient's unique situation. Such a system needs to be more flexible and diversified to meet the needs of different patients and improve the overall level of medical services. Summary of the invention
[0005] In response to the above problems, the purpose of the present invention is to provide a multimodal knowledge graph construction method for critical and emergency diseases, which can flexibly adapt to the conditions and needs of different patients, thereby collecting more and more accurate patient information during the entire consultation process, so as to provide more accurate and personalized medical advice.
[0006] The present invention provides a method for constructing a multimodal knowledge graph for critical and severe diseases, comprising:
[0007] Step S1, obtaining medical data, extracting medical entities and relationships between the medical entities from the medical data, and constructing a first graph based on the relationships between the medical entities; the medical data is all medical-related data;
[0008] Step S2, based on the first graph, associating the medical entities and the relationships between the medical entities with corresponding image data and text data to construct a second graph; the image data and text data are specific data of the medical entities;
[0009] Step S3, converting the medical entities and the relationships in the second graph into low-dimensional vectors to obtain a multimodal knowledge graph for critical and emergency diseases.
[0010] In a possible implementation, step S1 includes:
[0011] Acquire medical data and perform preprocessing to obtain first data; the preprocessing at least includes: data labeling, data cleaning, data deduplication, data conversion, data alignment, and data completion;
[0012] Extract medical entities in the first data and the relationships between the medical entities, and construct a first graph based on the medical entities and the relationships; the medical entities include at least users, organs, diseases, and drugs.
[0013] In a possible implementation, step S4, training a neural network model according to the medical entity to obtain an extraction model for identifying and extracting a conversation medical entity from the conversation;
[0014] Step S5: searching for and outputting related information from the multimodal knowledge graph according to the dialog medical entity.
[0015] In a possible implementation, step S2 includes:
[0016] Acquire image data and text data related to medical entities;
[0017] Based on the relationship between the medical entities, the image data and text data are associated with the medical entities to construct a second graph.
[0018] In a possible implementation, step S4 includes:
[0019] Initialize the parameters of the neural network model;
[0020] Inputting the medical entity into the neural network model, and calculating the predicted output through a forward propagation process;
[0021] Using a loss function to evaluate the difference between the predicted output and the true result;
[0022] Determining the gradient of the parameter by a back-propagation algorithm;
[0023] The parameters of the neural network model are adjusted according to the gradient to obtain the extraction model.
[0024] In a possible implementation, step S4 further includes:
[0025] The start position classifier, the end position classifier, and the span category matching classifier are used to respectively predict the start position, the end position, and the span category matching probability of the medical entity in the conversation;
[0026] The expression of the starting position classifier is as follows:
[0027] ;
[0028] ;
[0029] in, , represents a weight matrix or model parameter, which is used to perform linear transformation in neural networks; , represents the learnable parameters; represents element-wise multiplication, represents the sum of the input matrix, express x i For Category The probability of the starting position, X represents the position vector of the starting position classifier, f 1 represents the transformation function of the position vector, h x Indicates that a specific layer f 1 After treatment x i The expression, h xi Represents input x i Output after processing by the neural network;
[0030] The expression of the end position classifier is as follows:
[0031] ;
[0032] in, , represents a weight matrix or model parameter, which is used for linear transformation in neural network; , represents the learnable parameters; represents element-wise multiplication, represents the sum of the input matrix, express x j For Category The probability of the ending position of h xj express xij The output after being processed by the hidden layer of the neural network;
[0033] The expression of the span category matching classifier is as follows:
[0034] ;
[0035] in, express Sigmoid Function, when the function result , which means the sentence x Belongs to type c, threshold , and For Sentences x The set of all possible event types, i.e., the patient's diseases and symptoms, s c Representation and Category c The score or points associated with it.
[0036] In a possible implementation, step S3 includes:
[0037] The medical entities in the second graph are represented as multiple independent vectors by decoupling, and the embedded medical entities are obtained by a relationship-aware aggregation mechanism;
[0038] Determine the importance matrix of each of the relationships according to the topological structure of the second graph, and obtain the embedded relationship according to the graph partitioning method;
[0039] The embedded medical entity and the embedded relationship are imported into the second graph to obtain a multimodal knowledge graph.
[0040] In a possible implementation, step S3 further includes:
[0041] The medical entity is obtained according to the following formula:
[0042] ;
[0043] ;
[0044] in, N Indicates through relationship r With medical entities u Connected neighbors, k represents the number of spaces in which medical entity mappings are embedded, An aggregate of information representing embedded medical entities and embedded relationships, The projection matrix representing a specific relationship, Representing Entities u In space k The embedding representation of Represents a medical entity v In space k The embedding representation of, • represents the dot product of two embedding representations, e u,k Representing Entities u In Dimension k The embedding vector on Representing relationships r In Dimension k The weight on .
[0045] In a possible implementation, step S3 further includes:
[0046] The embedding relationship is obtained according to the following formula e r :
[0047] ;
[0048] in, represents the activation function ReLU, e r Representing relationships r The embedding of , ⊕ represents the connection of two embeddings, r N represents the topological structure embedding, is a trainable matrix.
[0049] In a possible implementation, step S3 further includes:
[0050] The graph information of the multimodal knowledge graph is completed according to the following formula:
[0051] ;
[0052] in, n represents the total number of samples, i Indicates i samples, pred represents the value of the model output, t represents the true value of the target, t [ i ] indicates the i The target true value of samples, pred [ i ] indicates the i The model output value of samples, L ( t, pred ) represents the predicted value pred and the true label t The difference between.
[0053] The multimodal knowledge graph construction method for critical and emergency diseases provided by the present invention realizes the guidance of medical consultation by combining the multimodal knowledge graph with the knowledge base large model, which can adapt to the conditions and needs of different patients, and collect accurate medical information of patients in a targeted manner during the medical consultation process, so as to avoid the situation where the doctor cannot obtain sufficiently accurate medical information in time, resulting in misdiagnosis, missed diagnosis or incomplete treatment plan; the first graph construction and knowledge base design of the present invention can collect more comprehensive and accurate medical information of patients, thereby improving the efficiency and accuracy of the medical consultation process; the present invention combines the flexibility of natural language understanding and the structured accuracy of multimodal knowledge graphs, and can better adapt to complex and diverse patient consultation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 A flowchart of a method for constructing a multimodal knowledge graph for critical and emergency diseases provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0055] The following detailed description of the embodiments of the present invention is further described in detail in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention, that is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.
[0056] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more than two; the terms “first”, “second”, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.
[0057] Figure 1 A schematic diagram of a process for constructing a multimodal knowledge graph for critical illnesses provided by an embodiment of the present invention, such as Figure 1 As shown, the multimodal knowledge graph construction method for critical and severe diseases provided by the present invention includes:
[0058] Step S1, obtaining medical data, extracting medical entities and relationships between medical entities from the medical data, and constructing a first graph based on the relationships between medical entities;
[0059] In a possible implementation, step S1 includes:
[0060] Acquire medical data and perform preprocessing to obtain first data; the preprocessing at least includes: data labeling, data cleaning, data deduplication, data conversion, data alignment, and data completion.
[0061] Extract medical entities and relationships between medical entities in the first data, and construct a first graph based on the medical entities and the relationships; the medical entities include at least users, organs, diseases, drugs, and diagnosis and treatment methods.
[0062] Among them, medical data include 120 center databases, medical databases, clinical trial registration databases, 120 center electronic medical records and clinical databases, full text of medical literature, clinical notes and reports, images and imaging data;
[0063] In one possible implementation, considering the privacy and security of the data, medical data containing sensitive patient information is desensitized, anonymized, or encrypted to protect patient privacy.
[0064] Specifically, after the medical data is collected, the data cleaning and preprocessing steps are carried out, which involves eliminating redundancy in the data, filling missing values, and standardizing the data format. Natural language processing technology is used to conduct in-depth analysis of the data to identify and mark key medical entities such as diseases, symptoms, and drugs. Then, these marked medical entities are further analyzed to explore their respective attributes. Through text analysis and relationship extraction technology, the mutual connections between medical entities are revealed. These medical entities, attributes and their mutual relationships are combined to construct a first map that comprehensively reflects medical knowledge.
[0065] In the medical field, the scope of medical entities covers specific entities such as people, organs, diseases, and drugs. For example, when a medical entity refers to a person, it may cover roles such as patients, doctors, nurses, and researchers; when an entity refers to an organ, it may include organs such as the heart, lungs, and kidneys; when an entity refers to a disease, it may cover diseases such as cancer, diabetes, and hypertension; and when an entity refers to a drug, it may include antibiotics, painkillers, anticancer drugs, and other drug types. These medical entities not only represent specific objects in the medical field, but also include related concepts and entities.
[0066] The attributes of a medical entity refer to the characteristics, properties or descriptions of a medical entity. These attributes specifically describe various aspects of a medical entity. For example, when a medical entity refers to an individual, its attributes may include personal information such as name, age, gender and height; if a medical entity refers to an organ, its attributes may include the specific location, shape, size and pathological changes of the organ in the body; for the medical entity of a disease, the attributes may cover symptoms, course of the disease and possible complications; and for a drug entity, the attributes may include dosage, route of administration and potential adverse reactions.
[0067] The relationships between medical entities refer to the connections or relationships between different medical entities in the medical field, including the association between diseases and symptoms, the interaction between drugs and diseases, and the interaction between doctors and patients. These relationships describe in detail the interactions, dependencies, and influences between medical entities.
[0068] The First Medical Graph builds a structured knowledge base that graphically presents various entities in the medical field, such as diseases, symptoms, examinations and treatments, as well as the relationships between these entities, such as the connection between diseases and symptoms. The First Medical Graph provides medical workers with a tool that helps them understand medical data more deeply, thereby improving efficiency and accuracy in the diagnosis and treatment process.
[0069] Step S2, based on the first atlas, associating the medical entities and the relationships between the medical entities with the corresponding image data and text data to construct a second atlas;
[0070] In a possible implementation, step S2 includes:
[0071] Acquire image data and text data related to medical entities; the text data at least includes standardized names, definitions, attributes and relationship explanations.
[0072] Based on the relationship between medical entities, the image data and text data are associated with the medical entities to construct a second graph.
[0073] In one possible implementation, the newly acquired knowledge can be integrated into the multimodal knowledge graph by first integrating the data to eliminate redundancy and inconsistency to form a unified data set, and then generating new knowledge through reasoning or other means to increase the coverage and completeness of the multimodal knowledge graph.
[0074] Taking into account the problems of unclear expressions and colloquial knowledge in critical situations, it is planned to design "alias" attributes for diseases and symptoms, so as to identify related symptoms and diseases more accurately and quickly. In addition, two attributes "image" and "video" are designed for the three entity types of disease, symptom and measure, and multimodal data such as images and video addresses are used as entity-specific attribute values to provide emergency personnel with sufficient and visual information, which is conducive to faster and more accurate emergency guidance.
[0075] Step S3, converting the medical entities and relationships in the second graph into low-dimensional vectors to obtain a multimodal knowledge graph for critical and emergency diseases.
[0076] Converting medical entities (such as diseases, symptoms, drugs) and relationships (such as "cause", "treatment", "side effects") into low-dimensional vectors can achieve efficient compression and expression of information. This representation makes it easier for computers to process and analyze large amounts of medical data.
[0077] Low-dimensional vector representation can capture complex relationships between entities, including the association between causes, symptoms, and treatments. This is particularly important for diagnosis and treatment reasoning, because doctors need to infer possible diseases based on symptoms during the diagnosis process and choose appropriate treatment plans. In low-dimensional space, the distance between vectors can represent the similarity between entities. This is of great significance for discovering the potential connection between symptoms and diseases, the adaptability between drugs and diseases, etc.
[0078] During the diagnosis and treatment process, it is crucial to quickly retrieve and match patient information with medical knowledge in the first graph. Low-dimensional vector representation can accelerate this process, allowing the system to respond quickly to emergencies and provide decision support for doctors. Graph embedding technology provides an effective feature representation method for machine learning models. These low-dimensional vectors can be used as input features to train classifiers, prediction models, etc. to assist in diagnosis and treatment decisions. In the field of critical and emergency diseases, cross-disciplinary knowledge fusion is often required. Graph embedding models can integrate knowledge from different fields (such as basic medicine, clinical medicine, and pharmacology) into a unified vector space to promote cross-domain reasoning. By analyzing the patient's specific information (such as genetic information and previous medical history), graph embedding models can help build a personalized first graph and provide patients with customized diagnosis and treatment recommendations. Graph embedding models can also reveal underutilized or unknown relationships in the first graph, providing new perspectives and insights for medical research and clinical practice.
[0079] In a possible implementation, step S3 includes:
[0080] By decoupling, the medical entities in the second graph are represented as multiple independent vectors, and the embedded medical entities are obtained through the relationship-aware aggregation mechanism;
[0081] Determine the importance matrix of each relationship according to the topological structure of the second graph, and obtain the embedded relationship according to the graph partitioning method;
[0082] The embedded medical entities and embedded relationships are imported into the second graph to obtain a multimodal knowledge graph.
[0083] In a possible implementation, step S3 further includes:
[0084] The medical entity is obtained according to the following formula:
[0085] ;
[0086] ;
[0087] in, N Indicates through relationship r With medical entities u Connected neighbors, k represents the number of spaces in which medical entity mappings are embedded, An aggregate of information representing embedded medical entities and embedded relationships, The projection matrix representing a specific relationship, Representing Entities u In space k The embedding representation of Represents a medical entity v In space k The embedding representation of, • represents the dot product of two embedding representations, e u,k Representing Entities u In Dimension k The embedding vector on Representing relationships r In Dimension k The weight on .
[0088] In a possible implementation, step S3 further includes:
[0089] The embedding relationship is obtained according to the following formula e r :
[0090] ;
[0091] in, represents the activation function ReLU, e r Representing relationships r The embedding of , ⊕ represents the connection of two embeddings, r N represents the topological structure embedding, is a trainable matrix. e r and topological embedding r N After concatenation, the embedding relationship is updated through the fully connected layer and activation function, and finally normalized e r .
[0092] In a possible implementation, step S3 further includes:
[0093] The graph information of the multimodal knowledge graph is completed according to the following formula:
[0094] ;
[0095] in, n represents the total number of samples, i Indicates i samples, pred represents the value of the model output, t represents the true value of the target, t [ i ] indicates the i The target true value of samples, pred [ i ] indicates the i The model output value of samples, L ( t, pred ) represents the predicted value pred and the true label t The difference between.
[0096] Step S4, training a neural network model according to the second graph to obtain an extraction model for extracting medical entities in the conversation;
[0097] Step S5, searching for related information from the multimodal knowledge graph based on the dialog medical entity and outputting it.
[0098] In the research and application of artificial intelligence, deep neural networks, especially those large models with more than 1 billion parameters, play a vital role. These large models can not only process huge data sets, but also handle a variety of complex tasks, such as natural language processing, computer vision, and speech recognition. In particular, large generative models, which rely on rich corpora, can achieve the generation, understanding, and reasoning of natural language. By learning from large amounts of text data, these models can perform a variety of tasks including text summarization, translation, and sentiment analysis.
[0099] With the improvement of hardware performance and the optimization of deep learning algorithms, the development of large models is accelerating. The number of parameters of these models continues to increase, and the training cycle is correspondingly extended, but at the same time their performance has also been significantly improved. Large models usually use deep learning architectures such as transformers, which makes them perform well in processing various natural language tasks. Currently, there are many large models in use, such as ChatGPT, GPT-4, ERNIE, Lingyi Bot, etc. The generative large models in this case are not limited to these, but also include open source compiled large models such as ChatGLM2-6B.
[0100] The extraction model is used to extract the relationship between entities, events and event relationships, attributes and keywords. The events here mainly refer to the key information in the conversation process with the patient.
[0101] In a possible implementation, step S4 includes:
[0102] Initialize the parameters of the neural network model;
[0103] The medical entity is input into the neural network model and the predicted output is calculated through the forward propagation process;
[0104] Use the loss function to evaluate the difference between the predicted output and the actual result;
[0105] Determine the gradient of the parameters through the back-propagation algorithm;
[0106] The parameters of the neural network model are adjusted according to the gradient to obtain the extraction model.
[0107] In one possible implementation, a prompt is constructed: the sequence labeling task of information extraction is converted into a generation task through prompt, and then input into the large model to generate preliminary information extraction identification results. The prompt is input into the large language model to obtain the generated text sequence; the text sequence is converted into an entity label sequence to obtain the final result. The model is trained based on span, which can identify entities embedded in the dialogue at one time. The implementation method is to train three classifiers to predict the probability of entity start position, end position, and span category matching respectively.
[0108] In a possible implementation, step S4 further includes:
[0109] The start position classifier, the end position classifier, and the span category matching classifier are used to respectively predict the start position, the end position, and the span category matching probability of the medical entity in the conversation;
[0110] The starting position classifier is generated based on the DisenKGAT model. It improves the first graph representation capability of the model by decoupling entities into multiple independent vectors, and updates entity embeddings through a relationship-aware information aggregation mechanism. In order to achieve different topics focused on in different spaces, the model adopts a relationship-aware information aggregation mechanism. When aggregating neighbor information, it considers the relationship between entities and neighbors, and gives higher weights to neighbors related to the topic.
[0111] The end position classifier adopts the method of dividing the subgraph. The subgraph is divided according to the batch size set in advance, which can effectively reduce the amount of calculation and speed up the training of the model. After completing the initialization, by querying the importance matrix of each relationship, the importance of the relationship in the original triple and the relationship in the subgraph can be obtained, and the relationship importance matrix corresponding to the topological structure can be constructed.
[0112] The span category matching classifier measures the embedding of the given head entity and relation through a scoring function to achieve the first graph completion. Through the fully connected layer, the inner product with the tail entity embedding is finally calculated to obtain the triple score.
[0113] The expression of the starting position classifier is as follows:
[0114] ;
[0115] ;
[0116] in, , represents a weight matrix or model parameter, which is used to perform linear transformation in neural networks; , represents the learnable parameters; represents element-wise multiplication, represents the sum of the input matrix, express x i For Category The probability of the starting position, X represents the position vector of the starting position classifier, f 1 represents the transformation function of the position vector, h x Indicates that a specific layer f 1 After treatment x i The expression, h xi Represents input x i Output after processing by the neural network;
[0117] The expression of the end position classifier is as follows:
[0118] ;
[0119] in, , represents a weight matrix or model parameter, which is used for linear transformation in neural network; , represents the learnable parameters; represents element-wise multiplication, represents the sum of the input matrix, express x j For Category The probability of the ending position of h xj express x ij The output after being processed by the hidden layer of the neural network;
[0120] The expression of the span category matching classifier is as follows:
[0121] ;
[0122] in, express Sigmoid Function, when the function result , which means the sentence x Belongs to type c, threshold , and For Sentences x The set of all possible event types, i.e., the patient's diseases and symptoms, s c Representation and Category c The score or points associated with it.
[0123] The multimodal knowledge graph construction method for critical and emergency diseases provided by the present invention realizes the guidance of medical consultation by combining the multimodal knowledge graph with the knowledge base large model, which can adapt to the conditions and needs of different patients, and collect accurate medical information of patients in a targeted manner during the medical consultation process, so as to avoid the situation where the doctor cannot obtain sufficiently accurate medical information in time, resulting in misdiagnosis, missed diagnosis or incomplete treatment plan; the first graph construction and knowledge base design of the present invention can collect more comprehensive and accurate medical information of patients, thereby improving the efficiency and accuracy of the medical consultation process; the present invention combines the flexibility of natural language understanding and the structured accuracy of multimodal knowledge graphs, and can better adapt to complex and diverse patient consultation scenarios.
[0124] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A method for constructing a multimodal knowledge graph for critical illnesses, characterized in that: include: Step S1, obtaining medical data, extracting medical entities and relationships between the medical entities from the medical data, and constructing a first graph based on the relationships between the medical entities; the medical data is all medical-related data; Step S2, based on the first atlas, associating the medical entities and the relationships between the medical entities with corresponding image data and text data to construct a second atlas; the image data and text data are specific data of the medical entities; Step S3, converting the medical entities and the relationships in the second graph into low-dimensional vectors to obtain a multimodal knowledge graph for critical and emergency diseases; Step S4, training a neural network model according to the medical entity to obtain an extraction model for identifying and extracting a dialogue medical entity from the dialogue; Step S5, searching for related information from the multimodal knowledge graph according to the dialog medical entity and outputting the related information; The step S4 comprises: Initialize the parameters of the neural network model; Inputting the medical entity into the neural network model, and calculating the predicted output through a forward propagation process; Using a loss function to evaluate the difference between the predicted output and the true result; Determining the gradient of the parameter by a back-propagation algorithm; Adjusting the parameters of the neural network model according to the gradient to obtain the extraction model; The step S4 further comprises: The start position classifier, the end position classifier, and the span category matching classifier are used to respectively predict the start position, the end position, and the span category matching probability of the medical entity in the conversation; The expression of the starting position classifier is as follows: h X =f1(X) in, Represents a weight matrix or model parameters used for linear transformation in a neural network: represents a learnable parameter; Indicates element-wise multiplication, f o represents the sum of the input matrix, Represents x i is the probability of the starting position of category c∈C, X represents the position vector of the starting position classifier, f1 represents the transformation function of the position vector, and h X represents x after being processed by a specific layer f1 i The expression, Represents input x i Output after processing by the neural network; The expression of the end position classifier is as follows: in, Represents a weight matrix or model parameter used for linear transformation in a neural network; represents a learnable parameter; represents the multiplication of the elements, f0 represents the sum of the input matrix, Represents x j is the probability of the end position of category c∈C, h xj Represents x ij The output after being processed by the hidden layer of the neural network; The expression of the span category matching classifier is as follows: Among them, σ represents the sigmoid function. When the function result Indicates that sentence x belongs to type c, threshold ξ1∈[0, 1], and p(c|x) is the set of all possible event types for sentence x, i.e., the patient’s disease and symptoms, s c represents the score or fraction associated with category c; The step S3 comprises: The medical entities in the second graph are represented as multiple independent vectors by decoupling, and the embedded medical entities are obtained by a relationship-aware aggregation mechanism; Determine the importance matrix of each of the relationships according to the topological structure of the second graph, and obtain the embedded relationship according to the graph partitioning method; Importing the embedded medical entity and the embedded relationship into the second graph to obtain a multimodal knowledge graph; The embedded medical entity is obtained according to the following formula: Where N represents the neighbors connected to medical entity u through relationship r, k represents the number of spaces where medical entity mapping is embedded, and φ(e v,k , e r ,θ r ) represents the information aggregation of embedded medical entities and embedded relations, θ represents the projection matrix of a specific relation, represents the embedding representation of entity u in space k, represents the embedded representation of medical entity v in space k, represents the dot product of two embedded representations, e u,k represents the embedding vector of entity u in dimension k, Represents the weight of relation r on dimension k; The embedding relationship e is obtained according to the following formula r : Among them, σ represents the activation function ReLU, e r represents the embedding of relation r, represents the concatenation of two embeddings, r N represents the topological structure embedding, W rel ∈R d×d is a trainable matrix.
2. The method for constructing a multimodal knowledge graph according to claim 1, characterized in that: The step S1 comprises: Acquire medical data and perform preprocessing to obtain first data; the preprocessing at least includes: data labeling, data cleaning, data deduplication, data conversion, data alignment, and data completion; Extract medical entities in the first data and the relationships between the medical entities, and construct a first graph based on the medical entities and the relationships; the medical entities include at least users, organs, diseases, and drugs.
3. The method for constructing a multimodal knowledge graph according to claim 1, characterized in that: The step S2 comprises: Acquiring image data and text data related to the medical entity; Based on the relationship between the medical entities, the image data and text data are associated with the medical entities to construct a second graph.
4. The method for constructing a multimodal knowledge graph according to claim 1, characterized in that: The step S3 further comprises: The graph information of the multimodal knowledge graph is completed according to the following formula: Where n represents the total number of samples, i represents the i-th sample, pred represents the value output by the model, t represents the true value of the target, t[i] represents the true value of the target for the f-th sample, pred[i] represents the model output value for the f-th sample, and L(t, pred) represents the difference between the predicted value pred and the true label t.
