A processing method and device of a multi-modal medical knowledge graph embedding coding model

Abstract

The embodiment of the application relates to a kind of multi-modal medical knowledge graph embedding coding model processing method and device, the method comprises: constructing patient database by collecting patient medical data;And based on the preset medical knowledge rule set and patient database, corresponding multi-modal medical knowledge graph is constructed, denoted as knowledge graph M;Corresponding embedding coding model is constructed for knowledge graph M;Based on knowledge graph M, embedding coding model is pre-trained;After pre-training ends, the medical atlas inference task of knowledge graph M is processed based on embedding coding model. Through the application, not only the complex entity relationship in the atlas can be deeply mined, but also the multi-modal entity relationship in the heterogeneous atlas can be deeply mined.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for processing a multimodal medical knowledge graph embedding coding model. Background Technology

[0002] As medical knowledge graphs grow in size, users expect to improve their semantic understanding capabilities from graph data containing massive amounts of prior knowledge, extracting dense vector representations from sparse graph triple relationships. This process from graph structure to vector representation is commonly known as knowledge graph embedding. Currently, most conventional knowledge graph embedding methods are based on translation distance models, such as TransH, TransR, and TransD models. However, through practice, we have found that these traditional methods struggle to deeply mine complex entity relationships within the graph, and are unable to deeply mine multimodal entity relationships (such as images and text) in heterogeneous graphs. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a method, apparatus, electronic device, and computer-readable storage medium for processing a multimodal medical knowledge graph embedding and encoding model. This invention first constructs a patient database by collecting patient medical data, and then constructs a multimodal medical knowledge graph, or knowledge graph M, based on a set of medical knowledge rules and the patient database. Next, it constructs an embedding and encoding model for knowledge graph M by combining graph attention networks (GATs) and a cross-modal fusion mechanism of image-text cross-attention. Then, using the positive / negative sample contrast loss of the graph triples as the target, the embedding and encoding model is pre-trained based on knowledge graph M. After pre-training, the embedding and encoding model is used to process medical graph reasoning tasks on knowledge graph M. This invention uses a deep learning model to replace the conventional translation distance model, which can achieve the technical effect of improving semantic richness. This invention can not only deeply mine complex entity relationships in the graph, but also deeply mine multimodal entity (e.g., image, text, etc.) relationships in multimodal heterogeneous graphs.

[0004] To achieve the above objectives, a first aspect of the present invention provides a method for processing a multimodal medical knowledge graph embedding and encoding model, the method comprising: A patient database is constructed by collecting patient medical data; and a corresponding multimodal medical knowledge graph, denoted as knowledge graph M, is constructed based on a preset set of medical knowledge rules and the patient database; the knowledge graph M consists of a first node set V and a first edge set E; the first node set V includes multiple first nodes v i 1 ≤ index i ≤ NV N V The total number of nodes; the first edge set E includes multiple first edges e k 1 ≤ index k ≤ N E N E Let e ​​be the total number of edges; for each of the first edges e k A directed edge connecting the two first nodes v i The starting node of a directed edge is denoted as the main entity node, and the ending node of a directed edge is denoted as the guest entity node. Construct a corresponding embedding encoding model for the knowledge graph M; the embedding encoding model is used to perform knowledge graph embedding encoding processing on the knowledge graph M as input to the model and output the corresponding feature graph M. * The feature map M * From the node feature set H * Edge feature set G * Composition; the node feature set H * Including N V Node feature vectors The edge feature set G * Including N E Each edge feature vector The node feature vector With the first node v i One-to-one correspondence; the edge feature vector With the first edge e k One-to-one correspondence; The embedding coding model is pre-trained based on the knowledge graph M; After pre-training, the medical graph reasoning task of the knowledge graph M is processed based on the embedding encoding model.

[0005] Preferably, the patient database includes multiple first patient records; each first patient record corresponds to a patient's medical data; the first patient record includes a patient identifier field, a diagnosed disease field, a diagnostic classification field, a diagnosis certificate field, an examination result field, and a laboratory result field; The patient identifier field is a unique identifier for the current patient; The diagnosed disease field is the name of the current patient's diagnosed disease; the name definition rule of the diagnosed disease field satisfies the disease definition rule of the medical knowledge rule set; The diagnostic classification field is the name of the diagnosis type for the current patient's current diagnosis; the naming rules of the diagnostic classification field satisfy the diagnostic classification rules of the medical knowledge rule set; The diagnosis report field contains the text of the current patient's medical diagnosis report. When the examination result field is not empty, it consists of one or more examination data items; each examination data item consists of a corresponding examination type and examination data; the name definition rule of the examination type satisfies the examination definition rule of the medical knowledge rule set; when the examination type is any type of medical imaging examination, the corresponding examination data is the corresponding medical examination image; when the examination type is not any type of medical imaging examination, the corresponding examination data is the corresponding medical examination report text. When the test result field is not empty, it consists of one or more test data items; each test data item consists of a corresponding test type and test data; the name definition rule of the test type satisfies the test definition rule of the medical knowledge rule set; the test data is the corresponding medical test report text; The set of medical knowledge rules includes medical entity definition rules and medical entity relationship definition rules; the medical entity definition rules include disease definition rules, symptom definition rules, drug definition rules, clinical protocol definition rules, diagnostic classification rules, examination definition rules, and laboratory test definition rules; the disease and symptom definition rules include the UMLS standard; the drug definition rules include the ATC classification standard; the clinical protocol definition rules include clinical guidelines for multiple diseases; the diagnostic classification rules include the ICD standard and the SNOMED CT standard; the examination definition rules include the LOINC standard and the RadLex standard; the laboratory test definition rules include the LOINC standard; the medical entity relationship definition rules include the UMLS standard, the FDA drug labeling standard, the LOINC standard, and the clinical guidelines for multiple diseases.

[0006] Preferably, the first node v i The node attributes include node identifier, modality type, entity type, entity name, and entity attributes; the node identifier is the unique identifier of the current node; the modality type includes image modality and text modality; the entity type includes disease, symptom, drug, clinical treatment plan, medical diagnosis, medical examination, and medical test. When the entity type is a disease, the entity name is the corresponding disease category name, and the entity attribute is the corresponding disease definition text; when the entity type is a symptom, the entity name is the symptom category name, and the entity attribute is the corresponding symptom definition text; when the entity type is a drug, the entity name is the drug category name, and the entity attribute is the corresponding drug definition text; when the entity type is a clinical treatment plan, the entity name is the plan category name, and the entity attribute is the corresponding treatment plan definition text; when the entity type is a medical diagnosis, the entity name is the diagnosis category name, and the entity attribute is the corresponding medical diagnosis report text; when the entity type is a medical examination, the entity name is the examination category name, and the entity attribute is the corresponding medical examination report text or medical examination image; when the entity type is a medical laboratory test, the entity name is the laboratory test category name, and the entity attribute is the corresponding medical laboratory test report text. The disease definition text, symptom definition text, drug definition text, and treatment plan definition text are all standardized based on the medical knowledge rule set and cannot be modified based on the patient's medical data; the medical diagnosis text, medical examination report text, medical examination image, and medical laboratory report text are set only based on the patient's medical data. The first side e k The edge attributes include edge identifier, main entity node identifier, guest entity node identifier, and main-guest entity relationship type; the edge identifier is the unique identifier of the current edge; the main and guest entity node identifiers are the node identifiers of the main and guest entity nodes connected by the current edge, respectively; the main-guest entity relationship type is the type of association relationship between the main and guest entity nodes connected by the current edge, including disease-symptom relationship, disease-drug relationship, disease-treatment plan relationship, disease-diagnosis relationship, diagnosis-examination relationship, and diagnosis-laboratory relationship; In the first node set V, the first node v with entity types of disease, symptom, drug, clinical treatment plan, medical diagnosis, medical examination, and medical test are respectively. i These are recorded as the corresponding disease nodes, symptom nodes, medication nodes, treatment plan nodes, diagnosis nodes, examination nodes, and laboratory test nodes; In the first node set V, each disease node can be directly connected as the main entity node to one or more symptom nodes, drug nodes, treatment plan nodes, or diagnostic nodes that are the object entities; each symptom node, drug node, treatment plan node, or diagnostic node that is the object entity node can be connected to one or more disease nodes that are the main entity nodes; however, there are no direct connections between the four types of nodes: symptom nodes, drug nodes, treatment plan nodes, and diagnostic nodes. In the first node set V, each diagnostic node may not be directly connected to any of the examination nodes or the test nodes, but may also act as the main entity node and be directly connected to one or more of the examination nodes or the test nodes that are the guest entity nodes; each examination node or the test node may only be directly connected to one of the diagnostic nodes that are the main entity node; there is no direct connection between the examination nodes and the test nodes. In the first node set V, each of the disease nodes, symptom nodes, drug nodes, and treatment plan nodes is configured based on the medical knowledge rule set; the first edge e between each disease node and its corresponding symptom node, drug node, or treatment plan node... k Then, refer to the medical entity relationship definition rules and the type categories of the subject-object entity relationship to set the corresponding directed edges; In the first node set V, each diagnostic node corresponds one-to-one with a first patient record in the patient database; each diagnostic node is configured based on the diagnostic classification field and the diagnostic certificate field of the corresponding first patient record; when the examination result field or laboratory result field corresponding to each diagnostic node is not empty, the examination node or laboratory node corresponding to the current diagnostic node is configured based on the non-empty examination result field or laboratory result field; the first edge e between each diagnostic node and its corresponding examination node or laboratory node... k Directed edges are set according to the configuration that the current diagnostic node is the primary entity node, the current examination node or laboratory node is the secondary entity node, and the primary-secondary entity relationship type is the corresponding diagnostic-examination relationship or diagnostic-laboratory relationship; the diagnostic disease field corresponding to each diagnostic node is matched with its corresponding disease node; the first edge e between each diagnostic node and its corresponding disease node... k Directed edges are set according to the configuration that the current disease node is the main entity node, the current diagnosis node is the guest entity node, and the main-guest entity relationship type is disease-diagnosis relationship.

[0007] Preferably, the model input of the embedding coding model is used to receive the knowledge graph M, and the model output is used to output the feature graph M. * ; The embedding coding model consists of a preprocessing layer, Nx first coding layers, a second GAT layer, a mapping layer, and a residual layer; Nx is a preset positive integer. The input of the preprocessing layer is connected to the input of the model, and the first and second outputs are connected to the inputs of the mapping layer and the first first encoding layer, respectively. The Nx first encoding layers are formed by sequentially connecting Nx first encoding layers, and each first encoding layer is formed by sequentially connecting a corresponding first GAT layer and a cross-modal fusion layer. The output of the Nxth first encoding layer is connected to the input of the second GAT layer. The output of the second GAT layer and the output of the mapping layer are connected to the first and second inputs of the residual layer, respectively. The output of the residual layer is connected to the model output. Each first GAT layer is implemented based on the GAT layer processing mechanism in the GATs network model, which concatenates the results of multi-head self-attention operations and uses the concatenated vector as the node update feature. The second GAT layer is implemented based on the GAT layer processing mechanism in the GATs network model, which calculates the average vector of the results of multi-head self-attention operations and uses the average vector as the node update feature. The cross-modal fusion layer is implemented based on the processing mechanism of the Co-TRM layer of the ViLBERT model. The preprocessing layer is used to process each of the first nodes v in the knowledge graph M. i Perform node feature vector transformation to obtain the corresponding node feature vector. And for each of the first edges e of the knowledge graph M k Perform edge feature vector transformation to obtain the corresponding edge feature vector. ; and from the obtained N V The node feature vectors Form the corresponding node feature set H 0 The obtained N E The edge feature vectors Form the corresponding edge feature set G 0 ; and the obtained node feature set H 0 and the edge feature set G 0 The corresponding feature map X is formed 0 Send to the mapping layer and the first first coding layer respectively; Among them, the feature map X 0 The node feature set H 0 and the edge feature set G 0 Composition; the node feature set H 0 Including N V The node feature vectors The edge feature set G 0 Including N E The edge feature vectors ; The qth first coding layer is used to process the feature map X input to the current layer. q-1The first GAT layer is input to perform feature encoding processing to obtain the corresponding feature map Y. q ; and the feature map Y q The corresponding cross-modal fusion layer is input for feature fusion processing, and the fusion result is used as the corresponding feature map X. q And identify whether the current first coding layer is the Nxth first coding layer; if not, then the current feature map X q＜N Input into the next first encoding layer; if so, then input the current feature map X. q=N Send to the second GAT layer; Where 1 ≤ index q ≤ Nx; the feature map X q-1 From the node feature set H q-1 Edge feature set G q-1 Composition; the node feature set H q-1 Including N V Node feature vectors The edge feature set G q-1 Including N E Each edge feature vector The feature map Y q From node feature set Hy q and edge feature set Gy q Composition; the node feature set Hy q Including N V Node feature vectors The edge feature set Gy q Including N E Each edge feature vector The feature map X q From the node feature set H q Edge feature set G q Composition; the node feature set H q Including N V The node feature vector hq i; the edge feature set G q Including N E Each edge feature vector ; The second GAT layer is used for the feature map X N Feature encoding is performed to obtain the corresponding feature map X. b Send to the residual layer; Among them, the feature map X b It consists of a node feature set Hb and an edge feature set Gb; the node feature set Hb includes N V The feature vector of each node hb i The edge feature set Gb includes N E Each edge feature vector gb k; The mapping layer is used to process the feature map X b The node feature vector space and edge feature vector space are used as the corresponding first and second target feature spaces; and the feature map X is... 0 The node feature set H 0 The feature vectors of each node Perform vector mapping processing on the first target feature space to obtain the corresponding node feature vector ha. i =W a1 ; and the feature map X 0 The edge feature set G 0 Each of the aforementioned edge feature vectors Perform vector mapping processing on the second target feature space to obtain the corresponding edge feature vector ga. k =W a2 ; and from the obtained N V The node feature vector ha i The corresponding node feature set Ha is formed, and the obtained N E The edge feature vector ga k The corresponding edge feature set Ga is formed; and the corresponding feature map X is formed by the node feature set Ha and the edge feature set Ga. a Send to the residual layer; Among them, W a1 W a2 These are the corresponding first and second feature mapping vectors; the feature map X a It consists of a node feature set Ha and an edge feature set Ga; the node feature set Ha includes N V The node feature vector ha i The edge feature set Ga includes N E The edge feature vector ga k ; The residual layer is used to process the feature map X. a X b The corresponding feature map M is obtained by performing residual calculation. * And output, M * =X a +X b ; Among them, the feature map M * The node feature set H * and the edge feature set G * Composition; the node feature set H * Including N V The node feature vectors , =hai +hb i The edge feature set G * Including N E The edge feature vectors , =ga k +gb k .

[0008] Furthermore, the first node v of the knowledge graph M i Perform node feature vector transformation to obtain the corresponding node feature vector. Specifically, it includes: The preprocessing layer processes each of the first nodes v in the knowledge graph M. i As the corresponding current node; and the modality type, entity type, entity name, and entity attribute of the current node as the corresponding current modality, current type, current name, and current attribute; The current modality and the current type are then subjected to one-hot vector encoding to obtain the corresponding modality embedding vector and type embedding vector, respectively. The current name is then subjected to word embedding encoding to obtain the corresponding name word embedding vector; And identify the current mode; If the current modality is an image modality, then the current attribute is processed by visual feature encoding based on the built-in visual feature encoding model to obtain the corresponding attribute feature vector; the visual feature encoding model is implemented based on a convolutional neural network model or a graph neural network model; If the current modality is a text modality, then the current attribute is processed by text feature encoding based on the built-in text feature encoding model to obtain the corresponding attribute feature vector; the text feature encoding model is implemented based on the BERT series of models; The corresponding node feature vector is composed of the modality embedding vector, the type embedding vector, the name word embedding vector, and the attribute feature vector corresponding to the current node. .

[0009] Furthermore, the first edge e of the knowledge graph M k Perform edge feature vector transformation to obtain the corresponding edge feature vector. Specifically, it includes: The preprocessing layer will process each of the first edges e of the knowledge graph M. k As the corresponding current edge; and perform one-hot vector encoding on the subject-object entity relationship type of the current edge, and use the encoding result as the corresponding edge feature vector. .

[0010] Furthermore, the feature map X input from the current layer... q-1 The first GAT layer is input to perform feature encoding processing to obtain the corresponding feature map Y. q Specifically, it includes: Step 71, the first GAT layer is used to process the feature map X q-1 Each of the aforementioned edge feature vectors This is used as the current edge feature vector; and the node feature vectors of the subject and object entity nodes corresponding to the current edge feature vector are recorded as the corresponding node feature vectors. , ; and derived from node feature vectors , The current edge feature vector is concatenated with it to form a corresponding triplet feature vector. ; Among them, the triplet feature vector The splicing method is as follows: , 1≤indexj≤N V j≠i, || is the vector concatenation operator; Step 72, and process the feature vectors of each triplet. Perform a linear transformation to obtain the corresponding triplet feature vector. ; Among them, the triplet feature vector The linear transformation method is as follows: ; Wq 1 is the weight vector of the first GAT layer of the first encoding layer q, used to process the linear transformation of the triplet feature vector; Step 73, connect each of the first nodes v in the knowledge graph M i As the current node; and for the set of neighboring nodes corresponding to the current node. Perform identification; and identify the current set of neighboring nodes. The total number of neighboring nodes n i Perform identification; and then identify the current set of neighboring nodes. The range of index values ​​for all neighboring nodes in a given set is denoted as the corresponding neighboring node index range. ; and the current neighboring node index range The edge index value corresponding to each index j in the graph is denoted as the corresponding neighborhood edge index k. i,j ; and the current set of neighboring nodes Each neighboring node in the array is denoted as ; and each of the aforementioned neighboring nodes The corresponding triplet feature vector Recorded as ; and denoted as M the total number of multi-head self-attention operations of the first GAT layer of the first coding layer in the qth first coding layer. q And assign a corresponding single-head index m to each single-head self-attention operation of the current first GAT layer; and assign each triplet feature vector Perform M q Perform head self-attention operation and record the self-attention operation result of the m-th single head during the operation as the corresponding single-head single-sided self-attention feature vector. And perform the corresponding n operations on the m-th head self-attention. i The single-head, single-sided self-attention feature vector Perform summation and nonlinear activation, and use the nonlinear activation result as the corresponding single-head neighborhood self-attention feature vector. ; and the obtained M q The single-head neighborhood self-attention feature vector Perform concatenation and use the concatenation result as the node feature vector corresponding to the current node. ; Among them, the neighborhood node set By n i The first node v is currently i The directly connected neighboring nodes Composition; Total number of neighboring nodes n i ≥1; The node feature vector The single-head neighborhood self-attention feature vector The single-head, single-sided self-attention feature vector The calculation method is as follows: , , , ; 1 ≤ Single-headed index m ≤ M q σ is a preset nonlinear activation function; In the m-th single-head self-attention operation of the first GAT layer of the first coding layer of the q-th layer, and with the j-th... The triplet feature vectors The corresponding self-attention weights; W q(m) is the linear transformation weight vector corresponding to the m-th single-head self-attention operation of the first GAT layer of the first coding layer of the q-th first coding layer; LeakyReLu is the LeakyReLu activation function; Step 74, and process the feature map Xq-1 Each of the aforementioned edge feature vectors Perform a linear transformation to obtain the corresponding edge feature vector. ; Wherein, the edge feature vector The calculation method is as follows: ; The weight vector of the first GAT layer of the qth first coding layer is used to process the linear transformation of the edge feature vector; Step 75, and from the obtained N V The node feature vectors The corresponding node feature set Hy is formed q ; The obtained N E The edge feature vectors The corresponding edge feature set Gy is formed q ; and by the node feature set Hy q and the edge feature set Gy q The corresponding feature map Y q .

[0011] Furthermore, the feature map Y q The corresponding cross-modal fusion layer is input for feature fusion processing, and the fusion result is used as the corresponding feature map X. q Specifically, it includes: Step 81, the cross-modal fusion layer is used to take each of the diagnostic nodes of the knowledge graph M as the current node; and take the subgraph of the knowledge graph M composed of the current node and all the corresponding object entity nodes as the current diagnostic subgraph; Step 82, and then the first node v in the current diagnostic sub-map, whose modality type is image modality and text modality respectively. i Record them as the corresponding image nodes and text nodes; and count the total number of image nodes. Step 83, and when the total number of image nodes is greater than zero, the feature map Y is... q The node feature vector corresponding to each of the image nodes The extracted feature vector is used as the corresponding image feature vector a, and the node feature vector corresponding to each of the text nodes is used as the node feature vector. The extracted text feature vectors are used as corresponding text feature vectors b; and the corresponding image feature tensor A is composed of all the obtained image feature vectors a, and the corresponding text feature tensor B is composed of all the obtained text feature vectors b. Step 84, and convert the text feature tensor B into the corresponding key vector K. 1Value vector V 1 The image feature tensor A is transformed into the corresponding query vector Q. 1 And according to the key vector K 1 Value vector V 1 Query vector Q 1 Perform cross-attention computation and use the result as the corresponding first fusion tensor A. * ; and based on the first fusion tensor A * Each of the first fusion vectors a * For the feature map Y q The node feature vector corresponding to the current fusion vector Reset; in, , , , , , Transform the first key vector into a weight vector, the first value vector into a weight vector, and the first query vector into a weight vector; The first fusion tensor A * The shape of the tensor is consistent with the image feature tensor A; the first fused tensor A * Consisting of one or more of the first fusion vectors a * Composition; the first fusion vector a * Each of the image feature vectors corresponds one-to-one with the image feature vector a; Step 85, and convert the image feature tensor A into the corresponding key vector K. 2 Value vector V 2 The text feature tensor B is transformed into the corresponding query vector Q. 2 And according to the key vector K 2 Value vector V 2 Query vector Q 2 Perform cross-attention computation and use the result as the corresponding second fusion tensor B. * ; and based on the second fusion tensor B * Each of the second fusion vectors b * For the feature map Y q The node feature vector corresponding to the current fusion vector Reset; in, , , , , , Transform the corresponding second key vector into a weight vector, the second value vector into a weight vector, and the second query vector into a weight vector; The second fusion tensor B * The shape of the tensor is consistent with the text feature tensor B; the second fused tensor B * Consisting of one or more of the second fusion vectors b * Composition; the second fusion vector b * Each corresponds one-to-one with the text feature vector b; Step 86, and in the feature map Y q The node feature vectors associated with all diagnostic submaps After all updates are complete, the latest feature map Y will be sent. q As the corresponding feature map X q And output it.

[0012] Furthermore, the feature map X N Feature encoding is performed to obtain the corresponding feature map X. b Specifically, it includes: Step 91, the second GAT layer is used to process the feature map X N The feature vectors of each edge This is used as the current edge feature vector; and the node feature vectors of the subject and object entity nodes corresponding to the current edge feature vector are recorded as the corresponding node feature vectors. , ; and derived from node feature vectors , The current edge feature vector is concatenated with it to form a corresponding triplet feature vector. ; Among them, the triplet feature vector The splicing method is as follows: , 1≤indexj≤N V j≠i, || is the vector concatenation operator; Step 92, and process the feature vectors of each of the triplets. Perform a linear transformation to obtain the corresponding triplet feature vector. ; Among them, the triplet feature vector The linear transformation method is as follows: ; This is the weight vector used by the second GAT layer to process the linear transformation of triplet feature vectors; Step 93, connect each of the first nodes v in the knowledge graph Mi As the current node; and for the set of neighboring nodes corresponding to the current node. Perform identification; and identify the current set of neighboring nodes. The total number of neighboring nodes n i Perform identification; and then identify the current set of neighboring nodes. The range of index values ​​for all neighboring nodes in a given set is denoted as the corresponding neighboring node index range. ; and the current neighboring node index range The edge index value corresponding to each index j in the graph is denoted as the corresponding neighborhood edge index k. i,j ; and the current set of neighboring nodes Each neighboring node in the array is denoted as ; and each of the aforementioned neighboring nodes The corresponding triplet feature vector Recorded as The total number of multi-head self-attention operations in the second GAT layer is denoted as M. N+1 And assign a corresponding single-head index m to each single-head self-attention operation of the second GAT layer; and assign each triplet feature vector Perform M N +1 Perform head self-attention operation and record the self-attention operation result of the m-th single head during the operation as the corresponding single-head single-sided self-attention feature vector. And perform the corresponding n operations on the m-th head self-attention. i The single-head, single-sided self-attention feature vector Perform summation and nonlinear activation, and use the nonlinear activation result as the corresponding single-head neighborhood self-attention feature vector. ; and the obtained M N+1 The single-head neighborhood self-attention feature vector Perform concatenation and use the concatenation result as the node feature vector corresponding to the current node. ; Among them, the neighborhood node set By n i The first node v is currently i The directly connected neighboring nodes Composition; Total number of neighboring nodes n i ≥1; The node feature vector The single-head neighborhood self-attention feature vector The single-head, single-sided self-attention feature vector The calculation method is as follows: , , , ; 1 ≤ Single-headed index m ≤ M N+1 σ is a preset nonlinear activation function; In the m-th single-head self-attention operation of the second GAT layer, and the j-th... The triplet feature vectors The corresponding self-attention weights; W N+1(m) is the linear transformation weight vector corresponding to the m-th single-head self-attention operation of the second GAT layer; LeakyReLu is the LeakyReLu activation function; Step 94, and process the feature map X N Each of the aforementioned edge feature vectors Perform a linear transformation to obtain the corresponding edge feature vector. ; Wherein, the edge feature vector The calculation method is as follows: ; This is the weight vector used by the second GAT layer to process the linear transformation of the edge feature vectors; Step 95, and from the obtained feature vectors of each node As a corresponding node feature vector hb i Each of the aforementioned edge feature vectors As a corresponding edge feature vector gb k ; and from the obtained N V The node feature vector hb i The corresponding node feature set Hb is composed, and N is obtained. E The edge feature vector gb k The corresponding edge feature set Gb is formed; and the corresponding feature graph X is formed by the node feature set Hb and the edge feature set Gb. b And output it.

[0013] Preferably, the pre-training of the embedding coding model based on the knowledge graph M specifically includes: Step 101: Construct a second edge set by traversing the first edge set E of the knowledge graph M; Wherein, the second edge set consists of one or more second edges e x Composed of, and any two of the second edges e in the second edge set x There is no connection between them; 1 ≤ index x ≤ N G NG This represents the total number of edges in the second edge set; Step 102: Use the knowledge graph M as a positive sample graph M + ; and copy the knowledge graph M to obtain the initialized negative sample graph M. - ; Step 103, set each of the second edges e in the second edge set x As the current sample edge; and the positive sample graph M is used as the current sample edge; + The negative sample map M - The corresponding edge of the current sample edge is used as the corresponding current positive sample edge and current negative sample edge; and the current positive sample edge and its corresponding subject and object entity nodes form a corresponding positive sample triple z. x ; and in the negative sample spectrum M - In this process, the subject and object entity nodes of the current negative sample edge are swapped, and a corresponding negative sample triple is formed by the current negative sample edge and its swapped subject and object entity nodes. ; Wherein, the positive sample triple z x for , , These are the subject and object entity nodes corresponding to the current positive sample, r. x The subject-object entity relationship corresponding to the current positive sample; the negative sample triple. for , , These are the subject and object entity nodes corresponding to the current negative sample, respectively. For the current negative sample, the subject-object entity relationship is... , , ; Step 104, transfer the positive sample spectrum M + The current knowledge graph M is input into the embedding encoding model for processing, and the feature graph M obtained from this processing is... * As the corresponding positive sample feature map ; and the negative sample map M - The current knowledge graph M is input into the embedding encoding model for processing, and the feature graph M obtained from this processing is... * As the corresponding negative sample feature map ; Step 105, the positive sample feature map is... In the above, each of the positive sample triples z x main entity node Customer entity node Subject-object entity relationship The corresponding two node feature vectors and one edge feature vector serve as the corresponding principal vector. relation vector g x Object vector ; and the negative sample feature map In the context of each of the negative sample triples... main entity node Customer entity node Subject-object entity relationship The corresponding two node feature vectors and one edge feature vector serve as the corresponding principal vector. relation vectors Object vector ; and composed of each of the aforementioned positive sample triples z x The corresponding main vector The relation vector g x The object vector Form a corresponding positive sample feature group p x ( ,g x , ); and each of the aforementioned negative sample triples The corresponding main vector The relation vector The object vector Form a corresponding negative sample feature group ( , , ); Step 106, obtain N G p, the positive sample feature groups x ( ,g x , ) and N G The negative sample feature group ( , , Substitute the preset model loss function L mod The corresponding first loss value is obtained through calculation; Wherein, the model loss function L model Based on the contrastive loss function: , , ; d() is the subject-object distance function for the triple, ⊙ is the Hadamard product, and 1 ≤ index x ’ ≤N G ; Step 107: Identify whether the obtained first loss value meets the preset first loss value range; if not, then based on the preset first model optimizer, move towards making the model loss function L... mod The model parameters of the embedding coding model are modulated once in the direction that reaches the minimum value, and the process returns to step 104 when the current modulation is completed; if the condition is not met, training is stopped and the pre-training is confirmed to be over. The first model optimizer includes the Adam optimizer and the SGD optimizer.

[0014] Furthermore, the construction of the second edge set by traversing the first edge set E of the knowledge graph M specifically includes: Step 111: Initialize the second edge set to empty; and set the first edge e of the knowledge graph M to empty. k As the current edge; Step 112: Check if there exists a second edge e in the current edge set. x The two nodes of the current edge intersect with the two nodes of the current edge to confirm their existence; if no intersection is found, the current edge is added as a new second edge e. x Add to the second edge set; Step 113: Check whether the current edge is the last first edge e of the knowledge graph M. k If identification is not performed, then the next first edge e of the knowledge graph M is... k As the new current edge, return to step 112; if so, output the current second edge set as the result of this construction.

[0015] Preferably, the step of processing the medical graph reasoning task based on the embedding encoding model of the knowledge graph M specifically includes: The knowledge graph M is input into the embedding coding model for processing, and the feature graph M obtained from this processing is... * As the current feature map; The system receives first query data input by the user; the first query data includes first query text and first query image; the first query text is the corresponding first diagnosis certificate, first examination report or first laboratory report; when the first query image is not empty, it is the corresponding first medical examination image. Based on the first query text, a virtual node of a specific text modality is constructed according to the node format of the diagnostic node, the inspection node, or the inspection node corresponding to the current query text, serving as the current text node. The preprocessing layer is then used to perform node feature vector transformation on the current text node, and the node feature vector obtained from this transformation is used as the corresponding first feature vector. Finally, the node feature vectors corresponding to each diagnostic node, inspection node, or inspection node in the current feature map that are relevant to the current query text are... As the corresponding first candidate vector; and the first candidate vector set is composed of all the obtained first candidate vectors; The system identifies whether the first query image is empty. If it is empty, the corresponding second candidate vector set is set to empty. If it is not empty, a virtual node representing an image modality is constructed based on the current query image and the node format of the inspection node, serving as the current image node. The preprocessing layer is then used to transform the node feature vector of the current image node, and the node feature vector obtained from this transformation is used as the corresponding second feature vector. The node feature vectors corresponding to each inspection node representing an image modality in the current feature map are then used as the second feature vector. As the corresponding second candidate vector, and all the obtained second candidate vectors form the corresponding second candidate vector set; The similarity between each candidate vector in the first candidate vector set and the first feature vector is calculated to obtain the corresponding first similarity; and the first similarity exceeding a preset first similarity threshold is taken as the corresponding second similarity; and the first node v corresponding to all second similarities is... i Form the corresponding first candidate node set; The system then identifies whether the second candidate vector set is empty; if so, it sets the corresponding second candidate node set to empty; otherwise, it calculates the similarity between each second candidate vector in the second candidate vector set and the second feature vector to obtain the corresponding third similarity, and takes the third similarity exceeding a preset second similarity threshold as the corresponding fourth similarity, and uses the first node v corresponding to all the fourth similarities as the fourth similarity. i Form the corresponding second candidate node set; The first and second candidate node sets are merged to obtain a corresponding merged node set; and the first node v with the highest similarity in the merged node set is selected. iAs the current node; and the disease nodes in the knowledge graph M that have a direct or indirect connection with the current node as the current screening nodes; and the current screening node and all the corresponding symptom nodes, drug nodes and treatment nodes constitute the corresponding current screening subgraph; The corresponding medical atlas reasoning report, composed of all node attributes of the currently selected sub-graph, is then fed back to the current user.

[0016] A second aspect of the present invention provides an apparatus for implementing the processing method of the multimodal medical knowledge graph embedding coding model described in the first aspect above. The apparatus includes: a knowledge graph construction module, a coding model construction module, a model pre-training module, and a graph task processing module. The knowledge graph construction module is used to construct a patient database by collecting patient medical data; and to construct a corresponding multimodal medical knowledge graph, denoted as knowledge graph M, based on a preset set of medical knowledge rules and the patient database; the knowledge graph M consists of a first node set V and a first edge set E; the first node set V includes multiple first nodes v i 1 ≤ index i ≤ N V N V The total number of nodes; the first edge set E includes multiple first edges e k 1 ≤ index k ≤ N E N E Let e ​​be the total number of edges; for each of the first edges e k A directed edge connecting the two first nodes v i The starting node of a directed edge is denoted as the main entity node, and the ending node of a directed edge is denoted as the guest entity node. The encoding model construction module is used to construct a corresponding embedding encoding model for the knowledge graph M; the embedding encoding model is used to perform knowledge graph embedding encoding processing on the knowledge graph M input to the model and output the corresponding feature graph M. * The feature map M * From the node feature set H * Edge feature set G * Composition; the node feature set H * Including N V Node feature vectors The edge feature set G * Including N E Each edge feature vector The node feature vector With the first node v i One-to-one correspondence; the edge feature vector With the first edge e k One-to-one correspondence; The model pre-training module pre-trains the embedding coding model based on the knowledge graph M; The graph task processing module is used to process the medical graph reasoning task of the knowledge graph M based on the embedding encoding model after pre-training.

[0017] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver; The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0019] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for processing a multimodal medical knowledge graph embedding encoding model. As described above, this invention first constructs a patient database by collecting patient medical data, and then constructs a multimodal medical knowledge graph, or knowledge graph M, based on a set of medical knowledge rules and the patient database. Next, it constructs an embedding encoding model for knowledge graph M by combining GATs networks and a cross-modal fusion mechanism of image-text cross-attention. Then, using the positive / negative sample contrast loss of the graph triples as the target, the embedding encoding model is pre-trained based on knowledge graph M. After pre-training, the medical graph reasoning task of knowledge graph M is processed based on the embedding encoding model. This invention uses a deep learning model to replace the conventional translation distance model, achieving the technical effect of improving semantic richness. This invention not only enables in-depth mining of complex entity relationships in the graph, but also in-depth mining of multimodal entity relationships in multimodal heterogeneous graphs. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of a processing method for a multimodal medical knowledge graph embedding and encoding model provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the nodes of the knowledge graph M provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the embedded coding model provided in Embodiment 1 of the present invention; Figure 4 This is a module structure diagram of a processing device for a multimodal medical knowledge graph embedding and encoding model provided in Embodiment 2 of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0022] Embodiment 1 of the present invention provides a method for processing a multimodal medical knowledge graph embedding encoding model, such as... Figure 1 The schematic diagram shows a processing method for a multimodal medical knowledge graph embedding and encoding model provided in Embodiment 1 of the present invention. The method mainly includes the following steps: Step 1: Construct a patient database by collecting patient medical data; and construct a corresponding multimodal medical knowledge graph, denoted as knowledge graph M, based on a preset set of medical knowledge rules and the patient database.

[0023] Here, the patient database in this embodiment of the invention includes multiple first patient records; each first patient record corresponds to a patient's medical data; the first patient record includes a patient identifier field, a diagnosed disease field, a diagnostic classification field, a diagnosis certificate field, an examination result field, and a laboratory result field. The patient identifier field is a unique identifier for the current patient. The diagnosed disease field is the name of the current patient's diagnosed disease, and the name definition rule of the diagnosed disease field satisfies the disease definition rules of the medical knowledge rule set. The diagnostic classification field is the name of the diagnostic category for the current patient's current diagnosis, and the name definition rule of the diagnostic classification field satisfies the diagnostic classification rules of the medical knowledge rule set. The diagnosis certificate field is the text of the current patient's medical diagnosis certificate. When the examination result field is not empty, it consists of one or more examination data items; each examination data item consists of a corresponding examination category and examination data; the name definition rule of the examination category satisfies the examination definition rules of the medical knowledge rule set; when the examination category is any type of medical imaging examination, the corresponding examination data is the corresponding medical examination image; when the examination category is not any type of medical imaging examination, the corresponding examination data is the corresponding medical examination report text. When the test result field is not empty, it consists of one or more test data items; each test data item consists of the corresponding test type and test data; the name definition rules of the test type satisfy the test definition rules of the medical knowledge rule set; the test data is the corresponding medical test report text.

[0024] The medical knowledge rule set in this invention includes medical entity definition rules and medical entity relationship definition rules. Specifically, the medical entity definition rules include disease definition rules, symptom definition rules, drug definition rules, clinical protocol definition rules, diagnostic classification rules, examination definition rules, and laboratory test definition rules. Disease and symptom definition rules include the UMLS standard. Drug definition rules include the ATC classification standard. Clinical protocol definition rules include clinical guidelines for multiple diseases. Diagnostic classification rules include the ICD standard and the SNOMED CT standard. Examination definition rules include the LOINC standard and the RadLex standard. Laboratory test definition rules include the LOINC standard. Medical entity relationship definition rules include the UMLS standard, the FDA drug labeling standard, the LOINC standard, and clinical guidelines for multiple diseases.

[0025] The knowledge graph M in this embodiment of the invention consists of a first node set V and a first edge set E.

[0026] The first node set V in this embodiment of the invention includes multiple first nodes v i 1 ≤ index i ≤ N V N V This represents the total number of nodes. The first node, v... i The node attributes include node identifier, modal type, entity type, entity name, and entity attributes.

[0027] The node identifier is a unique identifier for the current node. Modal types include image modality and text modality. Entity types include disease, symptom, drug, clinical treatment plan, medical diagnosis, medical examination, and medical test.

[0028] It should be noted that: 1) When the entity type is disease, the entity name is the corresponding disease category name, and the entity attribute is the corresponding disease definition text; 2) When the entity type is symptom, the entity name is the symptom category name, and the entity attribute is the corresponding symptom definition text; 3) When the entity type is drug, the entity name is the drug category name, and the entity attribute is the corresponding drug definition text; 4) When the entity type is clinical treatment plan, the entity name is the plan category name, and the entity attribute is the corresponding treatment plan definition text; 5) When the entity type is medical diagnosis, the entity name is the diagnosis category name, and the entity attribute is the corresponding medical diagnosis report text; 6) When the entity type is medical examination, the entity name is the examination category name, and the entity attribute is the corresponding medical examination report text or medical examination image; 7) When the entity type is medical laboratory test, the entity name is the laboratory test category name, and the entity attribute is the corresponding medical laboratory test report text.

[0029] It should be noted that the disease definition text, symptom definition text, drug definition text, and treatment plan definition text in this embodiment of the invention are all standardized based on a set of medical knowledge rules and cannot be modified based on the patient's medical data; while the medical diagnosis report text, medical examination report text, medical examination image, and medical laboratory report text in this embodiment of the invention are set only based on the patient's medical data.

[0030] The first edge set E in this embodiment of the invention includes a plurality of first edges e k 1 ≤ index k ≤ N E N E Let e ​​be the total number of edges. For each first edge e... k Let it be a directed edge connecting the two first nodes v i In this context, the starting node of a directed edge is denoted as the master entity node, and the ending node of a directed edge is denoted as the guest entity node. The first edge e... k The edge attributes include edge identifier, main entity node identifier, guest entity node identifier, and main-guest entity relationship type.

[0031] Here, the edge identifier is the unique identifier of the current edge. The subject and object entity node identifiers are the node identifiers of the subject and object entity nodes connected by the current edge, respectively. The subject-object entity relationship type is the type of association relationship between the subject and object entity nodes connected by the current edge, including disease-symptom relationship, disease-drug relationship, disease-treatment plan relationship, disease-diagnosis relationship, diagnosis-examination relationship, and diagnosis-laboratory relationship.

[0032] It should be noted that the first node set V contains first nodes of the following entity types: disease, symptom, drug, clinical treatment plan, medical diagnosis, medical examination, and medical test. i These are recorded as the corresponding disease node, symptom node, medication node, treatment plan node, diagnosis node, examination node, and laboratory test node. For example... Figure 2 The diagram shows a node representation of the knowledge graph M provided in Embodiment 1 of the present invention.

[0033] It should be noted that in the first node set V, each disease node can act as a primary entity node and directly connect to one or more symptom nodes, drug nodes, treatment plan nodes, or diagnosis nodes that act as secondary entities; each symptom node, drug node, treatment plan node, or diagnosis node that acts as a secondary entity node can connect to one or more disease nodes that act as primary entities; however, there are no direct connections between symptom nodes, drug nodes, treatment plan nodes, and diagnosis nodes. Figure 2 As shown.

[0034] It should be noted that each diagnostic node in the first node set V may not be directly connected to any inspection or testing node, but it can also act as a primary entity node and directly connect to one or more inspection or testing nodes that are secondary entity nodes. Each inspection or testing node can only be directly connected to one diagnostic node that is a primary entity node; there are no direct connections between inspection nodes and testing nodes. Figure 2 As shown.

[0035] It should be noted that the disease nodes, symptom nodes, drug nodes, and treatment plan nodes in the first node set V are all set based on a set of medical knowledge rules. The first edge e between each disease node and its corresponding symptom node, drug node, or treatment plan node... k Then, refer to the medical entity relationship definition rules and the type categories of subject-object entity relationships to set the corresponding directed edges.

[0036] It should be noted that in the first node set V, each diagnostic node corresponds one-to-one with the first patient record in the patient database. Each diagnostic node is configured based on the diagnostic classification field and diagnostic certificate field of the corresponding first patient record. If the examination result field or laboratory result field corresponding to each diagnostic node is not empty, then the corresponding examination node or laboratory node is configured based on the non-empty examination result field or laboratory result field. The first edge e between each diagnostic node and its corresponding examination node or laboratory node... k Directed edges are set according to the configuration where the current diagnosis node is the primary entity node, the current examination node or test node is the secondary entity node, and the primary-secondary entity relationship type corresponds to the diagnosis-examination relationship or diagnosis-test relationship. The diagnostic disease field of each diagnosis node is matched with its corresponding disease node. The first edge e between each diagnosis node and its corresponding disease node... k The corresponding directed edges are set according to the configuration that the current disease node is the main entity node, the current diagnosis node is the guest entity node, and the main-guest entity relationship type is disease-diagnosis relationship.

[0037] Step 2: Construct the corresponding embedding encoding model for the knowledge graph M.

[0038] Here, the embedding encoding model in this embodiment of the invention is used to perform knowledge graph embedding encoding processing on the knowledge graph M input to the model and output the corresponding feature graph M. * Among them, the feature map M * From the node feature set H * Edge feature set G * Composition; Node feature set H * Including N V Node feature vectors Edge feature set G* Including N E Each edge feature vector Node feature vectors With the first node v i One-to-one correspondence; edge feature vector With the first side e k One-to-one correspondence.

[0039] like Figure 3 As shown in the schematic diagram of the embedding coding model provided in Embodiment 1 of the present invention, the model input end of the embedding coding model of the present invention is used to receive the knowledge graph M, and the model output end is used to output the feature graph M. * .

[0040] like Figure 3 As shown, the model components of the embedded coding model include: a preprocessing layer, Nx first coding layers, a second GAT layer, a mapping layer, and a residual layer; Nx is a preset positive integer.

[0041] like Figure 3 As shown, the connection relationships of the model components in the embedded coding model are as follows: the input of the preprocessing layer is connected to the model input; the first and second outputs are connected to the inputs of the mapping layer and the first first coding layer, respectively; Nx first coding layers are formed by sequentially connecting Nx first coding layers, and each first coding layer is formed by sequentially connecting the corresponding first GAT layer and cross-modal fusion layer; the output of the Nxth first coding layer is connected to the input of the second GAT layer; the output of the second GAT layer and the output of the mapping layer are connected to the first and second inputs of the residual layer, respectively; the output of the residual layer is connected to the model output.

[0042] It should be noted that each first GAT layer is implemented based on the GAT layer processing mechanism in the GATs network model, which concatenates the results of multi-head self-attention operations and uses the concatenated vector as the node update feature. The second GAT layer is implemented based on the GAT layer processing mechanism in the GATs network model, which calculates the average vector of the results of multi-head self-attention operations and uses the average vector as the node update feature. The cross-modal fusion layer is implemented based on the processing mechanism of the Co-TRM (Co-Attentional Transformer Layers) of the ViLBERT model.

[0043] The model component functions of the embedded coding model in this embodiment of the invention are as follows.

[0044] (1) Preprocessing layer: The preprocessing layer is used to process each first node v of the knowledge graph M. i Perform node feature vector transformation to obtain the corresponding node feature vector. And for each first edge e of the knowledge graph Mk Perform edge feature vector transformation to obtain the corresponding edge feature vector. ; and from the obtained N V Node feature vectors Form the corresponding node feature set H 0 The obtained N E Each edge feature vector Form the corresponding edge feature set G 0 ; and the obtained node feature set H 0 Edge feature set G 0 The corresponding feature map X is formed 0 Send to the mapping layer and the first first coding layer respectively.

[0045] Among them, the feature map X 0 From the node feature set H 0 Edge feature set G 0 Composition; Node feature set H 0 Including N V Node feature vectors Edge feature set G 0 Including N E Each edge feature vector .

[0046] For each first node v of the knowledge graph M i Perform node feature vector transformation to obtain the corresponding node feature vector. Specifically, it includes: Step A1, the preprocessing layer processes each first node v of the knowledge graph M. i As the corresponding current node; and set the modality type, entity type, entity name, and entity attributes of the current node as the corresponding current modality, current type, current name, and current attributes; Step A2 involves performing one-hot vector encoding on the current modality and the current type to obtain the corresponding modality embedding vector and type embedding vector, respectively. Step A3, and perform word embedding encoding on the current name to obtain the corresponding name word embedding vector; Step A4, and identify the current modality; Step A5: If the current modality is an image modality, then the current attribute is processed by visual feature encoding based on the built-in visual feature encoding model to obtain the corresponding attribute feature vector. Here, the visual feature encoding model in this embodiment of the invention is implemented based on a convolutional neural network model or a graph neural network model; Step A6: If the current modality is a text modality, then the current attribute is processed by text feature encoding based on the built-in text feature encoding model to obtain the corresponding attribute feature vector. Here, the text feature encoding model in this embodiment of the invention is implemented based on the BERT series of models; Step A7, and the corresponding node feature vector is composed of the modality embedding vector, type embedding vector, name word embedding vector, and attribute feature vector corresponding to the current node. .

[0047] For each first edge e of the knowledge graph M k Perform edge feature vector transformation to obtain the corresponding edge feature vector. Specifically, this includes: the preprocessing layer processes each first edge e of the knowledge graph M. k This serves as the corresponding current edge; and the subject-object entity relationship type of the current edge is subjected to one-hot vector encoding, with the encoding result used as the corresponding edge feature vector. .

[0048] (2) Nx first coding layers: The q-th first encoding layer is used to process the feature map X input from the current layer. q-1 The corresponding first GAT layer is input for feature encoding to obtain the corresponding feature map Y. q ; and the feature map Y q Input the corresponding cross-modal fusion layer for feature fusion processing and use the fusion result as the corresponding feature map X. q It also identifies whether the current first coding layer is the Nxth first coding layer; if not, the current feature map X... q＜N Input into the next first encoding layer; if so, then input the current feature map X. q=N Send to the second GAT layer.

[0049] Where 1 ≤ index q ≤ Nx.

[0050] Feature Map X q-1 From the node feature set H q-1 Edge feature set G q-1 Composition; Node feature set H q-1 Including N V Node feature vectors Edge feature set G q-1 Including N E Each edge feature vector Feature map Y q From node feature set Hy q and edge feature set Gy q Composition; Node feature set Hy q Including N V Node feature vectors Edge feature set Gy q Including N E Each edge feature vector Feature map X q From the node feature set H q Edge feature set G q Composition; Node feature set H q Including N V Each node's feature vector hq i; edge feature set G q Including N E Each edge feature vector .

[0051] The feature map X input to the current layer q-1 The corresponding first GAT layer is input for feature encoding to obtain the corresponding feature map Y. q Specifically, it includes: Step C1, the first GAT layer is used to transform the feature map X q-1 The feature vectors of each edge This is used as the current edge feature vector; and the node feature vectors of the subject and object entity nodes corresponding to the current edge feature vector are recorded as the corresponding node feature vectors. , ; and derived from node feature vectors , The feature vector of the current edge is concatenated with the feature vector of the current edge to form a corresponding triplet feature vector. ; Among them, the triplet feature vector The splicing method is as follows: , 1≤indexj≤N V j≠i, || is the vector concatenation operator; Step C2, and process the feature vectors of each triplet. Perform a linear transformation to obtain the corresponding triplet feature vector. ; Among them, the triplet feature vector The linear transformation method is as follows: ; Wq 1 is the weight vector of the first GAT layer of the qth first coding layer used to process the linear transformation of triplet feature vectors; Step C3, connect each first node v of the knowledge graph M i As the current node; and for the set of neighboring nodes corresponding to the current node. Perform identification; and analyze the current neighborhood node set. The total number of neighboring nodes n i Perform identification; and set the current neighborhood node set The range of index values ​​for all neighboring nodes in a given set is denoted as the corresponding neighboring node index range. ; and index the current neighboring nodes. The edge index value corresponding to each index j in the graph is denoted as the corresponding neighborhood edge index k. i,j ; and the current neighborhood node set Each neighboring node in the array is denoted as ; and each neighboring node The corresponding triplet feature vector Recorded as And let M be the total number of multi-head self-attention operations of the first GAT layer of the qth first coding layer. q And assign a corresponding single-head index m to each single-head self-attention operation of the current first GAT layer; and assign each triplet feature vector Perform M q Perform head self-attention operation and record the self-attention operation result of the m-th single head during the operation as the corresponding single-head single-sided self-attention feature vector. And perform the corresponding n operations on the m-th head self-attention. i A single-head, single-sided self-attention feature vector Perform summation and nonlinear activation, and use the nonlinear activation result as the corresponding single-head neighborhood self-attention feature vector. ; and the obtained M q A single-head neighborhood self-attention feature vector Perform concatenation and use the concatenation result as the node feature vector corresponding to the current node. ; Among them, the neighborhood node set By n i The first node v i Directly connected neighbor nodes Composition; Total number of neighboring nodes n i ≥1; Node feature vectors Single-head neighborhood self-attention feature vector Single-head, single-sided self-attention feature vector The calculation method is as follows: , , , ; 1 ≤ Single-headed index m ≤ M q σ is a preset nonlinear activation function, such as ReLU activation function, ELU activation function, LeakyReLU activation function, etc. In the m-th single-head self-attention operation of the first GAT layer of the q-th first coding layer, with the j-th... eigenvectors of triplet pairs The corresponding self-attention weights; W q(m) is the linear transformation weight vector corresponding to the m-th single-head self-attention operation of the first GAT layer of the q-th first coding layer; LeakyReLu is the LeakyReLu activation function; Step C4, and process the feature map X q-1 The feature vectors of each edge Perform a linear transformation to obtain the corresponding edge feature vector. ; Among them, the edge feature vector The calculation method is as follows: ; The weight vector is used by the first GAT layer of the qth first encoding layer to process the linear transformation of the edge feature vector; Step C5, and the obtained N V Node feature vectors The corresponding node feature set Hy is formed q ; The obtained N E Each edge feature vector The corresponding edge feature set Gy is formed q ; and by the node feature set Hy q and edge feature set Gy q The corresponding feature map Y is formed q .

[0052] The feature map Y q Input the corresponding cross-modal fusion layer for feature fusion processing and use the fusion result as the corresponding feature map X. q Specifically, it includes: Step D1: The cross-modal fusion layer is used to take each diagnostic node of the knowledge graph M as the current node; and to take the subgraph in the knowledge graph M composed of the current node and all its corresponding object entity nodes as the current diagnostic subgraph. Step D2, and then select the first node v in the current diagnostic sub-map, whose modal type is either image modality or text modality. i Record them as the corresponding image nodes and text nodes; and count the total number of image nodes. Step D3, and when the total number of image nodes is greater than zero, the feature map Y is... q The node feature vectors corresponding to each image node Extracted as the corresponding image feature vector 'a', and the node feature vector corresponding to each text node. The extracted text feature vectors are used as the corresponding text feature vectors b; and the corresponding image feature tensor A is composed of all the obtained image feature vectors a, and the corresponding text feature tensor B is composed of all the obtained text feature vectors b. Step D4, and convert the text feature tensor B into the corresponding key vector K. 1 Value vector V 1 Transform the image feature tensor A into the corresponding query vector Q. 1 And based on the key vector K 1 Value vector V 1 Query vector Q 1 Perform cross-attention computation and use the result as the corresponding first fusion tensor A. * ; and based on the first fusion tensor A * Each of the first fusion vectors a * For the feature map Y q The node feature vector corresponding to the current fusion vector Reset; in, , , , , , Transform the first key vector into a weight vector, the first value vector into a weight vector, and the first query vector into a weight vector; First fusion tensor A * The shape of the tensor is consistent with that of the image feature tensor A; the first fused tensor A * Consisting of one or more first fusion vectors a * Composition; First fusion vector a * Each image feature vector a corresponds one-to-one with the image feature vector a. Step D5, and convert the image feature tensor A into the corresponding key vector K. 2 Value vector V 2 Transform the text feature tensor B into the corresponding query vector Q. 2 And based on the key vector K 2 Value vector V 2 Query vector Q 2 Perform cross-attention computation and use the result as the corresponding second fusion tensor B. * ; and based on the second fusion tensor B * Each of the second fusion vectors b * For the feature map Y q The node feature vector corresponding to the current fusion vector Reset; in, , , , , , Transform the corresponding second key vector into a weight vector, the second value vector into a weight vector, and the second query vector into a weight vector; Second fusion tensor B * The shape of the first fusion tensor is consistent with that of the text feature tensor B; the second fusion tensor B * Consisting of one or more second fusion vectors b * Composition; Second fusion vector b * Each corresponds one-to-one with the text feature vector b; Step D6, and in the feature map Y q Node feature vectors associated with all diagnostic submaps After all updates are complete, the latest feature map Y will be sent. q As the corresponding feature map X q And output it.

[0053] (3) Second GAT layer: The second GAT layer is used to process the feature map X. N Feature encoding is performed to obtain the corresponding feature map X. b Send to the residual layer.

[0054] Among them, the feature map X b It consists of a node feature set Hb and an edge feature set Gb; the node feature set Hb includes N V The feature vector of each node hb i The edge feature set Gb includes N E Each edge feature vector gb k .

[0055] For the feature map X N Feature encoding is performed to obtain the corresponding feature map X. b Specifically, it includes: Step E1, the second GAT layer is used to transform the feature map X N The feature vectors of each edge This is used as the current edge feature vector; and the node feature vectors of the subject and object entity nodes corresponding to the current edge feature vector are recorded as the corresponding node feature vectors. , ; and derived from node feature vectors , The feature vector of the current edge is concatenated with the feature vector of the current edge to form a corresponding triplet feature vector. ; Among them, the triplet feature vector The splicing method is as follows: , 1≤indexj≤N V j≠i, || is the vector concatenation operator; Step E2, and process the feature vectors of each triplet. Perform a linear transformation to obtain the corresponding triplet feature vector. ; Among them, the triplet feature vector The linear transformation method is as follows: ; The weight vectors used by the second GAT layer to process the linear transformation of triplet feature vectors; Step E3, connect each first node v of the knowledge graph M i As the current node; and for the set of neighboring nodes corresponding to the current node. Perform identification; and analyze the current neighborhood node set. The total number of neighboring nodes n i Perform identification; and set the current neighborhood node set The range of index values ​​for all neighboring nodes in a given set is denoted as the corresponding neighboring node index range. ; and index the current neighboring nodes. The edge index value corresponding to each index j in the graph is denoted as the corresponding neighborhood edge index k. i,j ; and the current neighborhood node set Each neighboring node in the array is denoted as ; and each neighboring node The corresponding triplet feature vector Recorded as The total number of multi-head self-attention operations in the second GAT layer is denoted as M. N+1 And assign a corresponding single-head index m to each single-head self-attention operation of the second GAT layer; and assign each triplet feature vector Perform M N+1 Perform head self-attention operation and record the self-attention operation result of the m-th single head during the operation as the corresponding single-head single-sided self-attention feature vector. And perform the corresponding n operations on the m-th head self-attention. i A single-head, single-sided self-attention feature vector Perform summation and nonlinear activation, and use the nonlinear activation result as the corresponding single-head neighborhood self-attention feature vector. ; and the obtained M N+1 A single-head neighborhood self-attention feature vector Perform concatenation and use the concatenation result as the node feature vector corresponding to the current node. ; Among them, the neighborhood node set By n i The first node v i Directly connected neighbor nodes Composition; Total number of neighboring nodes n i ≥1; Node feature vectors Single-head neighborhood self-attention feature vector Single-head, single-sided self-attention feature vector The calculation method is as follows: , , , ; 1 ≤ Single-headed index m ≤ M N+1 σ is a preset nonlinear activation function; In the m-th single-head self-attention operation of the second GAT layer, with the j-th... eigenvectors of triplet pairs The corresponding self-attention weights; W N +1(m) is the linear transformation weight vector corresponding to the m-th single-head self-attention operation in the second GAT layer; LeakyReLu is the LeakyReLu activation function. Step E4, and process the feature map X N The feature vectors of each edge Perform a linear transformation to obtain the corresponding edge feature vector. ; Among them, the edge feature vector The calculation method is as follows: ; This is the weight vector used by the second GAT layer to process the linear transformation of edge feature vectors; Step E5, and the resulting feature vectors of each node As a corresponding node feature vector hb i , each edge feature vector As a corresponding edge feature vector gb k ; and from the obtained N V The feature vector of each node hb i The corresponding node feature set Hb is formed, and N is obtained. E Each edge feature vector gb k The corresponding edge feature set Gb is formed; and the corresponding feature graph X is formed by the node feature set Hb and the edge feature set Gb. bAnd output it.

[0056] (4) Mapping layer: The mapping layer is used to map the feature map X b The node feature vector space and edge feature vector space are used as the corresponding first and second target feature spaces; and the feature map X is processed... 0 Node feature set H 0 The feature vectors of each node Perform vector mapping processing on the first target feature space to obtain the corresponding node feature vector ha. i =W a1 ; and the feature map X 0 edge feature set G 0 The feature vectors of each edge Perform vector mapping processing on the second target feature space to obtain the corresponding edge feature vector ga. k =W a2 ; and from the obtained N V The feature vector of each node ha i The corresponding node feature set Ha is formed, and the obtained N E The feature vector ga of each edge k The corresponding edge feature set Ga is formed; and the corresponding feature graph X is formed by the node feature set Ha and the edge feature set Ga. a Send to the residual layer.

[0057] Among them, W a1 W a2 These are the corresponding first and second feature mapping vectors; feature map X a It consists of a node feature set Ha and an edge feature set Ga; the node feature set Ha includes N V The feature vector of each node, ha i The edge feature set Ga includes N E The feature vector ga of each edge k .

[0058] (5) Residual layer: Residual layers are used for feature maps X a X b The corresponding feature map M is obtained by performing residual calculation. * And output, M * =X a +X b .

[0059] Among them, the feature map M * From the node feature set H * Edge feature set G * Composition; Node feature set H * Including NV Node feature vectors , =ha i +hb i Edge feature set G * Including N E Each edge feature vector , =ga k +gb k .

[0060] Step 3: Pre-train the embedding encoding model based on the knowledge graph M; Specifically, this includes: Step 31, constructing the second edge set by traversing the first edge set E of the knowledge graph M; The second edge set consists of one or more second edges e x Composed of, and any two second edges e in the second edge set x There is no connection between them; 1 ≤ index x ≤ N G N G Let be the total number of edges in the second edge set; Specifically, this includes: Step 311, initializing the second edge set to empty; and setting the first edge e of the knowledge graph M to be empty. k As the current edge; Step 312: Check if there exists a second edge e in the current edge set. x If the two nodes of the current edge intersect with the two nodes of the current edge, confirm that they do not exist; if no intersection exists, then add the current edge as a new second edge e. x Add to the second side set; Step 313: Check whether the current edge is the last first edge e of the knowledge graph M. k Perform identification; if not, then move the next first edge e of the knowledge graph M to... k As the new current edge, return to step 312; if so, output the current second edge set as the result of this construction. Step 32, use the knowledge graph M as the positive sample graph M + And copy the knowledge graph M to obtain the initialized negative sample graph M. - ; Step 33, combine each second edge e of the second edge set x As the current sample edge; and the positive sample graph M + Negative Sample Map M - The corresponding edge of the current sample edge is used as the corresponding current positive sample edge and current negative sample edge; and the current positive sample edge and its corresponding subject and object entity nodes form a corresponding positive sample triple z. x ; and in the negative sample spectrum M -In the process, the subject and object entity nodes of the current negative sample edge are swapped, and a corresponding negative sample triple is formed by the current negative sample edge and its swapped subject and object entity nodes. ; Among them, the positive sample triple z x for , , These are the subject and object entity nodes corresponding to the current positive sample, r. x The subject-object entity relationship corresponding to the current positive sample; the negative sample triplet. for , , These are the subject and object entity nodes corresponding to the current negative sample, respectively. For the current negative sample, the subject-object entity relationship is... , , ; Step 34, transfer the positive sample spectrum M + The current knowledge graph M is used as the input to the embedding encoding model for processing, and the feature graph M obtained from this processing is used as the input to the feature graph M. * As the corresponding positive sample feature map ; and the negative sample map M - The current knowledge graph M is used as the input to the embedding encoding model for processing, and the feature graph M obtained from this processing is used as the input to the feature graph M. * As the corresponding negative sample feature map ; Step 35, generate the positive sample feature map. In the middle, each positive sample triple z x main entity node Customer entity node Subject-object entity relationship The corresponding two node feature vectors and one edge feature vector serve as the corresponding principal vector. relation vector g x Object vector ; and the negative sample feature map In the middle, each negative sample triplet k's main entity node Customer entity node Subject-object entity relationship The corresponding two node feature vectors and one edge feature vector serve as the corresponding principal vector. relation vectors Object vector ; and composed of each positive sample triple z x Corresponding principal vector relation vector gx Object vector Form a corresponding positive sample feature group p x ( ,g x , ); and each negative sample triplet Corresponding principal vector relation vectors Object vector Form a corresponding negative sample feature group ( , , ); Step 36, obtain N G p positive sample feature groups x ( ,g x , ) and N G negative sample feature group ( , , Substitute the preset model loss function L mod The corresponding first loss value is obtained through calculation; Wherein, the model loss function L model Based on the contrastive loss function: , , ; d() is the subject-object distance function for the triple, ⊙ is the Hadamard product, and 1 ≤ index x ’ ≤N G ; Step 37: Identify whether the obtained first loss value meets the preset first loss value range; if not, then based on the preset first model optimizer, move towards making the model loss function L... mod The direction that reaches the minimum value modulates the model parameters of the embedding coding model in one round, and returns to step 34 when the modulation is completed; if the condition is not met, training is stopped and the pre-training is confirmed to be over. The first loss value range is a pre-set numerical range; the first model optimizer includes the Adam optimizer and the SGD optimizer.

[0061] Step 4: After pre-training, the medical graph reasoning task of knowledge graph M is processed based on the embedding coding model; Specifically, this includes: Step 41, inputting the knowledge graph M into the embedding encoding model for processing, and then using the feature graph M obtained from this processing...* As the current feature map; Step 42: Receive the first query data input by the user; The first query data includes the first query text and the first query image; the first query text is the corresponding first diagnosis certificate, first examination report or first laboratory report; when the first query image is not empty, it is the corresponding first medical examination image. Step 43: Based on the first query text, construct a virtual node in the specific text modality as the current text node according to the node format of the diagnostic node, check node, or check node corresponding to the current query text; use the preprocessing layer of the preprocessing layer to perform node feature vector transformation on the current text node, and use the node feature vector obtained from this transformation as the corresponding first feature vector; and use the node feature vectors corresponding to each diagnostic node, check node, or check node in the current feature map that correspond to the current query text. As the corresponding first candidate vector; and the first candidate vector set is composed of all the obtained first candidate vectors; Step 44 involves identifying whether the first query image is empty. If empty, the corresponding second candidate vector set is set to empty. If not empty, a virtual node representing an image modality is constructed based on the current query image and the node format of the inspection node, serving as the current image node. The preprocessing layer is then used to transform the node feature vector of the current image node, and the transformed node feature vector is used as the corresponding second feature vector. The node feature vectors corresponding to each inspection node representing an image modality in the current feature map are then used as the second feature vector. As the corresponding second candidate vector, and all the obtained second candidate vectors form the corresponding second candidate vector set; Step 45: Calculate the first similarity between each first candidate vector in the first candidate vector set and the first feature vector to obtain the corresponding first similarity; and take the first similarity exceeding the preset first similarity threshold as the corresponding second similarity; and use the first node v corresponding to all second similarities... i Form the corresponding first candidate node set; Here, the first similarity threshold is a pre-set threshold parameter; Step 46 involves identifying whether the second candidate vector set is empty; if so, the corresponding second candidate node set is set to empty; otherwise, the similarity between each second candidate vector and the second feature vector in the second candidate vector set is calculated to obtain the corresponding third similarity, and the third similarity exceeding the preset second similarity threshold is taken as the corresponding fourth similarity, and the first node v corresponding to all fourth similarities is used. i Form the corresponding second candidate node set; Here, the second similarity threshold is a pre-set threshold parameter; Step 47: Merge the first and second candidate node sets to obtain the corresponding merged node set; and select the first node v with the highest similarity in the merged node set. i As the current node; and the disease nodes in the knowledge graph M that have a direct or indirect connection with the current node are used as the current filtering nodes; and the current filtering subgraph is composed of the current filtering node and all its corresponding symptom nodes, drug nodes and treatment nodes. Step 48, and then provide feedback to the current user with a corresponding medical atlas reasoning report composed of all node attributes of the currently selected sub-graph.

[0062] Figure 4 This is a module structure diagram of a processing device for a multimodal medical knowledge graph embedding and encoding model provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 4 As shown, the device includes: a knowledge graph construction module 201, an encoding model construction module 202, a model pre-training module 203, and a graph task processing module 204.

[0063] The knowledge graph construction module 201 is used to construct a patient database by collecting patient medical data; and to construct a corresponding multimodal medical knowledge graph, denoted as knowledge graph M, based on a preset set of medical knowledge rules and the patient database; knowledge graph M consists of a first node set V and a first edge set E; the first node set V includes multiple first nodes v i 1 ≤ index i ≤ N V N V The total number of nodes; the first edge set E includes multiple first edges e. k 1 ≤ index k ≤ N E N E Let e ​​be the total number of edges; for each first edge e k Let it be a directed edge connecting the two first nodes v i The starting node of a directed edge is denoted as the main entity node, and the ending node of a directed edge is denoted as the guest entity node.

[0064] The encoding model construction module 202 is used to construct a corresponding embedding encoding model for the knowledge graph M; the embedding encoding model is used to perform knowledge graph embedding encoding processing on the knowledge graph M input to the model and output the corresponding feature graph M. * Feature map M * From the node feature set H * Edge feature set G * Composition; Node feature set H* Including N V Node feature vectors Edge feature set G * Including N E Each edge feature vector Node feature vectors With the first node v i One-to-one correspondence; edge feature vector With the first side e k One-to-one correspondence.

[0065] The model pre-training module 203 pre-trains the embedding coding model based on the knowledge graph M.

[0066] The graph task processing module 204 is used to process the medical graph reasoning task of the knowledge graph M based on the embedding coding model after pre-training.

[0067] The processing device for a multimodal medical knowledge graph embedding and encoding model provided in this embodiment of the invention can execute the method steps in the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0068] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the knowledge graph construction module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0069] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0070] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0071] Figure 5 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 5As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0072] exist Figure 5 The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include Non-Volatile Memory, such as at least one disk storage device.

[0073] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0074] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0075] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for processing a multimodal medical knowledge graph embedding encoding model. As described above, this invention first constructs a patient database by collecting patient medical data, and then constructs a multimodal medical knowledge graph, or knowledge graph M, based on a set of medical knowledge rules and the patient database. Next, it constructs an embedding encoding model for knowledge graph M by combining GATs networks and a cross-modal fusion mechanism of image-text cross-attention. Then, using the positive / negative sample contrast loss of the graph triples as the target, the embedding encoding model is pre-trained based on knowledge graph M. After pre-training, the medical graph reasoning task of knowledge graph M is processed based on the embedding encoding model. This invention uses a deep learning model to replace the conventional translation distance model, achieving the technical effect of improving semantic richness. This invention not only enables in-depth mining of complex entity relationships in the graph, but also in-depth mining of multimodal entity relationships in multimodal heterogeneous graphs.

[0076] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0077] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for processing a multimodal medical knowledge graph embedding and encoding model, characterized in that, The method includes: A patient database is constructed by collecting patient medical data; and a corresponding multimodal medical knowledge graph, denoted as knowledge graph M, is constructed based on a preset set of medical knowledge rules and the patient database; the knowledge graph M consists of a first node set V and a first edge set E; the first node set V includes multiple first nodes v i 1 ≤ index i ≤ N V N V The total number of nodes; the first edge set E includes multiple first edges e k 1 ≤ index k ≤ N E N E Let e ​​be the total number of edges; for each of the first edges e k A directed edge connecting the two first nodes v i The starting node of a directed edge is denoted as the main entity node, and the ending node of a directed edge is denoted as the guest entity node. Construct a corresponding embedding encoding model for the knowledge graph M; the embedding encoding model is used to perform knowledge graph embedding encoding processing on the knowledge graph M input to the model and output the corresponding feature graph M. * The feature map M * From the node feature set H * Edge feature set G * Composition; the node feature set H * Including N V Node feature vectors The edge feature set G * Including N E Each edge feature vector The node feature vector With the first node v i One-to-one correspondence; the edge feature vector With the first edge e k One-to-one correspondence; The embedding coding model is pre-trained based on the knowledge graph M; After pre-training, the medical graph reasoning task of the knowledge graph M is processed based on the embedding encoding model; The model input of the embedding coding model is used to receive the knowledge graph M, and the model output is used to output the feature graph M. * ; The embedding coding model consists of a preprocessing layer, Nx first coding layers, a second GAT layer, a mapping layer, and a residual layer; Nx is a preset positive integer. The input of the preprocessing layer is connected to the input of the model, and the first and second outputs are connected to the inputs of the mapping layer and the first first encoding layer, respectively. The Nx first encoding layers are formed by sequentially connecting Nx first encoding layers, and each first encoding layer is formed by sequentially connecting a corresponding first GAT layer and a cross-modal fusion layer. The output of the Nxth first encoding layer is connected to the input of the second GAT layer. The output of the second GAT layer and the output of the mapping layer are connected to the first and second inputs of the residual layer, respectively. The output of the residual layer is connected to the model output. Each first GAT layer is implemented based on the GAT layer processing mechanism in the GATs network model, which concatenates the results of multi-head self-attention operations and uses the concatenated vector as the node update feature. The second GAT layer is implemented based on the GAT layer processing mechanism in the GATs network model, which calculates the average vector of the results of multi-head self-attention operations and uses the average vector as the node update feature. The cross-modal fusion layer is implemented based on the processing mechanism of the Co-TRM layer of the ViLBERT model. The qth first coding layer is used to process the feature map X input to the current layer. q-1 The first GAT layer is input to perform feature encoding processing to obtain the corresponding feature map Y. q ; and the feature map Y q The corresponding cross-modal fusion layer is input for feature fusion processing, and the fusion result is used as the corresponding feature map X. q And identify whether the current first coding layer is the Nxth first coding layer; if not, then the current feature map X q＜N Input into the next first encoding layer; if so, then input the current feature map X. q=N Send to the second GAT layer; Where 1 ≤ index q ≤ Nx; the feature map X q-1 From the node feature set H q-1 Edge feature set G q-1 Composition; the node feature set H q-1 Including N V Node feature vectors The edge feature set G q-1 Including N E Each edge feature vector The feature map Y q From node feature set Hy q and edge feature set Gy q Composition; the node feature set Hy q Including N V Node feature vectors The edge feature set Gy q Including N E Each edge feature vector The feature map X q From the node feature set H q Edge feature set G q Composition; the node feature set H q Including N V The node feature vector hq i; the edge feature set G q Including N E Each edge feature vector ; The feature map Y q The corresponding cross-modal fusion layer is input for feature fusion processing, and the fusion result is used as the corresponding feature map X. q Specifically, it includes: Step 81, the cross-modal fusion layer is used to take each diagnostic node of the knowledge graph M as the current node; and take the subgraph of the knowledge graph M composed of the current node and all the corresponding object entity nodes as the current diagnostic subgraph; Step 82, and then select the first node v in the current diagnostic sub-map whose modal types are image modality and text modality respectively. i Record them as the corresponding image nodes and text nodes; and count the total number of image nodes. Step 83, and when the total number of image nodes is greater than zero, the feature map Y is... q The node feature vector corresponding to each of the image nodes The extracted feature vector is used as the corresponding image feature vector a, and the node feature vector corresponding to each of the text nodes is used as the node feature vector. The extracted text feature vectors are used as corresponding text feature vectors b; and the obtained image feature vectors a form the corresponding image feature tensor A, and the obtained text feature vectors b form the corresponding text feature tensor B. Step 84, and convert the text feature tensor B into the corresponding key vector K. 1 Value vector V 1 The image feature tensor A is transformed into the corresponding query vector Q. 1 And according to the key vector K 1 Value vector V 1 Query vector Q 1 Perform cross-attention computation and use the result as the corresponding first fusion tensor A. * ; and based on the first fusion tensor A * Each of the first fusion vectors a * For the feature map Y q The node feature vector corresponding to the current fusion vector Reset; in, ， ， ， , , Transform the first key vector into a weight vector, the first value vector into a weight vector, and the first query vector into a weight vector; The first fusion tensor A * The shape of the tensor is consistent with the image feature tensor A; the first fused tensor A * Consisting of one or more of the first fusion vectors a * Composition; the first fusion vector a * Each of the image feature vectors corresponds one-to-one with the image feature vector a; Step 85, and convert the image feature tensor A into the corresponding key vector K. 2 Value vector V 2 The text feature tensor B is transformed into the corresponding query vector Q. 2 And according to the key vector K 2 Value vector V 2 Query vector Q 2 Perform cross-attention computation and use the result as the corresponding second fusion tensor B. * ; and based on the second fusion tensor B * Each of the second fusion vectors b * For the feature map Y q The node feature vector corresponding to the current fusion vector Reset; in, ， ， ， , , Transform the corresponding second key vector into a weight vector, the second value vector into a weight vector, and the second query vector into a weight vector; The second fusion tensor B * The shape of the tensor is consistent with the text feature tensor B; the second fused tensor B * Consisting of one or more of the second fusion vectors b * Composition; the second fusion vector b * Each corresponds one-to-one with the text feature vector b; Step 86, and in the feature map Y q The node feature vectors associated with all diagnostic submaps After all updates are complete, the latest feature map Y will be sent. q As the corresponding feature map X q And output it.

2. The processing method for the multimodal medical knowledge graph embedding encoding model according to claim 1, characterized in that, The patient database includes multiple first patient records; each first patient record corresponds to a patient's medical data; the first patient record includes a patient identifier field, a diagnosed disease field, a diagnostic classification field, a diagnosis certificate field, an examination result field, and a laboratory result field. The patient identifier field is a unique identifier for the current patient; The diagnosed disease field is the name of the current patient's diagnosed disease; the name definition rule of the diagnosed disease field satisfies the disease definition rule of the medical knowledge rule set; The diagnostic classification field is the name of the diagnosis type for the current patient's current diagnosis; the naming rules of the diagnostic classification field satisfy the diagnostic classification rules of the medical knowledge rule set; The diagnosis report field contains the text of the current patient's medical diagnosis report. When the examination result field is not empty, it consists of one or more examination data items; each examination data item consists of a corresponding examination type and examination data; the name definition rule of the examination type satisfies the examination definition rule of the medical knowledge rule set; when the examination type is any type of medical imaging examination, the corresponding examination data is the corresponding medical examination image; when the examination type is not any type of medical imaging examination, the corresponding examination data is the corresponding medical examination report text. When the test result field is not empty, it consists of one or more test data items; each test data item consists of a corresponding test type and test data; the name definition rule of the test type satisfies the test definition rule of the medical knowledge rule set. The test data refers to the corresponding medical test report text. The set of medical knowledge rules includes medical entity definition rules and medical entity relationship definition rules; the medical entity definition rules include disease definition rules, symptom definition rules, drug definition rules, clinical protocol definition rules, diagnostic classification rules, examination definition rules, and laboratory test definition rules; Disease and symptom definition rules include the UMLS standard; The drug definition rules include the ATC classification criteria; The clinical protocol definition rules include clinical guidelines for multiple diseases; The diagnostic classification rules include ICD criteria and SNOMED CT criteria; The inspection definition rules include the LOINC standard and the RadLex standard; The laboratory test definition rules include the LOINC standard; The rules for defining medical entity relationships include the UMLS standard, the FDA drug labeling standard, the LOINC standard, and the clinical guidelines for the aforementioned diseases.

3. The processing method for the multimodal medical knowledge graph embedding encoding model according to claim 2, characterized in that, The first node v i The node attributes include node identifier, modality type, entity type, entity name, and entity attributes; the node identifier is the unique identifier of the current node; the modality type includes image modality and text modality; the entity type includes disease, symptom, drug, clinical treatment plan, medical diagnosis, medical examination, and medical test. When the entity type is a disease, the entity name is the corresponding disease category name, and the entity attribute is the corresponding disease definition text; when the entity type is a symptom, the entity name is the symptom category name, and the entity attribute is the corresponding symptom definition text; when the entity type is a drug, the entity name is the drug category name, and the entity attribute is the corresponding drug definition text; when the entity type is a clinical treatment plan, the entity name is the plan category name, and the entity attribute is the corresponding treatment plan definition text; when the entity type is a medical diagnosis, the entity name is the diagnosis category name, and the entity attribute is the corresponding medical diagnosis report text; when the entity type is a medical examination, the entity name is the examination category name, and the entity attribute is the corresponding medical examination report text or medical examination image; when the entity type is a medical laboratory test, the entity name is the laboratory test category name, and the entity attribute is the corresponding medical laboratory test report text. The disease definition text, symptom definition text, drug definition text, and treatment plan definition text are all standardized based on the medical knowledge rule set and cannot be modified based on the patient's medical data; the medical diagnosis text, medical examination report text, medical examination image, and medical laboratory report text are set only based on the patient's medical data. The first side e k The edge attributes include edge identifier, main entity node identifier, guest entity node identifier, and main-guest entity relationship type; the edge identifier is the unique identifier of the current edge; the main and guest entity node identifiers are the node identifiers of the main and guest entity nodes connected by the current edge, respectively; the main-guest entity relationship type is the type of association relationship between the main and guest entity nodes connected by the current edge, including disease-symptom relationship, disease-drug relationship, disease-treatment plan relationship, disease-diagnosis relationship, diagnosis-examination relationship, and diagnosis-laboratory relationship; In the first node set V, the first node v with entity types of disease, symptom, drug, clinical treatment plan, medical diagnosis, medical examination, and medical test are respectively. i These are recorded as the corresponding disease nodes, symptom nodes, medication nodes, treatment plan nodes, diagnosis nodes, examination nodes, and laboratory test nodes; In the first node set V, each disease node can be directly connected as the main entity node to one or more symptom nodes, drug nodes, treatment plan nodes, or diagnostic nodes that are the object entities; each symptom node, drug node, treatment plan node, or diagnostic node that is the object entity node can be connected to one or more disease nodes that are the main entity nodes; however, there are no direct connections between the four types of nodes: symptom nodes, drug nodes, treatment plan nodes, and diagnostic nodes. In the first node set V, each diagnostic node may not be directly connected to any of the examination nodes or the test nodes, but may also act as the main entity node and be directly connected to one or more of the examination nodes or the test nodes that are the guest entity nodes; each examination node or the test node may only be directly connected to one of the diagnostic nodes that are the main entity node; there is no direct connection between the examination nodes and the test nodes. In the first node set V, each of the disease nodes, symptom nodes, drug nodes, and treatment plan nodes is configured based on the medical knowledge rule set; the first edge e between each disease node and its corresponding symptom node, drug node, or treatment plan node... k Then, refer to the medical entity relationship definition rules and the type categories of the subject-object entity relationship to set the corresponding directed edges; In the first node set V, each diagnostic node corresponds one-to-one with a first patient record in the patient database; each diagnostic node is configured based on the diagnostic classification field and the diagnostic certificate field of the corresponding first patient record; when the examination result field or laboratory result field corresponding to each diagnostic node is not empty, the examination node or laboratory node corresponding to the current diagnostic node is configured based on the non-empty examination result field or laboratory result field; the first edge e between each diagnostic node and its corresponding examination node or laboratory node... k Directed edges are set according to the configuration that the current diagnostic node is the primary entity node, the current examination node or laboratory node is the secondary entity node, and the primary-secondary entity relationship type is the corresponding diagnostic-examination relationship or diagnostic-laboratory relationship; the diagnostic disease field corresponding to each diagnostic node is matched with its corresponding disease node; the first edge e between each diagnostic node and its corresponding disease node... k The corresponding directed edges are set according to the configuration that the current disease node is the main entity node, the current diagnosis node is the guest entity node, and the main-guest entity relationship type is disease-diagnosis relationship.

4. The processing method for the multimodal medical knowledge graph embedding encoding model according to claim 3, characterized in that, The preprocessing layer is used to process each of the first nodes v in the knowledge graph M. i Perform node feature vector transformation to obtain the corresponding node feature vector. And for each of the first edges e of the knowledge graph M k Perform edge feature vector transformation to obtain the corresponding edge feature vector. ; and from the obtained N V The node feature vectors Form the corresponding node feature set H 0 The obtained N E The edge feature vectors Form the corresponding edge feature set G 0 ; and the obtained node feature set H 0 and the edge feature set G 0 The corresponding feature map X is formed 0 Send to the mapping layer and the first first coding layer respectively; Among them, the feature map X 0 The node feature set H 0 and the edge feature set G 0 Composition; the node feature set H 0 Including N V The node feature vectors The edge feature set G 0 Including N E The edge feature vectors ; The second GAT layer is used for the feature map X N Feature encoding is performed to obtain the corresponding feature map X. b Send to the residual layer; Among them, the feature map X b It consists of a node feature set Hb and an edge feature set Gb; the node feature set Hb includes N V The feature vector of each node hb i The edge feature set Gb includes N E Each edge feature vector gb k ; The mapping layer is used to process the feature map X. b The node feature vector space and edge feature vector space are used as the corresponding first and second target feature spaces; and the feature map X is... 0 The node feature set H 0 The feature vectors of each node Perform vector mapping processing on the first target feature space to obtain the corresponding node feature vector ha. i =W a1 ; and the feature map X 0 The edge feature set G 0 Each of the aforementioned edge feature vectors Perform vector mapping processing on the second target feature space to obtain the corresponding edge feature vector ga. k =W a2 ; and from the obtained N V The node feature vector ha i The corresponding node feature set Ha is formed, and the obtained N E The edge feature vector ga k The corresponding edge feature set Ga is formed; and the corresponding feature map X is formed by the node feature set Ha and the edge feature set Ga. a Send to the residual layer; Among them, W a1 W a2 These are the corresponding first and second feature mapping vectors; the feature map X a It consists of a node feature set Ha and an edge feature set Ga; the node feature set Ha includes N V The node feature vector ha i The edge feature set Ga includes N E The edge feature vector ga k ; The residual layer is used to process the feature map X. a X b The corresponding feature map M is obtained by performing residual calculation. * And output, M * =X a +X b ; Among them, the feature map M * The node feature set H * and the edge feature set G * Composition; the node feature set H * Including N V The node feature vectors , =ha i +hb i The edge feature set G * Including N E The edge feature vectors , =ga k +gb k .

5. The processing method for the multimodal medical knowledge graph embedding encoding model according to claim 4, characterized in that, The first nodes v of the knowledge graph M i Perform node feature vector transformation to obtain the corresponding node feature vector. Specifically, it includes: The preprocessing layer processes each of the first nodes v in the knowledge graph M. i As the corresponding current node; and the modality type, entity type, entity name, and entity attribute of the current node as the corresponding current modality, current type, current name, and current attribute; The current modality and the current type are then subjected to one-hot vector encoding to obtain the corresponding modality embedding vector and type embedding vector, respectively. The current name is then subjected to word embedding encoding to obtain the corresponding name word embedding vector; And identify the current mode; If the current modality is an image modality, then the current attribute is processed by visual feature encoding based on the built-in visual feature encoding model to obtain the corresponding attribute feature vector; the visual feature encoding model is implemented based on a convolutional neural network model or a graph neural network model; If the current modality is a text modality, then the current attribute is processed by text feature encoding based on the built-in text feature encoding model to obtain the corresponding attribute feature vector; the text feature encoding model is implemented based on the BERT series of models; The corresponding node feature vector is composed of the modality embedding vector, the type embedding vector, the name word embedding vector, and the attribute feature vector corresponding to the current node. .

6. The processing method for the multimodal medical knowledge graph embedding encoding model according to claim 4, characterized in that, The first edge e of the knowledge graph M k Perform edge feature vector transformation to obtain the corresponding edge feature vector. Specifically, it includes: The preprocessing layer processes each of the first edges e of the knowledge graph M. k As the corresponding current edge; and perform one-hot vector encoding on the subject-object entity relationship type of the current edge, and use the encoding result as the corresponding edge feature vector. .

7. The processing method for the multimodal medical knowledge graph embedding encoding model according to claim 4, characterized in that, The feature map X input from the current layer q-1 The first GAT layer is input to perform feature encoding processing to obtain the corresponding feature map Y. q Specifically, it includes: Step 71, the first GAT layer is used to process the feature map X q-1 Each of the aforementioned edge feature vectors This is used as the current edge feature vector; and the node feature vectors of the subject and object entity nodes corresponding to the current edge feature vector are recorded as the corresponding node feature vectors. , ; and derived from node feature vectors , The current edge feature vector is concatenated with it to form a corresponding triplet feature vector. ; Among them, the triplet feature vector The splicing method is as follows: ， 1≤indexj≤N V j≠i, || is the vector concatenation operator; Step 72, and process the feature vectors of each triplet. Perform a linear transformation to obtain the corresponding triplet feature vector. ; Among them, the triplet feature vector The linear transformation method is as follows: ； Wq 1 is the weight vector of the first GAT layer of the first encoding layer q, used to process the linear transformation of the triplet feature vector; Step 73, connect each of the first nodes v in the knowledge graph M i As the current node; and for the set of neighboring nodes corresponding to the current node. Perform identification; and identify the current set of neighboring nodes. The total number of neighboring nodes n i Perform identification; and then identify the current set of neighboring nodes. The range of index values ​​for all neighboring nodes in a given set is denoted as the corresponding neighboring node index range. ; and the current neighboring node index range The edge index value corresponding to each index j in the graph is denoted as the corresponding neighborhood edge index k. i,j ; and the current set of neighboring nodes Each neighboring node in the array is denoted as ; and each of the aforementioned neighboring nodes Corresponding triplet feature vector Recorded as ; and denoted as M the total number of multi-head self-attention operations of the first GAT layer of the first coding layer in the qth first coding layer. q And assign a corresponding single-head index m to each single-head self-attention operation of the current first GAT layer; and assign each triplet feature vector Perform M q Perform head self-attention operation and record the self-attention operation result of the m-th single head during the operation as the corresponding single-head single-sided self-attention feature vector. And perform the corresponding n operations on the m-th head self-attention. i The single-head, single-sided self-attention feature vector Perform summation and nonlinear activation, and use the nonlinear activation result as the corresponding single-head neighborhood self-attention feature vector. ; and the obtained M q The single-head neighborhood self-attention feature vector Perform concatenation and use the concatenation result as the node feature vector corresponding to the current node. ; Among them, the neighborhood node set By n i The first node v is currently i The directly connected neighboring nodes Composition; Total number of neighboring nodes n i ≥1; The node feature vector The single-head neighborhood self-attention feature vector The single-head, single-sided self-attention feature vector The calculation method is as follows: , , ， ； 1 ≤ Single-headed index m ≤ M q σ is a preset nonlinear activation function; In the m-th single-head self-attention operation of the first GAT layer of the first coding layer of the q-th layer, and with the j-th... The triplet feature vectors The corresponding self-attention weights; W q(m) is the linear transformation weight vector corresponding to the m-th single-head self-attention operation of the first GAT layer of the first coding layer of the q-th first coding layer; LeakyReLu is the LeakyReLu activation function; Step 74, and process the feature map X q-1 Each of the aforementioned edge feature vectors Perform a linear transformation to obtain the corresponding edge feature vector. ; Wherein, the edge feature vector The calculation method is as follows: ； The weight vector of the first GAT layer of the qth first coding layer is used to process the linear transformation of the edge feature vector; Step 75, and from the obtained N V The node feature vectors The corresponding node feature set Hy is formed q ; The obtained N E The edge feature vectors The corresponding edge feature set Gy is formed q ; and by the node feature set Hy q and the edge feature set Gy q The corresponding feature map Y q .

8. The processing method for the multimodal medical knowledge graph embedding encoding model according to claim 4, characterized in that, The feature map X N Feature encoding is performed to obtain the corresponding feature map X. b Specifically, it includes: Step 91, the second GAT layer is used to process the feature map X N The feature vectors of each edge This is used as the current edge feature vector; and the node feature vectors of the subject and object entity nodes corresponding to the current edge feature vector are recorded as the corresponding node feature vectors. , ; and derived from node feature vectors , The current edge feature vector is concatenated with it to form a corresponding triplet feature vector. ; Among them, the triplet feature vector The splicing method is as follows: ， 1≤indexj≤N V j≠i, || is the vector concatenation operator; Step 92, and process the feature vectors of each of the triplets. Perform a linear transformation to obtain the corresponding triplet feature vector. ; Among them, the triplet feature vector The linear transformation method is as follows: ； This is the weight vector used by the second GAT layer to process the linear transformation of triplet feature vectors; Step 93, connect each of the first nodes v in the knowledge graph M i As the current node; and for the set of neighboring nodes corresponding to the current node. Perform identification; and identify the current set of neighboring nodes. The total number of neighboring nodes n i Perform identification; and then identify the current set of neighboring nodes. The range of index values ​​for all neighboring nodes in a given set is denoted as the corresponding neighboring node index range. ; and the current neighboring node index range The edge index value corresponding to each index j in the graph is denoted as the corresponding neighborhood edge index k. i,j ; and the current set of neighboring nodes Each neighboring node in the array is denoted as ; and each of the aforementioned neighboring nodes Corresponding triplet feature vector Recorded as The total number of multi-head self-attention operations in the second GAT layer is denoted as M. N+1 And assign a corresponding single-head index m to each single-head self-attention operation of the second GAT layer; and assign each triplet feature vector Perform M N+1 Perform head self-attention operation and record the self-attention operation result of the m-th single head during the operation as the corresponding single-head single-sided self-attention feature vector. And perform the corresponding n operations on the m-th head self-attention. i The single-head, single-sided self-attention feature vector Perform summation and nonlinear activation, and use the nonlinear activation result as the corresponding single-head neighborhood self-attention feature vector. ; and the obtained M N+1 The single-head neighborhood self-attention feature vector Perform concatenation and use the concatenation result as the node feature vector corresponding to the current node. ; Among them, the neighborhood node set By n i The first node v is currently i The directly connected neighboring nodes Composition; Total number of neighboring nodes n i ≥1; The node feature vector The single-head neighborhood self-attention feature vector The single-head, single-sided self-attention feature vector The calculation method is as follows: , , ， ； 1 ≤ Single-headed index m ≤ M N+1 σ is a preset nonlinear activation function; In the m-th single-head self-attention operation of the second GAT layer, and the j-th... The triplet feature vectors The corresponding self-attention weights; W N+1(m) Let be the linear transformation weight vector corresponding to the m-th single-head self-attention operation of the second GAT layer; LeakyReLu is the LeakyReLu activation function; Step 94, and process the feature map X N Each of the aforementioned edge feature vectors Perform a linear transformation to obtain the corresponding edge feature vector. ; Wherein, the edge feature vector The calculation method is as follows: ； This is the weight vector used by the second GAT layer to process the linear transformation of the edge feature vectors; Step 95, and from the obtained feature vectors of each node As a corresponding node feature vector hb i Each of the aforementioned edge feature vectors As a corresponding edge feature vector gb k ; and from the obtained N V The node feature vector hb i The corresponding node feature set Hb is composed, and N is obtained. E The edge feature vector gb k The corresponding edge feature set Gb is formed; and the corresponding feature graph X is formed by the node feature set Hb and the edge feature set Gb. b And output it.

9. The processing method for the multimodal medical knowledge graph embedding encoding model according to claim 1, characterized in that, The pre-training of the embedding encoding model based on the knowledge graph M specifically includes: Step 101: Construct a second edge set by traversing the first edge set E of the knowledge graph M; Wherein, the second edge set consists of one or more second edges e x Composed of, and any two of the second edges e in the second edge set x There is no connection between them; 1 ≤ index x ≤ N G N G This represents the total number of edges in the second edge set; Step 102: Use the knowledge graph M as a positive sample graph M + ; and copy the knowledge graph M to obtain the initialized negative sample graph M. - ; Step 103, set each of the second edges e in the second edge set x As the current sample edge; and the positive sample graph M is used as the current sample edge; + The negative sample map M - The corresponding edge of the current sample edge is used as the corresponding current positive sample edge and current negative sample edge; and the current positive sample edge and its corresponding subject and object entity nodes form a corresponding positive sample triple z. x ; and in the negative sample spectrum M - In this process, the subject and object entity nodes of the current negative sample edge are swapped, and a corresponding negative sample triple is formed by the current negative sample edge and its swapped subject and object entity nodes. ; Wherein, the positive sample triple z x for , , These are the subject and object entity nodes corresponding to the current positive sample, r. x The subject-object entity relationship corresponding to the current positive sample; the negative sample triple. for , , These are the subject and object entity nodes corresponding to the current negative sample, respectively. For the current negative sample, the subject-object entity relationship is... , , ; Step 104, transfer the positive sample spectrum M + The current knowledge graph M is input into the embedding encoding model for processing, and the feature graph M obtained from this processing is... * As the corresponding positive sample feature map ; and the negative sample map M - The current knowledge graph M is input into the embedding encoding model for processing, and the feature graph M obtained from this processing is... * As the corresponding negative sample feature map ; Step 105, the positive sample feature map is... In the above, each of the positive sample triples z x main entity node Customer entity node Subject-object entity relationship The corresponding two node feature vectors and one edge feature vector serve as the corresponding principal vector. relation vector g x Object vector ; and the negative sample feature map In the context of each of the negative sample triples main entity node Customer entity node Subject-object entity relationship The corresponding two node feature vectors and one edge feature vector serve as the corresponding principal vector. relation vectors Object vector ; and composed of each of the aforementioned positive sample triples z x The corresponding main vector The relation vector g x The object vector Form a corresponding positive sample feature group p x ( ,g x , ); and each of the aforementioned negative sample triples The corresponding main vector The relation vector The object vector Form a corresponding negative sample feature group ( , , ); Step 106, obtain N G p, the positive sample feature groups x ( ,g x , ) and N G The negative sample feature group ( , , Substitute the preset model loss function L mod The corresponding first loss value is obtained through calculation; Wherein, the model loss function L model Based on the contrastive loss function: ， , ; d() is the subject-object distance function for the triple, ⊙ is the Hadamard product, and 1 ≤ index x ’ ≤N G ; Step 107: Identify whether the obtained first loss value meets the preset first loss value range; if not, then based on the preset first model optimizer, move towards making the model loss function L... mod The model parameters of the embedding coding model are modulated once in the direction that reaches the minimum value, and the process returns to step 104 when the current modulation is completed; if the condition is not met, training is stopped and the pre-training is confirmed to be over. The first model optimizer includes the Adam optimizer and the SGD optimizer.

10. The processing method for the multimodal medical knowledge graph embedding encoding model according to claim 3, characterized in that, The medical graph reasoning task based on the embedded coding model for processing the knowledge graph M specifically includes: The knowledge graph M is input into the embedding coding model for processing, and the feature graph M obtained from this processing is... * As the current feature map; The system receives first query data input by the user; the first query data includes first query text and first query image; the first query text is the corresponding first diagnosis certificate, first examination report or first laboratory report; when the first query image is not empty, it is the corresponding first medical examination image. Based on the first query text, a virtual node of a specific text modality is constructed according to the node format of the diagnostic node, the inspection node, or the inspection node corresponding to the current query text, serving as the current text node. The preprocessing layer is then used to perform node feature vector transformation on the current text node, and the node feature vector obtained from this transformation is used as the corresponding first feature vector. Finally, the node feature vectors corresponding to each diagnostic node, inspection node, or inspection node in the current feature map that are relevant to the current query text are... As the corresponding first candidate vector; and the first candidate vector set is composed of all the obtained first candidate vectors; The system identifies whether the first query image is empty; if it is empty, the corresponding second candidate vector set is set to empty; if it is not empty, a virtual node representing an image modality is constructed based on the current query image and the node format of the inspection node as the current image node. The preprocessing layer is then used to perform node feature vector transformation on the current image node, and the node feature vector obtained from this transformation is used as the corresponding second feature vector. The node feature vectors corresponding to each inspection node representing an image modality in the current feature map are then used as the second feature vector. As the corresponding second candidate vector, and all the obtained second candidate vectors form the corresponding second candidate vector set; The similarity between each candidate vector in the first candidate vector set and the first feature vector is calculated to obtain the corresponding first similarity; and the first similarity exceeding a preset first similarity threshold is taken as the corresponding second similarity; and the first node v corresponding to all second similarities is... i Form the corresponding first candidate node set; The system then identifies whether the second candidate vector set is empty; if so, it sets the corresponding second candidate node set to empty; otherwise, it calculates the similarity between each second candidate vector in the second candidate vector set and the second feature vector to obtain the corresponding third similarity, and takes the third similarity exceeding a preset second similarity threshold as the corresponding fourth similarity, and uses the first node v corresponding to all the fourth similarities as the fourth similarity. i Form the corresponding second candidate node set; The first and second candidate node sets are merged to obtain a corresponding merged node set; and the first node v with the highest similarity in the merged node set is selected. i As the current node; and the disease nodes in the knowledge graph M that have a direct or indirect connection with the current node as the current screening nodes; and the current screening node and all the corresponding symptom nodes, drug nodes and treatment nodes constitute the corresponding current screening subgraph; The corresponding medical atlas reasoning report, composed of all node attributes of the currently selected sub-graph, is then fed back to the current user.

11. An apparatus for performing the processing method of the multimodal medical knowledge graph embedding coding model according to any one of claims 1-10, characterized in that, The device includes: a knowledge graph construction module, an encoding model construction module, a model pre-training module, and a graph task processing module; The knowledge graph construction module is used to construct a patient database by collecting patient medical data; and to construct a corresponding multimodal medical knowledge graph, denoted as knowledge graph M, based on a preset set of medical knowledge rules and the patient database; the knowledge graph M consists of a first node set V and a first edge set E; the first node set V includes multiple first nodes v i 1 ≤ index i ≤ N V N V The total number of nodes; the first edge set E includes multiple first edges e k 1 ≤ index k ≤ N E N E Let e ​​be the total number of edges; for each of the first edges e k A directed edge connecting the two first nodes v i The starting node of a directed edge is denoted as the main entity node, and the ending node of a directed edge is denoted as the guest entity node. The encoding model construction module is used to construct a corresponding embedding encoding model for the knowledge graph M; the embedding encoding model is used to perform knowledge graph embedding encoding processing on the knowledge graph M input to the model and output the corresponding feature graph M. * The feature map M * From the node feature set H * Edge feature set G * Composition; the node feature set H * Including N V Node feature vectors The edge feature set G * Including N E Each edge feature vector The node feature vector With the first node v i One-to-one correspondence; the edge feature vector With the first edge e k One-to-one correspondence; The model pre-training module pre-trains the embedding coding model based on the knowledge graph M; The graph task processing module is used to process the medical graph reasoning task of the knowledge graph M based on the embedding encoding model after pre-training.

12. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-10; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-10.

Images