Multi-modal knowledge graph privacy protection embedding method for diagnosis and treatment data

Abstract

The invention provides a multi-modal knowledge graph privacy protection embedding method for diagnosis and treatment data, and relates to the technical field of data processing, and the method comprises the steps: carrying out the initialization embedding processing of text information and numerical information based on the diagnosis and treatment data, and carrying out the initialization embedding processing of visual information based on medical image visual data; obtaining a multi-modal attention coefficient of a neighbor entity by using a multi-modal diagram self-attention layer; fusing the structured information of the multi-modal knowledge graph by stacking a plurality of encoder layers; and the multi-modal knowledge graph privacy protection embedding is scored by using the convolution embedding model. According to the method, efficient embedding and deep fusion of multi-modal information are effectively achieved, the processing capacity of heterogeneous data is enhanced, sensitive information is prevented from being leaked through the embedding process, an end-to-end privacy protection mechanism is provided, and the safety and reliability of diagnosis and treatment data in the analysis and reasoning process are ensured.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a privacy-preserving embedding method for a multimodal knowledge graph of medical data. Background Art

[0002] A knowledge graph is a model that organizes and displays data through a graph structure, where nodes represent entities and edges represent relationships between entities. As data complexity increases, knowledge graphs have gradually evolved into multimodal knowledge graphs, integrating data from different modalities such as text and images, enriching the description of entities and enhancing reasoning capabilities. In order to better process these structured data, knowledge graph embedding technology has emerged. By mapping entities and relationships into low-dimensional vector spaces, knowledge graphs can be processed through computation and applied to tasks such as knowledge completion and relationship prediction. Multimodal knowledge graph embedding further embeds modal data such as text and images into a unified vector space, improving the model's reasoning and prediction capabilities in a multimodal environment.

[0003] In the medical field, diagnosis and treatment data has a highly multimodal nature, including information such as text, images, and numerical values. The medical diagnosis and treatment data knowledge graph helps improve intelligent diagnosis and personalized medical services through structured modeling of data such as patients, diseases, and treatment plans. However, diagnosis and treatment data contains sensitive personal privacy information, such as patient medical records and images, which may pose a serious threat to the privacy of patients once leaked. Therefore, how to maintain data analysis capabilities while ensuring privacy security has become an important challenge in the application of medical knowledge graphs.

[0004] The existing technology combines latent features, relational features and numerical features to perform end-to-end knowledge graph embedding learning to improve the performance of knowledge graph completion tasks. The existing technology also uses different neural encoders for different data types and combines them with traditional relational models to enhance the embedding of entities and multimodal data. The existing technology also proposes an adversarial feature learning method, which effectively maps the text and image data of entities into a unified vector space through a multi-relational feature aggregation network to capture multimodal information. The existing technology also introduces a hierarchical attention network for multimodal medical knowledge graphs, providing a solution for explanatory medical question and answer.

[0005] However, due to the complex intra-modal and inter-modal interactions of entities containing multimodal data and the heterogeneity of multimodal knowledge graphs, existing multimodal knowledge graph embedding methods are not accurate enough in processing the complex interactions of multimodal data, cannot fully capture the relationship between different modalities, and are difficult to achieve end-to-end training. They are also limited in processing heterogeneous graph structures and multi-relationship chains, making it difficult to fully integrate the characteristics of multimodal data, resulting in limited model performance. In addition, existing multimodal knowledge graph embedding technologies are deficient in privacy protection and cannot effectively prevent potential reconstruction of data, leading to the leakage of sensitive information. There is no mechanism specifically designed for privacy protection in the process of multimodal data processing, making it difficult to ensure data security during analysis and sharing. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a privacy-preserving embedding method for a multimodal knowledge graph for medical data, so as to solve the technical problems existing in the existing multimodal knowledge graph embedding methods, such as insufficient complex interactive processing of multimodal data, limited heterogeneous graph processing capabilities, difficulty in achieving end-to-end training, lack of privacy protection mechanism, and difficulty in preventing leakage of sensitive information.

[0007] In order to achieve the above objectives, the technical solution adopted by the present invention is: a privacy-preserving embedding method for a multimodal knowledge graph of medical data, comprising the following steps:

[0008] S1. Initialize and embed text information and numerical information based on diagnosis and treatment data, and initialize and embed visual information based on medical image visual data;

[0009] S2. Based on the initial embedding result, the multimodal graph self-attention layer is used to obtain the multimodal attention coefficient of the neighbor entity;

[0010] S3, according to the multimodal attention coefficient, the structured information of the multimodal knowledge graph is fused by stacking multiple encoder layers;

[0011] S4. Based on the structured information of the fused multimodal knowledge graph, the convolutional embedding model is used to score the privacy-preserving embedding of the multimodal knowledge graph.

[0012] The beneficial effects of the present invention are: the present invention can efficiently process multimodal information such as text, images and numerical values ​​in medical data, and further deepen it through multi-layer encoders, collect multi-hop neighbor information and its related multimodal medical data, and reduce the risk of sensitive information exposure as a whole through the embedding process, thereby enhancing data privacy. In order to solve the technical problems of the existing multimodal knowledge graph embedding methods, such as insufficient complex interactive processing of multimodal data, limited heterogeneous graph processing capabilities, difficulty in achieving end-to-end training, lack of privacy protection mechanism, and difficulty in preventing sensitive information leakage, the present invention can effectively achieve efficient embedding and deep fusion of multimodal information, enhance the processing capabilities of heterogeneous data, and prevent sensitive information leakage through the embedding process, provide an end-to-end privacy protection mechanism, ensure the security and reliability of medical data in the analysis and reasoning process, and enable multimodal data and structural information to be jointly learned under a unified framework, simplify the training process and improve model performance.

[0013] Furthermore, the step S1 includes the following steps:

[0014] S101. Use the pre-trained language model to obtain the context information of the sentence-level description of entities and relations in the diagnosis and treatment data, and generate text information embedding t ;

[0015] S102: Extract visual information and embed it into medical image visual data v ;

[0016] S103, encode the entity numerical features in the diagnosis and treatment data and process them in the fully connected layer in turn to generate numerical data embedding e n , complete the initialization embedding of text information, visual information and numerical information.

[0017] The beneficial effect of the above further scheme is that the present invention reduces the risk of direct exposure of sensitive information by converting medical data into low-dimensional vector representation, effectively avoids the reverse derivation of original information from medical data during analysis, transmission and storage, and improves the effect of privacy protection. It can simultaneously process multimodal data such as text, images and numerical values ​​in the medical data knowledge graph, and provide a more comprehensive embedded representation for entities.

[0018] Furthermore, step S2 includes the following steps:

[0019] S201, using text information to embed e t and numerical data embedded in e n , construct an embedding matrix, and use visual information to embed e v , construct the modality matrix, where each modality graph attention layer processes two embedding matrices and one modality matrix;

[0020] S202. Utilize the multimodal graph self-attention layer to obtain a multimodal attention coefficient.

[0021] The beneficial effect of the above further scheme is that the present invention fuses multimodal information through a multimodal embedding matrix and a self-attention layer, thereby improving the comprehensiveness of feature representation and data comprehension capabilities.

[0022] Furthermore, the expression of the multimodal attention coefficient is as follows:

[0023]

[0024] c jk =W1[h j ||g k ||m k ]

[0025] Among them, b ijk and Both represent multimodal attention coefficients, c i Represents the central entity e i The eigenvector of jk Represents the central entity e i In comparison, the tail entity e j and the relationship k The composite entity vector, d k Represents the dimension of the vector, used to normalize the result of the vector dot product, exp() represents the natural exponential function, used to convert b ijk The value of b is mapped to the positive range, inr Represents entity e i The absolute attention parameter between the neighbor entity n through the relation r, n represents the entity e i A neighbor node of i Represents entity e i The first-order neighbor set of r represents the connection center entity e i A relation with a neighbor entity, R in Represents entity e i The relationship between the connection and the neighbors, W1 represents the learnable weight matrix, || represents the connection operation, and h j and g k Respectively represent the tail entity e j The vector and relationship r k The vector, m k Represents multimodal features.

[0026] The beneficial effect of the above further scheme is that the present invention can effectively distinguish the influence of different neighbors and modalities on the central entity through the calculation of the multimodal attention coefficient, thereby improving the representation accuracy of the multimodal knowledge graph.

[0027] Furthermore, step S3 includes the following steps:

[0028] S301, linearly fuse the multimodal attention coefficients to the central node to generate an embedded representation;

[0029] S302. Connect the embedding representations generated from M independent self-attention layers using the following formula:

[0030]

[0031] in, Represents the output after the multi-head self-attention mechanism, MultiHead() represents the multi-head self-attention mechanism, represents the input of the mth attention head, i.e. entity e i The representation of head m represents the output of the mth attention head, W O represents the matrix of input linear layer weights, j represents entity e i The first-order neighbor, N i Represents entity e i The first-order neighbor set of Represents the neighbor node e j For the central node e i The attention weight, Represents the neighbor node e j The feature representation combined with relation k is used in the mth attention head;

[0032] S303. According to the connection result, the structured information of the knowledge graph is integrated by stacking multiple encoder layers using the following formula:

[0033]

[0034] in, Represents entity e i In the embedding representation of the l+1th layer, σ represents a nonlinear activation function, and R represents the set of relations, that is, all relation types in the knowledge graph. Represents entity e i The neighbor set of c i,r represents the factor used for normalization, adjusting between different relationships, represents the attention weight from neighbor node j to center node i in layer l, Represents a combination function for embedding node i and the relationship between nodes j and i Perform feature transformation.

[0035] The beneficial effect of the above further scheme is: the present invention aggregates neighborhood information through a multi-head attention mechanism, which can better capture the diverse semantic features of nodes, and extract local features of complex relationships through convolution operations, making node embedding more expressive, thereby improving the reasoning performance of the multimodal knowledge graph and the accuracy of node representation.

[0036] Furthermore, the expression of the embedding vector is as follows:

[0037]

[0038] Among them, x i Represents entity e i The updated embedding vector, σ represents the nonlinear activation function, and j represents the entity e i The first-order neighbor, N i Represents entity e i The first-order neighbor set, k represents the relationship between neighbor entity j and another node, R jj represents the set of relationships between neighbor entity j and nodes other than neighbor entity j, α ijk represents the attention weight coefficient, which represents the contribution of neighbor node j and relationship k to the central node i, c jk Represents the feature representation of neighbor node j and relationship k.

[0039] The beneficial effect of the above further scheme is that the present invention can weightedly integrate the contributions of different neighbors to the central node by aggregating the embedded information of neighbor nodes and their relationships, thereby generating a more refined embedded representation and improving the accuracy and expressiveness of entity representation in the multimodal knowledge graph.

[0040] Furthermore, the expression of the scoring function of the convolutional embedding model is as follows:

[0041] f(h,r,t)=σ(vec(W([e h ;r]*ω))e t )

[0042] Among them, f() represents the scoring function of the triple, h represents the head entity, and r represents the connection center entity e i A relationship with a neighbor entity, t represents the tail entity, σ represents the sigmoid activation function, vec represents the operation of projecting the feature map into the vector space, W represents the transformation matrix, e h and e t They represent the embedding of the head entity and the tail entity respectively, r represents the embedding vector of the relationship, ω represents the weight matrix of the convolution kernel, and * represents the convolution operation.

[0043] The beneficial effect of the above further scheme is that the present invention can extract the complex interactive features of the head entity and the relationship through the fusion of convolution operation and multimodal features, and match them with the tail entity, thereby improving the accuracy of triple scoring and enhancing the reasoning ability of multimodal knowledge graph completion.

[0044] Furthermore, the loss function of the convolutional embedding model is expressed as follows:

[0045]

[0046] in, represents the loss function of the ConvE model, Represents the number of fact triples, y i' represents the calculated score of the i'th triple, f i' Represents a label, where 1 represents true facts and 0 represents false information.

[0047] The beneficial effect of the above further scheme is that the present invention can effectively measure the prediction error of the model on the authenticity of the triples by adopting the binary cross entropy loss function, thereby optimizing the training process of the multimodal knowledge graph completion model and improving the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The figure is a flow chart of the method of the present invention.

[0049] Figure 2 This is a brief flowchart of the SAGNN model.

[0050] Figure 3 A simplified schematic diagram of a multimodal graph self-attention encoder.

[0051] Figure 4 A simple schematic diagram of graph aggregation.

[0052] Figure 5 A simplified diagram of the ConvE decoder. DETAILED DESCRIPTION

[0053] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0054] Example

[0055] The present invention proposes a privacy-preserving embedding method for multimodal medical data knowledge graph based on self-attention graph neural network, referred to as SAGNN (Self-Attention Graph Neural Network model). The SAGNN encoder integrates neighborhood, global structure and multimodal information, and can efficiently process multimodal information such as text, images and numerical values ​​in medical data, and further deepens it through multi-layer encoders to collect multi-hop neighbor information and its related multimodal medical data. The risk of sensitive information exposure is reduced as a whole through the embedding process, thereby enhancing data privacy. The overall structure of the SAGNN model is as follows: Figure 1 As shown. In the encoder, two key modules are distinguished: the multimodal graph attention module and the multimodal aggregation module:

[0056] Multimodal Graph Self-Attention Encoder: This module dynamically updates composite entities and their features, including text, visual, and numerical information in medical data. It leads the embedding learning in the complex multimodal medical data graph structure, and the overall embedding process can effectively reduce the exposure risk of medical data, thereby improving privacy protection.

[0057] Graph aggregation module: Under the guidance of the attention layer scores (these scores take into account the multimodal information in the diagnosis and treatment data), this module integrates the neighbor information and its modality into the central node, and the generated node representation is the fusion result of the neighborhood and multimodal data. By embedding the data in a low-dimensional space, the direct exposure of sensitive information during reasoning and processing is avoided, thereby improving the privacy protection effect.

[0058] Before describing the present invention, the following description is given:

[0059] The multimodal knowledge graph is an enhanced directed graph G = (N, R, F, M), where N represents the entity set, R represents the relationship set, F represents the set of fact triples, and M represents the multimodal data related to the entity. Each triple in the graph is represented as {h, r, t}, where are the head entity and the tail entity, r∈R as the relationship between them, and M contains multimodal information related to the entities (e.g., text, images, and numerical data).

[0060] The Graph Attention Network (GAT) circumvents the limitation of traditional graph convolutional networks that use fixed weights when weighting neighborhoods. By leveraging the subtle complexity of the attention mechanism, GAT is able to assign different weights to nodes, thereby identifying the differential contributions of different nodes to the central node. The core of the graph attention network lies in its use of attention coefficients. These coefficients basically determine how a node distributes attention among its neighbors, ensuring that different weights are assigned to different nodes. For a pair of adjacent nodes i and j in the graph, the attention coefficient e ij The calculation method is as follows:

[0061]

[0062] Here, h i and h j Represent the feature vectors of nodes i and j respectively. The matrix W represents the weight matrix, and a represents the learnable weight vector. The symbol || represents concatenation, and LeakyReLU is the selected nonlinear activation function.

[0063] In order to seamlessly integrate these coefficients into the weighted information aggregation framework, they are normalized by the softmax function:

[0064]

[0065] Here, N(i) represents the neighborhood range of node i.

[0066] Each node integrates its neighborhood information by extracting information from the normalized attention coefficient:

[0067]

[0068] Here, σ represents an activation function, such as the ReLU function.

[0069] In order to enhance the expressiveness of the network and introduce a richer set of parameters, the graph attention network adopts a multi-head attention mechanism. Attention calculations are performed in parallel through different sets of parameters, and the results are then concatenated or averaged. In the evolving field of neural network architectures, Transformer is mainly used for sequence transformation tasks, symbolizing a shift from recursive methods to a new paradigm, with the core mechanism being the self-attention mechanism. This mechanism enables the model to access the importance of different positions in the input sequence without considering their relative positions. The self-attention mechanism first projects the input X into three different subspaces: query (Query, Q), key (Key, K), and value (Value, V):

[0070] Q=XW Q

[0071] K=XW K

[0072] V=XW V

[0073] Among them, W Q , W K and W V Represent the weight matrices associated with their respective subspaces.

[0074] Subsequently, the attention score calculation is performed, and the alignment of the query and the key produces an attention score, which is normalized by the softmax function to determine the importance of each value in the sequence:

[0075]

[0076] Among them, the normalization factor (d k is the dimension of the key vector) ensures the stability of the calculation.

[0077] The self-attention mechanism perceives information from multiple perspectives, called "heads". Each head projects the input into an independent subspace and computes an attention score. Their concatenated output lays the foundation for a richer representation:

[0078] MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O

[0079] For each head i:

[0080] head i =Atten(QW Qi ,KW Ki ,VW Vi )

[0081] Among them, W Qi , W Ki and W Vi Respectively represent the weight matrix of the i-th head, W O is the output linear layer weight matrix.

[0082] like Figure 1-2 As shown, the present invention provides a privacy-preserving embedding method for a multimodal knowledge graph of medical data, and its implementation method is as follows:

[0083] S1. Based on the diagnosis and treatment data, the text information and numerical information are initialized and embedded, and based on the medical image visual data, the visual information is initialized and embedded. The implementation method is as follows:

[0084] S101. Use the pre-trained language model to obtain the context information of the sentence-level description of entities and relations in the diagnosis and treatment data, and generate text information embedding t ;

[0085] S102: Extract visual information and embed it into medical image visual data v ;

[0086] S103, encode the entity numerical features in the diagnosis and treatment data and process them in the fully connected layer in turn to generate numerical data embedding e n , complete the initialization embedding of text information, visual information and numerical information.

[0087] In this embodiment, each modality information is learned independently through its respective modality-specific pre-trained encoder, as follows:

[0088] Text information: Use pre-trained language models to capture the contextual information of sentence-level descriptions of entities and relations in medical data. Specifically, Sentence-BERT is used to generate a 768-dimensional text information embedding t , reducing the exposure of sensitive information in the text by embedding it.

[0089] Visual information: For medical image visual data, the VGG-16 convolutional neural network model is used. By discarding the final fully connected layer and softmax layer, a 4096-dimensional visual information embedding is extracted. v , ensuring that patient privacy information will not be directly reconstructed during visual feature processing.

[0090] Numerical information: Numerical features of entities in the medical data are encoded by BERT and then processed by a fully connected layer. Here, attribute keys and values ​​are concatenated to generate a 768-dimensional numerical data embedding. n , effectively protecting sensitive numerical information in the representation of numerical data.

[0091] S2. According to the initial embedding result, the multimodal graph self-attention layer is used to obtain the multimodal attention coefficient of the neighbor entity. The implementation method is as follows:

[0092] S201, using text information to embed e t and numerical data embedded in e n , construct an embedding matrix, and use visual information to embed e v , construct the modality matrix, where each modality graph attention layer processes two embedding matrices and one modality matrix;

[0093] S202. Utilize the multimodal graph self-attention layer to obtain a multimodal attention coefficient.

[0094] In this embodiment, for a vertex e in the multimodal graph structure G i , learning entity e i (A vertex in the multimodal graph is an entity) i ′. Multiple graph attention layers are constructed, each of which processes two embedding matrices and an additional modality matrix. The entity embedding matrix is ​​represented as Among them, the i-th row corresponds to entity e i The embedding of N e represents the total number of entities, and T represents the feature dimension of each entity embedding. At the same time, the relationship embedding is expressed as Mode Matrix Encapsulate multimodal data. Since entities and relations are mainly composed of text information and numerical information, the embedding matrix of the two is obtained by embedding the text information into e t and numerical data embedding n Similarly, the modal matrix is ​​composed of the obtained visual information embedding e v After being processed by each attention layer, the output matrix is ​​H′, R′, M′.

[0095] In order to i Get the updated embedding h i ', learn from the above multimodal information matrix of entities, relations and associations. The learning depth of each source is determined by the multimodal attention coefficient α ijk Control, for entities (e i ,e j ) and its modality. The input features are transformed into higher linear combinations based on specific triples. The local feature connection is transformed, and the multimodal aspect is mathematically defined as follows:

[0096] c jk =W1[h j ||g k ||m k ]

[0097] Among them, c jk Represents the central entity e i In comparison, the tail entity e j and the relationship k The composite entity vector h j and g k Respectively represent the tail entity e j The vector and relationship r k The vector, m k Represents multimodal features, matrix represents a learnable weight matrix and || represents a connection operation.

[0098] For fine-grained feature associations between different modalities, the dot product between the central entity embedding and the composite entity embedding is calculated to determine the absolute weight. The weights are normalized by softmax to generate the relative weights of the triplets, taking into account multimodal data. This method captures the relative weights of the triplets iteratively through each attention layer of the graph, as Figure 3 shown.

[0099] The model relies primarily on immediate neighbors and the nodes themselves, leveraging their multimodal properties as sources of information (although multiple iterations through multi-hop features are captured by the attention layers entering the graph). The attention parameter b ijk and α ijk Both indicate entity e j For i The contribution of the triple (e i ,e j ,r k ), these weights are calculated as follows:

[0100]

[0101] Among them, N i Represents entity e i The first-order neighbor set, R in Represents entity e i The relationship between connections with neighbors.

[0102] S3. According to the multimodal attention coefficient, the structured information of the multimodal knowledge graph is fused by stacking multiple encoder layers. The implementation method is as follows:

[0103] S301, linearly fuse the multimodal attention coefficients to the central node to generate an embedded representation;

[0104] S302. Connect the embedding representations generated from M independent self-attention layers:

[0105] S303. According to the connection result, use the following formula to fuse the structured information of the knowledge graph by stacking multiple encoder layers.

[0106] In this embodiment, these features are linearly fused to the central node in combination with the attention coefficients of the neighboring entities calculated previously to generate a more refined embedding vector. Mathematically speaking, entity e i The updated embedding of is the weighted sum of the triplet embeddings, each determined by its individual contribution. It is expressed as follows:

[0107]

[0108] Among them, x i Represents entity e iThe updated embedding vector, σ represents the nonlinear activation function, and j represents the entity e i The first-order neighbor, N i Represents entity e i The first-order neighbor set, k represents the relationship between neighbor entity j and another node, R jj represents the set of relationships between neighbor entity j and nodes other than neighbor entity j, α ijk represents the attention weight coefficient, which represents the contribution of neighbor node j and relationship k to the central node i, c jk Represents the feature representation of neighbor node j and relationship k.

[0109] This model enhances a single attention layer and uses multiple attention heads to process, so that the encoder can absorb various semantic nuances from the context during the embedding learning process. Specifically, it generates embedding representations from M independent self-attention layers and concatenates these embeddings to provide a single vector. Figure 4 As shown, the process can be formalized as:

[0110]

[0111] For each head m :

[0112]

[0113] By stacking multiple encoder layers, each node aggregates data from its multi-hop neighbors, effectively fusing the structured information of the entire knowledge graph and generating node representations that are essentially consistent with the surrounding context.

[0114]

[0115] in, Represents entity e i In the embedding representation of the l+1th layer, σ represents a nonlinear activation function, and R represents the set of relations, that is, all relation types in the graph. Represents entity e i The neighbor set of c i,r represents the factor used for normalization, adjusting between different relationships, represents the attention weight from neighbor node j to center node i in layer l, Represents a combination function for embedding node i and the relationship between nodes j and i Perform feature transformation, Represents the output after the multi-head self-attention mechanism, MultiHead() represents the multi-head self-attention mechanism, represents the input of the mth attention head, i.e. entity e i The representation of head m represents the output of the mth attention head, W O represents the matrix of input linear layer weights, j represents entity e i The first-order neighbor, N i Represents entity e i The first-order neighbor set of Represents the neighbor node e j For the central node e i The attention weight, Represents the neighbor node e j The feature representation combined with relation k is used in the mth attention head.

[0116] S4. Based on the structured information of the fused multimodal knowledge graph, the convolutional embedding model is used to score the privacy-preserving embedding of the multimodal knowledge graph.

[0117] In this example, the first three steps have completed the multimodal knowledge graph embedding. The decoding step uses these embeddings to Figure 5 The convolutional embedding model ConvE shown is used to evaluate the score of fact triples, thereby enhancing the ability of multimodal knowledge graph completion (KGC). The scoring function is defined as follows:

[0118] f(h,r,t)=σ(vec(W([e h ;r]*ω))e t )

[0119] Among them, f() represents the scoring function of the triple, h represents the head entity, and r represents the connection center entity e i A relationship with a neighbor entity, t represents the tail entity, σ represents the sigmoid activation function, vec represents the operation of projecting the feature map into the vector space, W represents the transformation matrix, e h and e t They represent the embedding of the head entity and the tail entity respectively, r represents the embedding vector of the relationship, ω represents the weight matrix of the convolution kernel, and * represents the convolution operation.

[0120] By applying this function, the model evaluates the accuracy of facts based on specific inter-entity relationships. For model optimization, binary cross entropy is used as the loss function, expressed as:

[0121]

[0122] in, represents the loss function of the ConvE model, Represents the number of fact triples, y i'represents the calculated score of the i'th triple, f i' Represents a label, where 1 represents true facts and 0 represents false information.

[0123] In summary, through the above design, the present invention effectively realizes the efficient embedding and deep fusion of multimodal information, enhances the processing capability of heterogeneous data, prevents the leakage of sensitive information through the embedding process, provides an end-to-end privacy protection mechanism, and ensures the security and reliability of medical data during the analysis and reasoning process.

Claims

1. A privacy-preserving embedding method for multimodal knowledge graphs of medical data, characterized in that: The following steps are involved: S1. Initialize and embed text information and numerical information based on diagnosis and treatment data, and initialize and embed visual information based on medical image visual data; S2. Based on the initial embedding result, the multimodal graph self-attention layer is used to obtain the multimodal attention coefficient of the neighbor entity; S3, according to the multimodal attention coefficient, the structured information of the multimodal knowledge graph is fused by stacking multiple encoder layers; S4. Based on the structured information of the fused multimodal knowledge graph, the convolutional embedding model is used to score the privacy-preserving embedding of the multimodal knowledge graph.

2. The privacy-preserving embedding method for multimodal knowledge graphs of medical data according to claim 1 is characterized in that: The step S1 comprises the following steps: S101. Use the pre-trained language model to obtain the context information of the sentence-level description of entities and relations in the diagnosis and treatment data, and generate text information embedding t ; S102: Extract visual information and embed it into medical image visual data v ; S103, encode the entity numerical features in the diagnosis and treatment data and process them in the fully connected layer in turn to generate numerical data embedding e n , complete the initialization embedding of text information, visual information and numerical information.

3. The privacy-preserving embedding method for multimodal knowledge graphs of medical data according to claim 2 is characterized in that: The step S2 comprises the following steps: S201, using text information to embed e t and numerical data embedded in e n , construct an embedding matrix, and use visual information to embed e v , construct the modality matrix, where each modality graph attention layer processes two embedding matrices and one modality matrix; S202. Utilize the multimodal graph self-attention layer to obtain a multimodal attention coefficient.

4. The privacy-preserving embedding method for multimodal knowledge graphs of medical data according to claim 3 is characterized in that: The expression of the multimodal attention coefficient is as follows: c jk =W1[h j ||g k ||m k ] Among them, b ijk and Both represent multimodal attention coefficients, c i Represents the central entity e i The eigenvector of jk Represents the central entity e i In comparison, the tail entity e j and the relationship k The composite entity vector, d k Represents the dimension of the vector, used to normalize the result of the vector dot product, exp() represents the natural exponential function, used to convert b ijk The value of b is mapped to the positive range, inr Represents entity e i The absolute attention parameter between the neighbor entity n through the relation r, n represents the entity e i A neighbor node of i Represents entity e i The first-order neighbor set of r represents the connection center entity e i A relation with a neighbor entity, R in Represents entity e i The relationship between the connection and the neighbors, W1 represents the learnable weight matrix, || represents the connection operation, and h j and g k Respectively represent the tail entity e j The vector and relationship r k The vector, m k Represents multimodal features.

5. The privacy-preserving embedding method for multimodal knowledge graphs of medical data according to claim 4 is characterized in that: The step S3 comprises the following steps: S301, linearly fuse the multimodal attention coefficients to the central node to generate an embedded representation; S302. Connect the embedding representations generated from M independent self-attention layers using the following formula: in, Represents the output after the multi-head self-attention mechanism, MultiHead() represents the multi-head self-attention mechanism, Represents the input of the mth attention head, i.e., entity e i The representation of head m represents the output of the mth attention head, W O represents the matrix of input linear layer weights, j represents entity e i The first-order neighbor, N i Represents entity e i The first-order neighbor set of Represents the neighbor node e j For the central node e i The attention weight, Represents the neighbor node e j The feature representation combined with relation k is used in the mth attention head; S303. According to the connection result, the structured information of the knowledge graph is integrated by stacking multiple encoder layers using the following formula: in, Represents entity e i In the embedding representation of the l+1th layer, σ represents a nonlinear activation function, and R represents the set of relations, that is, all relation types in the knowledge graph. Represents entity e i The neighbor set of c i,r represents the factor used for normalization, adjusting between different relationships, represents the attention weight from neighbor node j to center node i in layer l, Represents a combination function for embedding node i and the relationship between nodes j and i Perform feature transformation.

6. The privacy-preserving embedding method for multimodal knowledge graphs of medical data according to claim 5 is characterized in that: The expression of the embedding vector is as follows: Among them, x i Represents entity e i The updated embedding vector, σ represents the nonlinear activation function, and j represents the entity e i The first-order neighbor, N i Represents entity e i The first-order neighbor set, k represents the relationship between neighbor entity j and another node, R jj represents the set of relationships between neighbor entity j and nodes other than neighbor entity j, α ijk represents the attention weight coefficient, which represents the contribution of neighbor node j and relationship k to the central node i, c jk Represents the feature representation of neighbor node j and relationship k.

7. The privacy-preserving embedding method for multimodal knowledge graphs of medical data according to claim 1 is characterized in that: The expression of the scoring function of the convolutional embedding model is as follows: f(h,r,t)=σ(vec(W([e h ;r]*ω))e t ) Among them, f() represents the scoring function of the triple, h represents the head entity, and r represents the connection center entity e i A relationship with a neighbor entity, t represents the tail entity, σ represents the sigmoid activation function, vec represents the operation of projecting the feature map into the vector space, W represents the transformation matrix, e h and e t They represent the embedding of the head entity and the tail entity respectively, r represents the embedding vector of the relationship, ω represents the weight matrix of the convolution kernel, and * represents the convolution operation.

8. The privacy-preserving embedding method for multimodal knowledge graphs of medical data according to claim 1 is characterized in that: The loss function of the convolutional embedding model is expressed as follows: in, represents the loss function of the ConvE model, Represents the number of fact triples, y i' represents the calculated score of the i'th triple, f i' Represents a label, where 1 represents true facts and 0 represents false information.

Images