Cross-modal data fusion method and system based on knowledge graph and deep learning

Abstract

The present application discloses a cross-modal data fusion method and system based on knowledge graph and deep learning. The method constructs a corresponding first knowledge graph for different modal data at each time node, and fuses the corresponding first knowledge graph of each modal data through a deep learning method to obtain a corresponding second knowledge graph. It not only retains the specific information of each modality, but also can deeply explore the complex relationship between different modalities, thereby achieving more comprehensive and accurate data analysis. The method also takes the time dimension into consideration and constructs a second knowledge graph for each time node. The graph corresponding to each time node represents the data state of the time node and can reflect the change of data over time. Through the graphs of different time nodes, the data that is continuous in time can be effectively associated. Combined with the fused second knowledge graphs at different time nodes, a more comprehensive and accurate data analysis can be achieved from the time series.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal data fusion, and in particular to a cross-modal data fusion method and system based on knowledge graph and deep learning. Background Art

[0002] Traditional multimodal data fusion methods fail to effectively utilize the deep interactions between modalities, limiting the comprehensiveness and accuracy of multimodal data analysis. Moreover, when processing data across time nodes, multimodal data fusion technology lacks consideration of the time dimension, making it impossible for the data integration process to reflect the characteristics of data changes over time, limiting the comprehensiveness and accuracy of multimodal data analysis. Summary of the invention

[0003] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application proposes a cross-modal data fusion method and system based on knowledge graph and deep learning, which can improve the comprehensiveness and accuracy of multimodal data analysis.

[0004] To achieve the above objectives, in a first aspect of the embodiments of the present application, the embodiments of the present application provide a cross-modal data fusion method based on knowledge graph and deep learning, the method comprising:

[0005] Acquire data to be processed at multiple time nodes, wherein the data to be processed includes multimodal data corresponding to each of the time nodes;

[0006] Extracting feature vectors corresponding to different modal data of each of the time nodes, and constructing a first knowledge graph corresponding to each modal data according to the feature vectors corresponding to each modal data;

[0007] Fusing the first knowledge graph corresponding to each modal data according to a deep learning method to obtain a second knowledge graph corresponding to the multimodal data of each of the time nodes;

[0008] The data to be processed is analyzed based on the second knowledge graph corresponding to the multimodal data of each time node.

[0009] The first aspect of the embodiment of the present application has the following beneficial effects:

[0010] The data fusion method provided in this embodiment constructs a corresponding first knowledge graph for different modal data at each time node. The data of each modality is converted into a graph structure. Such processing not only retains the unique information of each modality, but also lays the foundation for the identification of cross-modal relationships. In addition, based on the independent modal graph, the method fuses the corresponding first knowledge graph of each modal data according to a deep learning method to obtain the second knowledge graph corresponding to the multimodal data of each time node. Constructing the second knowledge graph not only retains the specific information of each modality, but also can deeply explore the complex relationship between different modalities, thereby achieving more comprehensive and accurate data analysis.

[0011] The data fusion method provided in this embodiment takes the time dimension into consideration and constructs a second knowledge graph for each time node. The second knowledge graph can be used to perform time series analysis on the data to be processed. Because the graph corresponding to each time node is independent, it represents the data state of the time node and can reflect the changes of data over time. Through the graphs of different time nodes, temporally continuous data can be effectively associated. Combined with the fused second knowledge graphs at different time nodes, the dynamics and real-time update capabilities of data processing are improved, thereby achieving more comprehensive and accurate data analysis.

[0012] In some embodiments of the present application, constructing a first knowledge graph corresponding to each type of modal data according to a feature vector corresponding to each type of modal data includes:

[0013] Mapping the feature vectors corresponding to the different modal data of each of the time nodes to the same vector space;

[0014] Calculate the similarity between the corresponding feature vectors of the same modality data;

[0015] According to the feature vectors corresponding to the same modal data and their similarities, a first knowledge graph corresponding to each modal data is constructed.

[0016] In some embodiments of the present application, after mapping the feature vectors corresponding to the different modal data of each time node to the same vector space, the method further includes:

[0017] Setting privacy budgets and sensitivities;

[0018] Adding a random noise vector to the feature vector according to the privacy budget and sensitivity;

[0019] The calculating the similarity between the feature vectors corresponding to the same modality data includes:

[0020] Calculate the similarity between the feature vectors of the same modality data with correspondingly increased random noise vectors.

[0021] In some embodiments of the present application, after calculating the similarity between the feature vectors of the same modality data corresponding to the added random noise vector, the method further includes:

[0022] Performing Paillie homomorphic encryption on the feature vector;

[0023] The step of constructing a first knowledge graph corresponding to each modal data according to the feature vectors corresponding to the same modal data and their similarities includes:

[0024] According to the feature vectors and their similarities that are homomorphically encrypted with Paillie for the same modal data, a first knowledge graph corresponding to each modal data is constructed.

[0025] In some embodiments of the present application, constructing a first knowledge graph corresponding to each modal data according to the homomorphically encrypted feature vector corresponding to each modal data and its similarity includes:

[0026] Constructing the initial graph;

[0027] By traversing each pair of the feature vectors in the feature vectors subjected to Paillie homomorphic encryption corresponding to each modal data through a nested loop, the similarity between each pair of the feature vectors subjected to Paillie homomorphic encryption is determined, and it is judged whether the similarity between each pair of the feature vectors subjected to Paillie homomorphic encryption meets a preset condition; if the preset condition is met, the corresponding pair of the feature vectors subjected to Paillie homomorphic encryption is respectively used as a node of the initial graph, and attribute construction edges between the nodes are constructed;

[0028] After traversing each pair of the feature vectors subjected to Paillie homomorphic encryption, a first knowledge graph is obtained.

[0029] In some embodiments of the present application, the method of fusing the first knowledge graph corresponding to each modal data according to the deep learning method to obtain the second knowledge graph corresponding to the multimodal data of each time node includes:

[0030] According to the graph neural network, the first knowledge graph corresponding to each modal data is fused with the following formula to obtain the second knowledge graph corresponding to the multimodal data of each time node:

[0031] ;

[0032] in, and are the weights and biases of the output layer of the graph neural network, is the set of all nodes in the first knowledge graph, express function, Is a node In the Characteristics of the layer.

[0033] In some embodiments of the present application, the step of analyzing the data to be processed based on the second knowledge graph corresponding to the multimodal data of each time node includes:

[0034] Sorting the second knowledge graph corresponding to the multimodal data of each time node according to the time series to obtain a sorting result;

[0035] According to the sorting result, the feature vector of the second knowledge graph corresponding to the multimodal data of the time node is input into a preset time series analysis model to obtain the analysis result of the data to be processed output by the time series analysis model.

[0036] To achieve the above purpose, in a second aspect of the embodiments of the present application, the embodiments of the present application provide a cross-modal data fusion system based on knowledge graph and deep learning, the system comprising:

[0037] A data acquisition unit, used to acquire data to be processed at multiple time nodes, wherein the data to be processed includes multimodal data corresponding to each of the time nodes;

[0038] A first knowledge graph construction unit, used to extract feature vectors corresponding to different modal data of each time node according to the modality type, and construct a first knowledge graph corresponding to each modal data according to the feature vectors corresponding to each modal data;

[0039] A second knowledge graph construction unit is used to fuse the first knowledge graph corresponding to each modal data according to a deep learning method to obtain a second knowledge graph corresponding to the multimodal data of each time node;

[0040] A data analysis unit is used to analyze the data to be processed based on the second knowledge graph corresponding to the multimodal data of each time node.

[0041] To achieve the above-mentioned purpose, in a third aspect of an embodiment of the present application, an embodiment of the present application provides an electronic device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the cross-modal data fusion method based on knowledge graph and deep learning described in the first aspect.

[0042] To achieve the above-mentioned purpose, in a fourth aspect of an embodiment of the present application, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the cross-modal data fusion method based on knowledge graph and deep learning described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 It is a flowchart of a cross-modal data fusion method based on knowledge graph and deep learning provided in an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of the data cleaning process provided by the embodiment of the present application;

[0046] Figure 3 It is a schematic diagram of the process of data standardization provided in the embodiment of the present application;

[0047] Figure 4 It is a process architecture diagram of cross-modal data fusion based on knowledge graph and deep learning provided in an embodiment of the present application;

[0048] Figure 5 It is a structural diagram of a cross-modal data fusion system based on knowledge graph and deep learning provided in an embodiment of the present application;

[0049] Figure 6 It is an architectural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0051] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0052] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0053] Before introducing the embodiments of the present application, the technical concepts of the present application are first introduced:

[0054] (1) BERT technology;

[0055] BERT (Bidirectional Encoder Representations from Transformers) is an advanced deep learning technology that is specifically designed to extract complex language features from large-scale corpora and understand the deep semantics of text. BERT processes text through the WordPiece algorithm, breaking it down into smaller tokens, each of which has a unique index in a given vocabulary to represent it. Given a text sequence , after BERT preprocessing and embedding, the sequence is transformed into ,in It is a word Embedding vector of . In BERT, the transformation of each layer can be described as:

[0056] ;

[0057] in ,and is the output of the last layer, representing the final embedding vector of each word in the sequence. Add a token at the beginning of the text sequence , and add at the end of the sequence Tags to help BERT understand the beginning and end of sentences.

[0058] (2) Differential privacy technology;

[0059] In multimodal data processing, the feature vector is usually Differential privacy technology is applied to protect individual privacy. Differential privacy is achieved by adding noise to the data query results. The purpose is to allow statistical analysis of the overall data while protecting user privacy. Suppose there is a feature vector , which represents the Defining sensitivity , that is, in adjacent data sets and The maximum possible change in the eigenvector calculated on . The sensitivity is defined as follows:

[0060] ;

[0061] To achieve - Differential privacy, usually in each dimension Add from The noise is independently drawn from the distribution, where

[0062] ;

[0063] Noise Vector Each component obey Distribution. Finally, the privacy-preserving feature vector is:

[0064] ;

[0065] in is the feature vector after noise is added, which can be used for subsequent analysis without leaking individual specific information. In the present invention, differential privacy not only ensures the privacy protection of individual data, but also maintains the statistical significance of the data, which is crucial for subsequent data analysis and knowledge graph construction. It can be safely used to build multimodal knowledge graphs without leaking any individual's specific information.

[0066] (3) Paillier homomorphic encryption technology;

[0067] The Paillier encryption system is an asymmetric encryption algorithm that supports additive homomorphic encryption, that is, the encrypted data can be directly used for arithmetic operations, and the calculation results are the same as the plaintext operation results. Assume that there is a eigenvector , hoping to encrypt it for secure remote computing.

[0068] First, choose two large prime numbers and ,calculate and .choose Make .in .

[0069] For the feature vector Each element of , choose a random number , and calculate the encrypted value:

[0070] ;

[0071] here yes The Paillier technique supports additive homomorphism, that is, for two encrypted numbers and , and its addition operation is expressed as , decrypt Will get As a result, this property allows direct vector addition in the encrypted state.

[0072] (4) Knowledge graph technology;

[0073] A knowledge graph is a structured semantic knowledge base that describes the knowledge of the real world through a graph structure of entities, attributes, and relationships between entities. The process of building a knowledge graph usually includes steps such as entity recognition, relationship extraction, and entity linking, involving a large amount of data processing and semantic analysis. Knowledge graphs usually contain a large number of entity types and relationship types and have high complexity. They often need to process information from different data sources, involving large-scale data integration and continuous updates. The nodes in the graph represent the identified entities, such as names, places, organizations, etc.; the edges in the graph represent the relationships between entities.

[0074] like Figure 1 An embodiment of the present application provides a cross-modal data fusion method based on knowledge graph and deep learning, the method comprising:

[0075] Step S110: Acquire data to be processed at multiple time nodes, wherein the data to be processed includes multimodal data corresponding to each time node.

[0076] Step S120: extract the feature vectors corresponding to the different modal data at each time node, and construct a first knowledge graph corresponding to each modal data based on the feature vectors corresponding to each modal data.

[0077] Step S130: fuse the first knowledge graph corresponding to each modal data according to the deep learning method to obtain the second knowledge graph corresponding to the multimodal data at each time node.

[0078] Step S140: Analyze the data to be processed based on the second knowledge graph corresponding to the multimodal data of each time node.

[0079] In step S110, the data to be processed may be multimodal data in the fields of medical care, elderly care, or rehabilitation (including but not limited to text data, medical images, and audio data); may be multimodal data from surveillance cameras (visual data), environmental sensors (numerical data), or social media (text data) in the management of smart cities; and may be multimodal data from radars, cameras, and GPS in the field of autonomous driving, without specific limitation here.

[0080] In step S110, the data to be processed is multimodal data composed of multiple time nodes. Taking medical data as an example, the medical records and medical records data (including but not limited to text data and image data) of the same patient at different time points are recorded in chronological order, and each data element is associated with a clear time label. Each time series data element is marked with a time label to ensure that each data element can be accurately attributed to a specific time node.

[0081] In step S120, feature vectors corresponding to different modal data of each time node are extracted according to the modality type. For example, text data of the same node can be feature extracted through the BERT model, image data can be processed through a convolutional neural network (CNN), and audio data can be processed through a recurrent neural network (RNN). Features extracted in different ways have different dimensions, and features extracted in different ways can be mapped to the same vector space for easy processing.

[0082] In step S120, a first knowledge graph corresponding to each modal data is constructed based on the feature vector corresponding to each modal data. For example, if there is text data, image and audio in the same time node, the text features, image features and audio features are extracted respectively. Then, for the text data, the corresponding knowledge graph is constructed using the text features; for the image data, the corresponding knowledge graph is constructed using the image features; and for the audio data, the corresponding knowledge graph is constructed using the audio features.

[0083] In step S130, the first knowledge graph corresponding to each modal data is fused according to a deep learning method. For example, the knowledge graph corresponding to the text data, the knowledge graph corresponding to the image data, and the knowledge graph corresponding to the audio data are fused, and finally a multimodal knowledge graph (called the second knowledge graph) composed of multimodal data at the same time node is obtained.

[0084] In step S130, for different time nodes, a second knowledge graph composed of multimodal data can be built for each time node.

[0085] In step S140, the data to be processed is analyzed based on the second knowledge graph corresponding to the multimodal data of each time node. For example, the health status of the patient is analyzed by analyzing the second knowledge graphs of multiple time nodes to capture the changing trends of the node features in the second knowledge graphs corresponding to the multiple time nodes. The probability distribution of the patient's health status at different times can be obtained, so that medical experts can make more data-supported decisions based on the analysis results.

[0086] This embodiment has at least the following beneficial effects:

[0087] The data fusion method provided in this embodiment constructs a corresponding first knowledge graph for different modal data at each time node. The data of each modality is converted into a graph structure. Such processing retains the unique information of each modality. On the basis of independent modal graphs, the method fuses the corresponding first knowledge graphs of each modal data according to a deep learning method to obtain the second knowledge graph corresponding to the multimodal data of each time node. The method constructs the second knowledge graph which not only retains the specific information of each modality, but also can deeply explore the complex relationship between different modalities, thereby achieving more comprehensive and accurate data analysis.

[0088] The data fusion method provided in this embodiment takes the time dimension into consideration and constructs a second knowledge graph for each time node. The second knowledge graph can be used to perform time series analysis on the data to be processed. Because the graph corresponding to each time node is independent, it represents the data state of the time node and can reflect the changes of data over time. Through the graphs of different time nodes, temporally continuous data can be effectively associated. Combined with the fused second knowledge graphs at different time nodes, time-related data can be dynamically updated and processed, thereby improving the dynamics of data processing and real-time update capabilities.

[0089] In some embodiments of the present application, a first knowledge graph corresponding to each type of modal data is constructed according to a feature vector corresponding to each type of modal data, including:

[0090] Step S1210: Map the feature vectors corresponding to the different modal data of each time node to the same vector space.

[0091] Step S1220: Calculate the similarity between feature vectors corresponding to the same modal data.

[0092] Step S1230: construct a first knowledge graph corresponding to each modal data based on the feature vectors corresponding to the same modal data and their similarities.

[0093] In step S1210, since the feature vectors corresponding to different modal data have different dimensions, mapping them to the same vector space facilitates subsequent processing.

[0094] In step S1220, the correlation between the two feature vectors can be represented by similarity, and the data can be aligned and matched by similarity; preferably, the correlation between the two feature vectors can be represented by cosine similarity. It should be noted that the similarity calculated here is the similarity calculation between the internal data of the same modality, and does not involve the similarity between the feature vectors across modalities.

[0095] In step S1230, the specific implementation of constructing the first knowledge graph corresponding to each modal data according to the feature vectors corresponding to the same modal data and their similarities includes:

[0096] Step S1231, constructing an initial graph.

[0097] Step S1232, traverse each pair of feature vectors in the feature vectors that are homomorphically encrypted for each modal data through a nested loop, determine the similarity corresponding to each pair of feature vectors that are homomorphically encrypted for Paillie, and judge whether the similarity corresponding to each pair of feature vectors that are homomorphically encrypted for Paillie meets a preset condition; if the preset condition is met, use the corresponding pair of feature vectors that are homomorphically encrypted for Paillie as nodes of the initial graph, and construct attribute construction edges between the nodes.

[0098] Step S1233: After traversing each pair of feature vectors subjected to Paillie homomorphic encryption, a first knowledge graph is obtained.

[0099] In some embodiments of the present application, after mapping the feature vectors corresponding to the different modal data of each time node to the same vector space, the method further includes:

[0100] Step S1240: construct a privacy budget and sensitivity.

[0101] Step S1250: Add a random noise vector to the feature vector according to the privacy budget and sensitivity.

[0102] Calculate the similarity between the corresponding feature vectors of the same modality data, including:

[0103] Step S1231, calculating the similarity between the feature vectors of the same modal data with the corresponding random noise vector added.

[0104] This embodiment introduces differential privacy technology to the extracted feature vectors, which can desensitize the data and protect data privacy. In addition, the amount of noise can be changed by adjusting the privacy budget and sensitivity, ensuring that the results of data processing and analysis will not be significantly different due to changes in a single data record.

[0105] In some embodiments of the present application, after calculating the similarity between the feature vectors of the same modality data and the corresponding feature vectors with the random noise added, the method further includes:

[0106] Step S1232: Perform Paillie homomorphic encryption on the feature vector.

[0107] According to the corresponding feature vectors and their similarities of the same modal data, the first knowledge graph corresponding to each modal data is constructed, including:

[0108] Step S1233: construct a first knowledge graph corresponding to each modal data based on the feature vectors and their similarities that are homomorphically encrypted using Paillie for the same modal data.

[0109] The traditional encryption technology in this embodiment needs to be decrypted before use, which increases the risk of data leakage. In addition, directly constructing a knowledge graph on unencrypted sensitive data may expose personal privacy. This method performs Paillie homomorphic encryption on the feature vector, and then constructs the first knowledge graph through the feature vector encrypted by Paillie homomorphic encryption. The data is encrypted using homomorphic encryption technology to ensure that the knowledge graph construction is carried out in an encrypted state, further ensuring the security of the data; this embodiment combines differential privacy and homomorphic encryption methods to allow complex data processing tasks (including similarity calculation, construction of the first knowledge graph, and construction of the second knowledge graph) to be completed without decrypting the data, thereby maintaining the availability and analysis depth of the data without sacrificing data privacy. It should be noted that since Paillie homomorphic encryption is a public key encryption system that supports additive homomorphic operations, it allows arithmetic operations on encrypted data without decryption. See the introduction of subsequent embodiments for details.

[0110] like Figure 4 As shown, for ease of understanding, an embodiment is provided. In this embodiment, medical and health care data is taken as an example. The cross-modal data fusion method based on knowledge graph and deep learning includes the following process:

[0111] Step S910: Data preprocessing of medical and health care data.

[0112] In this embodiment, the original medical and health care data is used To indicate that Represents raw text data, such as electronic health records, diagnosis reports, patient names, etc. Represents raw image data, such as CT images, pathological images, X-ray imaging data, etc.

[0113] Medical and health care data mainly include the same patient's medical records and medical history data at different time points. These data are recorded in chronological order, and each data point is associated with a clear time label. In the data preprocessing stage, each time series data point is first labeled with a time label to ensure that each data element can be accurately attributed to a specific time node.

[0114] The data is grouped according to these time tags, and independent multimodal knowledge graphs are constructed for each time node. These knowledge graphs represent the data status at different time points, including information extracted from text, images, and other possible modalities.

[0115] The data preprocessing stage includes the following steps:

[0116] (1) Data cleaning: First, the original data set De-noising is performed to extract valuable information. Noise can be grammatical errors, spelling errors, repeated information, ambiguous information, etc. Remove irrelevant values ​​and delete some values ​​that are not related to the original data set Remove irrelevant information from the subject, improve the quality of the data, and finally obtain the data after data cleaning .

[0117] (2) Data standardization: cleaned data Perform meticulous data standardization operations to ensure data consistency and model training efficiency.

[0118] like Figure 2 and Figure 3 As shown, for example, for text data, UTF-8 character encoding is used, which supports multilingual character sets and is the standard encoding method in modern text processing. At the same time, word form unification processing is also performed, including adjusting tenses and performing stem extraction. These operations help reduce the diversity of text data, thereby improving the consistency and accuracy of subsequent analysis.

[0119] For image data, the image size and color space are unified to ensure that all image data have the same resolution and color standard before entering the model. In addition, the image contrast is adjusted and the image pixel values ​​are normalized, which helps improve the convergence and stability of model training. After data cleaning and data standardization, data suitable for model training is obtained. .

[0120] Step S920: feature extraction and similarity calculation.

[0121] Step S921: For data of different modalities (such as images, texts, and audios), different methods are used to obtain features.

[0122] For example, after getting the preprocessed data Finally, the Clinical BERT model is used to extract features from the text data and encode the input text data into an embedding vector to represent its deep semantics.

[0123] For example, using a CNN model to extract features from image data can also yield an embedded vector that can represent the deep semantics of the image.

[0124] The dimensions of the embedded vectors obtained after feature extraction of data from different modalities are not necessarily the same. In this case, a linear transformation is needed to map the two vectors into the same vector space to prepare for subsequent cross-modal data fusion.

[0125] use Represents the embedded vector after encoding the text data. Use Represents the embedded vector after encoding the image data.

[0126] ;

[0127] Step S922: Add differential privacy.

[0128] In order to prevent sensitive data from being leaked, differential privacy technology is introduced during model training to enhance data privacy protection. Specifically, we first define the privacy budget. , which is a parameter that controls the strength of privacy protection. Next, we use the method of adding noise to protect these feature vectors. The data source and parameters are defined as follows:

[0129] (1) Text data vector .

[0130] (2) Image data vector .

[0131] (3) Privacy Budget , which controls the amount of noise added.

[0132] (4) Sensitivity , describing the maximum change of the query function in the worst case.

[0133] For text and image data, based on an assessment of the sensitivity of the specific functions and variables involved in the data processing, a maximum sensitivity of . While providing adequate noise coverage, avoid undue impact on data quality and accuracy of subsequent analyses.

[0134] The calculation of noise depends on the scale parameter of Laplace noise , which is calculated as follows:

[0135] ;

[0136] The noise in each dimension will be calculated based on the privacy budget and sensitivity To adjust, for each feature and , add independent Laplace noise:

[0137] ;

[0138] ;

[0139] in, and It is a random noise generated according to the Laplace distribution, ensuring that each feature dimension meets the privacy protection requirements. The embedding vectors of text data and image data after privacy protection are defined as follows:

[0140] ;

[0141] ;

[0142] Step S923: construct similarity.

[0143] Then, feature similarity calculation is performed on the privacy-protected data to achieve data matching. The following is the detailed process:

[0144] In order to achieve the data with privacy protection and The feature similarity calculation is performed using the cosine similarity method, because cosine similarity performs well when processing high-dimensional data and is insensitive to vector length, making it suitable for data affected by noise. The calculation formula is as follows:

[0145] ;

[0146] represents the dot product of two vectors, Represent the Euclidean norm of two vectors by Calculated. Set a threshold , when the cosine value between vectors exceeds the threshold, they are considered similar. Similarity Threshold No specific limitation is given.

[0147] After matching data of different modalities, different similarity matrices are obtained , used to represent the correlation between feature data. Taking text data as an example, ,The similarity matrices calculated here are all for the similarity calculation between the internal data of the same modality, and do not involve the similarity calculation between the ,feature vectors across modalities.

[0148] Step S924, add homomorphic encryption.

[0149] After calculating the correlation between the feature vectors, the data in the feature vectors is encrypted using homomorphic encryption technology to protect the security of the data, as the data will be shared among multiple institutions.

[0150] If the feature vectors are first homomorphically encrypted, and then the cosine similarity method is used to calculate the similarity matrix between vectors, the encryption technology needs to support both multiplication and addition (including inner product and norm operations). Due to the large scale of medical and health data, it is impossible to use fully homomorphic encryption technology to encrypt the data, so here we choose to perform similarity calculations first and then homomorphically encrypt the feature vectors. Calculating similarity directly on the original data is usually faster than calculating on encrypted data, because the processing of original data does not require additional computing resources to handle the encryption and decryption process. In addition, differential privacy technology has been used to effectively protect data privacy and security, which reduces the data security risks brought about by calculating similarity first and then using homomorphic encryption technology.

[0151] Here, the Paillier semi-homomorphic encryption algorithm is used to encrypt the data in the feature vector using the generated public key pk. The specific process is as follows:

[0152] ;

[0153] ;

[0154] When calculating the similarity of feature vectors, the cosine similarity is first calculated on the encrypted data to obtain the similarity matrix between the encrypted data. Since each encrypted vector is assigned a unique label before encryption, these labels remain associated during the encryption process. Therefore, when the similarity between vectors is calculated in an encrypted state, the encrypted similarity results can be mapped back to the original data through these retained labels, so that the similarity relationship between the original data vectors can be known without decryption. Specifically, each data vector is assigned an identifier in the form of a numeric value or string before differential privacy processing and homomorphic encryption. These identifiers are stored in a secure index database to track the processing history and status of each data vector. When the feature vector is encrypted, although the data content is encrypted, its corresponding identifier does not participate in the encryption calculation, but is stored in plain text or attached to the encrypted data. In this way, when performing similarity calculations or other analyses, the system can quickly retrieve and match the relationship between the original data and its encrypted data through these plain text identifiers, so that the similarity or other relationships between the original data can be inferred without decryption. This approach not only protects the privacy of data, but also maintains the continuity and traceability of data processing, effectively supporting cross-institutional data sharing and analysis.

[0155] Step S930: knowledge graph construction and fusion.

[0156] Step S931, constructing a unimodal data knowledge graph.

[0157] After getting the encrypted feature vector and Afterwards, we will use the similarity matrix that has been calculated To form knowledge graphs of different modalities. Here we take the construction of knowledge graphs of text data as an example to explain in detail.

[0158] First, each text feature vector Represents a node in the knowledge graph. The attributes of the node are determined by the characteristics and content of the vector, such as the topic of the text, keywords, etc. Use the similarity matrix To determine whether an edge is established between two nodes .

[0159] ;

[0160] if , then at the node and Create an edge between , and set the edge weight to , which is the cosine similarity value of two vectors.

[0161] ;

[0162] First initialize an empty graph Then, for each encrypted feature vector, the algorithm generates a Next, the algorithm iterates over each pair of feature vectors through a nested loop, checking the similarity matrix between them. If the similarity between two vectors meets the preset conditions , then add an edge between the two nodes and use their similarity measure as the edge weight. Finally, when all nodes and eligible edges are added to the graph After that, return to the constructed knowledge graph .

[0163] Step S932: Multimodal knowledge graph fusion based on GNN.

[0164] After constructing knowledge graphs of different modalities, the GNN graph neural network model is used to fuse knowledge graphs of different modalities and deeply explore the deep relationships between nodes of different modal graphs.

[0165] Use the attention mechanism to optimize the information integration between different modalities. Under the action of multi-head attention and gating layers, the node features of each modality are not only updated independently, but also interact and fuse with the node features of other modalities. This interaction allows the network to learn from one modality to supplement or enhance the information expression in another modality.

[0166] ;

[0167] in Is a node In the The characteristics of the layer, Is a node In modal The set of neighbor nodes under is the attention coefficient, is the weight matrix of this layer, is the activation function. The feature of each node in each layer is updated by aggregating the features of its adjacent nodes. , its Layer characteristics All adjacent nodes in the previous layer Features It is calculated in the form of weighted sum. The weight is the attention coefficient between nodes This allows the model to dynamically adjust the importance of different node information. This process is done through the activation function The introduction of nonlinearity enables the model to capture more complex feature relationships.

[0168] A normalized Laplacian matrix is ​​used in the convolutional layer to weight the features of neighbor nodes so that each node can maintain a consistent scale when aggregating neighbor node information.

[0169] ;

[0170] Represents the node features from the previous layer and updates the new node features through the convolutional layer .

[0171] A gated recurrent unit (GRU) layer is constructed to process sequence data, maintain the temporal continuity of sequence data, and dynamically adjust the relationship between different modalities. , assuming that in The feature representation of the layer is , then in The GRU operation of the layer can be expressed as:

[0172] Reset the gate:

[0173] ;

[0174] here, and is the weight matrix of the reset gate, is the bias term. Through the sigmoid activation function , reset the output of the gate Adjust the degree of information forgetting, which is a value between 0 and 1, and determines the previous state How much information will be retained in the past. Help the model determine which past information is relevant to the current task and needs to be passed on.

[0175] Update Gate:

[0176] ;

[0177] It is the gating signal that determines how much current node status is updated. and is the weight matrix of the update gate, is the bias term.

[0178] Candidate hidden states:

[0179] ;

[0180] is a candidate hidden state, used to update the new state of the node. and is the weight matrix of the update gate, is the bias term, represents the Hadamard product. This step takes into account the previous state of the node and the new input.

[0181] Final status update:

[0182] ;

[0183] Final node status It is a weighted sum of the previous state and the candidate state controlled by an update gate. This structure allows the model to capture long-term dependencies at each node while ignoring irrelevant information.

[0184] The output of the graph neural network is defined as follows:

[0185] ;

[0186] Final Output Represents the fusion features of each node after multi-layer graph convolution and attention mechanism processing. These features not only contain the information of the node itself, but also integrate the information from other modal nodes, and optimize the integration of information through attention mechanism and gating mechanism. and are the weights and biases of the output layer, is the set of all nodes in the graph, Function, used to convert real-number outputs into probabilities in multi-classification problems.

[0187] After deep fusion of the graph neural network, the relationship between nodes of different modal graphs is implicitly reflected in the features of the nodes. Due to the use of the attention mechanism, each node will be assigned an attention weight according to its correlation with other nodes. These weights reflect the strength of the relationship between nodes. In the end, a fusion graph that integrates multimodal knowledge graphs will be obtained, and the correlation between different modal data can be understood from the weights between different modal nodes, and the complementary information of each modality can be used to form a unified representation of multimodal data. By retaining these metadata tags during data encryption and feature extraction, it can help understand the relationship between different data without exposing the specific data content. Therefore, after constructing a unified representation, there is no need to perform the decryption operation corresponding to the homomorphic encryption operation on the feature vector. The focus is on the correlation between vectors, thereby constructing the correlation between the original multimodal data and forming a unified representation of the original multimodal data.

[0188] When processing multimodal data, traditional methods usually treat all data as static inputs, ignoring the characteristics of data changing dynamically over time. In order to overcome this limitation, this method in this embodiment classifies the data by time nodes, so that the data at each time node can be processed and analyzed independently, greatly improving the efficiency and accuracy of data processing.

[0189] Specifically, the technology proposed in this embodiment classifies the data according to time nodes during the data collection stage, which includes but is not limited to data in multiple modes such as text, images, and audio. Each time node is regarded as an independent entity, and the data contained in it is used to construct a corresponding multimodal knowledge graph. This method of classification by time nodes not only simplifies the data processing and fusion tasks within each individual time point, but also allows the temporal dynamics of the data to be effectively captured. For the multimodal data at each time node, the update of each node not only reflects the information within the modality, but also integrates the supplementary information of other modalities, thereby achieving deep cross-modal fusion.

[0190] These fused knowledge graphs not only contain the individual information of each modality, but also contain cross-modal comprehensive information through deep fusion of GNN. Each node in the fused graph not only saves its own modality information, but also contains the association information with other modalities. These fused nodes are marked and tracked to maintain the comparison and connection between nodes at different times.

[0191] Step S940: knowledge graph fusion of multimodal data at multiple time nodes.

[0192] The fused knowledge graphs at different time nodes, such as short-term memory networks (LSTMs) and hidden Markov models (HMMs), use these labeled, fused nodes as input. The purpose of the model is to analyze and predict how node features change over time, so as to infer the patient's health status and possible development trends, evaluate the treatment effect, and give the patient the best medical decision overall.

[0193] Taking LSTM as an example, first, determine and collect the features of the fusion graph nodes at each time node. These features include data from different modalities (text, image), arrange these nodes in time series, and use the feature vector of each node as the input of the LSTM model. LSTM can effectively manage the flow of information through its internal gating mechanism (including forget gate, input gate, and output gate), allowing the network to learn when to forget old data points and when to update new information. During the model training phase, LSTM will try to capture the trends and patterns of feature changes at different time nodes. The output layer uses a softmax function to convert the output of LSTM into a probability distribution of different health states of patients. In this way, medical experts can make more data-supported decisions based on the results predicted by the model, optimize treatment plans, and achieve personalized medical intervention.

[0194] In the multimodal data processing method proposed in this embodiment, by classifying the data at time nodes in the data collection stage, each time node is regarded as an independent entity. Such a design significantly simplifies the data processing process, making the data processing and fusion tasks at each time point more efficient. Compared with the traditional multimodal data processing method, this method avoids the complexity and high computational cost of processing all data into a static large-scale map. Traditional methods usually ignore the time dimension of data and cannot effectively capture the dynamic changes of data over time. The technology of this embodiment not only simplifies the complexity of data processing (because each smaller map is easier to process than a large map) by constructing an independent map for each time node and performing feature extraction, but also effectively retains the key information of the time dimension, which is crucial for coping with a dynamically changing medical environment. In addition, this data structure constructed by time nodes supports real-time updates and dynamic decision making, because the data at each time node can be updated independently without rebuilding or reprocessing the data structure of the entire system. This modular and dynamic update capability greatly enhances the flexibility and response speed of processing, enabling medical providers to quickly make accurate medical decisions based on the latest data.

[0195] This embodiment has at least the following beneficial effects:

[0196] (1) The method uses differential privacy encryption algorithm and Paillier encryption algorithm to encrypt multimodal data to protect the security and privacy of data during processing. This embodiment uses encrypted data to construct a knowledge graph. In this process, all operations such as entity recognition and relationship extraction are performed while keeping the data encrypted. Through graph neural network technology, these encrypted and protected data are deeply structured and learned, and cross-modal information is extracted and integrated, so as to realize the mining and utilization of deep relationships between multimodal data while protecting data privacy.

[0197] (2) The method proposes an innovative fusion strategy that combines multimodal knowledge graphs and graph neural networks to effectively integrate and analyze data from different modalities. This strategy not only retains the specific information of each modality, but also deeply explores the complex relationships between different modalities, thereby achieving more comprehensive and accurate data analysis. The method uses graph neural networks to perform cross-modal fusion. Through GNN, by processing information within the modality, building and strengthening the connection between modalities, and using the graph neural network with attention mechanism, the model allows the flow and integration strategy of information to be adaptively adjusted according to the importance of different entities and relationships, which improves the flexibility and dynamic adaptability of the fusion process. By utilizing these characteristics of graph neural networks, the method can dynamically update the knowledge graph based on new data inputs, reflecting the latest data features and interactions between modalities in real time. This dynamic update mechanism enables the fusion model to adapt to the continuously changing medical environment, such as new clinical trial results or updated medical images, so as to respond to clinical needs in real time while maintaining high accuracy and efficiency.

[0198] (3) This method can dynamically update and process time-related data, thereby effectively capturing and analyzing the changes in the patient's health status over time. First, this method introduces time tags in the data preprocessing stage to mark the time attributes of each data point, ensuring that the continuity and integrity of the time information are maintained during the data processing process. New data at any time point can be instantly integrated into the system, ensuring the timeliness and relevance of the data set. At each time node, the constructed multimodal data (including images, texts, sequences, etc.) is used to create a knowledge graph. Each graph is independent and represents the data state at that time node. By connecting the graphs of different time nodes to each other, a large-scale dynamic graph across time is formed, so that continuous data in time can be effectively associated. The graphs of different time nodes are analyzed and fused through graph neural network technology. By combining time series analysis technology with the fused multimodal knowledge graphs at different time nodes, future health trends or disease development can be predicted, providing decision support for doctors and researchers.

[0199] like Figure 5One embodiment of the present application provides a cross-modal data fusion system based on knowledge graph and deep learning, the system comprising:

[0200] The data acquisition unit 1100 is used to acquire the data to be processed at multiple time nodes, wherein the data to be processed includes the multimodal data corresponding to each time node;

[0201] The first knowledge graph construction unit 1200 is used to extract feature vectors corresponding to different modal data at each time node according to the modality type, and construct a first knowledge graph corresponding to each modal data according to the feature vector corresponding to each modal data;

[0202] The second knowledge graph construction unit 1300 is used to fuse the first knowledge graph corresponding to each modal data according to the deep learning method to obtain the second knowledge graph corresponding to the multimodal data at each time node;

[0203] The data analysis unit 1400 is used to analyze the data to be processed based on the second knowledge graph corresponding to the multimodal data of each time node.

[0204] It should be noted that the cross-modal data system based on knowledge graph and deep learning provided in this embodiment and the above-mentioned cross-modal data fusion method based on knowledge graph and deep learning are based on the same embodiment concept, so the relevant content of the above-mentioned cross-modal data fusion method based on knowledge graph and deep learning is also applicable to the content of the cross-modal data fusion system based on knowledge graph and deep learning, so it will not be repeated here.

[0205] like Figure 6 The embodiment of the present application further provides an electronic device, the electronic device comprising:

[0206] at least one memory;

[0207] at least one processor;

[0208] at least one program;

[0209] The program is stored in the memory, and the processor executes at least one program to implement the cross-modal data fusion method based on knowledge graph and deep learning as described above in the present disclosure.

[0210] The electronic device may include any intelligent terminal such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0211] The electronic device according to the embodiment of the present application is described in detail below.

[0212] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0213] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes a cross-modal data fusion method based on knowledge graph and deep learning in the embodiment of the present disclosure.

[0214] Input / output interface 1800, used to implement information input and output;

[0215] Communication interface 1900, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0216] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0217] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0218] An embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned cross-modal data fusion method based on knowledge graph and deep learning.

[0219] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0220] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0221] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0222] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0224] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0225] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0226] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0227] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0228] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0229] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0230] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A cross-modal data fusion method based on knowledge graph and deep learning, characterized in that: The method comprises: Acquire data to be processed at multiple time nodes, wherein the data to be processed includes multimodal data corresponding to each of the time nodes; the multimodal data includes at least two of text data, image data, and audio data; Extracting feature vectors corresponding to different modal data at each of the time nodes, and constructing a first knowledge graph corresponding to each modal data according to the feature vectors corresponding to each modal data; constructing a first knowledge graph corresponding to each modal data according to the feature vectors corresponding to each modal data includes: Mapping the feature vectors corresponding to the different modal data of each of the time nodes to the same vector space; Calculate the similarity between the corresponding feature vectors of the same modality data; Constructing a first knowledge graph corresponding to each modal data according to the feature vectors corresponding to the same modal data and the similarities thereof; According to the deep learning method, the first knowledge graph corresponding to each modal data is fused to obtain the second knowledge graph corresponding to the multimodal data of each time node; the first knowledge graph corresponding to each modal data is fused to obtain the second knowledge graph corresponding to the multimodal data of each time node according to the deep learning method, including: According to the graph neural network, the first knowledge graph corresponding to each modal data is fused with the following formula to obtain the second knowledge graph corresponding to the multimodal data of each time node: ； in, and are the weights and biases of the output layer of the graph neural network, is the set of all nodes in the first knowledge graph, express function, Is a node In the Characteristics of the layer; The data to be processed is analyzed based on the second knowledge graph corresponding to the multimodal data of each of the time nodes.

2. The cross-modal data fusion method based on knowledge graph and deep learning according to claim 1 is characterized in that: After mapping the feature vectors corresponding to the different modal data of each time node to the same vector space, the method further includes: Setting privacy budgets and sensitivities; Adding a random noise vector to the feature vector according to the privacy budget and sensitivity; The calculating the similarity between the feature vectors corresponding to the same modality data includes: Calculate the similarity between the feature vectors of the same modality data with correspondingly increased random noise vectors.

3. The cross-modal data fusion method based on knowledge graph and deep learning according to claim 2 is characterized in that: After calculating the similarity between the feature vectors of the same modal data and the corresponding random noise vector, the method further includes: Performing Paillie homomorphic encryption on the feature vector; The step of constructing a first knowledge graph corresponding to each modal data according to the feature vectors corresponding to the same modal data and their similarities includes: According to the feature vectors and their similarities that are homomorphically encrypted with Paillie for the same modal data, a first knowledge graph corresponding to each modal data is constructed.

4. The cross-modal data fusion method based on knowledge graph and deep learning according to claim 3 is characterized in that: The step of constructing a first knowledge graph corresponding to each type of modal data according to the homomorphically encrypted feature vector corresponding to each type of modal data and its similarity includes: Constructing the initial graph; By traversing each pair of the feature vectors in the feature vectors subjected to Paillie homomorphic encryption corresponding to each modal data through a nested loop, the similarity between each pair of the feature vectors subjected to Paillie homomorphic encryption is determined, and it is judged whether the similarity between each pair of the feature vectors subjected to Paillie homomorphic encryption meets a preset condition; if the preset condition is met, the corresponding pair of the feature vectors subjected to Paillie homomorphic encryption is respectively used as a node of the initial graph, and attribute construction edges between the nodes are constructed; After traversing each pair of the feature vectors subjected to Paillie homomorphic encryption, a first knowledge graph is obtained.

5. The cross-modal data fusion method based on knowledge graph and deep learning according to claim 1, characterized in that: The analyzing the data to be processed based on the second knowledge graph corresponding to the multimodal data of each time node includes: Sorting the second knowledge graph corresponding to the multimodal data of each time node according to the time series to obtain a sorting result; According to the sorting result, the feature vector of the second knowledge graph corresponding to the multimodal data of the time node is input into a preset time series analysis model to obtain the analysis result of the data to be processed output by the time series analysis model.

6. A cross-modal data fusion system based on knowledge graph and deep learning, characterized in that: The system comprises: A data acquisition unit, configured to acquire data to be processed at a plurality of time nodes, wherein the data to be processed includes multimodal data corresponding to each of the time nodes; the multimodal data includes at least two of text data, image data, and audio data; The first knowledge graph construction unit is used to extract the feature vectors corresponding to the different modal data of each time node according to the modality type, and construct the first knowledge graph corresponding to each modal data according to the feature vectors corresponding to each modal data; the first knowledge graph corresponding to each modal data according to the feature vectors corresponding to each modal data includes: Mapping the feature vectors corresponding to the different modal data of each of the time nodes to the same vector space; Calculate the similarity between the corresponding feature vectors of the same modality data; Constructing a first knowledge graph corresponding to each modal data according to the feature vectors corresponding to the same modal data and the similarities thereof; The second knowledge graph construction unit is used to fuse the first knowledge graph corresponding to each modal data according to the deep learning method to obtain the second knowledge graph corresponding to the multimodal data of each time node; the first knowledge graph corresponding to each modal data according to the deep learning method is fused to obtain the second knowledge graph corresponding to the multimodal data of each time node, including: According to the graph neural network, the first knowledge graph corresponding to each modal data is fused with the following formula to obtain the second knowledge graph corresponding to the multimodal data of each time node: ； in, and are the weights and biases of the output layer of the graph neural network, is the set of all nodes in the first knowledge graph, express function, Is a node In the Characteristics of the layer; A data analysis unit is used to analyze the data to be processed based on the second knowledge graph corresponding to the multimodal data of each time node.

7. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the cross-modal data fusion method based on knowledge graph and deep learning as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the cross-modal data fusion method based on knowledge graph and deep learning as described in any one of claims 1 to 5.

Images