Multi-mode scientific and technological intelligence knowledge graph completion method, device, equipment and medium

Abstract

The invention discloses a multi-mode scientific and technological intelligence knowledge graph completion method, device and equipment, a medium and a product. The method comprises the following steps: obtaining each piece of modal information in each entity node of a target triple in a multi-modal science and technology intelligence knowledge graph, wherein the modal information comprises image information, text information and voice information; inputting each piece of modal information into a pre-trained quality evaluation model, expressing each entity node and connection among the entity nodes as a matrix by using the model according to each piece of modal information, and checking by using a matrix contract transformation criterion to obtain a quality evaluation result of the target triple; and complementing the target triad according to the result under the condition that the result is that the connection of the target triad is missing or the connection is wrong. According to the embodiment of the invention, entity nodes and connections can be mapped into features containing more information to a certain extent, the accuracy of quality evaluation of the multi-modal science and technology intelligence knowledge graph is improved, and the quality and the application value of the science and technology intelligence knowledge graph are improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a method, device, equipment, medium and product for completing a multimodal scientific and technological intelligence knowledge graph. Background Art

[0002] The multimodal scientific and technological intelligence knowledge graph integrates different information together and can quickly capture and predict specific scientific and technological intelligence information. Therefore, the quality of the multimodal scientific and technological intelligence knowledge graph is particularly important, as it will directly affect the accuracy and credibility of intelligence information acquisition and prediction.

[0003] Scientific and technological intelligence information has the characteristics of high density and can carry a large amount of information in a small carrier. Therefore, there is a phenomenon in the multimodal scientific and technological intelligence knowledge graph that two closely related entities are easily confused. The effect of the quality assessment of the multimodal scientific and technological intelligence knowledge graph is not very ideal. In addition, the richness of modal categories also directly affects the quality of the knowledge graph. The current multimodal scientific and technological intelligence knowledge graph mainly focuses on the fusion of image and text modalities. There are fewer categories of modal fusion, and the information reflected in the graph is not sufficient. Summary of the invention

[0004] The embodiments of the present application provide a multimodal scientific and technological intelligence knowledge graph completion method, device, equipment, medium and product, which are used to at least solve the problems in related technologies such as high density of scientific and technological intelligence information and easy confusion between entities, resulting in low accuracy and credibility of scientific and technological intelligence knowledge graph prediction and quality.

[0005] In a first aspect, an embodiment of the present application provides a method for completing a multimodal scientific and technological intelligence knowledge graph, comprising:

[0006] Obtaining each modal information in each entity node of the target triple in the multimodal scientific and technological intelligence knowledge graph, the modal information includes image information, text information and voice information;

[0007] Input each modal information into the quality assessment model of the pre-trained multimodal scientific and technological intelligence knowledge graph, use the quality assessment model of the multimodal scientific and technological intelligence knowledge graph to represent each entity node and the connection between each entity node as a matrix according to each modal information, and use the matrix contract transformation criterion to check and obtain the quality assessment result of the target triple;

[0008] When the quality assessment result indicates that the target triplet is missing a connection or has an incorrect connection, the target triplet is completed according to the quality assessment result.

[0009] In a second aspect, an embodiment of the present application provides a multimodal scientific and technological intelligence knowledge graph completion device, the device comprising:

[0010] An acquisition module is used to acquire each modal information in each entity node of the target triple in the multimodal scientific and technological intelligence knowledge graph, and the modal information includes image information, text information and voice information;

[0011] An input module is used to input each modal information into a quality assessment model of a pre-trained multimodal scientific and technological intelligence knowledge graph, and use the quality assessment model of the multimodal scientific and technological intelligence knowledge graph to represent each entity node and the connection between each entity node as a matrix according to each modal information, and use the matrix contract transformation criterion to verify and obtain the quality assessment result of the target triple;

[0012] A completion module is used to complete the target triplet according to the quality assessment result when the quality assessment result is that the target triplet is missing a connection or has an incorrect connection.

[0013] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the processor implements the steps of the multimodal scientific and technological intelligence knowledge graph completion method as described in any one of the embodiments of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the steps of the multimodal scientific and technological intelligence knowledge graph completion method as described in any one of the embodiments of the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the multimodal scientific and technological intelligence knowledge graph completion method provided in the first aspect of the embodiment of the present application.

[0016] The multimodal scientific and technological intelligence knowledge graph completion method, device, equipment, medium and product of the embodiment of the present application introduces the fusion of voice modality, and based on the quality assessment model of the multimodal scientific and technological intelligence knowledge graph obtained by training, the nodes and connections of the multimodal scientific and technological intelligence knowledge graph are represented by a two-dimensional matrix. The two-dimensional matrix representation can not only reflect spatial information but also contain richer features, which is more conducive to distinguishing dense scientific and technological intelligence information; and a matrix contract transformation criterion is designed to represent the relationship between graph triples, so that different knowledge graph entity nodes and connections between nodes can be distinguished to a certain extent, and the accuracy of the quality assessment of the multimodal scientific and technological intelligence knowledge graph can be improved, so as to realize the corresponding completion of the scientific and technological intelligence knowledge graph according to the quality assessment results, and can adapt to more modalities and more complex data distributions, thereby improving the quality and application value of the scientific and technological intelligence knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 It is a flowchart of a multimodal scientific and technological intelligence knowledge graph completion method provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of a method for training a quality assessment model of a multimodal scientific and technological intelligence knowledge graph provided in an embodiment of the present application;

[0020] Figure 3 It is a flowchart of a method for constructing a multi-modal weighted node representation matrix provided in an embodiment of the present application;

[0021] Figure 4 It is a structural schematic diagram of a multimodal scientific and technological intelligence knowledge graph completion device provided in an embodiment of the present application;

[0022] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0023] Reference numerals:

[0024] Multimodal scientific and technological intelligence knowledge graph completion device 400, acquisition module 401, input module 402, completion module 403,

[0025] Electronic device 500 , processor 501 , memory 502 , communication interface 503 , bus 510 . DETAILED DESCRIPTION

[0026] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0028] Scientific and technological intelligence is used to represent useful knowledge about scientific development, technological innovation, and the latest trends obtained through public information channels.

[0029] Multimodal knowledge graphs are used to represent the all-round perception and understanding of the world through the integration of multiple information forms such as natural language, visual information, hearing, smell, taste, physiological signals, etc. Its core goal is to build a framework that can express the cognition of artificial intelligence systems in a more comprehensive and multi-dimensional way. It is an innovative form of knowledge representation.

[0030] The quality of the knowledge graph is used to indicate the accuracy, completeness, consistency and credibility of knowledge. The quality directly affects the application effect of the graph.

[0031] Knowledge graph quality assessment is used to refer to the quality assessment of the knowledge in the knowledge graph through certain methods and standards to determine the quality level of the knowledge.

[0032] Contract transformation is used to represent the contract transformation operation based on the connection matrix. Taking matrix A as an example, B = P T AP, where P represents the connectivity of the graph.

[0033] Knowledge graph completion is used to predict the missing parts of triples in the knowledge graph, making the knowledge graph more complete and better in quality. It is a method to improve the quality of the knowledge graph.

[0034] Knowledge graph embedding is used to encode entities and relations and preserve the relationship between two entities. It is one of the important methods to solve the problem of knowledge graph completion.

[0035] In a complex information environment, the emergence of big data has made the data of scientific and technological intelligence more diverse, more complex in structure, and larger in scale. In the new environment and new situation, governments, scientific research institutions, and enterprises have also put forward higher demands for scientific and technological intelligence services: for example, how to quickly discover intelligence clues related to themselves from massive amounts of data, how to accurately predict the trends and directions of future science and technology, etc. This has given rise to related research on multimodal scientific and technological intelligence knowledge graphs.

[0036] The multimodal scientific and technological intelligence knowledge graph integrates different information together and can quickly capture and predict specific scientific and technological intelligence information. Therefore, the quality of the multimodal scientific and technological intelligence knowledge graph is particularly important, as it will directly affect the accuracy and credibility of intelligence information acquisition and prediction.

[0037] Scientific and technological intelligence information has the characteristics of high density and can carry the largest amount of information with the smallest carrier. Therefore, there is a phenomenon in the multimodal scientific and technological intelligence knowledge graph that two closely related entities are easily confused.

[0038] Multimodal knowledge graphs represent a rich extension of traditional knowledge graphs, enriching structural triples in the form of (head entity, relationship, tail entity) with comprehensive multimodal entity attributes. Multimodal knowledge graphs are representative and have become an important infrastructure in many artificial intelligence fields, such as recommendation systems, computer vision, and natural language processing.

[0039] Studying how to exploit structural knowledge in knowledge graphs and multimodal knowledge graphs is beneficial to many areas of artificial intelligence research. However, knowledge graphs and multimodal knowledge graphs often face the critical incompleteness problem because they contain a lot of unobserved factual knowledge. This phenomenon makes knowledge graph completion an important task to automatically discover new knowledge in a given knowledge graph. Traditional knowledge graph completion methods usually emphasize learning structural embeddings to model triple structures and measure the rationality of triples. In addition, multimodal knowledge graphs extend the task to multimodal knowledge graph completion by integrating information from different modalities such as images and texts into the knowledge graph completion model, thereby completing the task more comprehensively.

[0040] Knowledge graph completion is an important task in knowledge graph quality assessment, which aims to discover unobserved triples in a given knowledge graph. Traditional knowledge graph completion methods are usually based on embedding, which embeds the entities and relations of the knowledge graph into a continuous vector space and learns the embedding based on the existing triple structure. This technology is also called knowledge graph embedding. Usually, traditional knowledge graph embedding models design different scoring functions to measure the rationality of triples, and their overall goal is to assign higher scores to positive triples and lower scores to negative triples.

[0041] Existing knowledge graph embedding models are mainly divided into two categories: translation-based methods and semantic matching-based methods. Translation-based methods such as TransE, TransD, RotatE, OTE, PairRE, etc. model the triple structure as a relational translation from the head entity to the tail entity, and design a distance-based scoring function as a rationality measure. Semantic matching methods such as DistMult, ComplEx, TuckER, etc. use a similarity scoring function based on tensor decomposition. Some methods also try to extract structural semantics using deep neural networks.

[0042] The TransE algorithm is an algorithm for knowledge representation learning, which aims to describe the triples in the knowledge base through distributed representation, so as to efficiently store and complete the knowledge base. It is based on the distributed vector representation of entities and relations, and regards the relation in each triple instance (head, relation, tail) as the translation from the entity head to the entity tail. By constantly adjusting the vectors of head, relation and tail (h, r and t), (h+r) is made as equal to t as possible, that is, h+r=t.

[0043] The implementation of the TransE algorithm includes three main parts: representation vector initialization, batch training dataset construction, and representation vector update. In the representation vector initialization stage, each entity and each relationship is initialized using a k-dimensional random uniform distribution. In the batch training dataset construction, positive samples (h, r, t) are randomly selected from the training set, and based on the positive samples, negative samples are obtained by changing t or h, and merged to form a batch training dataset. The representation vector update uses the stochastic gradient descent method to update the vector representations of positive and negative samples in the batch training dataset.

[0044] The algorithm logic of TransE includes: randomly initializing the head entity, tail entity and relationship vectors, and normalizing the randomly initialized vectors; sampling triplets in the data set, and replacing the sampled triplets with entities to form negative examples; and using the stochastic gradient descent method to continuously optimize the vector representation of entities and relationships.

[0045] Multimodal knowledge graph completion further considers leveraging the complex multimodal information in multimodal knowledge graphs to benefit knowledge graph completion. Current mainstream methods usually extend traditional knowledge graph embedding models to more flexible entity multimodal embeddings, such as visual embeddings and text embeddings. These embeddings are extracted through pre-trained models and represent entity features from multiple views. Multimodal knowledge graph completion methods can be further divided into three categories. The first category is modality fusion methods. These methods design elegant methods to achieve multimodal fusion in the same representation space. The second category is modality integration methods. These methods learn their own models for each modality separately and make joint predictions through ensemble learning. The third category is negative sampling enhancement methods. Negative sampling is an important technology for training knowledge graph embedding models. Therefore, some methods try to leverage multimodal information to enhance the negative sampling process and generate high-quality negative samples. The problem scenarios of existing multimodal knowledge graph completion methods are relatively simple, usually considering text and image modalities, while some works only consider numerical modalities.

[0046] It can be seen that the current knowledge graph embedding methods such as TransE have simple design ideas and are easy to implement. However, due to the high density and easy confusion of scientific and technological intelligence information, the effect of multimodal scientific and technological intelligence knowledge graph quality assessment is not very ideal, resulting in the current multimodal scientific and technological intelligence knowledge graph having problems with the accuracy and credibility of prediction and quality. There are mainly two points: First, the characteristic of scientific and technological intelligence information that a small carrier can carry a large amount of information leads to the poor quality of the constructed scientific and technological intelligence knowledge graph, and the entity nodes are easily confused; second, the current multimodal scientific and technological intelligence knowledge graph mainly focuses on the fusion of image and text modalities, with fewer categories of modal fusion, and the information reflected by the graph is not sufficient.

[0047] In order to solve the problems of related technologies, the embodiments of the present application provide a multimodal scientific and technological intelligence knowledge graph completion method, device, equipment, medium and product.

[0048] The multimodal scientific and technological intelligence knowledge graph completion method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in combination with the accompanying drawings.

[0049] Figure 1 FIG. 1 is a flow chart of a multimodal scientific and technological intelligence knowledge graph completion method 100 according to an embodiment of the present application. Figure 1 As shown, the multimodal scientific and technological intelligence knowledge graph completion method 100 may specifically include the following steps:

[0050] S101, obtaining each modal information in each entity node of the target triple in the multimodal scientific and technological intelligence knowledge graph, where the modal information includes image information, text information and voice information;

[0051] S102, input each modal information into the quality assessment model of the pre-trained multimodal scientific and technological intelligence knowledge graph, use the quality assessment model of the multimodal scientific and technological intelligence knowledge graph to represent each entity node and the connection between each entity node as a matrix according to each modal information, and use the matrix contract transformation criterion to check, so as to obtain the quality assessment result of the target triple;

[0052] S103: When the quality assessment result indicates that the target triplet is missing a connection or has an incorrect connection, complete the target triplet according to the quality assessment result.

[0053] Therefore, the fusion of speech modality is introduced. Based on the quality assessment model of the multimodal scientific and technological intelligence knowledge graph obtained through training, the nodes and connections of the multimodal scientific and technological intelligence knowledge graph are represented by a two-dimensional matrix. The two-dimensional matrix representation can not only reflect the spatial information but also contain richer features, which is more conducive to distinguishing dense scientific and technological intelligence information; and a matrix contract transformation criterion is designed to represent the relationship between graph triplets, so that different knowledge graph entity nodes and connections between nodes can be distinguished to a certain extent, and the accuracy of the quality assessment of the multimodal scientific and technological intelligence knowledge graph can be improved, so as to realize the corresponding completion of the scientific and technological intelligence knowledge graph according to the quality assessment results, which can adapt to more modalities and more complex data distribution, thereby improving the quality and application value of the scientific and technological intelligence knowledge graph.

[0054] The specific implementation methods of the above steps are introduced below.

[0055] It should be noted that the target triple is the triple to be tested in the multimodal science and technology intelligence knowledge graph. Specifically, each entity node of the target triple includes information of each modality, such as image information, text information, and voice information. In this embodiment, the target triple is tested using a pre-trained quality assessment model of the multimodal science and technology intelligence knowledge graph, including but not limited to: whether there is a connection between nodes, or whether the connection is wrong.

[0056] That is to say, before S102, this embodiment constructs and trains a quality assessment model for the multimodal scientific and technological intelligence knowledge graph, so as to use the model to provide strong support for the subsequent analysis and application of the multimodal scientific and technological intelligence knowledge graph, realize effective quality assessment and complete reasoning of the multimodal scientific and technological intelligence knowledge graph, and enhance the credibility of scientific and technological intelligence analysis.

[0057] Specifically, the training process of the quality assessment model of the multimodal scientific and technological intelligence knowledge graph provided in the embodiment of the present application can refer to Figure 2 .like Figure 2 As shown, a training method 200 for a quality assessment model of a multimodal scientific and technological intelligence knowledge graph provided in an embodiment of the present application may include the following steps:

[0058] S201 to S205.

[0059] S201. For the first entity node and the second entity node of any triple in the multimodal scientific and technological intelligence knowledge graph, extract the feature matrices corresponding to each modal information in the first entity node and the second entity node respectively.

[0060] In some embodiments, when there is image information in the entity node, features of different images of the entity node are extracted through convolution operations to obtain feature matrices corresponding to different images, and the feature matrices corresponding to different images are fused into image fusion feature matrices through weighted averaging; when there is text information in the entity node, features of the text information are extracted through word embedding operations to obtain a text feature matrix; when there is voice information in the entity node, Mel-frequency cepstral coefficient features are extracted for the voice information to obtain a voice feature matrix.

[0061] S202: Weightedly fuse the feature matrices corresponding to each modal information of the first entity node into a first node representation matrix, and weightedly fuse the feature matrices corresponding to each modal information of the second entity node into a second node representation matrix.

[0062] In some embodiments, the weights corresponding to the image fusion feature matrix, the text feature matrix, and the speech feature matrix are determined according to whether image information, text information, and speech information exist in the entity node; and according to the weights corresponding to the image fusion feature matrix, the text feature matrix, and the speech feature matrix, the image fusion feature matrix, the text feature matrix, and the speech feature matrix are weightedly fused into a first node representation matrix.

[0063] That is, the extracted image, text and speech representation matrices can be weighted and merged according to the following formula (1):

[0064] node=fig×w_f+text×w_t+speech×w_s; (1)

[0065] Among them, node represents the representation matrix of the knowledge graph entity node; fig represents the image fusion feature matrix; text represents the text feature matrix; speech represents the speech feature matrix; w_f represents the weight of the image fusion feature matrix, w_t represents the weight of the text feature matrix, and w_s represents the weight of the speech feature matrix.

[0066] The following describes the weight values ​​of the feature matrix corresponding to each modal information through a more specific embodiment.

[0067] In a specific implementation, when any one or any two of the modal information among image information, text information and voice information does not exist in the entity node, the feature matrix corresponding to the non-existent modal information is determined as a random matrix or a zero matrix, and the first weight of the feature matrix corresponding to the non-existent modal information is determined as a first preset value; based on the modal information that does not exist in the entity node and the first weight of its corresponding feature matrix, the second weight of the feature matrix corresponding to the modal information that exists in the entity node is determined.

[0068] For example, if the entity node only has text information, the weights of the speech feature matrix and the image feature matrix are both the first preset value, such as 0.02, and the weight of the text feature matrix is ​​0.96; if the entity node does not contain speech information, the weight of the speech feature matrix is ​​set to the first preset value, such as 0.02, and the weights of the image feature matrix and the text feature matrix are both 0.49.

[0069] In a specific implementation, when image information, text information and voice information exist in the entity node, the weights corresponding to the image fusion feature matrix, the text feature matrix and the voice feature matrix are determined to be the second preset value. That is, if image information, text information and voice information all exist in the entity node, the weights of their feature matrices are all 1 / 3.

[0070] In this case, when the feature matrix corresponding to the non-existent modal information is set to a random number, its weight is assigned to a very low value, so that its influence on the weighted fusion matrix can be basically ignored.

[0071] In another embodiment, taking the first entity node as an example, Figure 3 The specific process of constructing a multi-modal weighted node representation matrix by another method provided by this embodiment is shown.

[0072] like Figure 3 As shown in the figure, num images of n1×n1 dimension are convolved to obtain num m×m image feature matrices, and the image feature matrices are weighted averaged to obtain an m×m image fusion feature matrix. After the text is segmented, the word embedding operation is performed, and the maximum number of tokens of the text is set to n2, and the embedding dimension is dim, so that an n2×dim embedding matrix can be obtained. Then, the matrix is ​​feature extracted and compressed, and the feature dimension of the obtained fully connected layer (m×m nodes) is converted to obtain an m×m text representation matrix. Mel frequency cepstral coefficient feature extraction is performed on the speech information (the signal is divided into n3 frames) to obtain an n3×k dimensional speech feature matrix, where k is the speech feature dimension; feature extraction and compression are performed on the speech feature matrix to form an m×m speech representation matrix. Furthermore, the feature matrices of different modalities are weighted and merged to form a representation matrix of entity nodes.

[0073] S203 . Performing a contract transformation on the first node representation matrix using a connection matrix to obtain a contract transformation matrix, wherein the connection matrix represents a connection between the first entity node and the second entity node of the triple.

[0074] In some embodiments, the connection matrix P is introduced, and the first node representation matrix can be transformed according to the following formula (2):

[0075] B=P T AP; (2)

[0076] Wherein, A is the first node representation matrix; B is the second node representation matrix; and the condition restriction on the existence of an inverse matrix is ​​relaxed for the connection matrix P.

[0077] In this way, the nodes and connections of the multimodal science and technology intelligence knowledge graph are represented by a two-dimensional matrix. Compared with the vector representation used in the related art, the vector cannot reflect spatial information, but especially for images, a lot of information is two-dimensional. If a pure vector method is used, some information will be lost. This embodiment uses a two-dimensional matrix representation method to not only reflect spatial information, but also contain richer features, which is more conducive to distinguishing high-density science and technology intelligence information.

[0078] S204. Calculate the average Euclidean distance loss between the first entity node and the second entity node after the contract transformation is performed according to the contract transformation matrix and the second node representation matrix.

[0079] Specifically, the average Euclidean distance loss is the average Euclidean distance between the contract transformation matrix and the second node representation matrix. That is, the average Euclidean distance loss can be calculated according to the following formula (3):

[0080]

[0081] S205. Based on the average Euclidean distance loss, a quality assessment model of the multimodal scientific and technological intelligence knowledge graph is trained.

[0082] In some embodiments, the feature extraction network and the connection matrix P are optimized according to the average Euclidean distance loss. Specifically, the network parameters can be adjusted by back propagation until the preset training stop condition is met to obtain the final updated network. Optionally, the preset training stop condition may include: the maximum number of training iterations set according to actual application requirements, the model convergence condition, etc.

[0083] The above is a specific implementation method of the quality assessment model for training a multimodal science and technology intelligence knowledge graph provided in the embodiment of the present application. Thus, after completing the embedding training of the science and technology intelligence knowledge graph, an embedding matrix of knowledge graph entity nodes and connections can be obtained, that is, a trained quality assessment model can be obtained for subsequent knowledge graph completion and reasoning.

[0084] Back to Figure 1 The multimodal scientific and technological intelligence knowledge graph completion method 100 is shown. In some embodiments, in S102, first, the trained model is used to extract the features of each modal information in each entity node of the target triple, and the feature matrix of each modal information is formed; then the matrices corresponding to different modalities are weighted and fused to construct the matrix representation of the graph entity nodes, and the connection between each entity node is represented as a matrix; the contract transformation criterion is used to check whether the target triple is missing, and the quality assessment result of the target triple is obtained.

[0085] In specific implementation, it can be checked whether the average Euclidean distance between the representation matrices corresponding to the entity nodes of the target triple is less than the preset threshold value of 0.1. If it is less than 0.1, the criterion is met; otherwise, it is not met. It can be understood that meeting the criterion means that the quality assessment result of the target triple is that the target triple exists a connection. If there is no connection in the triple in the graph at this time, the knowledge graph is completed; otherwise, if the criterion is not met and the triple in the graph exists a connection, the connection is deleted.

[0086] Furthermore, in some embodiments, in S103, the scientific and technological intelligence knowledge graph is supplemented accordingly according to the quality assessment result, for example but not limited to: connecting the target triples or supplementing the entity nodes. On the contrary, if the target triples are not missing after verification, there is no need to supplement the graph.

[0087] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0088] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a multimodal scientific and technological intelligence knowledge graph completion device 400.

[0089] like Figure 4 As shown, the multimodal scientific and technological intelligence knowledge graph completion device 400 may include:

[0090] The acquisition module 401 is used to acquire each modal information in each entity node of the target triple in the multimodal scientific and technological intelligence knowledge graph, and the modal information includes image information, text information and voice information;

[0091] An input module 402 is used to input each modal information into a quality assessment model of a pre-trained multimodal scientific and technological intelligence knowledge graph, and use the quality assessment model of the multimodal scientific and technological intelligence knowledge graph to represent each entity node and the connection between each entity node as a matrix according to each modal information, and use the matrix contract transformation criterion to verify and obtain the quality assessment result of the target triple;

[0092] The completion module 403 is used to complete or delete the target triplet according to the quality assessment result when the quality assessment result shows that the target triplet is missing a connection or has an incorrect connection.

[0093] In some embodiments, the multimodal scientific and technological intelligence knowledge graph completion device 400 further includes a training module ( Figure 4 Specifically, the training module includes the following units:

[0094] An extraction unit, for extracting, for a first entity node and a second entity node of any triple in a multimodal scientific and technological intelligence knowledge graph, feature matrices corresponding to each modal information in the first entity node and the second entity node respectively;

[0095] A weighted fusion unit, used for weighted fusion of feature matrices corresponding to each modal information of the first entity node into a first node representation matrix, and weighted fusion of feature matrices corresponding to each modal information of the second entity node into a second node representation matrix;

[0096] A contract transformation unit, configured to perform a contract transformation on the first node representation matrix using a connection matrix to obtain a contract transformation matrix, wherein the connection matrix represents a connection between the first entity node and the second entity node of the triple;

[0097] A calculation unit, used for calculating the average Euclidean distance loss between the first entity node and the second entity node after the first entity node is contract-transformed according to the contract transformation matrix and the second node representation matrix;

[0098] A training unit is used to train a quality assessment model of a multimodal scientific and technological intelligence knowledge graph based on the average Euclidean distance loss.

[0099] Optionally, the contract transformation loss is the average Euclidean distance between the contract transformation matrix and the second node representation matrix.

[0100] In some optional embodiments, the extraction unit is specifically used to, when there is image information in the entity node, perform feature extraction on different images of the entity node through a convolution operation to obtain feature matrices corresponding to different images, and fuse the feature matrices corresponding to different images into an image fusion feature matrix through weighted averaging; when there is text information in the entity node, extract features of the text information through a word embedding operation to obtain a text feature matrix; when there is voice information in the entity node, perform Mel-frequency cepstral coefficient feature extraction on the voice information to obtain a voice feature matrix.

[0101] In some optional embodiments, the weighted fusion unit is specifically used to determine the weights corresponding to the image fusion feature matrix, the text feature matrix, and the speech feature matrix according to whether image information, text information, and speech information exist in the entity node; and according to the weights corresponding to the image fusion feature matrix, the text feature matrix, and the speech feature matrix, the image fusion feature matrix, the text feature matrix, and the speech feature matrix are weightedly fused into a first node representation matrix.

[0102] In some optional embodiments, the weighted fusion unit is further specifically used to determine the feature matrix corresponding to the non-existent modal information as a random matrix and determine the first weight of the feature matrix corresponding to the non-existent modal information as a first preset value when any one or any two of the modal information among image information, text information and voice information does not exist in the entity node; determine the second weight of the feature matrix corresponding to the modal information existing in the entity node based on the first weight of the modal information that does not exist in the entity node and its corresponding feature matrix; and determine that the weights corresponding to the image fusion feature matrix, the text feature matrix and the voice feature matrix are all second preset values ​​when image information, text information and voice information exist in the entity node.

[0103] It should be noted that, for the convenience of description, the above device is described in various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0104] The device of the above embodiment is used to implement the corresponding multimodal scientific and technological intelligence knowledge graph completion method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0105] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an electronic device.

[0106] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided by this embodiment is shown.

[0107] The electronic device 500 may include a processor 501 and a memory 502 storing computer program instructions.

[0108] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0109] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 502 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory.

[0110] In certain embodiments, the memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present application.

[0111] The processor 501 implements any one of the multimodal scientific and technological intelligence knowledge graph completion methods in the above-mentioned embodiments by reading and executing computer program instructions stored in the memory 502.

[0112] In some examples, the electronic device 500 may further include a communication interface 503 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.

[0113] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0114] Bus 510 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus 510 may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 510 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0115] Exemplarily, the electronic device 500 may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA).

[0116] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a non-transitory computer-readable storage medium. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any of the multimodal science and technology intelligence knowledge graph completion methods in the above-mentioned embodiments is implemented. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0117] Based on the same technical concept, corresponding to any of the above-mentioned embodiments, the present application also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the multimodal scientific and technological intelligence knowledge graph completion method. Corresponding to the execution subject corresponding to each step in each embodiment of the multimodal scientific and technological intelligence knowledge graph completion method, the processor that executes the corresponding step can belong to the corresponding execution subject.

[0118] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0119] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0120] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0121] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0122] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A multimodal scientific and technological intelligence knowledge graph completion method, characterized in that: include: Obtaining each modal information in each entity node of the target triple in the multimodal scientific and technological intelligence knowledge graph, the modal information includes image information, text information and voice information; Input each modal information into the quality assessment model of the pre-trained multimodal scientific and technological intelligence knowledge graph, use the quality assessment model of the multimodal scientific and technological intelligence knowledge graph to represent each entity node and the connection between each entity node as a matrix according to each modal information, and use the matrix contract transformation criterion to check and obtain the quality assessment result of the target triple; When the quality assessment result indicates that the target triplet is missing a connection or has an incorrect connection, the target triplet is completed according to the quality assessment result.

2. The method according to claim 1, characterized in that Before inputting each modal information into the quality assessment model of the pre-trained multimodal scientific and technological intelligence knowledge graph, the method further includes: For the first entity node and the second entity node of any triple in the multimodal scientific and technological intelligence knowledge graph, the feature matrices corresponding to each modal information in the first entity node and the second entity node are extracted respectively; The feature matrices corresponding to each modal information of the first entity node are weighted and fused into a first node representation matrix, and the feature matrices corresponding to each modal information of the second entity node are weighted and fused into a second node representation matrix; Performing a contract transformation on the first node representation matrix using a connection matrix to obtain a contract transformation matrix, wherein the connection matrix represents a connection between the first entity node and the second entity node of the triple; Calculating the average Euclidean distance loss between the first entity node and the second entity node after the first entity node is contract-transformed according to the contract transformation matrix and the second node representation matrix; According to the average Euclidean distance loss, a quality assessment model of the multimodal scientific and technological intelligence knowledge graph is trained.

3. The method according to claim 2, characterized in that The average Euclidean distance loss is the average Euclidean distance between the contract transformation matrix and the second node representation matrix.

4. The method according to claim 2, characterized in that: The extracting of the feature matrices corresponding to each modal information in the entity node respectively includes: When there is image information in the entity node, the features of different images of the entity node are extracted through convolution operation to obtain the feature matrices corresponding to the different images, and the feature matrices corresponding to the different images are fused into an image fusion feature matrix through weighted averaging; When there is text information in the entity node, the features of the text information are extracted through word embedding operations to obtain a text feature matrix; When there is voice information in the entity node, the Mel-frequency cepstral coefficient feature extraction is performed on the voice information to obtain a voice feature matrix.

5. The method according to claim 4, characterized in that The step of weighting and fusing the feature matrices corresponding to each modal information into a first node representation matrix includes: According to whether there is image information, text information and voice information in the entity node, the weights corresponding to the image fusion feature matrix, the text feature matrix and the voice feature matrix are determined; According to the weights corresponding to the image fusion feature matrix, the text feature matrix and the speech feature matrix, the image fusion feature matrix, the text feature matrix and the speech feature matrix are weighted and fused into a first node representation matrix.

6. The method according to claim 5, characterized in that The determining of the weights corresponding to the image fusion feature matrix, the text feature matrix and the voice feature matrix respectively according to whether there is image information, text information and voice information in the entity node comprises: In the case that any one or any two modal information among image information, text information and voice information does not exist in the entity node, the feature matrix corresponding to the non-existent modal information is determined as a random matrix, and the first weight of the feature matrix corresponding to the non-existent modal information is determined as a first preset value; according to the modal information that does not exist in the entity node and the first weight of the feature matrix corresponding to the modal information that exists in the entity node, the second weight of the feature matrix corresponding to the modal information that exists in the entity node is determined; When there is image information, text information and voice information in the entity node, it is determined that the weights corresponding to the image fusion feature matrix, the text feature matrix and the voice feature matrix are all second preset values.

7. A multimodal scientific and technological information knowledge graph completion device, characterized in that: The device comprises: An acquisition module is used to acquire each modal information in each entity node of the target triple in the multimodal scientific and technological intelligence knowledge graph, and the modal information includes image information, text information and voice information; An input module is used to input each modal information into a quality assessment model of a pre-trained multimodal scientific and technological intelligence knowledge graph, and use the quality assessment model of the multimodal scientific and technological intelligence knowledge graph to represent each entity node and the connection between each entity node as a matrix according to each modal information, and use the matrix contract transformation criterion to verify and obtain the quality assessment result of the target triple; A completion module is used to complete the target triplet according to the quality assessment result when the quality assessment result is that the target triplet is missing a connection or has an incorrect connection.

8. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; when the processor calls the computer program instructions, it implements the multimodal scientific and technological intelligence knowledge graph completion method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when called by a processor, implement the multimodal scientific and technological intelligence knowledge graph completion method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the multimodal scientific and technological intelligence knowledge graph completion method as described in any one of claims 1-6.

Images