Knowledge graph representation method and device based on similarity diffusion

Abstract

The invention provides a knowledge graph representation method and device based on similarity diffusion, and the method comprises the steps: constructing a target knowledge graph based on a multi-modal large model and a human-in-the-loop mechanism; constructing an adjacent graph matrix of the target knowledge graph based on the similarity relationship among the entities in the target knowledge graph; based on the adjacent graph matrix, performing similarity diffusion under local clustering constraint to obtain a first similarity matrix; and based on the first similarity matrix, performing geometric transmission on the target knowledge graph in a manifold graph to obtain relation representation of the target knowledge graph. Therefore, the more effective and robust relationship representation of the target knowledge graph is obtained, the construction and updating of the knowledge graph can be guided through the relationship representation, and the mining, fusion, reasoning and retrieval capabilities of the knowledge graph are improved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph technology, and in particular to a knowledge graph representation method and device based on similarity diffusion. Background Art

[0002] Knowledge graph is a structured semantic knowledge base that can effectively represent complex knowledge in information management systems by organizing knowledge entities and their relationships in the form of a graph. Specifically, each entity in the knowledge graph network is represented as a node, and the various connections between entities are represented by edges, so that the graph can systematically store rich entity information and their relationships, greatly improving the accessibility and comprehensibility of information.

[0003] The construction of knowledge graphs usually relies on a large number of data mining and information extraction technologies, including entity recognition, relationship extraction, and knowledge fusion. Although there are many knowledge graph modeling methods, the broader potential relationships between entities have not been fully explored, so the mining, fusion, reasoning, and retrieval capabilities of knowledge graphs still need to be improved. Summary of the invention

[0004] The present invention provides a knowledge graph representation method and device based on similarity diffusion, which is used to solve the defect that the broader potential relationships between entities in the prior art have not been fully explored. It can better guide the construction and update of knowledge graphs and enhance their data mining, fusion, reasoning and retrieval capabilities.

[0005] The present invention provides a knowledge graph representation method based on similarity diffusion, comprising the following steps: Build the target knowledge graph based on the multimodal big model and human-in-the-loop mechanism; Based on the similarity relationship between each entity in the target knowledge graph, construct an adjacency graph matrix of the target knowledge graph; Based on the adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix; Based on the first similarity matrix, the target knowledge graph is geometrically transmitted in a manifold graph to obtain a relational representation of the target knowledge graph.

[0006] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, based on the similarity relationship between entities in the target knowledge graph, an adjacency graph matrix of the target knowledge graph is constructed, specifically comprising: Based on the Euclidean distances between the entities in the target knowledge graph, an initial distance matrix of the entities is obtained; Based on the initial distance matrix and the nearest neighbor edges of each entity, an adjacency graph matrix of the target knowledge graph is constructed.

[0007] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, based on the first similarity matrix, the target knowledge graph is geometrically transmitted in a manifold graph to obtain a relational representation of the target knowledge graph, specifically comprising: Based on the neighbor nodes of the initial distance matrix, performing neighborhood-guided smoothing enhancement on the first similarity matrix to obtain a second similarity matrix; Based on the second similarity matrix, the target knowledge graph is geometrically transmitted in a manifold graph to obtain a relational representation of the target knowledge graph.

[0008] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, based on the adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix, which specifically includes: Based on the adjacency graph matrix and the k-nearest neighbor strategy, construct a local clustering graph of each entity; The local cluster graphs of the entities are diffused through a bidirectional propagation mechanism to obtain a first similarity matrix.

[0009] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, based on the second similarity matrix, the target knowledge graph is geometrically transmitted in a manifold graph to obtain a relational representation of the target knowledge graph, specifically comprising: Performing information propagation on the second similarity matrix to obtain a third similarity matrix; Based on the third similarity matrix, the target knowledge graph is geometrically transmitted in a manifold graph to obtain a relational representation of the target knowledge graph.

[0010] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, based on a first similarity matrix, a second similarity matrix or a third similarity matrix, the target knowledge graph is geometrically transmitted in a manifold graph to obtain a relational representation of the target knowledge graph, specifically comprising: Based on the initial distance matrix and the adjacency graph matrix, a geodesic distance matrix containing manifold space information is obtained; Based on the geodesic distance matrix, an optimal transfer matrix between distribution representations of each entity in the target knowledge graph in the manifold space is calculated; wherein the distribution representations of each entity in the target knowledge graph in the manifold space are obtained based on the first similarity matrix, the second similarity matrix or the third similarity matrix; Determining the geometric transmission distance between each entity based on the optimal transmission matrix and the distribution representation of each entity in the manifold space; Based on the geometric transmission distance, a relational representation of the target knowledge graph is obtained.

[0011] The present invention also provides a knowledge graph representation device based on similarity diffusion, comprising the following modules: A construction module for building a target knowledge graph based on a multimodal large model and a human-in-the-loop mechanism; An adjacency module, used to construct an adjacency graph matrix of the target knowledge graph based on the similarity relationship between each entity in the target knowledge graph; A diffusion module, used for performing similarity diffusion under local clustering constraints based on the adjacency graph matrix to obtain a first similarity matrix; A representation module is used to geometrically transmit the target knowledge graph in a manifold graph based on the first similarity matrix to obtain a relational representation of the target knowledge graph.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the above-described knowledge graph representation methods based on similarity diffusion.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described knowledge graph representation methods based on similarity diffusion.

[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned knowledge graph representation methods based on similarity diffusion.

[0015] The knowledge graph representation method and device based on similarity diffusion provided by the present invention obtain a target knowledge graph by constructing it based on a multimodal large model and a human-in-the-loop mechanism, and construct an adjacency graph matrix of the target knowledge graph according to the similarity relationship between each entity in the target knowledge graph. Then, based on the adjacency graph matrix, similarity diffusion is performed under local clustering constraints, which can mine a wider range of potential relationships between entities and obtain a first similarity matrix. The target knowledge graph is geometrically transmitted in a manifold graph according to the first similarity matrix, thereby obtaining a more effective and robust relational representation of the target knowledge graph. Through this relational representation, the construction and update of the knowledge graph can be guided, and the mining, fusion, reasoning and retrieval capabilities of the knowledge graph can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of the flow of the knowledge graph representation method based on similarity diffusion provided by the present invention.

[0018] Figure 2 A schematic diagram of a process for obtaining more robust relationships between knowledge graph entities based on similarity diffusion provided by the present invention.

[0019] Figure 3 A schematic diagram of the structure of the knowledge graph representation device based on similarity diffusion provided by the present invention.

[0020] Figure 4 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Figure 1 A schematic diagram of the process flow of the knowledge graph representation method based on similarity diffusion provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 100: Build a target knowledge graph based on a multimodal large model and a human-in-the-loop mechanism.

[0023] Specifically, the target knowledge graph in the embodiment of the present invention is the knowledge graph that the user wants to construct.

[0024] Multimodal big models refer to AI models that can process and understand multiple types of data (e.g., text, images, audio, video, etc.). These models are designed to fuse information from different modalities together to facilitate more comprehensive and accurate task processing. Human-in-the-loop mechanisms refer to the interaction and feedback mechanisms between humans and automated systems in certain systems.

[0025] Therefore, we can first use the human-in-the-loop prompt learning technology combined with a multimodal large model to extract the text description information of the image, and then use a method that combines neural network-based named entity recognition and relationship classification models to realize the extraction of triple information of text descriptions in all database images, thereby constructing the target knowledge graph.

[0026] In the process of using the multimodal big model to build the target knowledge graph, the multimodal big model can be used to automatically extract the text description of the image library and build an attribute knowledge graph based on these descriptions, which requires obtaining the entity, relationship, and attribute information of the target knowledge graph.

[0027] The human-in-the-loop algorithm manually filters images, corpora, and triple information based on intelligent algorithms and achieves unified extraction. In the field of knowledge extraction, the human-in-the-loop mechanism can integrate human prior knowledge to improve the reliability of the knowledge graph knowledge base while retaining the high generalization and high robustness of the multimodal large model.

[0028] Step 101: Based on the similarity relationship between each entity in the target knowledge graph, construct an adjacency graph matrix of the target knowledge graph.

[0029] Specifically, after constructing the target knowledge graph, for each entity in the target knowledge graph, the entity set can be defined as ,in is the number of entities in the target knowledge graph, for the set Each entity in , we can construct a knowledge representation for it and use the powerful expression ability of the deep learning network to encode it into a Characteristics of Dimensions ,These features are used in the subsequent knowledge graph mining, fusion, reasoning and retrieval processes.

[0030] Can calculate the set Any pair of entities and For example, the similarity between entities is calculated by Euclidean distance, which is used to measure the distance between two entities in the feature space. A smaller distance indicates a higher similarity. Another example is cosine similarity, which is used to calculate the cosine value of the angle between two vectors. It is used to indicate the similarity between entity features. The closer the value is to 1, the higher the similarity.

[0031] After calculating the similarity between each entity, we can construct the adjacency graph matrix of the target knowledge graph based on the similarity calculation results. The adjacency graph can be represented as , in this adjacency graph, the vertex set ,correspond The feature encoding of each entity in Representing the connection relationship between entity vertices, the edge weight of the adjacency graph can be determined according to the similarity calculation results between each entity, and then the weight matrix of the adjacency graph is used as the adjacency graph matrix.

[0032] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, an adjacency graph matrix of a target knowledge graph is constructed based on the similarity relationship between entities in the target knowledge graph, specifically including: Based on the Euclidean distance between each entity in the target knowledge graph, the initial distance matrix of each entity is obtained; Based on the initial distance matrix and the nearest neighbor edges of each entity, the adjacency graph matrix of the target knowledge graph is constructed.

[0033] Specifically, in the feature encoding space of the target knowledge graph, any pair of entities and The similarity between them can be calculated by the Euclidean distance of their feature vectors, and the calculation formula is as follows:

[0034] After calculating the distance between each pair of entities, an initial distance matrix can be obtained. , where the matrix elements Defined as . Each row of the matrix Representing Entities With the dataset The distance between them can be used to sort the data set.

[0035] Based on such prior knowledge: similar entities tend to cluster in the distance matrix In the low-dimensional manifold structure implied by The expressiveness of this structure is enhanced by mapping it to a new space. This mapping ensures that entities with similar knowledge semantics are not only close to each other in the traditional Euclidean space, but also show higher similarity in the manifold space.

[0036] In order to approximate this manifold structure, the embodiment of the present invention can use the k-nearest neighbor (k-NN) method to select each node nearest neighbor nodes and assign weights to the edges between them according to the similarity, thus constructing a Nearest Neighbor Graph , in this graph, the vertex set correspond The feature encoding of each entity in Represents the connection relationship between entity vertices, each edge Connect Vertices and , the edge weight is the entity and The similarity between them, the weight matrix It can be expressed as:

[0037] The matrix is the indicator matrix used to connect Nearest neighbor, formally, when hour, , Representation and Node Most similar A collection of nodes; It is a parameter that controls the attenuation of similarity, so that the entities with smaller distances have higher similarities and greater weights.

[0038] The constructed adjacency graph can be constructed using the weight matrix It is represented by the adjacency graph matrix. Each row of the adjacency graph matrix represents the similarity relationship between an entity and all other entities, forming the basic connection structure in the knowledge graph.

[0039] Step 102: Based on the adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix.

[0040] Specifically, after obtaining the adjacency graph matrix, a more robust similarity matrix can be obtained by diffusing the similarity information under the constraint of local clustering, thus avoiding the influence of noise and outliers on the construction and update of the overall graph.

[0041] Local clustering means defining a local group containing similar neighbors for each entity. Therefore, we can first determine the local clusters of each entity, and then perform the diffusion process only in the entities of the local clusters, so as to achieve smooth transfer of similarity within the local group.

[0042] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, based on an adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix, which specifically includes: Based on the adjacency graph matrix and k-nearest neighbor strategy, a local clustering graph of each entity is constructed; The local cluster graph of each entity is diffused through a bidirectional propagation mechanism to obtain a first similarity matrix.

[0043] Specifically, we can first find the most trusted neighbors of each entity in the adjacency graph matrix through the k-reciprocal neighbors strategy, which ensures the bidirectional relationship between entities. That is, if the entity Is Entity Neighbors of It is also an entity This process avoids the negative impact of untrustworthy neighbors and constructs a higher confidence local clustering set. .

[0044] -The mutual nearest neighbor formalization is defined as follows:

[0045] Specifically, given an entity , its -The neighbor set is defined as , for any element in this set, only when the entity Also in this element -The element belongs to of - Mutual neighbor sets , this set can be regarded as a local cluster .

[0046] After obtaining the local clustering graph of each entity, the local clustering graph of each entity can be diffused through a bidirectional propagation mechanism to obtain a first similarity matrix.

[0047] Traditional similarity diffusion methods often propagate similarity in the entire graph, which is easily disturbed by outliers and noise samples. To this end, a bidirectional similarity diffusion method under local clustering constraints is proposed, which only propagates similarity in the local graph (i.e., the local cluster set The goal of the diffusion process is to achieve a smooth transfer of similarities among local clusters and ensure the symmetry of the similarity matrix through a bidirectional propagation mechanism.

[0048] In the embodiment of the present invention, an objective function is defined, which aims to minimize the difference between the similarity matrix and the local graph structure, while taking into account the local similarity between entities. The definition is as follows:

[0049] in, is the smoothing control parameter. Represents the adjacency graph matrix. For simplicity, a symmetric matrix can be used in its construction. Replace the original adjacency matrix. The matrix is a diagonal matrix, whose The diagonal elements are equal to No. The sum of the rows, in the In the guided regularization term, the matrix is positive definite, it can be used to prevent Too smooth, you can adopt different strategies according to your needs For entity triples in local clustering and , using adjacency weights To constrain and At the same time, we also consider and The similarity of this pair makes the smoothing strategy bidirectional, ensuring that the obtained similarity matrix It has symmetry.

[0050] For the constrained objective function mentioned above, it is very difficult to directly solve the optimization goal. Therefore, in some implementations, in order to simplify the optimization problem, the complex objective function can be decomposed into two sub-problems that are easier to solve. First, for bidirectional diffusion without clustering constraints, the optimization goal is:

[0051] Then, only the In order to more conveniently solve the optimization problem in the above formula, the embodiment of the present invention assumes that the optimization target is , using the mathematical tools in graph theory, the target formula can be reconstructed as:

[0052] in is an identity matrix, The regularization matrix of ,matrix About The mean Kronecker product of is defined as , operator is a vectorization function that can combine the column vectors of a matrix into a new column vector, and is It can be proved that the objective function after relaxing the constraints is is a convex function, that is, its Hessian matrix is ​​positive definite. Taking partial derivatives, we can get:

[0053] By setting the value of this expression to zero, we can obtain the closed-form solution as follows:

[0054] Here you can use Replace hyperparameters In simplified form, without In the more general case of symmetry, similar expressions can also be derived. The process of matrix inversion in the closed-form solution is an operation with very high time complexity. In order to find the optimal solution more quickly within a limited time, the problem can be converted to solving the following Lyapunov equation:

[0055] This equation is equivalent to replacing the graph Considered as a Asymptotically stable linear system defined as Represents the state of each vertex, by solving the The induced Lyapunov energy can obtain the manifold information contained in the adjacency graph. This equation can be solved numerically, and the time complexity of the calculation is reduced to , making the whole process more feasible and efficient, the iterative process of approximate optimal solution is as follows:

[0056] After the iteration converges, the similarity matrix with relaxed constraints can be obtained Next, each point is selectively retained along with its local cluster The similarity between them is then approximated to obtain the similarity matrix under local clustering constraints. , specifically, for the entity For example, only The corresponding row in The items are reserved for constructing , then, Apply to each row of The normalization operation is performed to obtain the final smoothed similarity matrix (i.e., the first similarity matrix).

[0057] Step 103: Based on the first similarity matrix, geometrically transmit the target knowledge graph in the manifold graph to obtain a relational representation of the target knowledge graph.

[0058] Specifically, in an embodiment of the present invention, after obtaining the first similarity matrix, the target knowledge graph can be geometrically transferred in the manifold space based on the first similarity matrix to more accurately represent the similarity relationship between entities of the target knowledge graph. By considering the geometric characteristics of the data in the manifold space, it can be ensured that the entity relationship is more robust in the knowledge graph.

[0059] The knowledge graph representation method based on similarity diffusion provided by the present invention obtains a target knowledge graph by constructing it based on a multimodal large model and a human-in-the-loop mechanism, constructs an adjacency graph matrix of the target knowledge graph according to the similarity relationship between each entity in the target knowledge graph, and then based on the adjacency graph matrix, performs similarity diffusion under local clustering constraints, which can mine a wider range of potential relationships between entities and obtain a first similarity matrix, and geometrically transmits the target knowledge graph in a manifold graph according to the first similarity matrix, thereby obtaining a more effective and robust relational representation of the target knowledge graph. Through this relational representation, the construction and update of the knowledge graph can be guided, and the mining, fusion, reasoning and retrieval capabilities of the knowledge graph can be improved.

[0060] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, based on a first similarity matrix, geometrically transmitting the target knowledge graph in a manifold graph to obtain a relational representation of the target knowledge graph specifically includes: Based on the neighbor nodes of the initial distance matrix, the first similarity matrix is ​​subjected to neighborhood-guided smooth enhancement to obtain a second similarity matrix; Based on the second similarity matrix, the target knowledge graph is geometrically transmitted in the manifold graph to obtain the relational representation of the target knowledge graph.

[0061] Specifically, local neighbors can provide a more accurate estimate of the relationship between classes than a single node. In order to further enhance the expression ability of the similarity matrix, the embodiment of the present invention can use the local neighbor set to The heuristic information of constrains the similarity consistency between entities and their neighbors, thereby further smoothing and enhancing the similarity matrix.

[0062] Therefore, after obtaining the first similarity matrix, we can first perform neighborhood-guided smoothing enhancement on the first similarity matrix through the neighbor nodes found from the initial distance matrix to obtain the second similarity matrix, and then based on the second similarity matrix, perform geometric transmission on the target knowledge graph in the manifold graph to obtain the relational representation of the target knowledge graph.

[0063] In the embodiment of the present invention, the process can be expressed as first using neighbor nodes according to the first similarity matrix Smooth to get the smoothed neighborhood consistent matrix , and then use the neighbor nodes to get its enhanced representation, that is, the second similarity matrix .

[0064] In the embodiment of the present invention, - Mutual nearest neighbor strategy to determine local neighbors, specifically, for entities , its neighborhood Defined as , here In the entity In the neighborhood space of For Entity To Entity The average similarity of the local neighbors, defining the local neighborhood The average similarity between pairs of entities is , the larger This means that the neighbors are more likely to belong to the same cluster, reflecting the reliability of local neighbors. With entity The similarity should be the same as the factor Consistent, in giving weight After adding the regularization term, the optimization objective can be expressed as:

[0065] where operator You can multiply the corresponding elements of two vectors to get a new vector. is a local clustering set used to constrain the similarity diffusion, with a weight of The regular term can be used to constrain the target To prevent it from deviating too much from the initial value, you can usually set a lower value to obtain better numerical stability. In addition, when constructing the vector When the embodiment of the present invention uses a truncation operation to ensure that all its values ​​are not greater than .

[0066] After solving the KKT conditions (Karush-Kuhn-Tucker Conditions) of the optimization problem, the neighborhood consistent matrix can be obtained The elements are as follows:

[0067] In addition to increasing consistency, local neighbors can further enhance the representation of the similarity matrix in other ways. For example, higher weights can be assigned to neighbors with similar The relevant edges emphasize the role of local neighbors in the diffusion process. Formally, when When the indicator matrix Item becomes , where the hyperparameters is the importance weight. In addition, since entities in the same neighborhood are likely to belong to the same category, the embodiment of the present invention can aggregate the neighborhood The entities within the , which can be realized by weighted average This can be achieved by using the manifold distribution of entities in:

[0068] In the formula, Used to enhance information from the neighborhood The information of can balance the relative weights of the neighborhood-guided similarity distribution and the original similarity distribution, where Used for normalization to ensure that each row of the target similarity matrix is Normalized. By adjusting the hyperparameters The adjustment of can control the influence of the neighborhood and nearest neighbor on the final similarity matrix, ensuring that a certain global smoothness is maintained on the basis of local enhancement.

[0069] By adding the node The similarity information of and nearest neighbor The similarity distribution of the nodes is combined to enhance the local similarity consistency. The second similarity matrix obtained in this way can better represent the node The relationship with surrounding nodes can improve the accuracy of the similarity matrix.

[0070] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, based on a second similarity matrix, geometrically transmitting the target knowledge graph in a manifold graph to obtain a relational representation of the target knowledge graph specifically includes: Perform information propagation on the second similarity matrix to obtain a third similarity matrix; Based on the third similarity matrix, the target knowledge graph is geometrically transferred in the manifold graph to obtain the relational representation of the target knowledge graph.

[0071] Specifically, after completing the local similarity enhancement to obtain the second similarity matrix, if it is necessary to better improve the global similarity, an additional step can be taken to perform a global information propagation on the entire graph, thereby further smoothing the similarity matrix of the entire graph.

[0072] The second similarity matrix It can be regarded as the best representation of the data manifold in local clustering. In order to achieve better global expression capability, the embodiment of the present invention can perform an additional information propagation on the whole graph to obtain the third similarity matrix , the third similarity matrix The second similarity matrix The transfer matrix is ​​calculated by multiplying the transpose of To get:

[0073] In this process, the present embodiment can also combine the sparse operation to process the transfer matrix , filter out the similarity values ​​that are too low, and retain the most significant similarity relationships to obtain the third similarity matrix , thereby avoiding noise interference and better reducing the impact of outliers.

[0074] According to a knowledge graph representation method based on similarity diffusion provided by the present invention, based on the first similarity matrix, the second similarity matrix or the third similarity matrix, the target knowledge graph is geometrically transmitted in the manifold graph to obtain the relational representation of the target knowledge graph, specifically including: Based on the initial distance matrix and the adjacency graph matrix, a geodesic distance matrix containing manifold space information is obtained; Based on the geodesic distance matrix, the optimal transfer matrix between the distribution representations of each entity in the target knowledge graph in the manifold space is calculated; wherein the distribution representations of each entity in the target knowledge graph in the manifold space are obtained based on the first similarity matrix, the second similarity matrix or the third similarity matrix; Based on the optimal transmission matrix and the distribution representation of each entity in the manifold space, the geometric transmission distance between each entity is determined; Based on the geometric transmission distance, the relational representation of the target knowledge graph is obtained.

[0075] Specifically, traditional similarity metrics (such as Euclidean distance) cannot fully express the complex relationships between entities, especially when the data is in a manifold structure. Manifold-aware geometric transmission methods are based on Wasserstein distance or optimal transmission theory to find more appropriate similarity metrics in manifold space. This method not only considers the direct distance between entities, but also makes full use of the structural information of the manifold space to calculate the optimal transmission path.

[0076] First, the geodesic distance matrix containing the manifold space information can be obtained based on the initial distance matrix and the adjacency graph matrix (the constructed nearest neighbor graph).

[0077] This embodiment models the target as a transport of each entity in the manifold space. Distribution representation in manifold space The problem, It can be the first similarity matrix, the second similarity matrix or the third similarity matrix, so that the calculated Wasserstein distance can fully consider the manifold information of the overall space. In order to simplify the optimal transmission process, the embodiment of the present invention defines the transmission path as the shortest path between two nodes in the graph. The nodes in and For example, the shortest path between them for:

[0078] Which belongs to the set Each path in are composed of a series of nodes, such as arrive As shown, this node sequence starts from Start to End. Node and The edge between them is defined as , from the manifold diagram arrive The geodesic of is equivalent to the shortest distance between these two points. This path contains the overall information in the data manifold space. arrive The geodesic distance The calculation method is:

[0079] The calculation method of this distance is equivalent to summing the weights of the edges on the geodesic path. The geodesic distances between all pairs of entity nodes form the geodesic matrix In the field of graph theory, calculating the shortest distance between all pairs of nodes on a graph is a mature problem that has been discussed for decades. For different data sizes, many solutions can be found to efficiently obtain the geodesic distance matrix. .

[0080] After obtaining the geodesic distance matrix, we can use it to calculate the optimal transfer matrix between the distribution representations of each entity in the target knowledge graph in the manifold space.

[0081] The geodesic distance matrix can be used as the cost of optimal transmission. The optimal transmission matrix between two manifold distributions can be obtained by solving the following optimization objective:

[0082] The matrix That is, it means that the distribution on the manifold space To distribution The optimal transmission matrix is For Entity Distribution representation in manifold space, For Entity Distribution representation in manifold space. In order to calculate the optimal distance more efficiently, the embodiment of the present invention adds an entropy regularization term to the optimization objective , so that the optimal transmission problem can be numerically solved by the Sinkhorn-Knopp algorithm.

[0083] After obtaining the optimal transmission matrix, the geometric transmission distance between the distribution representations of each entity in the manifold space can be determined based on the optimal transmission matrix and the Euclidean distance between each entity.

[0084] First, after obtaining arrive After the optimal transmission matrix is ​​obtained, their distance in the manifold graph can be calculated by the following formula:

[0085] In order to preserve the important neighbor relationship in the original distance space, the embodiment of the present invention also takes the initial Euclidean distance into consideration, so that the geometric transmission distance between the two distribution representations finally obtained can be expressed as:

[0086] In the formula is the initial Euclidean distance, and is the optimal transmission distance on the manifold, and these two distances are determined by a hyperparameter to balance.

[0087] Finally, the relational representation of the target knowledge graph can be obtained based on the geometric transmission distance. The geometric transmission distance obtained in this way can better measure the similarity relationship between entities, so it can be used to guide the fusion construction of knowledge graphs and improve the retrieval and reasoning efficiency of knowledge graphs.

[0088] The following further illustrates the knowledge graph representation method based on similarity diffusion provided by the present invention through embodiments in specific application scenarios.

[0089] Figure 2 A schematic diagram of a process for obtaining more robust relationships between knowledge graph entities based on similarity diffusion provided by the present invention. Figure 2 As shown, this embodiment first constructs a knowledge graph based on a multimodal large model using a human-in-the-loop mechanism, then constructs an adjacency graph matrix using the similarity relationship between entities, performs similarity diffusion under local clustering constraints, and then enhances the distribution structure using neighborhood information. Finally, a more effective relationship representation for knowledge graph modeling is obtained with the help of a manifold-aware geometric transmission module.

[0090] The following describes each step in detail.

[0091] 1. Construction of knowledge graph based on human in the loop.

[0092] The purpose of constructing a knowledge graph based on human-in-the-loop is to use a multimodal large model to automatically extract text descriptions of the image library and build an attribute knowledge graph based on these descriptions. Taking the knowledge graph of pedestrian description as an example, its core is to obtain the entity, relationship, and attribute information about the pedestrian description. First, the human-in-the-loop prompt learning technology is combined with a multimodal large model to extract the text description information of the image, and the method based on neural network-based named entity recognition and relationship classification model is used to realize the extraction of triple information of the text description in all database images. It should be understood that the algorithm based on human-in-the-loop is to manually filter the image, corpus and triple information on the basis of intelligent algorithms and realize unified extraction. In the field of knowledge extraction, the human-in-the-loop mechanism can integrate human prior knowledge to improve the reliability of the knowledge graph knowledge base while retaining the high generalization and high robustness of the multimodal large model.

[0093] In order to make the generated description targeted, a human-in-the-loop mechanism can be used to generate a biased description of pedestrian images using a specific manually designed prompt template ('the person is wearing [xxxx]'). Taking advantage of the text extraction advantages of the multimodal large model, the algorithm based on the multimodal large model can be used to extract relevant descriptive information of the image using a specific manually designed prompt template. In this process, since the image often contains a large number of noisy images or low-quality images, the image library can be manually screened to filter out some low-quality images or images without information so as to retain only high-quality images rich in information. Then a multimodal large model, such as a large language model (LLM), can be used to generate a corresponding text description for each image.

[0094] After generating text descriptions of all images, the knowledge graph semantic network of pedestrian descriptions can be obtained by constructing triplets of the "entity-relationship-entity" and "entity-attribute-attribute value" categories based on the entity, relationship, and attribute information network. Among them, the intelligent word segmentation algorithm can be used to extract the entities and relationships in each description through the "entity-relationship-entity" model. Specifically, given the text description of the image, the word segmentation processing algorithm performs a structured analysis on the content of the attribute description corpus, decomposes the sentence components based on the part of speech and semantics of each word or phrase, and identifies the nouns, colors, entities and other proper nouns related to the pedestrian attribute description in the sentence.

[0095] Since the generated "entities" or "relationships" contain a large number of invalid or low-quality meaningless phrases (for example, modal particles, auxiliary words, pronouns), some irrelevant or low-quality "entity" and "relationship" phrases can be further screened out artificially to further improve the accuracy of the knowledge graph corpus. It should be understood that the attribute text description of the pedestrian attribute knowledge graph is converted into a form, segmented, and invalid words are deleted to construct a cleaner pedestrian attribute knowledge graph semantic library. Through the above operations, the triple knowledge containing "entity-relationship-entity" and other data of the text description library can be finally generated, and a knowledge graph can be constructed. Therefore, this embodiment constructs a pedestrian attribute knowledge graph containing 112,323 entities and 2,877,187 relationships in total. An "entity-relationship-entity" pattern relationship in the knowledge graph is such as: 'woman'-'is wearing'-'black jacket with jeans'.

[0096] 2. Construction of adjacency relationship graph in knowledge graph.

[0097] Define entity collections as ,in is the number of entities in the knowledge graph, for the set Each entity in , a knowledge representation can be constructed for it, and the powerful expression ability of deep learning network can be used to encode it into a Characteristics of Dimensions , these features are used in the subsequent knowledge graph mining, fusion, reasoning and retrieval process. In this feature encoding space, any pair of entities and The similarity between them can be calculated by the Euclidean distance of their feature vectors as follows:

[0098] After calculating the distance between each pair of entities, an initial distance matrix is ​​obtained. , where the matrix elements Defined as . Each row of the matrix Representing Entities With the dataset The distance between them can be used to sort the data set. Based on such prior knowledge: similar entities tend to cluster in the distance matrix In the low-dimensional manifold structure implied by Mapping to a new space enhances the expressiveness of this structure. This mapping ensures that entities with similar knowledge semantics are not only close to each other in the traditional Euclidean space, but also show higher similarity in the manifold space. In order to approximate this manifold structure, a feature set can be used to construct a Nearest Neighbor Graph , in this graph, the vertex set correspond The feature encoding of each entity in Represents the connection relationship between entity vertices, each edge Connect Vertices and , whose weight is expressed as:

[0099] The matrix is the indicator matrix used to connect Nearest neighbor, formally, when hour, .

[0100] 3. Bidirectional similarity diffusion under local clustering constraints.

[0101] After constructing the affinity weight matrix of Nearest Neighbor Graph Finally, in order to use the manifold information contained in the neighbor graph to obtain a more robust similarity relationship, this embodiment adopts a diffusion-based method to transmit information in the graph, thereby obtaining a similarity matrix The traditional diffusion process propagates similarity across the entire graph, resulting in a similarity matrix It is easily affected by outliers and nearby manifolds, resulting in the similarity matrix Lack of expressiveness.

[0102] To solve this problem, this embodiment proposes bidirectional similarity diffusion under local clustering constraints, which aims to perform similarity propagation on a local graph rather than the entire graph. This method can reduce the propagation of misleading information from other noise samples, thereby improving the accuracy of similarity inference. For each entity, its local graph can be regarded as a local cluster containing the entity itself and its similar neighbors. , such as entity The local clustering of In order to obtain a smooth similarity matrix , this embodiment defines an objective function, which aims to minimize the difference between the similarity matrix and the local graph structure, while taking into account the local similarity between entities, and its definition is as follows:

[0103] in The weight matrix representing the adjacency graph can be constructed using a symmetric matrix for simplicity. Replace the original adjacency matrix. In addition, the matrix is a diagonal matrix, whose The diagonal elements are equal to No. The sum of the rows, in the In the guided regularization term, the matrix is positive definite, it can be used to prevent Too smooth, you can adopt different strategies according to your needs For entity triples in local clustering and , using adjacency weights To constrain and At the same time, we also consider and The similarity of this pair makes the smoothing strategy bidirectional, ensuring that the obtained similarity matrix It has symmetry.

[0104] In addition, in order to construct local clusters , this embodiment uses -The mutual nearest neighbor strategy is used to find the neighbor node with the highest confidence. This method takes into account the reverse information and can be regarded as -A stricter case of the nearest neighbor, the local clustering constructed can effectively exclude erroneous neighbors, thereby minimizing their negative impact on the final diffusion process. -The mutual nearest neighbor formalization is defined as follows:

[0105] Specifically, given an entity , its -The neighbor set is defined as , for any element in this set, only when the entity Also in this element -The element belongs to of - Mutual neighbor sets , this set can be regarded as a local cluster In order to avoid ambiguity and reduce the number of parameters, this embodiment will be used to construct local clustering The value is set to In addition, when When the number of elements in reaches a certain threshold, the entity of - Nearest neighbor reciprocity extended to larger sets To improve the expressiveness of the model.

[0106] However, it is very difficult to directly solve the optimization target for the constrained objective function mentioned above. In the bidirectional similarity diffusion process under local clustering constraints proposed in this embodiment, this embodiment simplifies the optimization problem into two easily solvable sub-problems by relaxing the constraints. First, consider the bidirectional diffusion process without clustering constraints, and its objective function is as follows:

[0107] Then, only the In order to more conveniently solve the optimization problem in the above formula, this embodiment assumes that the optimization target is , using the mathematical tools in graph theory, the target formula can be reconstructed as:

[0108] in is an identity matrix, The regularization matrix of ,matrix About The mean Kronecker product of is defined as , operator is a vectorization function that can combine the column vectors of a matrix into a new column vector, and is It can be proved that the objective function after relaxing the constraints is is a convex function, that is, its Hessian matrix is ​​positive definite. Taking partial derivatives, we can get:

[0109] By setting the value of this expression to zero, we can obtain the closed-form solution as follows:

[0110] Here you can use Replace hyperparameters In simplified form, without In the more general case of symmetry, similar expressions can also be derived. The process of matrix inversion in the closed-form solution is an operation with very high time complexity. In order to find the optimal solution more quickly within a limited time, the problem can be converted to solving the following Lyapunov equation:

[0111] This equation is equivalent to replacing the graph Considered as a Asymptotically stable linear system defined as Represents the state of each vertex, by solving the The induced Lyapunov energy can obtain the manifold information contained in the adjacency graph. This equation can be solved numerically, and the time complexity of the calculation is reduced to , making the whole process more feasible and efficient, the iterative process of approximate optimal solution is as follows:

[0112] After the iteration converges, the similarity matrix with relaxed constraints can be obtained Next, each point is selectively retained along with its local cluster The similarity between them is then approximated to obtain the similarity matrix under local clustering constraints. , specifically, for the entity For example, only The corresponding row in The items are reserved for constructing , then, Apply to each row of The normalization operation is performed to obtain the final smoothed similarity matrix.

[0113] 4. Neighborhood-guided distribution construction smoothing enhancement.

[0114] Local neighbors can provide a more accurate estimate of the relationship between classes than a single node. In order to further enhance the expression ability of the similarity matrix, this embodiment uses the local neighbor set to The heuristic information of the entity is used to constrain the similarity consistency between the entity and the neighborhood, thereby further enhancing the smoothness of the similarity matrix, and then adjusting it as needed to have better global characteristics. This step is called domain-guided distribution construction smooth enhancement. The steps of this process can be in the form of, first, according to the initial similarity matrix Smoothing to get a neighborhood consistent matrix Then use the neighbor nodes to get its enhanced representation Finally, according to the need Expanded to the entire image generation ' to achieve better global characteristics.

[0115] This embodiment can be again - Mutual nearest neighbor strategy to determine local neighbors, specifically, for entities , its neighborhood Defined as , here In the entity In the neighborhood space of For Entity To Entity The average similarity of the local neighbors, defining the local neighborhood The average similarity between pairs of entities is , the larger This means that the neighbors are more likely to belong to the same cluster, reflecting the reliability of local neighbors. With entity The similarity should be the same as the factor Consistent, in giving weight After adding the regularization term, the optimization objective can be expressed as:

[0116] where operator You can multiply the corresponding elements of two vectors to get a new vector. is the local cluster set introduced in the previous section to constrain the similarity diffusion, and the weight is The regular term can be used to constrain the target To prevent it from deviating too much from the initial value, you can usually set a lower value to obtain better numerical stability. In addition, when constructing the vector In this embodiment, a truncation operation is performed to ensure that all values ​​are not greater than After solving the KKT condition of the optimization problem, we can get the neighborhood consistent matrix The elements are as follows:

[0117] In addition to increasing consistency, local neighbors can also further For example, when constructing the nearest neighbor graph, higher weights can be assigned to The relevant edges emphasize the role of local neighbors in the diffusion process. Formally, when When the indicator matrix Item becomes , where the hyperparameters is the importance weight. In addition, since entities in the same neighborhood are likely to belong to the same category, this embodiment can aggregate the neighborhood The entities within the , which can be realized by weighted average This can be achieved by using the manifold distribution of entities in:

[0118] The similarity matrix obtained in this step It can be regarded as the best representation of the data manifold in local clustering. If better global expression ability is required, this embodiment can perform an additional information propagation on the whole graph to obtain the final similarity matrix , the propagation matrix can be expressed as During the calculation process, this embodiment can also combine the sparse operation to process the transfer matrix, so as to better reduce the influence of outliers.

[0119] 5. Manifold-aware geometry transfer.

[0120] Next, the goal of this embodiment is to establish a distance metric in an efficient way, which can be used to better measure the relationship between entities in the manifold space. Some traditional methods lack consideration of the geometric characteristics of the entire manifold space when calculating the distance between sample pairs. To solve this problem, this embodiment models the problem as a transmission of each entity on the manifold space. Distribution representation in manifold space The Wasserstein distance calculated in this way can fully consider the manifold information of the entire space. In order to simplify the optimal transmission process, this embodiment defines the transmission path as the shortest path between two nodes in the graph. Formally speaking, for the manifold graph The nodes in and For example, the shortest path between them is defined as :

[0121] Which belongs to the set Each path in are composed of a series of nodes, such as arrive As shown, this node sequence starts from Start to End. Node and The edge between them is defined as , from the manifold diagram arrive The geodesic of is equivalent to the shortest distance between these two points. This path contains the overall information in the data manifold space. Formally speaking, arrive The geodesic distance Defined as:

[0122] The calculation method of this distance is equivalent to summing the weights of the edges on the geodesic path. The geodesic distances between all pairs of entity nodes form a set In the field of graph theory, calculating the shortest distance between all pairs of nodes on a graph is a mature problem that has been discussed for decades. For different data sizes, many solutions can be found to efficiently obtain the geodesic distance matrix. Next, taking the geodesic distance matrix as the cost of optimal transmission, the optimal transmission matrix between two manifold distributions can be obtained by solving the following optimization objective:

[0123] The matrix That is, it means that the distribution on the manifold space To distribution In order to calculate the optimal distance more efficiently, this embodiment adds an entropy regularization term to the optimization objective. , so that the optimal transmission problem can be solved numerically by iterative Sinkhorn-Knopp algorithm. arrive After the optimal transmission matrix is ​​obtained, their distance in the manifold graph can be calculated by the following formula

[0124] Furthermore, in order to preserve the important neighbor relationship in the original distance space, this embodiment also takes the initial Euclidean distance into consideration, so that the geometric transmission distance between the two distributions can be expressed as:

[0125] In the formula is the initial Euclidean distance, and is the optimal transmission distance on the manifold, and these two distances are determined by a hyperparameter To balance, the final geometric transmission distance can better measure the similarity relationship between entities, which can be used to guide the fusion construction of knowledge graphs or improve the retrieval and reasoning efficiency of knowledge graphs.

[0126] The method proposed in this embodiment has the advantages of low time complexity, strong generalization, and the ability to be implemented in parallel on a graphics processing unit (GPU). It can effectively improve the construction, update, mining, fusion, reasoning and retrieval capabilities of knowledge graphs.

[0127] The knowledge graph representation device based on similarity diffusion provided by the present invention is described below. The knowledge graph representation device based on similarity diffusion described below and the knowledge graph representation method based on similarity diffusion described above can be referenced to each other.

[0128] Figure 3 A schematic diagram of the structure of the knowledge graph representation device based on similarity diffusion provided by the present invention, such as Figure 3 As shown, the device includes the following modules: A construction module 300 is used to construct a target knowledge graph based on a multimodal large model and a human-in-the-loop mechanism; An adjacency module 310 is used to construct an adjacency graph matrix of the target knowledge graph based on the similarity relationship between each entity in the target knowledge graph; A diffusion module 320 is used to perform similarity diffusion under local clustering constraints based on the adjacency graph matrix to obtain a first similarity matrix; The representation module 330 is used to geometrically transmit the target knowledge graph in the manifold graph based on the first similarity matrix to obtain a relational representation of the target knowledge graph.

[0129] According to a knowledge graph representation device based on similarity diffusion provided by the present invention, an adjacency graph matrix of a target knowledge graph is constructed based on the similarity relationship between entities in the target knowledge graph, specifically comprising: Based on the Euclidean distance between each entity in the target knowledge graph, the initial distance matrix of each entity is obtained; Based on the initial distance matrix and the nearest neighbor edges of each entity, the adjacency graph matrix of the target knowledge graph is constructed.

[0130] According to a knowledge graph representation device based on similarity diffusion provided by the present invention, based on a first similarity matrix, geometrically transmitting a target knowledge graph in a manifold graph to obtain a relational representation of the target knowledge graph specifically includes: Based on the neighbor nodes of the initial distance matrix, the first similarity matrix is ​​subjected to neighborhood-guided smooth enhancement to obtain a second similarity matrix; Based on the second similarity matrix, the target knowledge graph is geometrically transmitted in the manifold graph to obtain the relational representation of the target knowledge graph.

[0131] According to a knowledge graph representation device based on similarity diffusion provided by the present invention, based on an adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix, specifically including: Based on the adjacency graph matrix and k-nearest neighbor strategy, a local clustering graph of each entity is constructed; The local cluster graph of each entity is diffused through a bidirectional propagation mechanism to obtain a first similarity matrix.

[0132] According to a knowledge graph representation device based on similarity diffusion provided by the present invention, based on a second similarity matrix, geometrically transmitting the target knowledge graph in a manifold graph to obtain a relational representation of the target knowledge graph specifically includes: Perform information propagation on the second similarity matrix to obtain a third similarity matrix; Based on the third similarity matrix, the target knowledge graph is geometrically transferred in the manifold graph to obtain the relational representation of the target knowledge graph.

[0133] According to a knowledge graph representation device based on similarity diffusion provided by the present invention, based on the first similarity matrix, the second similarity matrix or the third similarity matrix, geometrically transmitting the target knowledge graph in the manifold graph to obtain the relational representation of the target knowledge graph specifically includes: Based on the initial distance matrix and the adjacency graph matrix, a geodesic distance matrix containing manifold space information is obtained; Based on the geodesic distance matrix, the optimal transfer matrix between the distribution representations of each entity in the target knowledge graph in the manifold space is calculated; wherein the distribution representations of each entity in the target knowledge graph in the manifold space are obtained based on the first similarity matrix, the second similarity matrix or the third similarity matrix; Based on the optimal transmission matrix and the distribution representation of each entity in the manifold space, the geometric transmission distance between each entity is determined; Based on the geometric transmission distance, the relational representation of the target knowledge graph is obtained.

[0134] Figure 4 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the knowledge graph representation method based on similarity diffusion, and the method includes: Build the target knowledge graph based on the multimodal big model and human-in-the-loop mechanism; Based on the similarity relationship between each entity in the target knowledge graph, an adjacency graph matrix of the target knowledge graph is constructed; Based on the adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix; Based on the first similarity matrix, the target knowledge graph is geometrically transmitted in the manifold graph to obtain a relational representation of the target knowledge graph.

[0135] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0136] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the knowledge graph representation method based on similarity diffusion provided by the above methods, the method including: Build the target knowledge graph based on the multimodal big model and human-in-the-loop mechanism; Based on the similarity relationship between each entity in the target knowledge graph, an adjacency graph matrix of the target knowledge graph is constructed; Based on the adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix; Based on the first similarity matrix, the target knowledge graph is geometrically transmitted in the manifold graph to obtain a relational representation of the target knowledge graph.

[0137] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the knowledge graph representation method based on similarity diffusion provided by the above methods, the method comprising: Build the target knowledge graph based on the multimodal big model and human-in-the-loop mechanism; Based on the similarity relationship between each entity in the target knowledge graph, an adjacency graph matrix of the target knowledge graph is constructed; Based on the adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix; Based on the first similarity matrix, the target knowledge graph is geometrically transmitted in the manifold graph to obtain a relational representation of the target knowledge graph.

[0138] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge graph representation method based on similarity diffusion, characterized in that: include: Build the target knowledge graph based on the multimodal big model and human-in-the-loop mechanism; Based on the similarity relationship between each entity in the target knowledge graph, construct an adjacency graph matrix of the target knowledge graph; Based on the adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix; Based on the first similarity matrix, the target knowledge graph is geometrically transmitted in a manifold graph to obtain a relational representation of the target knowledge graph.

2. The knowledge graph representation method based on similarity diffusion according to claim 1 is characterized in that: Based on the similarity relationship between the entities in the target knowledge graph, an adjacency graph matrix of the target knowledge graph is constructed, specifically including: Based on the Euclidean distances between the entities in the target knowledge graph, an initial distance matrix of the entities is obtained; Based on the initial distance matrix and the nearest neighbor edges of each entity, an adjacency graph matrix of the target knowledge graph is constructed.

3. The knowledge graph representation method based on similarity diffusion according to claim 2 is characterized in that: Based on the first similarity matrix, geometrically transmitting the target knowledge graph in a manifold graph to obtain a relational representation of the target knowledge graph specifically includes: Based on the neighbor nodes of the initial distance matrix, performing neighborhood-guided smoothing enhancement on the first similarity matrix to obtain a second similarity matrix; Based on the second similarity matrix, the target knowledge graph is geometrically transmitted in a manifold graph to obtain a relational representation of the target knowledge graph.

4. The knowledge graph representation method based on similarity diffusion according to claim 1 is characterized in that: Based on the adjacency graph matrix, similarity diffusion is performed under local clustering constraints to obtain a first similarity matrix, which specifically includes: Based on the adjacency graph matrix and the k-nearest neighbor strategy, construct a local clustering graph of each entity; The local cluster graphs of the entities are diffused through a bidirectional propagation mechanism to obtain a first similarity matrix.

5. The knowledge graph representation method based on similarity diffusion according to claim 3 is characterized in that: Based on the second similarity matrix, geometrically transmitting the target knowledge graph in a manifold graph to obtain a relational representation of the target knowledge graph specifically includes: Performing information propagation on the second similarity matrix to obtain a third similarity matrix; Based on the third similarity matrix, the target knowledge graph is geometrically transmitted in a manifold graph to obtain a relational representation of the target knowledge graph.

6. The knowledge graph representation method based on similarity diffusion according to claim 2, 3 or 5, characterized in that: Based on the first similarity matrix, the second similarity matrix or the third similarity matrix, the target knowledge graph is geometrically transmitted in the manifold graph to obtain a relational representation of the target knowledge graph, specifically including: Based on the initial distance matrix and the adjacency graph matrix, a geodesic distance matrix containing manifold space information is obtained; Based on the geodesic distance matrix, an optimal transfer matrix between distribution representations of each entity in the target knowledge graph in the manifold space is calculated; wherein the distribution representations of each entity in the target knowledge graph in the manifold space are obtained based on the first similarity matrix, the second similarity matrix or the third similarity matrix; Determining the geometric transmission distance between each entity based on the optimal transmission matrix and the distribution representation of each entity in the manifold space; Based on the geometric transmission distance, a relational representation of the target knowledge graph is obtained.

7. A knowledge graph representation device based on similarity diffusion, characterized in that: include: A construction module for building a target knowledge graph based on a multimodal large model and a human-in-the-loop mechanism; An adjacency module, used to construct an adjacency graph matrix of the target knowledge graph based on the similarity relationship between each entity in the target knowledge graph; A diffusion module, used for performing similarity diffusion under local clustering constraints based on the adjacency graph matrix to obtain a first similarity matrix; A representation module is used to geometrically transmit the target knowledge graph in a manifold graph based on the first similarity matrix to obtain a relational representation of the target knowledge graph.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the knowledge graph representation method based on similarity diffusion as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the knowledge graph representation method based on similarity diffusion as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the knowledge graph representation method based on similarity diffusion as described in any one of claims 1 to 6 is implemented.

Images