Heterogeneous knowledge graph fusion method based on cross-modal manifold alignment

Abstract

The application discloses a heterogeneous knowledge graph fusion method based on cross-modal manifold alignment, and relates to the technical field of knowledge graph, comprising: for each pair of first modal entity and second modal entity, calculating coordinate deviation according to the difference between the twisted first modal spectral coordinates and the second modal spectral coordinates, calculating geometric structure deviation according to the difference between the first modal local tangent space orthogonal basis and the second modal local tangent space orthogonal basis, forming a matching cost from the coordinate deviation and the geometric structure deviation, and obtaining a cross-modal alignment entity pair set through optimal matching solution. The application combines the matching entities into cross-modal unified entity nodes and generates fusion entity features, hangs the unified relationship representation to the corresponding nodes to form an initial fusion graph, and then adopts a link prediction model with manifold distance regularization to complete the missing links, so that a cross-modal fusion knowledge graph with complete structure and rich information is generated.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph technology, and in particular to a heterogeneous knowledge graph fusion method based on cross-modal manifold alignment. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, knowledge graphs, as a structured semantic knowledge representation method, have been widely applied in various fields such as information retrieval, intelligent question answering, and recommendation systems. However, knowledge in the real world often exists in multiple modalities, such as text, images, audio, and video. Data from different modalities are semantically related but also heterogeneous. Traditional knowledge graph construction methods usually focus only on information from a single modality (such as plain text or purely structured data), ignoring the rich semantic relationships between cross-modal data. On the other hand, knowledge graphs from different sources and domains often adopt their own independent entity and relation systems, leading to a serious phenomenon of knowledge silos. Therefore, how to effectively integrate knowledge graphs from different modalities, break down modal barriers, and achieve unified representation and sharing of heterogeneous knowledge resources has become an important problem that urgently needs to be solved in the field of knowledge engineering.

[0003] Currently, research on knowledge graph fusion mainly focuses on entity alignment and knowledge merging within the same modality. Typical methods include entity name string similarity-based alignment, graph structure-based embedding alignment, and external dictionary-based matching. For cross-modal knowledge graph fusion, existing technologies often map data from different modalities to the same embedding space using pre-trained models (such as CLIP and BERT), and then perform entity similarity calculation and alignment within that space. Furthermore, some studies attempt to use graph neural networks to encode the graph structures of different modalities separately, and then achieve intermodal information interaction through adversarial training or cross-modal attention mechanisms. However, these methods typically assume a global linear transformation between the embedding spaces of different modalities, ignoring the local geometric characteristics of the manifold structure within and between modalities. Traditional methods often struggle to maintain the local topological structure of entity neighborhoods when processing cross-modal heterogeneous data, leading to alignment results that are highly sensitive to global embedding quality and easily affected by semantic gaps and nonlinear distortions between modalities.

[0004] Existing cross-modal knowledge graph fusion techniques still suffer from the following key problems: First, the entity embedding spaces of different modal knowledge graphs often exhibit nonlinear geometric distortions. Simply mapping the embeddings of one modality to another cannot effectively maintain the local diffusion distance between entities, leading to a decrease in alignment accuracy. Second, most existing methods only utilize entity-level embedding similarity for matching, lacking the use of the geometric structure of the entity's neighborhood (such as the local tangent space), making it difficult to distinguish entities with similar structures but different semantics. Third, the fusion of cross-modal relationships lacks a unified representation mechanism. Relationship triples from the first modality are difficult to naturally map into the relation space of the second modality, easily resulting in missing relationships or semantic distortion. Finally, the knowledge graph after preliminary fusion usually has a large number of missing links, and traditional link prediction models fail to fully consider the consistency constraints of the cross-modal manifold structure when completing the link, causing the completion result to destroy the original cross-modal geometric structure.

[0005] Therefore, it is essential to invent a heterogeneous knowledge graph fusion method based on cross-modal manifold alignment to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a heterogeneous knowledge graph fusion method based on cross-modal manifold alignment to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a heterogeneous knowledge graph fusion method based on cross-modal manifold alignment, comprising the following steps: S1. Obtain a first modality knowledge graph and a second modality knowledge graph. The first modality knowledge graph contains first modality entities and first modality relation triples. The second modality knowledge graph contains second modality entities and second modality relation triples. Each entity has a preset modality embedding vector. S2. Based on the preset modal embedding vector, construct the first modal entity adjacency graph and the second modal entity adjacency graph respectively, calculate the local diffusion distance between entities in the entity adjacency graph, and use the local diffusion distance to generate the first modal spectral coordinates and the first modal local tangent space orthogonal basis, the second modal spectral coordinates and the second modal local tangent space orthogonal basis; S3. Using a pre-set cross-modal anchor point entity pair as a constraint, construct a weighted non-rigid transformation in the spectral coordinate space. The weighted non-rigid transformation aims to minimize the diffusion distance error of the anchor point pair and improve the smoothness of the transformation. Solve to obtain the twisted mapping function from the first modal spectral coordinate space to the second modal spectral coordinate space. Substitute the first modal spectral coordinates into the twisted mapping function to obtain the twisted first modal spectral coordinates. S4. For each pair of first modal entities and second modal entities, calculate the coordinate deviation based on the difference between the distorted first modal spectrum coordinates and the second modal spectrum coordinates, and calculate the geometric structure deviation based on the difference between the first modal local tangent space orthogonal basis and the second modal local tangent space orthogonal basis. The coordinate deviation and the geometric structure deviation constitute the matching cost. The set of cross-modal aligned entity pairs is obtained by solving the optimal matching solution. S5. Extract the first modal relation triplet involved in the cross-modal aligned entity pair set, obtain the first modal spectrum coordinates of the head entity and tail entity of the relation triplet, transform it through the twist mapping function, embed and merge it with the first modal relation, map it to the second modal relation space to obtain a unified relation representation, and integrate the second modal relation representation to form a unified relation representation set; S6. Merge the matching entities in the cross-modal aligned entity pair set into cross-modal unified entity nodes, generate fused entity features for each cross-modal unified entity node, attach the relations in the unified relation representation set to the corresponding cross-modal unified entity nodes to form an initial fused graph, and use a link prediction model with manifold distance regularization to complete the missing links in the initial fused graph, and output a cross-modal fused knowledge graph.

[0008] The technical effects and advantages of this invention are as follows: 1. This invention constructs an entity adjacency graph based on modal embedding vectors and calculates local diffusion distances to generate spectral coordinates and local tangent space orthogonal bases, effectively preserving the local manifold geometry of entities within their respective modalities and providing a consistent geometric basis for cross-modal alignment; 2. This invention constructs a twisted mapping function from the first modal spectral coordinate space to the second modal spectral coordinate space by using cross-modal anchor point entity pairs as constraints and employing weighted non-rigid transformation, thereby correcting nonlinear geometric distortions between modes and enhancing the flexibility and accuracy of cross-modal manifold alignment. 3. This invention combines the coordinate deviation between the first and second modal spectral coordinates and the geometric structural deviation between the orthogonal bases of the local tangent space to form a matching cost, and uses the optimal matching solution to obtain the cross-modal aligned entity pair set, thereby improving the accuracy and robustness of entity alignment in heterogeneous knowledge graphs. 4. This invention achieves semantic unification of cross-modal relations by merging the head and tail entity spectrum coordinates of the first modal relation triplet involved in the cross-modal alignment entity pair with the relation embedding after distortion mapping transformation, and mapping them to the second modal relation space to obtain a unified relation representation, thereby avoiding the loss of relation information and semantic distortion. 5. This invention merges matching entities into cross-modal unified entity nodes and generates fused entity features. It then attaches unified relation representations to the corresponding nodes to form an initial fused graph. Finally, it uses a link prediction model with manifold distance regularization to complete missing links, thus generating a cross-modal fused knowledge graph with complete structure and rich information. 6. This invention introduces a manifold distance regularization term into the link prediction model and calculates the squared difference of the diffusion distance between the first modal entity and the second modal entity corresponding to the cross-modal unified entity node in the distorted spectral coordinates as a penalty, so that the completed graph maintains the consistency of the original cross-modal manifold structure. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating the overall method of the present invention; Figure 2 A flowchart is generated to represent the unified relationship in this invention; Figure 3 This is a flowchart of the initial fusion map construction and link prediction completion process of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] This invention provides, for example Figure 1 The heterogeneous knowledge graph fusion method based on cross-modal manifold alignment shown includes the following steps: S1. Obtain a first modality knowledge graph and a second modality knowledge graph. The first modality knowledge graph contains first modality entities and first modality relation triples. The second modality knowledge graph contains second modality entities and second modality relation triples. Each entity has a preset modality embedding vector. In one specific implementation, the execution process of step S1 is as follows: First, the specific types of the first and second modalities are clarified. In this embodiment, the first modality is set as the text modality, and the second modality is set as the image modality. Accordingly, the first modality knowledge graph originates from a large-scale Wikipedia text knowledge base. Each entity in this knowledge base (e.g., "cat", "Eiffel Tower") is defined by the text description information of its corresponding Wikipedia page and organized in the form of a triple of "head entity, relation, tail entity", such as (cat, belongs to, mammal). The second modality knowledge graph originates from a publicly available image annotation dataset, such as ImageNet or VisualGenome. Each entity in this knowledge graph (e.g., "picture of a cat", "picture of the Eiffel Tower") is defined by one or more representative image instances, and its relation triple describes the visual semantic association between image entities, such as (picture of a cat, has the feature, furry).

[0012] Secondly, modal embedding vectors are pre-defined for each first modal entity (text entity) and second modal entity (image entity). Specifically, a pre-trained cross-modal model CLIP (Contrastive Language-Image Pre-training) is used to extract the embedding vectors. For each text entity in the first modal knowledge graph, its corresponding Wikipedia text description is used as input and passed to the CLIP model's text encoder. The text encoder outputs a fixed-dimensional feature vector as the modal embedding vector of that text entity. For each image entity in the second modal knowledge graph, one or more representative images (if multiple, the average of all image embedding vectors is taken) are used as input and passed to the CLIP model's image encoder. The image encoder outputs a feature vector of the same fixed dimension as the modal embedding vector of that image entity. In this embodiment, the fixed dimension is set to 512 dimensions. Thus, each entity in the first modality knowledge graph and the second modality knowledge graph obtains an initial embedding vector representation that corresponds to its modality and can be computed across modalities. These embedding vectors are then associated with and stored with the corresponding entity identifiers for use in subsequent steps.

[0013] S2. Based on the preset modal embedding vector, construct the first modal entity adjacency graph and the second modal entity adjacency graph respectively, calculate the local diffusion distance between entities in the entity adjacency graph, and use the local diffusion distance to generate the first modal spectral coordinates and the first modal local tangent space orthogonal basis, the second modal spectral coordinates and the second modal local tangent space orthogonal basis; Furthermore, in the above technical solution, the preset modal embedding vector is extracted by the cross-modal pre-trained model CLIP to obtain text modal embedding vector and image modal embedding vector respectively.

[0014] Furthermore, in the above technical solution, the construction of the first modal entity adjacency graph and the second modal entity adjacency graph includes: For each modality, the cosine similarity between pairs of entities is calculated using the modality embedding vector of the entity, and a hybrid similarity matrix is ​​constructed by combining the structural co-occurrence relationship in the knowledge graph of that modality. The hybrid similarity matrix is ​​then converted into an entity adjacency graph by a preset threshold.

[0015] Furthermore, in the above technical solution, the calculation of the local diffusion distance specifically includes: Perform a preset t-step random walk on the entity adjacency graph to obtain the diffusion distance d(i,j) between entity i and entity j. Construct a diffusion kernel matrix K, and then perform symmetric normalization on K to obtain matrix P, where K(i,j) = exp(-d(i,j)). 2 / σ), where σ is the median diffusion distance between each entity and its h-th nearest neighbor, and h is the preset number of nearest neighbors; Perform eigenvalue decomposition on matrix P, and take the eigenvectors corresponding to the first d largest eigenvalues ​​as the spectral coordinates, where d is the preset spectral coordinate dimension; For each entity's spectral coordinates, select its k nearest neighbors. Perform principal component analysis on the k nearest neighbors and use the resulting k principal component directions as the local tangent space orthogonal basis of the entity, where k is the preset dimension of the local tangent space.

[0016] For the first modality (text modality) and the second modality (image modality), the following sub-steps are performed independently: Calculate cosine similarity: Suppose a modality has N entities, and its modality embedding vector is {v1, v2, ..., v...} N Each vector has a dimension of 512. For any two entities i and j, calculate the cosine similarity S between their embedding vectors. cos (i, j) = (v i ·v j ) / (||v i ||·||v j The value of || is in the range of [-1, 1].

[0017] Extracting structural co-occurrence relations: Extracting the structural co-occurrence relation weights S between entities from the triples of this modality knowledge graph. struc (i, j). If there is at least one relation triple (i.e., a direct edge with i and j as the head and tail entities) between entity i and entity j, then S struc(i, j) = 1; if there is no direct edge, then S struc (i, j) = 0. To further enhance the structural information, a two-hop path can be considered: if i and j are connected by an intermediate entity (i.e., i—k—j), then S can be... struc (i, j) is set to 0.5. In this embodiment, a direct edge definition is preferred, that is, it is set to 1 only when a direct relationship exists, and 0 otherwise.

[0018] Constructing a hybrid similarity matrix: weighted fusion of cosine similarity and structural co-occurrence relationship yields the hybrid similarity M(i,j) = α*S cos (i, j) + (1-α)*S struc (i, j), where α is the balance coefficient, ranging from [0, 1]. The value of α is adaptively determined based on the topological density of the current modal knowledge graph: first, the average degree (deg) of entities in the modal knowledge graph is calculated. avg and average clustering coefficient cc avg Define the density index ρ = (deg) avg / (N-1))·(1+cc avg ), where N is the total number of entities; then, α is obtained by mapping through the sigmoid function: α = 1 / (1 + exp(-k·(ρ0-ρ))), where k = 10, and ρ0 = 0.3 is an empirical reference point, such that when ρ is low (sparse graph), α approaches 0.8~0.9 (emphasizing semantic similarity), and when ρ is high (dense graph), α approaches 0.4~0.5 (enhancing structural constraints). If the actually calculated α exceeds the range [0, 1], it is truncated to the boundary value.

[0019] Threshold transformation generates an adjacency graph: Set a threshold θ (θ = 0.65 in this embodiment). Traverse all entity pairs (i, j). If M(i, j) ≥ θ, then establish an undirected edge between entity i and entity j, and the weight of the edge is the value of M(i, j); if M(i, j) < θ, then there is no edge. Finally, a weighted undirected entity adjacency graph G = (V, E, W) is formed, where V is the entity node, E is the edge set, and W is the edge weight.

[0020] On the obtained entity adjacency graph G, calculate the local diffusion distance d(i,j) between any two entities i and j. The specific steps are as follows: Construct the transition probability matrix: Based on the weight W of graph G, calculate the one-step transition probability P1(i→j) = W(i,j) / If i and j have no boundaries, then W(i, j) = 0.

[0021] Perform a t-step random walk: Set the number of steps t (t=4 in this example). The t-step transition probability matrix P t =(P1)^t, where Pt (i, j) represents the probability of starting from i and reaching j after exactly t random walks.

[0022] Calculate the diffusion distance: Define the diffusion distance d(i,j) = , where φ(z) is the stationary distribution of node z (i.e., the degree percentage of node z in the graph). This distance measures the difference in probability distribution between two nodes under t-step random walks. To simplify the calculation, this embodiment uses the equivalent formula: d(i,j)=||ψ i -ψ j ||2, where ψ i =[λ1^t*φ1(i), λ2^t*φ2(i),...,λ N ^t*φ N [i], where λk and φk are the eigenvalues ​​and eigenvectors of the graph Laplacian operator, respectively. In practice, these can be pre-computed through eigenvalue decomposition.

[0023] Construct the diffusion kernel matrix K: K(i,j) = exp(-d(i,j)^2 / σ), where σ is the scale parameter. To adaptively determine σ, for each entity i, find the diffusion distance d of its h-th nearest neighbor (h=10 in this embodiment). h (i), then take the d of all entities. h The median of (i) is taken as σ. That is, σ = median{d h (i) | i=1...N}.

[0024] Symmetric normalization: Compute the diagonal matrix D, where D(i, i) = Then calculate the normalized matrix P = D^(-1 / 2) * K * D^(-1 / 2). This matrix P is a symmetric positive semi-definite matrix.

[0025] Eigenvalue decomposition: Perform eigenvalue decomposition on matrix P to obtain eigenvalues ​​λ1≥λ2≥...≥λ N and their corresponding eigenvectors u1, u2, ..., u N The eigenvectors corresponding to the first d largest eigenvalues ​​are taken as the spectral coordinates of each entity. In this embodiment, the dimension d of the spectral coordinates is set to d=20. Specifically, the spectral coordinates of entity i are a d-dimensional vector: x i =[u1(i),u2(i),...,u d (i)].

[0026] Calculate the orthogonal basis of the local tangent space: For each entity i, find its k nearest neighbors (k=15) in the spectral coordinate space, using Euclidean distance. Let the spectral coordinates of these k nearest neighbors (including entity i itself) form a k×d matrix X. Center X by subtracting the mean, and then perform principal component analysis (PCA). Take the first k principal component directions, i.e., the first k eigenvectors, obtained from PCA. Since these directions are orthogonal to each other, they constitute a set of orthogonal bases for the local tangent space of entity i. Store the orthogonal basis T of the local tangent space for each entity. i Its dimension is d×k, and in actual storage, the first k principal component directions can be taken, k=15.

[0027] S3. Using a pre-set cross-modal anchor point entity pair as a constraint, construct a weighted non-rigid transformation in the spectral coordinate space. The weighted non-rigid transformation aims to minimize the diffusion distance error of the anchor point pair and improve the smoothness of the transformation. Solve to obtain the twisted mapping function from the first modal spectral coordinate space to the second modal spectral coordinate space. Substitute the first modal spectral coordinates into the twisted mapping function to obtain the twisted first modal spectral coordinates. Furthermore, in the above technical solution, the distortion mapping function is obtained through the following steps: Thin plate spline functions are selected as the transformation model; The displacement field is constructed using the difference between the first and second modal spectral coordinates of the cross-modal anchor point entity pairs, and a weighted thin-plate spline energy functional is constructed using the negative exponential function of the local diffusion distance between the anchor point pairs as the weight; the preset cross-modal anchor point entity pairs are obtained by cross-modal semantic similarity matching of entity names and supplemented by manual verification. Finding the extreme value of the energy functional yields a system of linear equations. Solving this system of linear equations yields the thin-plate spline coefficients, forming a twisted mapping function from the first modal spectral coordinate space to the second modal spectral coordinate space.

[0028] In this embodiment, the preset cross-modal anchor point entity pairs are obtained in the following way: Candidate Anchor Point Generation: For each entity in the first modality (text), obtain its name, such as extracting it from a Wikipedia page title. For each entity in the second modality (image), obtain its image filename or associated text tag, such as the category name from ImageNet. Leveraging the cross-modal capabilities of the CLIP model, calculate the cosine similarity between the text embedding of each text entity name and the image embedding of each image entity name or tag. Set a high similarity threshold, such as 0.85, and filter text-image entity pairs exceeding this threshold as candidate anchor point pairs.

[0029] Manual verification: At least two human annotators independently verify the candidate anchor point pairs to determine whether the text entity and the image entity do indeed point to the same objective entity. For example, the text entity "cat" and the image entity "cat picture" should match. Only entity pairs that are correctly determined by both annotators are adopted as the final cross-modal anchor point entity pairs. To improve the transformation quality, this embodiment requires that the number of anchor point pairs be no less than 20. If there are fewer, the similarity threshold is appropriately lowered to 0.80, and the pairs are regenerated and verified. The final set of anchor point pairs is denoted as A = {(p q q q )|q=1...Q}, where p q Let q be the spectral coordinates (d-dimensional vector) of the first modality entity. q Let Q be the spectral coordinates (d-dimensional vector) of the corresponding second modal entity, and let Q be the number of anchor points. In this embodiment, Q=30.

[0030] A thin plate spline (TPS) function is selected as the twisting mapping function f: R^d → R^d from the first modal spectral coordinate space to the second modal spectral coordinate space. For d-dimensional space (d=20 in this embodiment), the thin plate spline transformation consists of a linear affine transformation plus a nonlinear term of the radial basis function: f(x) = Ax + b + Where A is a d×d affine matrix, b is a d-dimensional translation vector, and w q It is a d-dimensional coefficient vector, and φ(r) = r^2 * ln(r) (when r > 0, and φ(0) = 0 is defined) is the radial basis function of the thin plate spline.

[0031] For each anchor pair (p q q q Assign a weight λ q The weight is based on the consistency of the local neighborhood structure of the two anchor entities in their respective modal knowledge graphs. Specifically, the nearest neighbor parameter m is first set to max(5, floor(0.02·N1)), where N1 is the total number of entities in the first modality, and the minimum value of m is 5, and the maximum value is 50, to ensure the statistical validity of the local neighborhood. The anchor entity p is then calculated. q The average diffusion distance between the corresponding first modal entity and its m-th nearest neighbor first modal entity is denoted as D1. q Similarly, calculate the second modal anchor point entity q. q The average diffusion distance D2 between the second modal entity and its m-th nearest neighbor q Define the local structural difference δ q =|D1 q -D2 q | / max(D1) q D2 q, ε), where ε = 1e-6 to prevent division by zero.

[0032] Bandwidth parameter σ λ Using all anchor point pairs δ q Median adaptive setting: σ λ =median({δ q |q=1...Q}) / sqrt(2). If the median is zero, then take σ λ =0.1. The weight calculation formula is λ. q =exp(-δ q ^2 / (2·σ λ ^2). For δ q >3·σ λ Anchor point pairs with a weight less than 0.011 can be considered outliers; in this embodiment, if λ q If the value is less than 0.01, then its weight λ is directly applied. q Setting it to zero means that this anchor point will not participate in subsequent fitting (equivalent to removing outlier anchor points) to avoid negative impacts on the transformation. If the number of effective anchor points (λ) q If the number of anchor points is less than 10 (>0), then you need to return to step S2 to reduce the spectral coordinate dimension d or increase the initial anchor point candidate set until the minimum anchor point requirement is met.

[0033] Define the weighted thin-plate spline energy functional E(f) = E fit (f) + γ*E smooth (f), Among them, E fit (f) = , which is the weighted fitting error term, requires that the transformed anchor point coordinates be as close as possible to the target coordinates, and that anchor points with higher weights contribute a greater penalty.

[0034] E smooth (f) = Let be the bending energy (smoothness constraint) of the thin plate spline, where α is a multi-order derivative index. This integral can be simplified to a quadratic form of the coefficient vector in thin plate spline theory: E smooth (f) = trace( W is the coefficient matrix (each w) q Composition), Φ is the radial basis function matrix, where Φ(i,j) = φ(||p i -p j ||).

[0035] γ is the regularization coefficient, used to balance fitting accuracy and smoothness. In this embodiment, γ = 0.01.

[0036] The energy functional E(f) with respect to the transformation parameters (A, b, w) qTo find the extreme values, the following system of linear equations can be derived using the variational method: For each anchor point q, we have: f(p) q =A*p q +b+ =q q (In the sense of weighted least squares).

[0037] Based on the orthogonality condition: =0 (zero vector) =0 (zero matrix).

[0038] Construct a matrix K, where K(i,j) = φ(||p i -p j (i, j=1...Q). Construct a matrix P whose i-th row is [1, p...Q]. i ^T] (1 is a scalar 1). Construct a matrix. Where I is a Q×Q identity matrix (due to the addition of a regularization term). Construct the right-hand side Y = [q1^T; q2^T; ...; q Q ^T;0;0] (The dimensions of the last two zero vectors must match the number of columns in P^T). To solve this, a dimension-by-dimensional approach is used: for the r-th dimension (r=1...d) of the output space, take all q... q The r-th component constitutes the vector Y r To solve a linear system, this embodiment employs a dimension-by-dimensional solution method: for the first dimension of the output space, take all q... q The r-th component constitutes the vector Y r Then solve the linear system of equations L*[w1^(r;...;w Q ^(r);a^(r);b^(r)]=[Y r [;0;0]. Obtain the coefficients for this dimension. Repeat d times to obtain all coefficients. Solve the linear system using Gaussian elimination or LU decomposition to obtain the thin-plate spline coefficients w. q (q=1...Q) Affine matrix A and translation vector b.

[0039] Substituting the coefficients obtained from the solution into the expression for the thin plate spline, we obtain the torsion mapping function f(x) from the first modal spectral coordinate space to the second modal spectral coordinate space. This function can calculate the mapping value for any first modal spectral coordinate x (a d-dimensional vector): f(x) = Ax + b + Where φ(r) = r^2 * ln(r) (when r > 0, r < 10) -8 (Assign a value of 0 directly to ensure numerical stability).

[0040] For each entity in the first modality knowledge graph, let its spectral coordinate be x. i(i=1...N1), substituting these values ​​into the aforementioned distortion mapping function, we can calculate the distortion first mode spectrum coordinates x'. i =f(x i These distorted spectral coordinates will be located in the same space as the spectral coordinates of the second modality entity, providing a basis for subsequent matching cost calculation and entity alignment.

[0041] In this embodiment, all steps involving the calculation of thin plate splines are implemented using double-precision floating-point numbers to ensure numerical stability. If the number of anchor points Q is large (e.g., exceeding 200), a sparse matrix solver can be considered for acceleration. The regularization coefficient γ = 0.01 and the weight bandwidth σ... λ =0.2 can be finely adjusted within the range of 0.001~0.1 according to the actual alignment effect.

[0042] S4. For each pair of first modal entities and second modal entities, calculate the coordinate deviation based on the difference between the distorted first modal spectrum coordinates and the second modal spectrum coordinates, and calculate the geometric structure deviation based on the difference between the first modal local tangent space orthogonal basis and the second modal local tangent space orthogonal basis. The coordinate deviation and the geometric structure deviation constitute the matching cost. The set of cross-modal aligned entity pairs is obtained by solving the optimal matching solution. Furthermore, in the above technical solution, the cross-modal aligned entity pair set is obtained through the following steps: The coordinate deviation is calculated as the Euclidean distance between the distorted first modal spectrum coordinates and the second modal spectrum coordinates; The geometric deviation is calculated as the Grieman distance between the first modal local tangent space orthogonal basis and the second modal local tangent space orthogonal basis; The coordinate deviation and geometric structure deviation are weighted and summed using preset weight coefficients to obtain the matching cost of candidate entity pairs, thus forming a cost matrix; The Hungarian algorithm is used to solve the minimum matching of the cost matrix, and the set of cross-modal aligned entity pairs is output.

[0043] For each entity i in the first mode (whose distorted first mode spectrum coordinates are x') i (where is a d-dimensional vector, d=20) and each entity j in the second mode (whose second mode spectral coordinates are y) j (also a d-dimensional vector), the coordinate deviation is defined as the Euclidean distance between the two: d coord (i, j) = ||x' i -y j ||2= , The smaller the value, the closer the spectral coordinates of the two entities are after the distortion alignment.

[0044] For a first modal entity i, its local tangent space orthogonal basis is denoted as T. i It is a d×k matrix (d=20, k=15), and each column is orthonormal. For the second modal entity j, its local tangent space orthogonal basis is denoted as U. j This is also a d×k orthogonal matrix. Geometric deviation is measured using the Grassmann distance between the two subspaces, and the specific calculation steps are as follows: Calculate matrix Its size is k×k.

[0045] Perform singular value decomposition (SVD) on M: M = S * diag(σ1, σ2, ..., σ k )*V^T, where σ1≥σ2≥...≥σ k Values ​​≥0 are singular values, and since T i and U j All columns are orthogonal, and singular values ​​satisfy 0 ≤ σ. p ≤1.

[0046] Calculate the principal angle θ p =arccos(σ) p (For p=1...k). The principal angle reflects the angle between two subspaces in various directions; the larger the angle, the greater the difference between the subspaces.

[0047] The Griman distance is defined as the square root of the sum of the squares of the principal angles (i.e., the subspace distance in the sense of the Frobenius norm): d geom (i, j) = Alternatively, the more common d can be used. geom = The two are similar in value, and this embodiment uses the former, d. geom = .

[0048] If d=20 and k=15, then the calculated d geom (i, j) is a non-negative real number, with values ​​approximately ranging from 0 to 1. Between these two values, the smaller the value, the more similar the local tangent space structures around the two entities are.

[0049] For each pair of first modal entity i and second modal entity j, their coordinate deviations and geometric deviations are weighted and summed to obtain the matching cost C(i,j): C(i,j)=β*(d coord (i,j) / σ coord )+(1-β)*(d geom (i,j) / σ geom ), Wherein: β is a preset weight coefficient, used to balance the importance of coordinate deviation and geometric structure deviation. In this embodiment, β = 0.6 is taken, that is, the coordinate deviation contributes 60% and the geometric structure deviation contributes 40%. If the geometric structure information is more reliable in the application, β can be appropriately reduced to 0.4.

[0050] σ coord and σ geom are the normalization factors of the coordinate deviation and the geometric structure deviation respectively, used to scale the two deviation values to a similar scale. σ coord takes the mean or median of all (d coord (i, j)), and σ geom is the same. In this embodiment, 1000 pairs of entities are randomly sampled first to calculate d coord and d geom , and the median of each is taken as the normalization factor. By default, σ coord = 2.0, σ geom = 0.8 (dynamically calculated according to the data during actual execution).

[0051] To avoid numerical instability, if a certain σ value is less than 1e-6, it is directly set to 1.0.

[0052] Let the number of entities in the first modality be N1, and the number of entities in the second modality be N2. Construct a cost matrix M cost , where M cost (i, j) = C(i, j). Since the Hungarian algorithm usually requires a square matrix (i.e., N1 = N2), and in practice N1 and N2 may not be equal, this embodiment adopts one of the following two processing methods: Method 1: Take N = min(N1, N2), and only match the first N entities (or the first N entities sorted by some importance), ignoring the entities exceeding the quantity. This method is suitable for the case where it is expected that most entities can find alignments.

[0053] Method 2: Construct a square matrix of size max(N1, N2) × max(N1, N2). If N1 < N2, then add (N2 - N1) virtual first modality entities, and set the matching cost of them with all second modality entities to a large constant (such as 1e6); if N1 > N2, then add virtual second modality entities. The matching of virtual nodes will be discarded in the final result.

[0054] This embodiment defaults to using Method 1, taking N = min(N1, N2), and sorting the entities of the two modalities in descending order of their degrees (or importance) in the knowledge graph, and selecting the first N entities to construct an N×N cost matrix. If all entities need to be retained in actual applications, then Method 2 is adopted.

[0055] The Hungarian algorithm (also known as the Kuhn-Munkres algorithm) is used to find the minimum weight perfect matching for the above cost matrix. The specific execution steps (applicable to dense matrices) are as follows: Initialization: For the cost matrix M cost Subtract the minimum value of each row of the (N×N) array, and then subtract the minimum value of each column.

[0056] Cover all zero elements in the matrix with the minimum number of lines.

[0057] If the number of lines equals N, then the optimal match is found; otherwise, adjust the matrix (subtract the minimum value of the uncovered line from the matrix, and add the matrix to the intersection of the covering lines) and repeat step 2.

[0058] To obtain the matching result: For each first modal entity index i (1≤i≤N), find the matching second modal entity index match(i) (1≤match(i)≤N).

[0059] This embodiment can use existing open-source implementations of the Hungarian algorithm (such as the `linear_sum_assignment` function in the SciPy library) or a custom implementation. The algorithm has a time complexity of O(N^3). When N is large (e.g., exceeding 5000), the Jonker-Volgenant algorithm can be used to accelerate it.

[0060] Based on the matching results obtained from the Hungarian algorithm, a cross-modal aligned entity pair set, AlignSet, is generated. The specific rules are as follows: For each matching pair (i, match(i)), if its original matching cost C(i, match(i)) is less than the preset maximum allowed cost threshold τ (τ=1.2 in this embodiment), then the entity pair is added to AlignSet.

[0061] If C(i, match(i)) ≥ τ, then the match is considered unreliable and will not be added to the alignment set. The value of τ can be adjusted according to the actual alignment quality, typically ranging from 0.8 to 1.5.

[0062] The final output is AlignSet={(ent1)} i ent2 j ) | i matches j and C(i, j) < τ}.

[0063] Each aligned entity pair contains a unique identifier (such as entity ID or name) for the first modality entity and a unique identifier for the second modality entity, and may also include a matching cost as a confidence score for subsequent steps. In this embodiment, the default method one (N=min(N1, N2)) is used and τ=1.2 is set, resulting in approximately N aligned entity pairs. To obtain higher quality alignment, τ can be reduced to 0.8, but this will result in a loss of some matching pairs.

[0064] S5. Extract the first modal relation triplet involved in the cross-modal aligned entity pair set, obtain the first modal spectrum coordinates of the head entity and tail entity of the relation triplet, transform it through the twist mapping function, embed and merge it with the first modal relation, map it to the second modal relation space to obtain a unified relation representation, and integrate the second modal relation representation to form a unified relation representation set; Furthermore, in the above technical solutions, such as Figure 2 As shown, the unified relation representation is obtained through the following steps: Obtain the relation type embedding vector of the first modal relation triplet; The spectral coordinates of the head and tail entities after being transformed by the twisting mapping function are concatenated with the relationship type embedding vector and input into a pre-trained two-layer fully connected mapping network to obtain the mapping embedding in the second modality relationship space. Calculate the cosine similarity between the mapping embedding and each relation embedding in the second modality knowledge graph, and take the relation with the highest similarity that exceeds a preset threshold as the corresponding unified relation, and use the second modality relation embedding as the unified relation representation.

[0065] Let AlignSet = {a} be the set of cross-modal aligned entity pairs output by step S4. p b p )|p=1...M}, where a p b is the first modal entity. p For the second modal entity to be aligned, M is the number of aligned entity pairs. This step extracts all relation triples (h, r, t) from the first modal knowledge graph that satisfy the following condition: h (head entity) and t (tail entity) are both first modal entities, and both h and t appear in the first modal entity set of AlignSet (i.e., both h and t have corresponding aligned second modal entities). If only one end is aligned, the triple is not processed temporarily because it cannot be completely mapped to the second modal relation space. The extracted relation triples constitute a set R. triples ={(h i r i , t i Let |i=1...L}, where L is the number of triples that satisfy the condition.

[0066] Each relation type in the first-modality knowledge graph, such as "belongs to," "located in," and "possesses attribute," requires a learnable embedding vector. This embodiment obtains this embedding vector in the following way: Pre-training is performed on the first-modality knowledge graph using a knowledge graph embedding model (such as TransE or RotatE). Specifically, all triples in the first-modality knowledge graph are used as training data, the embedding dimension is set to 64 dimensions, and the training is performed for 100 rounds with a learning rate of 0.01 to obtain the embedding vector for each entity and each relation type.

[0067] Extract the embedding vector corresponding to relation type r from the trained model. The dimension is 64. For relations in the first modality that did not appear in the training, a 64-dimensional vector is randomly initialized and normalized.

[0068] Similarly, the same knowledge graph embedding model (such as TransE) was pre-trained on the second modality knowledge graph (image modality), with the embedding dimension also set to 64 dimensions, to obtain the embedding vector for each relation type in the second modality. The dimensions are 64. These embedding vectors will serve as the target space for the unified relation representation.

[0069] To map the relation triples from the first modality to the relation space of the second modality, a two-layer fully connected mapping network (MLP) is constructed. The structure and training method of this network are as follows: Network structure: Input layer: The input vector is composed of three parts concatenated: The spectral coordinates x' of the head entity after distortion mapping h Dimension d=20; The spectral coordinates x' of the tail entity after distortion mapping t Dimension 20; First modality relation type embedding vector The dimension is 64. Therefore, the total input dimension = 20 + 20 + 64 = 104.

[0070] First hidden layer: Fully connected layer with 128 neurons, using ReLU activation function, and adding Dropout (dropout rate 0.2) to prevent overfitting.

[0071] The second hidden layer is a fully connected layer with 64 neurons and ReLU activation function.

[0072] Output layer: Fully connected layer, with the number of neurons equal to the dimension of the second modality relation embedding, i.e., 64. No activation function is used, and the output vector is v. out ∈R^64.

[0073] Construction of training data: This implementation uses a contrastive learning framework to train the mapping network, and does not require that the second modality must contain data from b. h to b t The direct relationship. The specific steps are as follows: (1) Positive sample pairs: For each first modal relation triple (h, r, t), its head entity h is aligned to b. h The tail entity t is aligned to b. t In the second modality knowledge graph, if there exists at least one entry from b... h to b t If the direct relation (relation type denoted as s) is defined, then s is treated as a positive label, and positive sample pairs are constructed: Input vector X = [x' h , x' t , The output target is relation embedding. If multiple direct relationships exist, one of them is randomly selected, and the weight of that sample is recorded as 1.

[0074] (2) Negative Sample Pairs: For the same first-modal triplet, randomly sample a relation type s' different from s from the second-modal relation set S (if there is no direct relation, randomly sample any relation type as the negative label) to construct a negative sample pair. The input vectors are the same, and the output target is... The ratio of positive to negative samples is 1:1.

[0075] (3) Processing of triples with no direct relationship: If b h With b t If there is no direct relationship between them, the triplet is only used for constructing negative samples (i.e., treated as a negative example, requiring the distance between the mapping network output and any real relation embedding to be greater than a threshold). However, to avoid sample imbalance, the sampling rate of such samples should not exceed 30% of the total samples. Furthermore, virtual relationships can be utilized: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] h With b t The relationship sequences on the shortest path between them are aggregated (e.g., by averaging or LSTM encoding) as a soft target. The specific method is not limited to this invention.

[0076] Loss function: The contrastive loss (InfoNCE) is used. For each positive sample pair (X, e) in the batch... + ) and their corresponding negative sample pairs (X, e) - The loss is defined as: L triplet =max(0, ||F θ (X)-e + ||^2-||F θ (X)-e -(||^2+margin), where margin=0.2, F θ It is a two-layer fully connected mapping network. Additionally, for samples with no direct relationship, a regularization term is added: L consistency =||F θ (X)||^2 (Encourage outputs close to the origin, indicating no correspondence). The total loss is L. triplet +0.1·L consistency This training method does not require all possible alignment relationships to exist in the second modality beforehand and can generalize to unseen relationship pairs.

[0077] Optimizer: The Adam optimizer is used, with an initial learning rate of 0.001, a batch size of 32, 200 training epochs, and an early stopping strategy (stopping if the loss does not decrease for 10 consecutive epochs). After training, the network weights are saved.

[0078] For R triples For each first-modal relation triple (h, r, t) in the table, perform the following steps: Obtain the first mode spectral coordinates x' of the twisted head entity h. h (Calculated and stored in step S3), the first modal spectral coordinates x' of the tail entity t's torsion. t and the embedding vector of relation type r (Dimension 64).

[0079] Concatenate the three to obtain the input vector X=[x' h , x' t , (Dimension 104).

[0080] Inputting X into a pre-trained two-layer fully connected mapping network yields the predicted mapping embedding v. map ∈R^64.

[0081] Calculate v map Embedded with each relation in the second modality knowledge graph Cosine similarity between (s∈S, where S is the set of all relation types in the second modality): sim(s) = (v map · ) / (||v map ||·|| Since all embeddings have been normalized, the cosine similarity is the dot product.

[0082] Set a similarity threshold θ rel =0.75. Find the relation type s* that maximizes sim(s) if sim(s) ≥ θ. rel Then, the first modal relation triple is mapped to the second modal relation s*, resulting in a unified relation representation e.unified = . If sim(s*)<θ rel If the first modality relation has no suitable correspondence in the second modality, then the triplet is discarded.

[0083] Record the mapping results to form a mapping record: (h, r, t) -> (b h ,s*,b t ), where b h and b t These are the second modal entities that h and t are aligned in AlignSet, respectively.

[0084] Iterate through all successfully mapped first-modal relation triples and collect all occurrences of second-modal relation embeddings (which may have duplicates). Since the same second-modal relation may correspond to multiple first-modal relations, in this embodiment, the unified relation representation set is defined as all activated second-modal relation embeddings and their corresponding source information. The specific output format is as follows: UnifiedRelSet={(s, ,source_triples_list)}, Where s is the identifier for the second modal relation type. The embedding vector is defined by `source_triples_list`, which is a list of all first-modal relation triples mapped to this relation. If only the relation representation itself is needed for subsequent fusion graph construction, the output can be simplified to all... The set, along with a one-to-one mapping table from the first modal triples to the second modal relations.

[0085] In this embodiment, the final unified relation representation set includes a subset of the second modal relation space (approximately 60% to 80% of the original number of second modal relations), as well as mappings from first modal relation triples to these relations. These unified relation representations will be used in step S6 to attach to cross-modal unified entity nodes, forming relation edges in the initial fusion graph.

[0086] like Figure 3 As shown: S6, merge the matching entities in the cross-modal aligned entity pair set into cross-modal unified entity nodes, generate the fused entity features of each cross-modal unified entity node, attach the relations in the unified relation representation set to the corresponding cross-modal unified entity nodes to form an initial fused graph, and use a link prediction model with manifold distance regularization to complete the missing links in the initial fused graph, and output the cross-modal fused knowledge graph.

[0087] Furthermore, in the above technical solution, the link prediction model with manifold distance regularization is a graph autoencoder, and its training steps include: The adjacency matrix and fused entity features of the initial fused graph are input into the encoder to obtain node embeddings, and the decoder generates link probabilities. Calculate the reconstruction loss and add a manifold distance regularization term; for each predicted link, the manifold distance regularization term obtains the first modal entity and the second modal entity corresponding to the cross-modal unified entity nodes at both ends, calculates the diffusion distance of the two first modal entities under the distorted first modal spectrum coordinates, and the diffusion distance of the two second modal entities under the second modal spectrum coordinates, and uses the square difference of the two diffusion distances as a penalty. The graph autoencoder is trained using a total loss that includes reconstruction loss and manifold distance regularization. After training, missing links are predicted and filled in to obtain a cross-modal fusion knowledge graph.

[0088] Let AlignSet = {a p b p )|p=1...M}, where a p b is the first modal entity. p For each pair of aligned entities, merge them into a single cross-modal unified entity node u. p The specific method for generating fused entity features is as follows: Obtain the first modal entity a p modal embedding vector eva p ; Obtain the second modal entity b p modal embedding vector evb p .

[0089] Each cross-modal unified entity node u p fused entity features f u Features are generated using a learnable projection matrix, which is then jointly trained end-to-end with a subsequent graph autoencoder to ensure that the features are optimized for the link prediction task. The specific implementation is as follows: First, embed the first mode into the vector eva. p Second mode embedding vector evb p Features of the same dimension (256 dimensions) are obtained by linear projection: va proj =W a ·eva p +b a ,vb proj =W_b·evb p +b b , Among them W a and W b The learnable weight matrix is ​​256×512, b a and b b A 256-dimensional learnable bias. Fusion features f u =va proj +vb proj (Element-wise addition). This fusion feature serves as node u. p The initial features are input into the subsequent graph autoencoder.

[0090] Joint training strategy: When training the graph autoencoder (GAE), W... a W b b a b b These projection matrices are also used as trainable parameters and updated along with the encoder and decoder parameters. The total loss function (reconstruction loss + manifold distance regularization term) simultaneously optimizes these projection matrices, making the fused features more conducive to cross-modal link prediction. To maintain training stability, the projection matrices can be fixed for the first 10 epochs of training (training only GAE), and then unfrozen for joint optimization. There is no situation where "X is treated as a fixed input". If a pre-trained projection matrix is ​​needed, W can be pre-trained on a small number of anchor entity pairs using contrastive loss. a and W b Then connect to GAE for fine-tuning.

[0091] Each cross-modal unified entity node u p It also records the corresponding original entity ID (a p ID and b p The ID), as well as basic attribute information from both modalities (such as name, text description, image path, etc.).

[0092] In the unified relation representation set output by step S5, each mapping record is: (h, r, t) -> (b h ,s,b t ), where h is the first modal head entity, t is the first modal tail entity, and b h and b t These are the second modal entities corresponding to them in the AlignSet, and s is the type of the second modal relationship obtained through mapping. Since b h and b t Each corresponds to a cross-modal unified entity node, denoted as u. head and u tail Therefore, it is possible to use u head and u tailEstablish a directed edge between them, with the edge labeled s* (second modal relation type). The edge's attributes also include the source first modal relation type r (optional, used for tracing).

[0093] For all successfully mapped records, perform the following operations: Search u head =unified_node_map(b h ) and u tail =unified_node_map(b t ).

[0094] Add a directed edge (u) to the edge set of the initial fused graph. head ,s*,u tail If multiple relationships of the same type exist between the same pair of nodes, they can be merged into one edge, and the number of relationship instances can be recorded as the weight; if different types of relationships exist, different edges should be added for each.

[0095] After completing the above operations, the initial fusion map G is obtained. init Includes the following elements: Node set: V unified ={u1, u2, ..., u M There are a total of M unified entity nodes.

[0096] Node feature matrix: X∈R^{M×256}, the p-th row is the fused feature f u .

[0097] Edge set: E init Each edge is (u i ,s,u j ), where s is the second modal relation type identifier.

[0098] To fill in the missing links (i.e., potentially omitted inter-entity relationships) in the initial fused graph, this embodiment employs a graph auto-encoder (GAE) model and adds a manifold distance regularization term to the loss function.

[0099] Its model structure specifically includes: Directed Graph Encoder: Since relationships in knowledge graphs often have a clear direction (from head entity to tail entity), simply treating the graph as an undirected graph would lose important semantic information. This implementation uses a Directed Graph Convolutional Network (GCN), as detailed below: Construct two adjacency matrices respectively: A out and A in A out (i, j) = 1 indicates that there exists a directed edge from node i to node j, Ain (i, j) = 1 indicates that there exists a directed edge from node j to node i (i.e., A). in =A out The encoder employs a two-layer directed graph convolution to aggregate information about the incoming and outgoing neighbors for each node: , , , Where D out and D in These are the out-degree matrix and the in-degree matrix, respectively (the diagonal elements are the out-degree and in-degree of the corresponding nodes), W out W in W combine The learning weight matrix is ​​[;], which represents the concatenation operation, and σ is the activation function (ReLU in the first layer and linear in the second layer). The input of the first layer, H^(0), is the fused entity feature matrix X (dimension M×256), which outputs the node embedding Z (dimension M×64) after passing through two layers of directed GCN.

[0100] Directed decoder: Employs a directed inner product model, where the probability of a link existing from node i to node j is: p(i→j) = sigmoid(z) i ^T·W dir ·z j ), Among them W dir A learnable matrix of 64×64 is used to capture directionality. For undirected symmetric relations, this can be achieved by using W... dir The constraints can be implemented using symmetric matrices or shared parameters, but this embodiment is not limited to these.

[0101] If undirected simplification must be used due to special scenarios, it must satisfy the premise that all relationships in the two modal knowledge graphs are symmetrical (such as "similar" or "related") and that the direction does not affect the semantics. This embodiment defaults to using the above-mentioned directed processing scheme to ensure its universality for general knowledge graphs.

[0102] The total loss consists of two parts: reconstruction loss and manifold distance regularization term. (1) Reconstruction loss, using cross-entropy loss. Let the set of training edges be E. train (i.e., the edges already present in the initial fused graph), the set of negative sampled edges is E. neg (Random sampling is performed from unconnected node pairs, with the number of samples roughly equivalent to the number of positive edges). The reconstruction loss is: Loss rec = .

[0103] (2) Manifold distance regularization term For each predicted link (i.e., each trained positive edge or candidate edge to be predicted), consider the cross-modal unified entity nodes u at both ends. i and u j Each unified entity node u i Corresponding to a first modal entity a i (Original text entity) and a second modal entity b i (Original image entities). Calculate separately: d first (i, j) = two first-modal entities a i and a j The diffusion distance in the distorted first-mode spectral coordinates. Since step S2 has already calculated the local diffusion distance matrix D between entities in the first mode. first Furthermore, step S3 distorts the spectral coordinates, but the diffusion distance is based on the original graph structure, so the diffusion distance d calculated by S2 can be used directly. first (i, j) is sufficient (already stored).

[0104] d second (i, j) = two second-modal entities b i and b j The diffusion distance in the second modal spectral coordinates (also calculated and stored in step S2).

[0105] The purpose of the manifold distance regularization term is to ensure that if two first-modal entities are close on the manifold (d first If the size is smaller, then the corresponding second modal entity should also be close to (d) in terms of manifold. second (smaller), and vice versa. The specific penalty term is defined as the squared difference of the diffusion distances of the two: Reg(i,j)=(d first (i,j)-d second (i, j)^2.

[0106] For all edges involved in training (including positive and negative sampling edges), the manifold distance regularization term is: Loss reg =λ*(1 / |E all |)* , Where E all =E train ∪E neg λ is the regularization coefficient, which is set to 0.1 in this embodiment. This regularization term penalizes inconsistencies in the cross-modal manifold structure, enabling the node embeddings learned by the encoder to simultaneously maintain the local geometric relationships of the two modes.

[0107] (3) Total loss Total loss = Loss rec +Lossreg .

[0108] 3. Model Training Training set: The existing edges (positive edges) of the initial fused graph plus an equal number of randomly sampled negative edges. Note: Positive edges may contain bidirectional edges; in this embodiment, they are uniformly treated as undirected.

[0109] Optimizer: Adam, learning rate 0.001, weight decay 1e-5.

[0110] Training rounds: up to 300 rounds. Early stopping mechanism: if the reconstruction loss of the validation set (randomly selected from the positive edges as validation edges) does not decrease for 20 consecutive rounds, the training will stop.

[0111] Batch training: Full-batch training is used because M (the number of uniform entity nodes) is generally in the thousands, which can be accommodated.

[0112] During training, the node feature matrix X follows the projection matrix W. a W b It is dynamically calculated based on updates, and is not fixed individually; After training is complete, the encoder parameters are saved to obtain the node embedding matrix Z.

[0113] For all entity pairs in the initial fusion graph that are not directly connected (including those relationships that may exist in the first or second modality but have not yet been mapped to the fusion graph), calculate the probability that they are linked: p(i,j) = sigmoid(z i ^T*z j Set the link prediction threshold θ. link =0.7. If p(i,j)≥θ link If so, then it is assumed that there is a missing link between (i, j).

[0114] Relationship type prediction mechanism: Since the graph autoencoder only predicts the probability of the existence of links, for node pairs predicted to have links (u i u j Further inference of the specific relationship type is needed. This implementation design employs a type prediction network, jointly trained with the encoder, as follows: (1) Type prediction network structure: based on the embedding z of two unified entity nodes i and z j The 64-dimensional vector output by the GAE encoder is used as input, concatenated, and then fed into a two-layer fully connected network (the first layer is 128-dimensional, ReLU; the second layer is |S|-dimensional, where |S| is the total number of second modality relation types). The output is a probability distribution p(s|u i u j =softmax(MLP([zi ;z j ])).

[0115] (2) Training data: The existing edges in the initial fused graph (i.e., the unified relation representation set obtained in S5) are used as the supervision signal. For each existing edge (u i ,s,u j The type label is s. Simultaneously, negative sampling (random node pairs) is performed on node pairs without edges, and their type label is "unrelated" (which can be considered as the |S|+1th class). The loss function is cross-entropy loss.

[0116] (3) Prediction phase: For the missing links predicted by GAE (i.e., p... link (i, j) ≥ θ_ link ), the node pair (z i , z j The input type prediction network obtains the relation type s with the highest probability. If the probability of s is less than a preset threshold θ... type (In this embodiment, we take 0.6), then the type prediction is considered unreliable, and the link is not added; otherwise, an edge (u) is added to the final fused graph. i ,s,u j If s is of the "no relation" class, then mark it as "complete link". If s is of the "no relation" class, then no link is added.

[0117] The predicted links that meet the thresholds and whose relationship types can be determined are added to the initial fusion graph to form the final cross-modal fusion knowledge graph G. final .

[0118] The final output is a cross-modal fusion knowledge graph G. final Include: Nodes: All cross-modal unified entity nodes (M nodes), each node has fused entity features (256 dimensions) and original metadata from both modalities.

[0119] Edges: Original edges in the initial fused graph (from the S5 relation mapping) plus the completed missing links (confirmed by relation type). Each edge has a relation type label (second modality relation identifier).

[0120] Optional: Retain the original entity ID mapping relationship for each node to facilitate subsequent tracing and updates.

[0121] The graph can be output in standard formats (such as RDF triples, CSV files, or graph database import formats). In this embodiment, the output consists of three files: entities.csv contains the unified entity ID, the original text entity ID, the original image entity ID, the fused feature vector (the vector is stored as a string), and metadata.

[0122] relations.csv contains relation type ID, relation name, and embedding vector.

[0123] edges.csv: contains header entity ID, relation type ID, tail entity ID, and whether to pad (flag).

[0124] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A heterogeneous knowledge graph fusion method based on cross-modal manifold alignment, characterized in that, Includes the following steps: S1. Obtain a first modality knowledge graph and a second modality knowledge graph. The first modality knowledge graph contains first modality entities and first modality relation triples. The second modality knowledge graph contains second modality entities and second modality relation triples. Each entity has a preset modality embedding vector. S2. Based on the preset modal embedding vector, construct the first modal entity adjacency graph and the second modal entity adjacency graph respectively, calculate the local diffusion distance between entities in the entity adjacency graph, and use the local diffusion distance to generate the first modal spectral coordinates and the first modal local tangent space orthogonal basis, the second modal spectral coordinates and the second modal local tangent space orthogonal basis; S3. Using a pre-set cross-modal anchor point entity pair as a constraint, construct a weighted non-rigid transformation in the spectral coordinate space. The weighted non-rigid transformation aims to minimize the diffusion distance error of the anchor point pair and improve the smoothness of the transformation. Solve to obtain the twisted mapping function from the first modal spectral coordinate space to the second modal spectral coordinate space. Substitute the first modal spectral coordinates into the twisted mapping function to obtain the twisted first modal spectral coordinates. S4. For each pair of first modal entities and second modal entities, calculate the coordinate deviation based on the difference between the distorted first modal spectrum coordinates and the second modal spectrum coordinates, and calculate the geometric structure deviation based on the difference between the first modal local tangent space orthogonal basis and the second modal local tangent space orthogonal basis. The coordinate deviation and the geometric structure deviation constitute the matching cost. The set of cross-modal aligned entity pairs is obtained by solving the optimal matching solution. S5. Extract the first modal relation triplet involved in the cross-modal aligned entity pair set, obtain the first modal spectrum coordinates of the head entity and tail entity of the relation triplet, transform it through the twist mapping function, embed and merge it with the first modal relation, map it to the second modal relation space to obtain a unified relation representation, and integrate the second modal relation representation to form a unified relation representation set; S6. Merge the matching entities in the cross-modal aligned entity pair set into cross-modal unified entity nodes, generate fused entity features for each cross-modal unified entity node, attach the relations in the unified relation representation set to the corresponding cross-modal unified entity nodes to form an initial fused graph, and use a link prediction model with manifold distance regularization to complete the missing links in the initial fused graph, and output a cross-modal fused knowledge graph.

2. The heterogeneous knowledge graph fusion method based on cross-modal manifold alignment according to claim 1, characterized in that, The preset modality embedding vectors are extracted using the cross-modal pre-trained model CLIP to obtain text modality embedding vectors and image modality embedding vectors, respectively.

3. The heterogeneous knowledge graph fusion method based on cross-modal manifold alignment according to claim 1, characterized in that, The construction of the first modal entity adjacency graph and the second modal entity adjacency graph includes: For each modality, the cosine similarity between pairs of entities is calculated using the modality embedding vector of the entity, and a hybrid similarity matrix is ​​constructed by combining the structural co-occurrence relationship in the knowledge graph of that modality. The hybrid similarity matrix is ​​then converted into an entity adjacency graph by a preset threshold.

4. The heterogeneous knowledge graph fusion method based on cross-modal manifold alignment according to claim 1, characterized in that, The calculation of the local diffusion distance specifically includes: Perform a preset t-step random walk on the entity adjacency graph to obtain the diffusion distance d(i,j) between entity i and entity j. Construct a diffusion kernel matrix K, and then perform symmetric normalization on K to obtain matrix P, where K(i,j) = exp(-d(i,j)). 2 / σ), where σ is the median diffusion distance between each entity and its h-th nearest neighbor, and h is the preset number of nearest neighbors; Perform eigenvalue decomposition on matrix P, and take the eigenvectors corresponding to the first d largest eigenvalues ​​as the spectral coordinates, where d is the preset spectral coordinate dimension; For each entity's spectral coordinates, select its k nearest neighbors. Perform principal component analysis on the k nearest neighbors and use the resulting k principal component directions as the local tangent space orthogonal basis of the entity, where k is the preset dimension of the local tangent space.

5. The heterogeneous knowledge graph fusion method based on cross-modal manifold alignment according to claim 1, characterized in that, The twisted mapping function is obtained through the following steps: Thin plate spline functions are selected as the transformation model; The displacement field is constructed using the difference between the first and second modal spectral coordinates of the cross-modal anchor point entity pairs, and a weighted thin-plate spline energy functional is constructed using the negative exponential function of the local diffusion distance between the anchor point pairs as the weight; the preset cross-modal anchor point entity pairs are obtained by cross-modal semantic similarity matching of entity names and supplemented by manual verification. Finding the extreme value of the energy functional yields a system of linear equations. Solving this system of linear equations yields the thin-plate spline coefficients, forming a twisted mapping function from the first modal spectral coordinate space to the second modal spectral coordinate space.

6. The heterogeneous knowledge graph fusion method based on cross-modal manifold alignment according to claim 1, characterized in that, The cross-modal aligned entity pair set is obtained through the following steps: The coordinate deviation is calculated as the Euclidean distance between the distorted first modal spectrum coordinates and the second modal spectrum coordinates; The geometric deviation is calculated as the Grieman distance between the first modal local tangent space orthogonal basis and the second modal local tangent space orthogonal basis; The coordinate deviation and geometric structure deviation are weighted and summed using preset weight coefficients to obtain the matching cost of candidate entity pairs, thus forming a cost matrix; The Hungarian algorithm is used to solve the minimum matching of the cost matrix, and the set of cross-modal aligned entity pairs is output.

7. The heterogeneous knowledge graph fusion method based on cross-modal manifold alignment according to claim 1, characterized in that, The unified relation representation is obtained through the following steps: Obtain the relation type embedding vector of the first modal relation triplet; The spectral coordinates of the head and tail entities after being transformed by the twisting mapping function are concatenated with the relationship type embedding vector and input into a pre-trained two-layer fully connected mapping network to obtain the mapping embedding in the second modality relationship space. Calculate the cosine similarity between the mapping embedding and each relation embedding in the second modality knowledge graph, and take the relation with the highest similarity that exceeds a preset threshold as the corresponding unified relation, and use the second modality relation embedding as the unified relation representation.

8. The heterogeneous knowledge graph fusion method based on cross-modal manifold alignment according to claim 1, characterized in that, The link prediction model with manifold distance regularization is a graph autoencoder, and its training steps include: The adjacency matrix and fused entity features of the initial fused graph are input into the encoder to obtain node embeddings, and the decoder generates link probabilities. Calculate the reconstruction loss and add a manifold distance regularization term; for each predicted link, the manifold distance regularization term obtains the first modal entity and the second modal entity corresponding to the cross-modal unified entity nodes at both ends, calculates the diffusion distance of the two first modal entities under the distorted first modal spectrum coordinates, and the diffusion distance of the two second modal entities under the second modal spectrum coordinates, and uses the square difference of the two diffusion distances as a penalty. The graph autoencoder is trained using a total loss that includes reconstruction loss and manifold distance regularization. After training, missing links are predicted and filled in to obtain a cross-modal fusion knowledge graph.

Images