Cross-modal data retrieval method, system and device based on multi-modal knowledge graph

Abstract

The application discloses a cross-modal data retrieval method, system and device based on a multi-modal knowledge graph. The method decouples features based on a visual feature vector and a structural feature vector, and determines a target visual projection vector. A path constraint contrast learning loss function is constructed based on the target visual projection vector and a text feature vector. A multi-modal knowledge graph is constructed, and effective paths between entities in the multi-modal knowledge graph are mined. The effective paths are encoded into path encoding features, and a multi-scale path perception repulsion loss function is constructed based on the path encoding features. The path constraint contrast learning loss function and the multi-scale path perception repulsion loss function are jointly optimized to determine a target path encoding feature subset. The text feature vector, the target visual projection vector and the target path encoding feature subset are fused to obtain a joint vector. Cross-modal data retrieval is performed based on the joint vector to obtain a retrieval result. The application can improve the accuracy of image-text retrieval.

Description

Technical Field

[0001] This application relates to the field of cross-modal retrieval technology, and in particular to a cross-modal data retrieval method, system and device based on multimodal knowledge graph. Background Technology

[0002] Multimodal knowledge graph vectorization is a core topic in the intersection of artificial intelligence and knowledge graph technology. Its goal is to model heterogeneous modal data (e.g., text, images, and videos) and their complex relationships through a unified vector space, supporting cross-modal semantic understanding and intelligent reasoning. Current multimodal knowledge graph construction technologies mainly revolve around two paradigms: independent embedding followed by fusion (such as the MCEAS model), which uses dedicated encoders (such as BERT for text and ResNet for images) to generate single-modal vectors, and then fuses them into entity vectors through an attention mechanism; and graph structure extension methods (such as the M3-Agent framework), which directly aggregates multimodal neighborhood information based on graph neural networks (such as GAT / R-GCN), preserving local graph structure.

[0003] Despite the progress made by the two types of methods mentioned above in heterogeneous data fusion, the bottleneck of alignment-structure mutual exclusion still exists. That is, cross-modal contrastive learning dilutes key topological relationships when it forces text / image vectors to be close together, while graph structure injection contaminates visual features, resulting in a decrease in the accuracy of image-text retrieval. Summary of the Invention

[0004] This application aims to propose a cross-modal data retrieval method, system, and device based on multimodal knowledge graphs, which can improve the accuracy of image and text retrieval.

[0005] In a first aspect, embodiments of this application provide a cross-modal data retrieval method based on a multimodal knowledge graph, the method comprising:

[0006] The acquired text modal data and image modal data are preprocessed to obtain standardized text modal data and standardized image modal data;

[0007] Extract text feature vectors from the standardized text modal data, and extract visual feature vectors and structural feature vectors from the standardized image modal data;

[0008] Based on the visual feature vector and the structural feature vector, feature decoupling is performed to determine the target visual projection vector;

[0009] Based on the target visual projection vector and the text feature vector, a path constraint contrastive learning loss function is constructed.

[0010] Construct a multimodal knowledge graph between text and images, and mine effective paths between entities in the multimodal knowledge graph;

[0011] The effective path is encoded as a path encoding feature, and a multi-scale path-aware rejection loss function is constructed based on the path encoding feature;

[0012] The path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function are jointly optimized to determine the target path encoding feature subset;

[0013] The text feature vector, the target visual projection vector, and the target path encoding feature subset are fused to obtain a joint vector;

[0014] Cross-modal data retrieval is performed based on the joint vector to obtain retrieval results.

[0015] Compared with the prior art, the first aspect of this application has the following beneficial effects:

[0016] This method preprocesses the acquired text and image modal data to obtain standardized text and image modal data; extracts text feature vectors from the standardized text modal data, and visual and structural feature vectors from the standardized image modal data; performs feature decoupling based on the visual and structural feature vectors to determine the target visual projection vector; constructs a path constraint contrastive learning loss function based on the target visual projection vector and text feature vectors; constructs a multimodal knowledge graph between text and images, and mines effective paths between entities in the multimodal knowledge graph; encodes the effective paths into path encoding features, and constructs a multi-scale path-aware rejection loss function based on the path encoding features; jointly optimizes the path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function to determine the target path encoding feature subset; fuses the text feature vector, the target visual projection vector, and the target path encoding feature subset to obtain a joint vector; and performs cross-modal data retrieval based on the joint vector to obtain the retrieval results. Thus, feature decoupling ensures semantic fidelity between text and images; path-constrained contrastive learning loss optimizes the similarity distribution of multimodal features to enable semantic alignment of multimodal data; multi-scale path-aware rejection loss forces the path feature distance between semantically associated positive entity pairs to be less than the path feature distance between positive and negative sample pairs, thereby enhancing the discriminativeness of path semantics; joint optimization of the path-constrained contrastive learning loss function and the multi-scale path-aware rejection loss function determines the optimal path encoding feature subset. By fusing text feature vectors, target visual projection vectors, and target path encoding feature subsets, and performing cross-modal data retrieval based on the joint vector, the inherent contradiction between cross-modal semantic alignment and graph structure preservation in multimodal knowledge graphs can be resolved, thus improving the accuracy of text-image retrieval.

[0017] In some implementations, the step of decoupling features based on the visual feature vector and the structural feature vector to determine the target visual projection vector includes:

[0018] The visual feature vector and the structural feature vector are mapped to an orthogonal subspace by an orthogonal projection operation to obtain an initial visual projection vector and an initial structural projection vector.

[0019] Construct a mutual information constraint loss function for feature decoupling, and determine whether the initial visual projection vector and the initial structural projection vector satisfy the mutual information constraint loss function;

[0020] If the initial visual projection vector and the initial structural projection vector do not satisfy the mutual information constraint loss function, then the gradient of the mutual information constraint loss function is updated through backpropagation to optimize the visual feature vector and structural feature vector extracted during the forward propagation process.

[0021] Based on the optimized visual feature vector and structural feature vector, an optimized initial visual projection vector and an optimized initial structural projection vector are obtained by orthogonal projection operation. The optimized initial visual projection vector is used as the target visual projection vector after the mutual information constraint loss function is satisfied.

[0022] In some implementations, constructing a path-constrained contrastive learning loss function based on the target visual projection vector and the text feature vector includes:

[0023] ;

[0024] in, This represents the path constraint contrastive learning loss function. This represents the set of positive sample pairs that satisfy the path constraints. Represents the text feature vector. Represents the target visual projection vector. The cosine similarity function represents the feature pairs. Indicates the temperature coefficient. Indicates the number of negative samples. Represents text entities, Represents an image entity.

[0025] In some implementations, mining effective paths between entities in the multimodal knowledge graph includes:

[0026] Preset semantic constraints;

[0027] Depth-first search and breadth-first search are used to mine potential connection paths between entities in the multimodal knowledge graph;

[0028] From the potential connection paths, extract the valid paths between entities that satisfy the semantic constraints.

[0029] In some implementations, constructing a multi-scale path-aware repulsion loss function based on the path encoding features includes:

[0030] ;

[0031] in, This represents the multi-scale path-aware rejection loss function. This represents the sampling weights for paths with different hop counts. Represents anchor point entity go through Path encoding features after skipping relational paths. Represents positive sample entities go through Path encoding features after skipping relational paths. Represents negative sample entities go through Path encoding features after skipping relational paths. Indicates the boundary margin.

[0032] In some implementations, fusing the text feature vector, the target visual projection vector, and the target path encoding feature subset to obtain a joint vector includes:

[0033] The text feature vector, the target visual projection vector, and the target path encoding feature subset are each subjected to layer normalization to obtain the layer normalized text feature vector, the layer normalized target visual projection vector, and the layer normalized target path encoding feature subset.

[0034] The text feature vector after layer normalization, the target visual projection vector after layer normalization, and the target path encoding feature subset after layer normalization are concatenated to obtain a joint vector.

[0035] In some implementations, the cross-modal data retrieval based on the joint vector to obtain retrieval results includes:

[0036] The joint vector generated from each piece of data is stored in a vector database with the same spatial vector representation, and an approximate nearest neighbor index is established for the vector database.

[0037] The acquired modal data to be retrieved is encoded into a query vector represented by the same spatial vector;

[0038] In a vector database with an approximate nearest neighbor index, an approximate nearest neighbor search is used to perform cross-modal data retrieval on the query vector to obtain retrieval results.

[0039] Secondly, embodiments of this application also provide a cross-modal data retrieval system based on a multimodal knowledge graph, the system comprising:

[0040] The data processing unit is used to preprocess the acquired text modal data and image modal data to obtain standardized text modal data and standardized image modal data.

[0041] The feature extraction unit is used to extract text feature vectors from the standardized text modal data, and to extract visual feature vectors and structural feature vectors from the standardized image modal data;

[0042] A feature decoupling unit is used to perform feature decoupling based on the visual feature vector and the structural feature vector to determine the target visual projection vector;

[0043] The first construction unit is used to construct a path constraint contrastive learning loss function based on the target visual projection vector and the text feature vector;

[0044] The path mining unit is used to construct a multimodal knowledge graph between text and images, and to mine effective paths between entities in the multimodal knowledge graph.

[0045] The second construction unit is used to encode the effective path into path encoding features and construct a multi-scale path-aware rejection loss function based on the path encoding features.

[0046] The joint optimization unit is used to jointly optimize the path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function to determine the target path encoding feature subset.

[0047] The feature fusion unit is used to fuse the text feature vector, the target visual projection vector, and the target path encoding feature subset to obtain a joint vector;

[0048] The data retrieval unit is used to perform cross-modal data retrieval based on the joint vector and obtain retrieval results.

[0049] Thirdly, embodiments of this application also provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform a cross-modal data retrieval method based on a multimodal knowledge graph as described above.

[0050] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a cross-modal data retrieval method based on a multimodal knowledge graph as described above.

[0051] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0052] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0053] Figure 1 This is a flowchart illustrating an embodiment of the cross-modal data retrieval method based on multimodal knowledge graphs provided in this application;

[0054] Figure 2 This is a schematic diagram of the overall process in the best embodiment of the cross-modal data retrieval method based on multimodal knowledge graph provided in this application;

[0055] Figure 3 This is a schematic diagram of the structure of an embodiment of the cross-modal data retrieval system based on multimodal knowledge graph provided in this application;

[0056] Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0057] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0058] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0059] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0060] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0061] While existing methods have made progress in heterogeneous data fusion, the bottleneck of alignment-structure mutual exclusion still exists. That is, cross-modal contrastive learning dilutes key topological relationships when it forces text / image vectors to be close together, while graph structure injection contaminates visual features, leading to a decrease in the accuracy of image-text retrieval.

[0062] To address the issue of low accuracy in existing image and text retrieval technologies, this application proposes a cross-modal data retrieval method, system, and device based on multimodal knowledge graphs.

[0063] Reference Figure 1 This application provides a schematic flowchart of a cross-modal data retrieval method based on a multimodal knowledge graph. This method is applied to electronic devices, such as servers or mobile terminals. Figure 1 As shown, the cross-modal data retrieval method based on multimodal knowledge graphs may include the following steps:

[0064] Step S101: Preprocess the acquired text modal data and image modal data to obtain standardized text modal data and standardized image modal data;

[0065] Step S102: Extract text feature vectors from the standardized text modal data, and extract visual feature vectors and structural feature vectors from the standardized image modal data;

[0066] Step S103: Decouple features based on visual feature vectors and structural feature vectors to determine the target visual projection vector;

[0067] Step S104: Construct a path constraint contrastive learning loss function based on the target visual projection vector and text feature vector;

[0068] Step S105: Construct a multimodal knowledge graph between text and images, and mine effective paths between entities in the multimodal knowledge graph;

[0069] Step S106: Encode the effective path into path encoding features, and construct a multi-scale path-aware rejection loss function based on the path encoding features;

[0070] Step S107: Jointly optimize the path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function to determine the target path encoding feature subset;

[0071] Step S108: Fuse the text feature vector, the target visual projection vector, and the target path encoding feature subset to obtain a joint vector;

[0072] Step S109: Perform cross-modal data retrieval based on joint vectors to obtain retrieval results.

[0073] In this embodiment, the acquired text modal data and image modal data are preprocessed to obtain standardized text modal data and standardized image modal data; text feature vectors are extracted from the standardized text modal data, and visual feature vectors and structural feature vectors are extracted from the standardized image modal data; feature decoupling is performed based on the visual feature vectors and structural feature vectors to determine the target visual projection vector; a path constraint contrastive learning loss function is constructed based on the target visual projection vector and text feature vectors; a multimodal knowledge graph between text and images is constructed, and effective paths between entities in the multimodal knowledge graph are mined; effective paths are encoded as path encoding features, and a multi-scale path-aware rejection loss function is constructed based on the path encoding features; the path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function are jointly optimized to determine the target path encoding feature subset; the text feature vector, the target visual projection vector, and the target path encoding feature subset are fused to obtain a joint vector; cross-modal data retrieval is performed based on the joint vector to obtain the retrieval results. Thus, feature decoupling ensures semantic fidelity between text and images; path-constrained contrastive learning loss optimizes the similarity distribution of multimodal features to enable semantic alignment of multimodal data; multi-scale path-aware rejection loss forces the path feature distance between semantically associated positive entity pairs to be less than the path feature distance between positive and negative sample pairs, thereby enhancing the discriminativeness of path semantics; joint optimization of the path-constrained contrastive learning loss function and the multi-scale path-aware rejection loss function determines the optimal path encoding feature subset. By fusing text feature vectors, target visual projection vectors, and target path encoding feature subsets, and performing cross-modal data retrieval based on the joint vector, the inherent contradiction between cross-modal semantic alignment and graph structure preservation in multimodal knowledge graphs can be resolved, thus improving the accuracy of text-image retrieval.

[0074] The extraction of text feature vectors from standardized text modal data, and the extraction of visual and structural feature vectors from standardized image modal data, can be achieved by using the BERT-base model to extract text feature vectors from standardized text modal data, the ResNet-50 model to extract visual feature vectors from standardized image modal data, and a graph neural network to extract structural feature vectors from standardized image modal data. It should be noted that the BERT-base model, ResNet-50 model, and graph neural network used in this embodiment are models well-known to those skilled in the art, and will not be described in detail in this embodiment.

[0075] The above-mentioned encoding of effective paths into path encoding features can be achieved by first mapping each relation in the effective path to an embedding vector, and then using a bidirectional long short-term memory network (BiLSTM) to fuse the forward and backward sequence semantics of the path (such as the logical order of "birthplace → dynasty") to generate a unified vector that can represent the path semantics (i.e., path encoding features).

[0076] The aforementioned joint optimization of the path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function can be achieved by using projective gradient descent and backpropagation to jointly optimize the path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function.

[0077] In some implementations, feature decoupling is performed based on visual feature vectors and structural feature vectors to determine the target visual projection vector, including:

[0078] The visual feature vector and the structural feature vector are mapped to an orthogonal subspace by an orthogonal projection operation to obtain the initial visual projection vector and the initial structural projection vector.

[0079] Construct a mutual information constraint loss function for feature decoupling, and determine whether the initial visual projection vector and the initial structural projection vector satisfy the mutual information constraint loss function;

[0080] If the initial visual projection vector and the initial structural projection vector do not satisfy the mutual information constraint loss function, the gradient of the mutual information constraint loss function is updated through backpropagation to optimize the visual feature vector and structural feature vector extracted during the forward propagation process.

[0081] Based on the optimized visual feature vector and structural feature vector, orthogonal projection operation is used to obtain optimized initial visual projection vector and optimized initial structural projection vector. After the mutual information constraint loss function is satisfied, the optimized initial visual projection vector is used as the target visual projection vector.

[0082] In this embodiment, by constructing a mutual information constraint loss function for feature decoupling, it is possible to effectively decouple structural features and visual features in orthogonal space. By eliminating the coupling between structural features and visual features, the semantic fidelity of text and images can be guaranteed.

[0083] The above-mentioned orthogonal projection operation can refer to the projection of image space U and null space W into mutually orthogonal subspaces. It should be noted that the orthogonal projection operation in this embodiment is an operation known to those skilled in the art, and will not be described in detail in this embodiment.

[0084] In some implementations, a path-constrained contrastive learning loss function is constructed based on the target visual projection vector and the text feature vector, including:

[0085] ;

[0086] in, This represents the path constraint contrastive learning loss function. This represents the set of positive sample pairs that satisfy the path constraints. Represents the text feature vector. Represents the target visual projection vector. The cosine similarity function represents the feature pairs. Indicates the temperature coefficient. Indicates the number of negative samples. Represents text entities, Represents an image entity.

[0087] In this embodiment, the similarity distribution of multimodal features is optimized by constructing a path constraint contrastive learning loss function so as to enable semantic alignment of multimodal data.

[0088] In some implementations, mining effective paths between entities in a multimodal knowledge graph includes:

[0089] Preset semantic constraints;

[0090] We employ depth-first search and breadth-first search to mine potential connection paths between entities in a multimodal knowledge graph;

[0091] Extract effective paths between entities that satisfy semantic constraints from potential connection paths.

[0092] In this embodiment, effective paths that satisfy semantic constraints between entities are mined from potential connection paths. This ensures semantic relevance, filters relationships irrelevant to the target domain, and excludes some invalid entity type chains, thereby selecting effective paths and laying a good data foundation for improving data retrieval in the later stage.

[0093] In some implementations, a multi-scale path-aware repulsion loss function is constructed based on path encoding features, including:

[0094] ;

[0095] in, This represents the multi-scale path-aware rejection loss function. This represents the sampling weights for paths with different hop counts. Represents anchor point entity go through Path encoding features after skipping relational paths. Represents positive sample entities go through Path encoding features after skipping relational paths. Represents negative sample entities go through Path encoding features after skipping relational paths. Indicates the boundary margin.

[0096] In this embodiment, the path feature distance of positive sample entity pairs that are forced to be semantically associated by multi-scale path-aware exclusion loss is smaller than the path feature distance between positive and negative sample pairs, thereby enhancing the discriminativeness of path semantics.

[0097] In some implementations, the text feature vector, the target visual projection vector, and a subset of target path encoded features are fused to obtain a joint vector, including:

[0098] The text feature vector, the target visual projection vector, and the target path encoding feature subset are each subjected to layer normalization to obtain the layer normalized text feature vector, the layer normalized target visual projection vector, and the layer normalized target path encoding feature subset.

[0099] The text feature vector after layer normalization, the target visual projection vector after layer normalization, and the target path encoding feature subset after layer normalization are concatenated to obtain the joint vector.

[0100] In this embodiment, a joint vector is obtained by concatenating the layer-normalized text feature vector, the layer-normalized target visual projection vector, and the layer-normalized target path encoding feature subset. This maps all modal data (text and images) into vector representations in the same space, achieving deep semantic alignment and laying a solid data foundation for improving data retrieval in the future.

[0101] In some implementations, cross-modal data retrieval is performed based on joint vectors to obtain retrieval results, including:

[0102] The joint vector generated from each piece of data is stored in a vector database with the same spatial vector representation, and an approximate nearest neighbor index is established for the vector database.

[0103] The acquired modal data to be retrieved is encoded into a query vector represented by the same spatial vector;

[0104] In a vector database with an approximate nearest neighbor index, an approximate nearest neighbor search is used to perform cross-modal data retrieval on the query vector to obtain the retrieval results.

[0105] In this embodiment, the joint vector generated from each piece of data is stored in a vector database with the same spatial vector representation. In the vector database with an approximate nearest neighbor index, the approximate nearest neighbor search is used to perform cross-modal data retrieval on the query vector, which can improve the accuracy of image and text retrieval.

[0106] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0107] Multimodal knowledge graph vectorization is a core topic in the intersection of artificial intelligence and knowledge graph technology. Its goal is to model heterogeneous modal data (e.g., text, images, and videos) and their complex relationships through a unified vector space, supporting cross-modal semantic understanding and intelligent reasoning. Current multimodal knowledge graph construction technologies mainly revolve around two paradigms: independent embedding followed by fusion (such as the MCEAS model), which uses dedicated encoders (such as BERT for text and ResNet for images) to generate single-modal vectors, and then fuses them into entity vectors through an attention mechanism; and graph structure extension methods (such as the M3-Agent framework), which directly aggregates multimodal neighborhood information based on graph neural networks (such as GAT / R-GCN), preserving local graph structure.

[0108] While the two aforementioned methods have made progress in heterogeneous data fusion, they still suffer from the bottleneck of alignment-structure mutual exclusion. Specifically, cross-modal contrastive learning, by forcing text / image vectors closer together, dilutes key topological relationships, while graph structure injection contaminates visual features. Furthermore, cross-modal alignment requires minimizing the distance between text and image vectors, while structure preservation requires maximizing the vector distance between path-differentiated entities, leading to a metric conflict within the unified space. Therefore, existing methods result in a decrease in the accuracy of text-image retrieval.

[0109] To address the core contradiction in multimodal knowledge graph construction—namely, the disruption of graph structure by cross-modal semantic alignment and the injection of distorted visual features into the graph structure—this embodiment aims to design a topology-aware multimodal dynamic joint coding framework. Through a spatially decoupled path constraint alignment mechanism and intent-aware feature reorganization technology, it achieves accurate cross-modal semantic understanding and improves the accuracy of text and image retrieval. (Refer to...) Figure 2 The method in this embodiment specifically includes the following:

[0110] The system consists of two main steps: multimodal data preprocessing and feature decoupling, and dual-space joint modeling.

[0111] 1. Multimodal data preprocessing and feature decoupling.

[0112] This step is characterized by eliminating the coupling between structural features and visual features, ensuring the semantic fidelity of text and images, and laying a good data foundation for subsequent dual-space modeling.

[0113] (1) Multimodal data normalization: First, data preprocessing is performed to remove HTML / XML tags and functionally irrelevant noise characters (such as garbled characters), but special symbols with semantic value are retained (such as #, &, *, etc. to avoid destroying entity structure, for example, retaining "AT&T" or "C#"); Pointer Network is used to extract core information paragraphs of the text (i.e., text modal data), and TF-IDF (Term Frequency-Inverse Document Frequency) algorithm is introduced to achieve refined extraction of key semantics, resulting in normalized text modal data. Bilinear interpolation is used to scale the image (i.e., image modal data) to 224×224 to obtain normalized image modal data.

[0114] (2) Independent feature extraction: Text feature vectors are extracted from canonical text modal data using the BERT_base model. Image feature encoding is implemented based on the ResNet-50 model. The process begins by forward propagating the preprocessed input image (i.e., canonical image modality data) through layers 1 to 4 of the ResNet-50 model. Finally, the last convolutional layer outputs a high-dimensional feature map. Subsequently, global average pooling is performed on this feature map, aggregating the two-dimensional feature matrices of each channel into a single scalar value, ultimately generating a fixed-dimensional vector representation. To balance the model's general feature extraction capability with domain adaptability, a hierarchical parameter freezing strategy is adopted: the pre-trained parameters of the lower layers (layers 1 to 3) in ResNet-50 are fixed to retain its ability to extract general low-level features (such as texture and edges); at the same time, the parameters of the higher layer (layer 4) are allowed to be fine-tuned during training to learn high-level semantic features highly relevant to specific domains, thereby enhancing the model's adaptability to the target task; then, topological association information of entities is obtained through knowledge extraction and encoded using graph neural networks (such as GAT or R-GCN). First, the original node information and association relationships of the entities are embedded and mapped into initial vectors. Then, by aggregating the structural semantic information of the entity's 1-2 hop neighborhood, a structural feature vector that can represent the entity's topological attributes is generated. .

[0115] (3) Feature orthogonal decoupling: The original structural feature vectors are decoupled by orthogonal projection operation. With visual feature vectors Mapping each vector to an orthogonal subspace yields the projection vector. (i.e., the initial structure projection vector) and (i.e., the initial visual projection vectors), ensuring that the two satisfy orthogonality constraints in the spatial dimension (i.e., mutual information constraint loss). A mutual information constraint loss function is introduced. To control for statistical correlation between features:

[0116] ;

[0117] in, For loss weighting coefficients, express and The Jensen-Shannon divergence is used to measure the difference between the probability distributions of two objects. The calculation of the Jensen-Shannon divergence is based on the Kullback-Leibler divergence, and therefore can be defined as:

[0118] ;

[0119] in, and They are respectively and The probability distribution, The average distribution of the two, Let KL divergence be the threshold value. This loss function penalizes cases where the JSD divergence exceeds a threshold of 0.25, ensuring that the mutual information of the projected orthogonal features is below this threshold. Ultimately, this achieves effective decoupling of structural and visual features in the orthogonal subspace, resulting in... and (i.e., the target visual projection vector).

[0120] 2. Dual-space joint modeling.

[0121] This step aims to resolve the inherent contradiction between cross-modal semantic alignment and graph structure preservation in multimodal knowledge graphs.

[0122] (1) Alignment Subspace Modeling: Using the relationship paths between entities in the Knowledge Graph (KG) as constraints, positive sample pairs with semantic associations are selected, and the feature discriminativeness is enhanced by an adaptive temperature coefficient. A path constraint contrastive learning loss is constructed. To optimize the similarity distribution of multimodal features:

[0123] ;

[0124] in, The cosine similarity function for feature pairs; This is a temperature coefficient used to control the sharpness of the similarity distribution; The number of negative samples; The set of positive sample pairs that satisfy the path constraints is defined by the relational path constraints of entities in the knowledge graph:

[0125] ;

[0126] That is, if and only if the text entity With image entities When there are relationship paths in a knowledge graph with a length of no more than 5 hops... These are identified as positive sample pairs to ensure their semantic relevance. To enhance the discriminative power of features in contrastive learning, a temperature coefficient is designed. Follow The obtained similarity value The dynamically adjusted strategy is calculated using the following formula:

[0127] ;

[0128] When similarity variance When the feature discrimination is reduced, then Reducing the similarity variance enhances the contrastive learning's ability to distinguish features; conversely, increasing the similarity variance... When the feature discrimination is reduced (enhanced), then Increase the size to avoid overfitting.

[0129] (2) Structural subspace modeling: First, from unstructured multimodal data such as text and images, atomic facts (such as "Li Bai-birthplace-Suyab" and "Suyab-dynasty-Tang Dynasty") are extracted using entity extraction and relation extraction techniques to form symbolic triples in the form of "entity-relationship-entity". Then, based on the initial "entity-relationship" graph composed of these triples, potential connection paths between entities are mined using depth-first search (DFS) and breadth-first search (BFS) graph traversal algorithms. Semantic constraints (such as path length ≤ 5 hops to ensure semantic relevance, filtering relations such as "height" and "friends" that are irrelevant to the target domain, and excluding invalid entity type chains such as "person-relationship-number-relationship-color") are combined to select effective paths. Then, the semantic abstraction stage is entered. Each relation in the effective path is mapped to an embedding vector. The forward and backward sequence semantics of the path (such as the logical order of "birthplace → dynasty") are fused through a bidirectional long short-term memory network (BiLSTM) to generate a unified vector that can represent the semantics of the path. Then, relying on the domain ontology and pre-defined logical rules, specific paths are abstracted into general logical patterns (e.g., unifying "Li Bai - birthplace - Suyab - dynasty - Tang Dynasty" and "Du Fu - birthplace - Gongxian - dynasty - Tang Dynasty" into "person - birthplace - location - dynasty - dynasty"). Simultaneously, implicit relationships within the paths are extracted (e.g., abstracting the indirect association of "person → dynasty" from the above paths). Finally, these abstracted structured logical knowledge are integrated with the original symbolic triples to form a knowledge graph structure that combines topological connectivity and semantic logic. This structure is then injected into the vector space to enhance the distinguishability of entity paths in the knowledge graph, thereby reducing the loss of graph topological information. A multi-scale path-aware rejection loss mechanism is constructed. For more than two relational paths, apply exclusion constraints to entity features, forcing semantically related positive sample entity pairs. The path feature distance is smaller than that between positive and negative samples. The path feature distance is used to enhance the distinguishability of path semantics:

[0130] ;

[0131] in, Represents anchor point entity go through Path encoding features after skipping relational paths. Represents positive sample entities go through Path encoding features after skipping relational paths. Represents negative sample entities go through Path encoding features after skipping relational paths. These are the sampling weights for paths with different hop counts; This is the boundary margin, used to control the strictness of the exclusion constraint. Since multi-hop relation paths of entities in a knowledge graph have sequential semantic attributes, BiLSTM is used to encode the paths. For a relation path of length k for entity e... (from k relations) Composition), first, each relation Mapped to embedding vector Then, the embedding vectors are fused using BiLSTM. Finally, the encoded features of the path are output. The formula is:

[0132] ;

[0133] To cover relationship paths of different lengths and balance their impact, differentiated sampling weights are set for paths with 2–5 hops. and boundary margin The specific parameters are shown in Table 1 below:

[0134] Table 1 shows the path setting parameters for 2–5 hops.

[0135]

[0136] Sampling weight Decreasing with increasing path length (shorter paths have more direct semantics and contribute more significantly to discriminability); boundary margin The value increases with the path length (longer paths have more complex semantics and require stricter exclusion constraints to ensure feature discriminability).

[0137] Relying solely on relation embedding to encode paths can lead to the loss of semantic information about entity types (e.g., the type difference between "person" and "organization" can affect path semantics). Therefore, by incorporating entity type embeddings, the semantic expression of relation paths can be enhanced. (The final embedding calculation connecting the source entity src and the target entity dst is as follows:)

[0138] ;

[0139] in, , The weight matrix to be learned. It is a relationship The original embedding, , These are the type embeddings of the source entity `src` and the target entity `dst`, respectively. Entity type information is fused through vector concatenation operations, enabling the relationship embedding to more accurately reflect the semantic attributes of the path.

[0140] In multi-loss joint optimization scenarios and The gradients of the two types of losses may have severe directional conflicts, i.e. This leads to unstable training. To address this, Projected Gradient Descent (PGD) is used to reconcile gradients, updating the gradient... for:

[0141] ;

[0142] in, For dynamic parameters, t represents the training round, which gradually decreases from 0.7 as the training progresses, in order to achieve strong coordination of gradient direction in the early stage of training and weak intervention to ensure convergence in the later stage, thus balancing the gradient update of the two types of loss.

[0143] It should be noted that the projection gradient descent in this embodiment not only optimizes and The parameters in both types of loss functions can be automatically adjusted through global optimization based on projective gradient descent. This means that parameters in mutual information constraint loss, ResNet-50 models, graph neural networks, and bidirectional long short-term memory networks (BiLSTM) can also be adjusted via backpropagation combined with projective gradient descent. For example, when the projected features (i.e., the projected vectors) are... and When the mutual information constraint loss is not satisfied, the projection operation is not simply re-performed. Instead, the parameters are globally optimized through backpropagation and projection gradient descent. The gradient calculated by the mutual information constraint loss function simultaneously updates the front-end feature encoder (i.e., the ResNet-50 model and the graph neural network) and the learnable projection matrix parameters (i.e., the parameters in the orthogonal projection operation). This guides them to learn how to generate less coupled original features (i.e., the extracted visual feature vectors and structural feature vectors) and to find better projection directions, thereby guiding the orthogonal projection operation to obtain better projection vectors. and And make the projection vector satisfy the mutual information constraint loss.

[0144] 3. Dual-space fusion output and cross-modal retrieval.

[0145] Joint vector It is constructed by segmenting and concatenating text features, visual features, and optimal path structure features, and then using layer normalization. Its expression is as follows:

[0146] ;

[0147] in, This represents the subset of target path encoding features, i.e., the set of optimal path structure features.

[0148] Finally, generate a joint vector for each data point. All joint vectors are stored in the efficient vector database Milvus, and an Approximate Nearest Neighbor (ANN) index is created.

[0149] Through the above steps, all modal data (text and images) are mapped to vector representations within the same space, achieving deep semantic alignment. In practical applications, by encoding user queries (text, images, or mixed inputs) into query vectors within the same space and utilizing approximate nearest neighbor search technology, fast retrieval can be performed directly within this vector space (i.e., the vector database), thereby efficiently and accurately returning the most relevant cross-modal retrieval results and improving the efficiency and accuracy of cross-modal retrieval.

[0150] Compared with the prior art, the technical solution of this embodiment has the following advantages:

[0151] In independent subspaces, mutually exclusive objectives (cross-modal alignment and graph structure preservation) are optimized separately to improve retrieval accuracy. Path-constrained contrastive learning ensures that cross-modal semantic alignment does not compromise valuable topological relationships. Multi-scale path-aware exclusion loss is employed to explicitly encode 2- to 5-hop relational paths. BiLSTM aggregates path semantics and widens the vector distance between terminal entities of divergent paths, enabling the model to possess deep reasoning capabilities and effectively distinguish locally similar but globally different entities, thereby improving the accuracy of multimodal retrieval.

[0152] Reference Figure 3 This application also provides a cross-modal data retrieval system based on a multimodal knowledge graph, the system comprising:

[0153] The data processing unit 100 is used to preprocess the acquired text modal data and image modal data to obtain standardized text modal data and standardized image modal data.

[0154] The feature extraction unit 200 is used to extract text feature vectors from canonical text modal data, and to extract visual feature vectors and structural feature vectors from canonical image modal data.

[0155] The feature decoupling unit 300 is used to decouple features based on visual feature vectors and structural feature vectors to determine the target visual projection vector.

[0156] The first building unit 400 is used to construct a path constraint contrastive learning loss function based on the target visual projection vector and the text feature vector.

[0157] The path mining unit 500 is used to construct a multimodal knowledge graph between text and images, and to mine effective paths between entities in the multimodal knowledge graph.

[0158] The second building unit 600 is used to encode the effective path into path encoding features and to construct a multi-scale path-aware rejection loss function based on the path encoding features.

[0159] The joint optimization unit 700 is used to jointly optimize the path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function to determine the target path encoding feature subset.

[0160] The feature fusion unit 800 is used to fuse the text feature vector, the target visual projection vector, and the target path encoding feature subset to obtain a joint vector;

[0161] The data retrieval unit 900 is used to perform cross-modal data retrieval based on joint vectors and obtain retrieval results.

[0162] It should be noted that since the cross-modal data retrieval system based on multimodal knowledge graph in this embodiment is based on the same inventive concept as the cross-modal data retrieval method based on multimodal knowledge graph described above, the corresponding content in the method embodiment is also applicable to this system embodiment, and will not be described in detail here.

[0163] Reference Figure 4 This application also provides an electronic device, which includes:

[0164] At least one memory;

[0165] At least one processor;

[0166] At least one program;

[0167] The program is stored in memory, and the processor executes at least one program to implement the cross-modal data retrieval method based on multimodal knowledge graph described above in this disclosure.

[0168] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0169] The electronic devices according to embodiments of this application will now be described in detail.

[0170] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0171] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to execute the cross-modal data retrieval method based on multimodal knowledge graphs according to the embodiments of this disclosure.

[0172] The input / output interface 1800 is used to implement information input and output.

[0173] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0174] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0175] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0176] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described cross-modal data retrieval method based on a multimodal knowledge graph.

[0177] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0178] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0179] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0180] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0182] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0183] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0186] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The embodiments of this application have been described in detail above with reference to the accompanying drawings, but this application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of this application.

[0188] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A cross-modal data retrieval method based on multimodal knowledge graphs, characterized in that, The method includes: The acquired text modal data and image modal data are preprocessed to obtain standardized text modal data and standardized image modal data; Extract text feature vectors from the standardized text modal data, and extract visual feature vectors and structural feature vectors from the standardized image modal data; Based on the visual feature vector and the structural feature vector, feature decoupling is performed to determine the target visual projection vector; Based on the target visual projection vector and the text feature vector, a path constraint contrastive learning loss function is constructed. Construct a multimodal knowledge graph between text and images, and mine effective paths between entities in the multimodal knowledge graph; The effective path is encoded as a path encoding feature, and a multi-scale path-aware rejection loss function is constructed based on the path encoding feature; The path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function are jointly optimized to determine the target path encoding feature subset; The text feature vector, the target visual projection vector, and the target path encoding feature subset are fused to obtain a joint vector; Cross-modal data retrieval is performed based on the joint vector to obtain retrieval results.

2. The cross-modal data retrieval method based on multimodal knowledge graphs according to claim 1, characterized in that, The step of decoupling features based on the visual feature vector and the structural feature vector to determine the target visual projection vector includes: The visual feature vector and the structural feature vector are mapped to an orthogonal subspace by an orthogonal projection operation to obtain an initial visual projection vector and an initial structural projection vector. Construct a mutual information constraint loss function for feature decoupling, and determine whether the initial visual projection vector and the initial structural projection vector satisfy the mutual information constraint loss function; If the initial visual projection vector and the initial structural projection vector do not satisfy the mutual information constraint loss function, then the gradient of the mutual information constraint loss function is updated through backpropagation to optimize the visual feature vector and structural feature vector extracted during the forward propagation process. Based on the optimized visual feature vector and structural feature vector, an optimized initial visual projection vector and an optimized initial structural projection vector are obtained by orthogonal projection operation. After the mutual information constraint loss function is satisfied, the optimized initial visual projection vector is used as the target visual projection vector.

3. The cross-modal data retrieval method based on multimodal knowledge graphs according to claim 1, characterized in that, The construction of a path-constrained contrastive learning loss function based on the target visual projection vector and the text feature vector includes: ； in, This represents the path constraint contrastive learning loss function. This represents the set of positive sample pairs that satisfy the path constraints. Represents the text feature vector. Represents the target visual projection vector. The cosine similarity function represents the feature pairs. Indicates the temperature coefficient. Indicates the number of negative samples. Represents text entities, Represents an image entity.

4. The cross-modal data retrieval method based on multimodal knowledge graphs according to claim 1, characterized in that, The process of mining effective paths between entities in the multimodal knowledge graph includes: Preset semantic constraints; Depth-first search and breadth-first search are used to mine potential connection paths between entities in the multimodal knowledge graph; From the potential connection paths, extract the valid paths between entities that satisfy the semantic constraints.

5. The cross-modal data retrieval method based on multimodal knowledge graphs according to claim 1, characterized in that, The construction of a multi-scale path-aware rejection loss function based on the path encoding features includes: ； in, This represents the multi-scale path-aware rejection loss function. This represents the sampling weights for paths with different hop counts. Represents anchor point entity go through Path encoding features after skipping relational paths. Represents positive sample entities go through Path encoding features after skipping relational paths. Represents negative sample entities go through Path encoding features after skipping relational paths. Indicates the boundary margin.

6. The cross-modal data retrieval method based on multimodal knowledge graphs according to claim 1, characterized in that, The step of fusing the text feature vector, the target visual projection vector, and the target path encoding feature subset to obtain a joint vector includes: The text feature vector, the target visual projection vector, and the target path encoding feature subset are each subjected to layer normalization to obtain the layer normalized text feature vector, the layer normalized target visual projection vector, and the layer normalized target path encoding feature subset. The text feature vector after layer normalization, the target visual projection vector after layer normalization, and the target path encoding feature subset after layer normalization are concatenated to obtain a joint vector.

7. The cross-modal data retrieval method based on multimodal knowledge graphs according to claim 1, characterized in that, The cross-modal data retrieval based on the joint vector yields retrieval results, including: The joint vector generated from each piece of data is stored in a vector database with the same spatial vector representation, and an approximate nearest neighbor index is established for the vector database. The acquired modal data to be retrieved is encoded into a query vector represented by the same spatial vector; In a vector database with an approximate nearest neighbor index, an approximate nearest neighbor search is used to perform cross-modal data retrieval on the query vector to obtain retrieval results.

8. A cross-modal data retrieval system based on a multimodal knowledge graph, characterized in that, The system includes: The data processing unit is used to preprocess the acquired text modal data and image modal data to obtain standardized text modal data and standardized image modal data. The feature extraction unit is used to extract text feature vectors from the standardized text modal data, and to extract visual feature vectors and structural feature vectors from the standardized image modal data; A feature decoupling unit is used to perform feature decoupling based on the visual feature vector and the structural feature vector to determine the target visual projection vector; The first construction unit is used to construct a path constraint contrastive learning loss function based on the target visual projection vector and the text feature vector; The path mining unit is used to construct a multimodal knowledge graph between text and images, and to mine effective paths between entities in the multimodal knowledge graph. The second construction unit is used to encode the effective path into path encoding features and construct a multi-scale path-aware rejection loss function based on the path encoding features. The joint optimization unit is used to jointly optimize the path constraint contrastive learning loss function and the multi-scale path-aware rejection loss function to determine the target path encoding feature subset. The feature fusion unit is used to fuse the text feature vector, the target visual projection vector, and the target path encoding feature subset to obtain a joint vector; The data retrieval unit is used to perform cross-modal data retrieval based on the joint vector and obtain retrieval results.

9. An electronic device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the cross-modal data retrieval method based on a multimodal knowledge graph as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the cross-modal data retrieval method based on a multimodal knowledge graph as described in any one of claims 1 to 7.

Images