Multi-modal knowledge graph construction method in cross-media retrieval

Abstract

The embodiment of the invention provides a multi-modal knowledge graph construction method in cross-media retrieval. The method comprises the steps that extracted multi-modal features are mapped to a multi-modal feature space through a linear transformation layer; in the multi-modal feature space, the intra-modal attention weight of each modal feature is calculated according to a self-attention mechanism, the cross-modal attention weight of different modal features is calculated according to a cross attention mechanism, the two weights are fused to obtain a final fusion weight, each modal feature is weighted and input into a graph attention network, and the multi-modal feature is obtained. Obtaining a multi-modal fusion graph structure; performing semantic analysis on each modal feature, matching with a preset multi-modal semantic knowledge base, determining potential semantic association, performing semantic alignment on the multi-modal fusion graph structure according to a preset graph matching algorithm and the potential semantic association to obtain a multi-modal knowledge graph, performing cross-modal data retrieval according to the multi-modal knowledge graph, and performing cross-modal data retrieval according to the multi-modal knowledge graph. The multi-modal data cross-media retrieval method and device can improve the efficiency and accuracy of multi-modal data cross-media retrieval.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a method for constructing a multimodal knowledge graph in cross-media retrieval. Background Art

[0002] In today's era of rapid information development, cross-media data such as text, images, and audio are exploding, containing a wealth of information and knowledge. However, due to the unique feature representations and semantic structures of various media modalities, traditional single-modality retrieval methods face severe challenges and struggle to meet the comprehensiveness and accuracy requirements of cross-media retrieval.

[0003] As an emerging semantic representation structure, multimodal knowledge graphs provide a new solution for cross-media retrieval. However, building a multimodal knowledge graph is not an easy task and faces many technical challenges. First, how to effectively fuse the features of different modalities is a key issue. Data of different modalities have different representation methods and feature dimensions. How to unify and fuse these features so that they can jointly support cross-media retrieval is an urgent problem to be solved. Secondly, the semantic gap between modalities is also a difficult problem that needs to be overcome in building a multimodal knowledge graph. Data of different modalities may have semantic differences and gaps. How to narrow these differences and ensure the semantic accuracy of the knowledge graph in cross-media retrieval is another important technical challenge.

[0004] In summary, there is an urgent need for a method to construct a multimodal knowledge graph in cross-media retrieval to improve the efficiency and accuracy of cross-media retrieval of multimodal data. Summary of the Invention

[0005] In response to the problems in the prior art, the present application provides a method and device for constructing a multimodal knowledge graph in cross-media retrieval, which can improve the efficiency and accuracy of cross-media retrieval of multimodal data.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions: In a first aspect, the present application provides a method for constructing a multimodal knowledge graph in cross-media retrieval, comprising: Performing feature extraction on the multimodal features respectively, mapping each modal feature obtained after the feature extraction to a unified feature space through a linear transformation layer, and determining a corresponding multimodal feature space; In the multimodal feature space, self-attention calculation is performed on each modal feature according to the self-attention mechanism to determine the corresponding intra-modal attention weight, association calculation is performed on different modal features according to the cross-attention mechanism to determine the corresponding cross-modal attention weight, the intra-modal attention weight and the cross-modal attention weight are fused, and the modal features are weighted based on the final fused weight to determine the corresponding multimodal fusion graph structure; Perform semantic analysis on the modal features, match the semantic features after the semantic analysis with a preset multimodal semantic knowledge base, determine the potential semantic associations between the modal features, semantically align the multimodal fusion graph structure according to a preset graph matching algorithm and the potential semantic associations, determine the corresponding multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph.

[0007] Furthermore, the extracting features from the multimodal features separately includes: Performing semantic extraction on the text modality data according to the pre-trained language model to determine corresponding text modality features, wherein the text modality features include vocabulary, grammatical structure, and global semantic features; Performing multi-layer convolution and pooling operations on the image modality data according to a preset convolutional neural network to determine corresponding image modality features, wherein the image modality features include local image features and global image features; Acoustic features are extracted from the audio modal data according to a preset time-frequency transformation algorithm, and semantic features are extracted from the acoustic features obtained through the acoustic feature extraction according to a preset time series modeling network to determine the corresponding audio modal features.

[0008] Furthermore, mapping each modal feature obtained after the feature extraction to a unified feature space through a linear transformation layer to determine the corresponding multimodal feature space includes: Normalizing the text modal features, the image modal features, and the audio modal features, and projecting the normalized modal features into a shared feature space of the same dimension according to a learnable weight matrix; In the shared feature space, an activation function is applied to each modal feature after the projection to determine a corresponding multimodal feature space.

[0009] Furthermore, performing self-attention calculation on each modal feature according to the self-attention mechanism to determine the corresponding intra-modal attention weight includes: Performing matrix calculations on the modal features using trainable parameters to determine query vectors, key vectors, and value vectors corresponding to the modal features; Perform self-attention calculation on the feature elements within the same modality to determine the corresponding same-modality attention score; The value vector of the modality is weighted according to the same-modal attention score to determine the corresponding intra-modal attention weight.

[0010] Furthermore, the cross-attention mechanism is used to perform correlation calculations on features of different modalities to determine corresponding cross-modal attention weights, including: Different modalities are divided into target modalities and source modalities, and the query vector of the target modality is interacted with the key vector of the source modality to determine the corresponding cross-modal attention score; The value vector of the source modality is weighted according to the cross-modal attention score to determine the cross-modal attention weight of the corresponding target modality.

[0011] Furthermore, the semantic parsing of each modal feature includes: Perform named entity recognition and relationship extraction on text modal features to determine the corresponding structured semantic representation; Analyze the image modal features according to the preset visual semantic parsing model to determine the corresponding visual entities and spatial relationships; Based on acoustic event detection and speech semantic understanding, the audio modal features are analyzed to determine the corresponding high-level semantic labels.

[0012] Furthermore, the semantic alignment of the multimodal fusion graph structure according to the preset graph matching algorithm and the potential semantic association to determine the corresponding multimodal knowledge graph includes: Calculating semantic similarity between entities of different modalities on the multimodal fusion graph structure according to the latent semantic association, and determining a corresponding cross-modal association weight matrix; Performing cross-modal entity alignment on the cross-modal association weight matrix according to a preset graph matching algorithm to determine a corresponding optimal alignment path; The multimodal fusion graph structure is semantically aligned according to the optimal alignment path to determine the corresponding multimodal knowledge graph.

[0013] In a second aspect, the present application provides a multimodal knowledge graph construction device for cross-media retrieval, comprising: A multimodal feature extraction module is used to extract features from the multimodal features respectively, map the features of each modality obtained after the feature extraction to a unified feature space through a linear transformation layer, and determine the corresponding multimodal feature space; A multimodal fusion graph construction module is used to perform self-attention calculations on the modal features in the multimodal feature space according to the self-attention mechanism to determine the corresponding intra-modal attention weights, perform association calculations on the different modal features according to the cross-attention mechanism to determine the corresponding cross-modal attention weights, fuse the intra-modal attention weights and the cross-modal attention weights, and weight the modal features based on the final fused weights to determine the corresponding multimodal fusion graph structure; A multimodal knowledge graph determination module is used to perform semantic analysis on the modal features, match the semantic features after the semantic analysis with a preset multimodal semantic knowledge base, determine the potential semantic associations between the modal features, semantically align the multimodal fusion graph structure according to a preset graph matching algorithm and the potential semantic associations, determine the corresponding multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph.

[0014] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for constructing a multimodal knowledge graph in cross-media retrieval are implemented.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for constructing a multimodal knowledge graph in cross-media retrieval.

[0016] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for constructing a multimodal knowledge graph in cross-media retrieval.

[0017] It can be seen from the above technical solution that the present application provides a method and device for constructing a multimodal knowledge graph in cross-media retrieval, which maps the extracted multimodal features to the multimodal feature space through a linear transformation layer; in the multimodal feature space, the intra-modal attention weight of each modal feature is calculated according to the self-attention mechanism, and the cross-modal attention weights of different modal features are calculated according to the cross-attention mechanism, and the two are fused to obtain the final fusion weight, which is weighted for each modal feature and input into the graph attention network to obtain a multimodal fusion graph structure; each modal feature is semantically parsed and matched with a preset multimodal semantic knowledge base to determine the potential semantic association, and the multimodal fusion graph structure is semantically aligned according to the preset graph matching algorithm and the potential semantic association to obtain a multimodal knowledge graph, and cross-modal data retrieval is performed based on the multimodal knowledge graph, thereby improving the efficiency and accuracy of cross-media retrieval of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 This is a flow chart of a method for constructing a multimodal knowledge graph in cross-media retrieval in an embodiment of the present application; Figure 2 A structural diagram of a multimodal knowledge graph construction device for cross-media retrieval in an embodiment of the present application; Figure 3 Schematic diagram of the structure of the electronic device in the embodiment of the present application.

[0020] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0021] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] The acquisition, storage, use, and processing of data in the technical solution of this application comply with relevant regulations.

[0023] Considering that the semantic gap between data modalities of different modalities is a difficult problem that needs to be overcome in constructing a multimodal knowledge graph. The present application provides a method and device for constructing a multimodal knowledge graph in cross-media retrieval, which maps the extracted multimodal features to the multimodal feature space through a linear transformation layer; in the multimodal feature space, the intra-modal attention weight of each modal feature is calculated according to the self-attention mechanism, and the cross-modal attention weight of different modal features is calculated according to the cross-attention mechanism, and the two are fused to obtain the final fusion weight, which is weighted for each modal feature and input into the graph attention network to obtain a multimodal fusion graph structure; semantically parse each modal feature and match it with a preset multimodal semantic knowledge base to determine the potential semantic association, and semantically align the multimodal fusion graph structure according to the preset graph matching algorithm and the potential semantic association to obtain a multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph, thereby improving the efficiency and accuracy of cross-media retrieval of multimodal data.

[0024] In order to improve the efficiency and accuracy of cross-media retrieval of multimodal data, this application provides an embodiment of a method for constructing a multimodal knowledge graph in cross-media retrieval. Figure 1 The multimodal knowledge graph construction method in cross-media retrieval specifically includes the following contents: Step S101: performing feature extraction on the multimodal features respectively, mapping each modal feature obtained after the feature extraction to a unified feature space through a linear transformation layer, and determining a corresponding multimodal feature space; Optionally, in this embodiment, in order to construct a multimodal knowledge graph, we first perform multimodal feature extraction and map all features to the same vector space for subsequent feature alignment.

[0025] Specifically, for text modalities, text preprocessing is first performed, including lexical analysis, syntactic analysis, and part-of-speech tagging, to clean and standardize text data; a pre-trained word vector model (such as BERT) is used to convert the text into a vector representation, and at the same time, key semantic information in the text, such as entities and relationships, is extracted.

[0026] Specifically, for image modalities, convolutional neural networks (CNNs) are used to extract features from images. For example, network structures such as ResNet are used to extract local and global features of images.

[0027] Specifically, for audio modalities, audio feature extraction algorithms (such as Mel-frequency cepstral coefficients (MFCC)) are used to obtain the spectral features of the audio, and then deep learning models (such as long short-term memory networks (LSTM)) are used to further extract the semantic features of the audio.

[0028] Next, a fusion framework is constructed to map the extracted text, image, and audio features into a unified feature space.

[0029] Specifically, a linear transformation layer performs a weighted summation on the feature vectors of each modality, where the weight parameters are learned through training. Through the mapping of the linear transformation layer, features originally belonging to different modalities with different dimensions and semantic spaces are integrated into a unified feature space. In this space, features from different modalities can be represented and compared in a unified manner, providing a foundation for subsequent multimodal fusion operations.

[0030] Constructing a unified feature space is a key step in multimodal fusion. Only when features from different modalities are in the same space can we effectively mine the complementary information in multimodal data and improve the performance of multimodal tasks such as multimodal classification and retrieval.

[0031] Step S102: In the multimodal feature space, self-attention calculation is performed on each modal feature according to the self-attention mechanism to determine the corresponding intra-modal attention weight, correlation calculation is performed on different modal features according to the cross-attention mechanism to determine the corresponding cross-modal attention weight, the intra-modal attention weight and the cross-modal attention weight are fused, and the modal features are weighted based on the final fused weight to determine the corresponding multimodal fusion graph structure; Optionally, in this embodiment, an attention mechanism is used to dynamically assign weights according to the importance of different modal features, so that important modal features occupy a larger proportion in the fusion process.

[0032] Optionally, in this embodiment, a self-attention mechanism is employed for each modal feature. Its core is to calculate the similarity of each feature in a unified feature set. Specifically, each feature vector in the modal feature set is used as a query vector (Query), while all feature vectors in the entire modal feature set are used as key vectors (Key) and value vectors (Value).

[0033] By calculating the degree of similarity between the query vector and the key vector (using a dot product operation followed by scaling by the inverse of the square root of the key vector's dimension), an attention score matrix is ​​generated. This score matrix represents the degree of correlation between each feature vector and the other feature vectors. For example, in the text modality, the feature vector of a word is similar to the feature vectors of other words in the same text, thereby understanding the importance of this word in the entire text context.

[0034] The attention score matrix is ​​then normalized using the softmax function so that the sum of the attention scores corresponding to each eigenvector is 1, resulting in the intra-modality attention weight. This reflects the importance of each eigenvector relative to other eigenvectors within the modality, highlighting important eigenvectors while suppressing less important ones.

[0035] Optionally, in this embodiment, a cross-attention mechanism is employed between features of different modalities. The core of this mechanism is to calculate the similarity between different feature vectors. For example, in the text-image modality, if a word in the text has a high cross-modal attention weight with a feature of an object region in the image, then this indicates that the word is likely describing the object.

[0036] The cross-attention mechanism establishes semantic associations between different modalities. It enables the model to understand the correspondence between data from different modalities. For example, in image-text retrieval tasks, cross-attention allows the model to understand which parts of the image match the content in the text description, allowing it to more accurately retrieve images that match the text description.

[0037] Optionally, in this embodiment, the calculated intra-modal attention weight and cross-modal attention weight are fused. The weight coefficients are adjusted according to the actual task and data. For example, in some multimodal tasks with text as the main information, the weight coefficient of the intra-modal attention weight may be set relatively high, while in some tasks that emphasize the mutual complementarity between modalities, the weight coefficient of the cross-modal attention weight may be increased.

[0038] Optionally, in this embodiment, each modal feature is weighted according to the final fusion weight. Each modal feature vector is multiplied by the corresponding final fusion weight, so that the resulting weighted feature vector is more consistent with the overall semantic structure of the multimodal data.

[0039] The weighted feature vectors are input into the graph attention network, which treats the weighted modal features as node features in the graph. The connections between nodes are determined by the associations between modalities, such as the association between words in text and regions in an image. The graph attention network aggregates information from neighboring nodes by calculating attention coefficients between nodes, thereby updating the features of each node. For example, for a text node in a text-image fusion graph, its own features are updated based on the features of the image nodes connected to it and the attention coefficients between them, allowing the text features to better integrate with the image information.

[0040] Finally, the updated node feature set is obtained, which constitutes a graph structure of multimodal fusion of text, image and audio, which can more accurately represent the entity relationship and structural information between multimodal data, and provide effective data representation for subsequent multimodal tasks.

[0041] Step S103: Perform semantic analysis on the modal features, match the semantic features after the semantic analysis with a preset multimodal semantic knowledge base, determine the potential semantic associations between the modal features, semantically align the multimodal fusion graph structure according to a preset graph matching algorithm and the potential semantic associations, determine the corresponding multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph.

[0042] Optionally, in this embodiment, we deeply explore the intrinsic correlation between the features of each modality from a semantic level, and update the multimodal fusion graph structure based on this.

[0043] Optionally, in this embodiment, the core goal is to first convert the feature vectors of each modality after the previous feature extraction and fusion processing into a feature expression form with clear semantic meaning.

[0044] Specifically, for text modal features, semantic analysis uses natural language processing technology to abstract text words, sentences, etc. into semantic vectors to obtain semantic entities and relationships between entities.

[0045] Specifically, for image modal features, semantic analysis of the image is performed by combining computer vision technologies such as object detection, image segmentation, and scene understanding to form semantic features. For example, in an image of a city street scene, the semantic features obtained after semantic analysis may include elements such as "road," "building," "pedestrian," and "vehicle," their relative positions, and the semantic concept that the overall scene is a "city street."

[0046] Specifically, based on audio modal features, speech recognition technology first converts speech into text, then performs semantic analysis on the text to extract the core semantic information conveyed by the speech, such as command intent, conversation topic, etc.

[0047] Optionally, in this embodiment, after semantic parsing is completed, the obtained semantic features of each modality will be matched with a preset multimodal semantic knowledge base.

[0048] Specifically, the matching process is achieved by calculating the similarity between the semantic features and the semantic entries in the knowledge base.

[0049] For text semantic features, the cosine similarity measurement method is used to compare the closeness between the text semantic vector and the text topic vector in the knowledge base; For image semantic features, we use technologies such as the bag-of-visual-words model and similarity measurement networks in deep learning to match the semantics of image scenes or object categories in the knowledge base; For audio semantic features, the judgment is based on the similarity between the audio features and the audio category features in the knowledge base.

[0050] Specifically, based on the semantic matching results, we further explore the potential semantic associations between the features of each modality. In multimodal scenarios, different modalities often describe the same object or scene from different perspectives, and there are rich semantic associations between them. For example, in a multimodal dataset containing product images, product introduction text, and product promotional audio, the description of the product's "high-end and atmospheric" appearance mentioned in the text is interrelated with the product's exquisite appearance scene shown in the image, and the semantic concepts such as "excellent quality" promoted in the audio.

[0051] By analyzing the relationship between the matched knowledge base semantic entries, the parts with similar or related semantic concepts in each modal feature are identified.

[0052] Optionally, in this embodiment, after determining the potential semantic association between the features of each modality, a preset graph matching algorithm is used to perform semantic alignment on the multimodal fusion graph structure.

[0053] The core function of the graph matching algorithm is to adjust the connection mode of nodes and edges in the fusion graph structure based on semantic association information, so that the graph structure can more accurately reflect the semantic consistency between the features of each modality.

[0054] Specifically, the algorithm compares the strength of semantic associations between nodes with different modal features. For nodes with close semantic associations, the weight of the connection edges between them will be strengthened, and even new connection edges will be added when necessary. For nodes with weaker semantic associations or irrelevant nodes, the connection edge weights may be reduced or the connection edges may be deleted.

[0055] For example, in a multimodal data set containing news report text, news scene pictures and news anchor commentary audio, if the text and pictures semantically point to the theme of "major sporting events", and the audio also mentions this theme, the graph matching algorithm will strengthen the connection between the text, picture and audio nodes, ensuring that they are closely connected in the fusion graph structure, forming a semantically consistent subgraph structure around the theme of "major sporting events".

[0056] We integrate and optimize the nodes and edges in the fusion graph structure according to semantic hierarchy and semantic relationships. The semantically aligned multimodal fusion graph is ultimately transformed into a multimodal knowledge graph. Nodes with identical or similar semantic concepts are merged or grouped to avoid duplication and redundancy. For complex semantic relationships, we add intermediate semantic nodes or redefine the semantic types of edges to more clearly and accurately express semantic associations.

[0057] In cross-modal retrieval scenarios, users can enter a query in one modality (such as entering a picture), and the system can quickly locate other modal data associated with the semantics of the picture (such as related text descriptions, audio introductions, etc.) based on the knowledge graph, greatly improving the accuracy and efficiency of retrieval, allowing users to obtain information in multimodal data more comprehensively and deeply.

[0058] This example demonstrates how this embodiment constructs a multimodal knowledge graph through multimodal feature extraction, fusion, and semantic similarity matching, thereby improving the efficiency and accuracy of cross-media retrieval of multimodal data.

[0059] From the above description, it can be seen that the multimodal knowledge graph construction method in cross-media retrieval provided in the embodiment of the present application can map the extracted multimodal features to the multimodal feature space through a linear transformation layer; in the multimodal feature space, the intra-modal attention weight of each modal feature is calculated according to the self-attention mechanism, and the cross-modal attention weights of different modal features are calculated according to the cross-attention mechanism. The two are fused to obtain the final fusion weight, and each modal feature is weighted and input into the graph attention network to obtain a multimodal fusion graph structure; each modal feature is semantically parsed and matched with a preset multimodal semantic knowledge base to determine the potential semantic association, and the multimodal fusion graph structure is semantically aligned according to the preset graph matching algorithm and the potential semantic association to obtain a multimodal knowledge graph, and cross-modal data retrieval is performed based on the multimodal knowledge graph, thereby improving the efficiency and accuracy of cross-media retrieval of multimodal data.

[0060] In one embodiment of the method for constructing a multimodal knowledge graph in cross-media retrieval of this application, see Figure 2 , and can also include the following: Step S201: performing semantic extraction on text modality data according to a pre-trained language model to determine corresponding text modality features, wherein the text modality features include vocabulary, grammatical structure, and global semantic features; Step S202: performing multi-layer convolution and pooling operations on the image modality data according to a preset convolutional neural network to determine corresponding image modality features, wherein the image modality features include local image features and global image features; Step S203: performing acoustic feature extraction on the audio modal data according to a preset time-frequency transformation algorithm, performing semantic feature extraction on the acoustic features obtained through the acoustic feature extraction according to a preset time series modeling network, and determining corresponding audio modal features.

[0061] Optionally, in this embodiment, a pre-trained language model is used to process text modality data.

[0062] Specifically, the pre-trained language model is trained based on a large amount of text corpus.

[0063] First, the language model focuses on lexical features, including information about each word that appears in the text and its part of speech.

[0064] Secondly, the model analyzes the grammatical structure features, namely the combination relationship between words and the structural form of sentences.

[0065] Finally, the model also extracts global semantic features, that is, understanding the overall semantic information from the perspective of the entire text segment (such as a paragraph or an article).

[0066] Through this step, the obtained text modal features can more comprehensively reflect the semantic information of the text, providing a rich text semantic basis for subsequent multimodal fusion and other operations.

[0067] Optionally, in this embodiment, a preset convolutional neural network is used to extract image modality features.

[0068] Specifically, in the feature extraction process, multi-layer convolution operations are first performed. In the shallow convolution layer, local features such as the edges and textures of the image are mainly extracted; as the convolution layer deepens, the model will gradually extract more abstract and advanced features.

[0069] Next, a pooling operation is performed. The purpose of the pooling operation is to downsample the convolutional feature map, reducing the spatial size of the feature map while retaining important feature information.

[0070] Specifically, after multiple layers of convolution and pooling operations, the obtained image modal features contain both rich local image features, such as edges and textures, and global image features that can reflect the overall semantics of the image, such as object shape and spatial layout.

[0071] Optionally, in this embodiment, in terms of audio modal data processing, acoustic features of the audio data are first extracted using a preset time-frequency transformation algorithm to extract the acoustic features of the audio.

[0072] Specifically, after obtaining the acoustic features, semantic features are extracted from them using a preset time series modeling network. Audio signals are sequential data that changes over time. Therefore, a time series modeling network is used to process the temporal relationships in the audio data, identify the textual content corresponding to the speech, and thus extract the semantic features of the speech. Semantic features enable audio modal data to express its semantic information in a manner that is closer to how humans understand language and sound.

[0073] Through step S203, this embodiment extracts features from text, image, and audio data respectively, determines the features of each modality, and lays a solid data foundation for the subsequent construction of the knowledge graph.

[0074] In one embodiment of the method for constructing a multimodal knowledge graph in cross-media retrieval of the present application, the following contents may also be specifically included: Step S301: normalizing the text modal features, the image modal features, and the audio modal features, and projecting the normalized modal features into a shared feature space of the same dimension according to a learnable weight matrix; Step S302: In the shared feature space, an activation function is applied to each modal feature after the projection to determine a corresponding multimodal feature space.

[0075] Optionally, in this embodiment, normalization processing is performed on the text modal features, image modal features, and audio modal features respectively, and the numerical ranges of different modal features are adjusted to a relatively uniform interval.

[0076] Specifically, data from different modalities may have different scales and dimensions in their original state. Without normalization, subsequent processing may be affected by these numerical differences, causing features from certain modalities to dominate the calculation process while features from other modalities are neglected. For example, image modality features may contain large values ​​such as pixel values, while text modality features may be relatively small values ​​after encoding.

[0077] Optionally, in this embodiment, the normalized modal features are projected into a shared feature space of the same dimension according to a learnable weight matrix.

[0078] Specifically, a learnable weight matrix is ​​a parameterized method that learns the importance and correlation between features from different modalities. During the projection process, the learnable weight matrix automatically adjusts based on the training data to find the optimal way to map features from different modalities into a shared space.

[0079] Optionally, in this embodiment, in the shared feature space, an activation function is applied to each modal feature after projection to determine the corresponding multimodal feature space.

[0080] Specifically, the activation function introduces nonlinear factors into the model, enabling it to learn more complex feature representations and mapping relationships. For example, the semantic relationship between text and images is not a simple linear relationship; activation functions can reveal this complex semantic connection. Features processed by activation functions can better reflect the inherent characteristics of each modal feature in the shared space, thereby forming a richer and more effective multimodal feature space.

[0081] Through step S302 , this embodiment successfully constructs a multimodal feature space, laying a foundation for subsequent feature processing in the feature space.

[0082] In one embodiment of the method for constructing a multimodal knowledge graph in cross-media retrieval of the present application, the following contents may also be specifically included: Step S401: performing matrix calculation on each modal feature using trainable parameters to respectively determine a query vector, a key vector, and a value vector corresponding to each modal feature; Step S402: Perform self-attention calculation on the feature elements in the same modality to determine the corresponding same-modality attention score; Step S403: Weight the value vector of the modality according to the same-modal attention score to determine the corresponding intra-modal attention weight.

[0083] Optionally, in this embodiment, during multimodal feature processing, each modal feature must first be vectorized to facilitate subsequent attention mechanism calculations. Specifically, matrix calculations are performed on each modal feature using trainable parameters to determine the query vector, key vector, and value vector corresponding to each modal feature.

[0084] Specifically, for each mode’s characteristic matrix (Where m represents the modality type, n is the number of features, and d is the feature dimension) Calculate the query matrix Qm, key matrix Km, and value matrix Vm respectively.

[0085] Query Matrix , is a trainable parameter.

[0086] The query vector represents the query intent of the current feature when searching for related information. It is obtained by performing a linear transformation on the original modal features. The parameters of the linear transformation are trainable. Bond Matrix , is a trainable parameter.

[0087] The key vector is used to represent the "key information" contained in the feature element, which is used to match the query vector. Similarly, the key vector is obtained by linearly transforming the original modal features; Value Matrix , is a trainable parameter.

[0088] The role of the value vector is to represent the actual "value" or "content" of the feature element, which will be weighted and summed according to the attention weight to generate the final feature representation. The value vector is also obtained from the original modality feature through linear transformation.

[0089] Optionally, in this embodiment, after the query vector, key vector, and value vector are determined, self-attention calculation is performed on the feature elements within the same modality to determine the corresponding same-modality attention scores.

[0090] Specifically, calculate the self-attention score matrix , which is used to characterize the dependency between feature elements within the same modality, so that features within the same modality can influence and complement each other.

[0091] Specifically, the output modality enhancement features , multiply the value vector of each feature element by its corresponding intra-modal attention weight, and then sum the weighted value vectors of all feature elements to obtain a comprehensive feature representation of the modality.

[0092] Through step S403, this embodiment successfully realizes the self-attention calculation of each modal feature and the determination of the intra-modal attention weight, generates a comprehensive feature representation that can better represent the features of each modal, and provides an effective intra-modal feature foundation for subsequent cross-modal interaction and fusion.

[0093] In one embodiment of the method for constructing a multimodal knowledge graph in cross-media retrieval of the present application, the following contents may also be specifically included: Step S501: Different modalities are divided into target modalities and source modalities, and the query vector of the target modality is interacted with the key vector of the source modality to determine the corresponding cross-modal attention score; Step S502: Weight the value vector of the source modality according to the cross-modal attention score to determine the cross-modal attention weight of the corresponding target modality.

[0094] Optionally, in this embodiment, in order to achieve effective fusion and interaction between different modalities, these modalities are divided into "target modalities" and "Source Modality" The target modality is the main modality that we want to obtain information from, while the source modality is the modality used to assist the target modality in obtaining information.

[0095] The query matrix Q of the target modality m Bond matrix with source modality Interact and calculate the cross-modal attention score matrix ; Aggregate the value matrix of the source modality based on the cross-modal attention score , generate cross-modal context features:

[0096] Repeat the above steps for all source modalities to obtain multiple sets of cross-modal context features of the target modality m.

[0097] The modal attention weight reflects the most relevant and useful information that the target modality extracts from the source modality. It not only retains important information from the source modality but also optimizes it based on the needs of the target modality. This weighting mechanism makes multimodal data fusion more efficient and targeted, better meeting the information needs of the target modality.

[0098] Through step S502, this embodiment successfully determines the cross-modal attention weight, so that the target modality can clearly express the demand for source modality information, enhancing the ability to understand and utilize different modal data.

[0099] In one embodiment of the method for constructing a multimodal knowledge graph in cross-media retrieval of the present application, the following contents may also be specifically included: Step S601: performing named entity recognition and relationship extraction on text modal features to determine corresponding structured semantic representations; Step S602: parsing the image modal features according to a preset visual semantic parsing model to determine the corresponding visual entities and spatial relationships; Step S603: Analyze the audio modal features based on acoustic event detection and speech semantic understanding to determine the corresponding high-level semantic labels.

[0100] Optionally, in this embodiment, named entity recognition and relationship extraction are performed on the text modal features. Through these two steps of named entity recognition and relationship extraction, the key information in the text can be presented in the form of a structured semantic representation. For example, sentence information can be represented as structured data to facilitate subsequent processing and application.

[0101] Optionally, in this embodiment, the visual semantic parsing model is trained based on a large amount of image data, and it can understand various semantic information in the image.

[0102] First, the model extracts features from the image. These features can be low-level visual features such as color, texture, and shape, or higher-level semantic features. Using these features, the model can identify visual entities in the image. Visual entities can be various objects in the image, such as people, animals, vehicles, and buildings. For example, in an image of a city street, the model can identify different visual entities such as pedestrians, cars, streetlights, and buildings. The recognition of these entities is achieved by the model learning and matching image features. During training, the model learns the characteristic patterns of different objects and can find matching areas in new images to determine the presence of these objects.

[0103] In addition to identifying visual entities, the model also determines the spatial relationships between these entities. Spatial relationships refer to the positional relationships between different objects in an image, such as "to the left of," "above," or "around." For example, in an image of a living room, the model can identify objects such as a sofa, a coffee table, and a TV, and can determine that the coffee table is in front of the sofa and the TV is opposite the sofa. These spatial relationships are determined by analyzing information such as the coordinate position and relative size of the objects in the image. This visual semantic parsing allows the content of the image to be expressed in a semantic form, making the image not just a collection of pixels but visual information with clear semantic meaning. This plays a crucial role in image understanding and image retrieval.

[0104] Optionally, in this embodiment, the processing of audio data involves two aspects: acoustic event detection and speech semantic understanding.

[0105] Acoustic event detection primarily identifies various sound events in audio. It analyzes the audio signal's characteristics to extract time-domain features (such as energy and zero-crossing rate) and frequency-domain features (such as spectrum and Mel-Frequency Cepstral Coefficients (MFCCs). These features are then used to determine whether a specific acoustic event is present in the audio.

[0106] Speech semantic understanding processes the speech portion of audio. When the audio contains human speech, the goal of speech semantic understanding is to understand the semantic meaning of the speaker's words. This requires first performing speech recognition to convert the speech signal into text, and then using natural language processing techniques to perform semantic analysis on the text.

[0107] Through step S603, this embodiment successfully implements semantic analysis of multimodal features, laying the foundation for subsequent updating of the knowledge graph based on semantic associations.

[0108] In one embodiment of the method for constructing a multimodal knowledge graph in cross-media retrieval of the present application, the following contents may also be specifically included: Step S701: Calculating semantic similarity between entities of different modalities in the multimodal fusion graph structure according to the latent semantic association, and determining a corresponding cross-modal association weight matrix; Step S702: performing cross-modal entity alignment on the cross-modal association weight matrix according to a preset graph matching algorithm to determine a corresponding optimal alignment path; Step S703: semantically align the multimodal fusion graph structure according to the optimal alignment path to determine the corresponding multimodal knowledge graph.

[0109] Optionally, in this embodiment, semantic similarity is calculated for each pair of entities in different modalities within the multimodal fusion graph structure based on latent semantic associations. A cross-modal association weight matrix is ​​constructed based on these semantic similarity values. The rows and columns of the matrix correspond to entities in different modalities, and each element in the matrix represents the semantic similarity between the corresponding pairs of entities in different modalities. This matrix intuitively reflects the strength of associations between entities in different modalities within the multimodal fusion graph structure, providing a precise quantitative basis for subsequent alignment operations.

[0110] Optionally, in this embodiment, after obtaining the cross-modal association weight matrix, the next task is to use a preset graph matching algorithm to process this matrix to determine the optimal cross-modal entity alignment path.

[0111] In practice, we treat entities from different modalities as two sets of a bipartite graph, using the elements in the association weight matrix as edge weights. We iteratively search for augmenting paths using the Hungarian algorithm, gradually constructing the optimal matching solution. This allows us to select the optimal alignment path from among numerous possible cross-modal entity alignment solutions.

[0112] For example, in a multimodal data scenario containing text descriptions and image content, this step can ensure that the objects mentioned in the text are matched with the corresponding object entities in the image in the best way, laying a solid foundation for the subsequent construction of an accurate multimodal knowledge graph.

[0113] Optionally, in this embodiment, the different modal entities in the multimodal fusion graph structure are reorganized and adjusted according to the optimal alignment path. Under the guidance of the optimal alignment path, different modal entities with high semantic relevance are aligned and connected in the graph structure to form a semantically coherent and clearly structured multimodal knowledge graph.

[0114] For example, if the optimal alignment path indicates that "car" in the text is associated with the "car" shaped object in the image and the "car horn" segment in the audio, then in the multimodal knowledge graph, these three entities of different modalities will be tightly connected together to form a complete semantic unit.

[0115] Through step S702, this embodiment successfully obtains a multimodal knowledge graph, improving the efficiency and accuracy of cross-media retrieval of multimodal data.

[0116] In order to improve the efficiency and accuracy of cross-media retrieval of multimodal data, the present application provides an embodiment of a multimodal knowledge graph construction device in cross-media retrieval for realizing all or part of the content of the multimodal knowledge graph construction method in cross-media retrieval, see Figure 2 The multimodal knowledge graph construction device in cross-media retrieval specifically includes the following contents: The multimodal feature extraction module 10 is used to extract features from the multimodal features respectively, map the features of each modality obtained after the feature extraction to a unified feature space through a linear transformation layer, and determine the corresponding multimodal feature space; A multimodal fusion graph construction module 20 is configured to perform self-attention calculations on each modal feature in a multimodal feature space according to a self-attention mechanism to determine corresponding intra-modal attention weights, perform association calculations on different modal features according to a cross-attention mechanism to determine corresponding cross-modal attention weights, fuse the intra-modal attention weights and the cross-modal attention weights, and weight each modal feature based on the final fused weights to determine a corresponding multimodal fusion graph structure. The multimodal knowledge graph determination module 30 is used to perform semantic analysis on the modal features, match the semantic features after the semantic analysis with a preset multimodal semantic knowledge base, determine the potential semantic associations between the modal features, semantically align the multimodal fusion graph structure according to a preset graph matching algorithm and the potential semantic associations, determine the corresponding multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph.

[0117] From the above description, it can be seen that the multimodal knowledge graph construction device in cross-media retrieval provided by the embodiment of the present application can map the extracted multimodal features to the multimodal feature space through a linear transformation layer; in the multimodal feature space, the intra-modal attention weight of each modal feature is calculated according to the self-attention mechanism, and the cross-modal attention weights of different modal features are calculated according to the cross-attention mechanism. The two are fused to obtain the final fusion weight, and each modal feature is weighted and input into the graph attention network to obtain a multimodal fusion graph structure; each modal feature is semantically parsed and matched with a preset multimodal semantic knowledge base to determine the potential semantic association, and the multimodal fusion graph structure is semantically aligned according to the preset graph matching algorithm and the potential semantic association to obtain a multimodal knowledge graph, and cross-modal data retrieval is performed based on the multimodal knowledge graph, thereby improving the efficiency and accuracy of cross-media retrieval of multimodal data.

[0118] From a hardware perspective, in order to improve the efficiency and accuracy of cross-media retrieval of multimodal data, this application provides an embodiment of an electronic device for implementing all or part of the method for constructing a multimodal knowledge graph in cross-media retrieval. The electronic device specifically includes the following: Processor, memory, communication interface, and bus; wherein the processor, memory, and communication interface communicate with each other via the bus; the communication interface is used to implement information transmission between the multimodal knowledge graph construction method in cross-media retrieval and related devices such as core business systems, user terminals, and related databases; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited to this. In this embodiment, the logic controller can be implemented with reference to the embodiment of the multimodal knowledge graph construction method in cross-media retrieval and the embodiment of the multimodal knowledge graph construction method in cross-media retrieval in the embodiment, and their contents are merged here, and repeated parts are not repeated.

[0119] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0120] In practical applications, part of the multimodal knowledge graph construction method in cross-media retrieval can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0121] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a distributed server configuration.

[0122] Figure 3 Schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that the Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0123] In one embodiment, the multimodal knowledge graph construction method function in cross-media retrieval can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control: Step S101: performing feature extraction on the multimodal features respectively, mapping each modal feature obtained after the feature extraction to a unified feature space through a linear transformation layer, and determining a corresponding multimodal feature space; Step S102: In the multimodal feature space, self-attention calculation is performed on each modal feature according to the self-attention mechanism to determine the corresponding intra-modal attention weight, correlation calculation is performed on different modal features according to the cross-attention mechanism to determine the corresponding cross-modal attention weight, the intra-modal attention weight and the cross-modal attention weight are fused, and the modal features are weighted based on the final fused weight to determine the corresponding multimodal fusion graph structure; Step S103: Perform semantic analysis on the modal features, match the semantic features after the semantic analysis with a preset multimodal semantic knowledge base, determine the potential semantic associations between the modal features, semantically align the multimodal fusion graph structure according to a preset graph matching algorithm and the potential semantic associations, determine the corresponding multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph.

[0124] From the above description, it can be seen that the electronic device provided in the embodiment of the present application maps the extracted multimodal features to the multimodal feature space through a linear transformation layer; in the multimodal feature space, the intra-modal attention weight of each modal feature is calculated according to the self-attention mechanism, and the cross-modal attention weights of different modal features are calculated according to the cross-attention mechanism. The two are fused to obtain the final fusion weight, which is weighted for each modal feature and input into the graph attention network to obtain a multimodal fusion graph structure; each modal feature is semantically parsed and matched with a preset multimodal semantic knowledge base to determine the potential semantic association, and the multimodal fusion graph structure is semantically aligned according to the preset graph matching algorithm and the potential semantic association to obtain a multimodal knowledge graph, and cross-modal data retrieval is performed based on the multimodal knowledge graph, thereby improving the efficiency and accuracy of cross-media retrieval of multimodal data.

[0125] In another embodiment, the multimodal knowledge graph construction method in cross-media retrieval can be configured separately from the central processing unit 9100. For example, the multimodal knowledge graph construction method in cross-media retrieval can be configured as a chip connected to the central processing unit 9100, and the function of the multimodal knowledge graph construction method in cross-media retrieval is realized through the control of the central processing unit.

[0126] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 3 In addition, the electronic device 9600 may also include all components shown in Figure 3 For components not shown, reference may be made to the prior art.

[0127] like Figure 3 As shown, the central processing unit 9100 is sometimes also referred to as a controller or operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0128] Memory 9140 can be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store the aforementioned failure-related information and also store programs that execute the relevant information. The CPU 9100 can execute the programs stored in memory 9140 to implement information storage or processing.

[0129] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 may be, for example, a keypad or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0130] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), or SIM card. Alternatively, it may be a memory that retains information even when power is off, can be selectively erased, and is capable of storing additional data. Examples of such memory are sometimes referred to as EPROMs. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs, or processes used by the central processing unit 9100 to execute operations of the electronic device 9600.

[0131] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, images, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various driver programs for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0132] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0133] Based on different communication technologies, multiple communication modules 9110 can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module. The communication module 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing common telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is also coupled to the central processing unit 9100, enabling local recording via the microphone 9132 and playback of stored audio via the speaker 9131.

[0134] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the method for constructing a multimodal knowledge graph in cross-media retrieval in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the computer program implements all steps of the method for constructing a multimodal knowledge graph in cross-media retrieval in the above-mentioned embodiment, where the execution subject is a server or a client. For example, when the processor executes the computer program, the following steps are implemented: Step S101: performing feature extraction on the multimodal features respectively, mapping each modal feature obtained after the feature extraction to a unified feature space through a linear transformation layer, and determining a corresponding multimodal feature space; Step S102: In the multimodal feature space, self-attention calculation is performed on each modal feature according to the self-attention mechanism to determine the corresponding intra-modal attention weight, correlation calculation is performed on different modal features according to the cross-attention mechanism to determine the corresponding cross-modal attention weight, the intra-modal attention weight and the cross-modal attention weight are fused, and the modal features are weighted based on the final fused weight to determine the corresponding multimodal fusion graph structure; Step S103: Perform semantic analysis on the modal features, match the semantic features after the semantic analysis with a preset multimodal semantic knowledge base, determine the potential semantic associations between the modal features, semantically align the multimodal fusion graph structure according to a preset graph matching algorithm and the potential semantic associations, determine the corresponding multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph.

[0135] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application maps the extracted multimodal features to the multimodal feature space through a linear transformation layer; in the multimodal feature space, the intra-modal attention weight of each modal feature is calculated according to the self-attention mechanism, and the cross-modal attention weights of different modal features are calculated according to the cross-attention mechanism. The two are fused to obtain the final fusion weight, which is weighted for each modal feature and input into the graph attention network to obtain a multimodal fusion graph structure; each modal feature is semantically parsed and matched with a preset multimodal semantic knowledge base to determine the potential semantic association, and the multimodal fusion graph structure is semantically aligned according to the preset graph matching algorithm and the potential semantic association to obtain a multimodal knowledge graph, and cross-modal data retrieval is performed based on the multimodal knowledge graph, thereby improving the efficiency and accuracy of cross-media retrieval of multimodal data.

[0136] The embodiments of the present application also provide a computer program product capable of implementing all steps of the method for constructing a multimodal knowledge graph in cross-media retrieval in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instructions are executed by a processor, the computer program / instructions implement the steps of the method for constructing a multimodal knowledge graph in cross-media retrieval. For example, the computer program / instructions implement the following steps: Step S101: performing feature extraction on the multimodal features respectively, mapping each modal feature obtained after the feature extraction to a unified feature space through a linear transformation layer, and determining a corresponding multimodal feature space; Step S102: In the multimodal feature space, self-attention calculation is performed on each modal feature according to the self-attention mechanism to determine the corresponding intra-modal attention weight, correlation calculation is performed on different modal features according to the cross-attention mechanism to determine the corresponding cross-modal attention weight, the intra-modal attention weight and the cross-modal attention weight are fused, and the modal features are weighted based on the final fused weight to determine the corresponding multimodal fusion graph structure; Step S103: Perform semantic analysis on the modal features, match the semantic features after the semantic analysis with a preset multimodal semantic knowledge base, determine the potential semantic associations between the modal features, semantically align the multimodal fusion graph structure according to a preset graph matching algorithm and the potential semantic associations, determine the corresponding multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph.

[0137] From the above description, it can be seen that the computer program product provided in the embodiment of the present application maps the extracted multimodal features to the multimodal feature space through a linear transformation layer; in the multimodal feature space, the intra-modal attention weight of each modal feature is calculated according to the self-attention mechanism, and the cross-modal attention weight of different modal features is calculated according to the cross-attention mechanism. The two are fused to obtain the final fusion weight, which is weighted for each modal feature and input into the graph attention network to obtain a multimodal fusion graph structure; each modal feature is semantically parsed and matched with a preset multimodal semantic knowledge base to determine the potential semantic association, and the multimodal fusion graph structure is semantically aligned according to the preset graph matching algorithm and the potential semantic association to obtain a multimodal knowledge graph, and cross-modal data retrieval is performed based on the multimodal knowledge graph, thereby improving the efficiency and accuracy of cross-media retrieval of multimodal data.

[0138] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatuses, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0139] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0142] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for constructing a multimodal knowledge graph in cross-media retrieval, characterized in that: The method comprises: Perform feature extraction on the multimodal features respectively, map the features of each modality obtained after feature extraction to a unified feature space through a linear transformation layer, and determine the corresponding multimodal feature space; In the multimodal feature space, self-attention is calculated for each modal feature according to the self-attention mechanism to determine the corresponding intra-modal attention weight, and correlation calculation is performed on different modal features according to the cross-attention mechanism to determine the corresponding cross-modal attention weight. The intra-modal attention weight and the cross-modal attention weight are fused, and the modal features are weighted based on the final fused weight to determine the corresponding multimodal fusion graph structure; Perform semantic analysis on the modal features, match the semantic features after the semantic analysis with a preset multimodal semantic knowledge base, determine the potential semantic associations between the modal features, semantically align the multimodal fusion graph structure according to a preset graph matching algorithm and the potential semantic associations, determine the corresponding multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph.

2. The method for constructing a multimodal knowledge graph in cross-media retrieval according to claim 1, characterized in that: The extracting features of the multimodal features respectively includes: Performing semantic extraction on the text modality data according to the pre-trained language model to determine corresponding text modality features, wherein the text modality features include vocabulary, grammatical structure, and global semantic features; Performing multi-layer convolution and pooling operations on the image modality data according to a preset convolutional neural network to determine corresponding image modality features, wherein the image modality features include local image features and global image features; Acoustic features are extracted from the audio modal data according to a preset time-frequency transformation algorithm, and semantic features are extracted from the acoustic features obtained through the acoustic feature extraction according to a preset time series modeling network to determine the corresponding audio modal features.

3. The method for constructing a multimodal knowledge graph in cross-media retrieval according to claim 2, characterized in that: Mapping each modal feature obtained after the feature extraction to a unified feature space through a linear transformation layer to determine the corresponding multimodal feature space includes: Normalizing the text modal features, the image modal features, and the audio modal features, and projecting the normalized modal features into a shared feature space of the same dimension according to a learnable weight matrix; In the shared feature space, an activation function is applied to each modal feature after the projection to determine a corresponding multimodal feature space.

4. The method for constructing a multimodal knowledge graph in cross-media retrieval according to claim 1, characterized in that: The self-attention calculation is performed on each modal feature according to the self-attention mechanism to determine the corresponding intra-modal attention weight, including: Performing matrix calculations on the modal features using trainable parameters to determine query vectors, key vectors, and value vectors corresponding to the modal features; Perform self-attention calculation on the feature elements within the same modality to determine the corresponding same-modality attention score; The value vector of the modality is weighted according to the same-modal attention score to determine the corresponding intra-modal attention weight.

5. The method for constructing a multimodal knowledge graph in cross-media retrieval according to claim 4, characterized in that: The cross-attention mechanism is used to associate features of different modalities and determine corresponding cross-modal attention weights, including: Different modalities are divided into target modalities and source modalities, and the query vector of the target modality is interacted with the key vector of the source modality to determine the corresponding cross-modal attention score; The value vector of the source modality is weighted according to the cross-modal attention score to determine the cross-modal attention weight of the corresponding target modality.

6. The method for constructing a multimodal knowledge graph in cross-media retrieval according to claim 2, characterized in that: The semantic parsing of each modal feature includes: Perform named entity recognition and relationship extraction on text modal features to determine the corresponding structured semantic representation; Analyze the image modal features according to the preset visual semantic parsing model to determine the corresponding visual entities and spatial relationships; Based on acoustic event detection and speech semantic understanding, the audio modal features are analyzed to determine the corresponding high-level semantic labels.

7. The method for constructing a multimodal knowledge graph in cross-media retrieval according to claim 6, characterized in that: The semantic alignment of the multimodal fusion graph structure according to the preset graph matching algorithm and the potential semantic association to determine the corresponding multimodal knowledge graph includes: Calculating semantic similarity between entities of different modalities on the multimodal fusion graph structure according to the latent semantic association, and determining a corresponding cross-modal association weight matrix; Performing cross-modal entity alignment on the cross-modal association weight matrix according to a preset graph matching algorithm to determine a corresponding optimal alignment path; The multimodal fusion graph structure is semantically aligned according to the optimal alignment path to determine the corresponding multimodal knowledge graph.

8. A multimodal knowledge graph construction device for cross-media retrieval, characterized in that: The device comprises: A multimodal feature extraction module is used to extract features from the multimodal features respectively, map the features of each modality obtained after the feature extraction to a unified feature space through a linear transformation layer, and determine the corresponding multimodal feature space; A multimodal fusion graph construction module is used to perform self-attention calculations on each modal feature in the multimodal feature space according to the self-attention mechanism to determine the corresponding intra-modal attention weights, perform association calculations on different modal features according to the cross-attention mechanism to determine the corresponding cross-modal attention weights, fuse the intra-modal attention weights and the cross-modal attention weights, and weight the modal features based on the final fused weights to determine the corresponding multimodal fusion graph structure; A multimodal knowledge graph determination module is used to perform semantic analysis on the modal features, match the semantic features after the semantic analysis with a preset multimodal semantic knowledge base, determine the potential semantic associations between the modal features, semantically align the multimodal fusion graph structure according to a preset graph matching algorithm and the potential semantic associations, determine the corresponding multimodal knowledge graph, and perform cross-modal data retrieval based on the multimodal knowledge graph.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for constructing a multimodal knowledge graph in cross-media retrieval according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing a multimodal knowledge graph in cross-media retrieval according to any one of claims 1 to 7 are implemented.

Images