Media asset retrieval method and system realized by adopting multi-modal fusion technology

Abstract

The invention discloses a media asset retrieval method and system realized by adopting a multi-modal fusion technology, and relates to the technical field of knowledge maps, and the method comprises the following steps: obtaining video, audio and text original features in a media asset high-quality data set, and identifying a target entity from the original features; mapping a target entity as a semantic anchor point to a knowledge graph, obtaining a logic association relationship between nodes, and performing cross-modal semantic completion on the original features; injecting the logic association relationship as an offset parameter into a cross attention model, and performing weight remodeling and deep fusion on the aligned multi-modal features to generate a fusion feature vector; and constructing a semantic index based on the fused feature vector, responding to a retrieval instruction and outputting a retrieval result after semantic generalization matching. According to the method, semantic anchor point capture driven by the knowledge graph is introduced, so that logic faults and retrieval precision bottlenecks of the media asset data in the multi-modal fusion process are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph technology, and in particular to a media asset retrieval method and system that employs multimodal fusion technology. Background Technology

[0002] With the acceleration of global digital transformation and the widespread adoption of high-bandwidth network technology, internet media asset data is experiencing explosive growth. Heterogeneous data, encompassing visual images, audio signals, and unstructured text, has become the main carrier of information. Traditional media asset retrieval technologies have evolved from keyword retrieval based on manual annotation to feature vector retrieval based on deep learning. Current common solutions mostly rely on attention mechanisms under large-scale pre-trained models for feature interaction, attempting to mine statistical correlations between modalities in massive amounts of data through end-to-end learning, thereby constructing a high-dimensional joint representation space.

[0003] However, existing multimodal fusion methods suffer from a mismatch between perceptual features and deep logical semantics. The lack of precise anchoring of the mapping relationship between underlying physical signals and high-level logical entities leads to a significant decrease in alignment accuracy between modalities when processing media asset data with complex scenes or strong noise interference, easily resulting in semantic shifts. At the retrieval and matching level, existing technologies are mostly limited to Euclidean distance or cosine similarity calculations in vector space, with limited ability to capture users' fuzzy search intentions and a lack of semantic generalization capabilities based on knowledge topology structures. This results in retrieval results often limited to literal matching or shallow feature similarity, failing to meet the increasingly complex needs of deep semantic retrieval. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a media asset retrieval method using multimodal fusion technology to solve the problems of existing media asset retrieval technologies, such as fragile cross-modal alignment logic, lack of completion ability when information is damaged, lack of knowledge guidance in the fusion process, and insufficient generalization ability of retrieval intent.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a media asset retrieval method using multimodal fusion technology, which includes acquiring the original features of video, audio and text in a high-quality media asset dataset and identifying target entities therefrom; The target entity is used as a semantic anchor point to map to the knowledge graph, the logical relationship between nodes is obtained, and cross-modal semantic completion is performed on the original features; The logical relationship is injected into the cross-attention model as a bias parameter, and the weights of the aligned multimodal features are reshaped and deeply fused to generate a fused feature vector. A semantic index is constructed based on fused feature vectors, which responds to retrieval commands and outputs retrieval results after semantic generalization matching.

[0007] As a preferred embodiment of the media asset retrieval method using multimodal fusion technology described in this invention, the original features include: extracting data features using multi-path parallel sensing, activating three independent sensing operators, and mapping the data in the high-quality media asset dataset to a high-dimensional vector space. The extraction of visual features utilizes a visual Transformer model, dividing video frames into multiple patches for self-attention computation, and outputting visual feature vectors containing spatial object distribution and scene information. ; Audio feature extraction utilizes an acoustic convolutional neural network to process the Mel spectrum, capturing the acoustic features of ambient sounds, intonation, and background music, and outputting an audio feature vector. ; Text feature extraction utilizes a pre-trained language model to encode the title, tags, and ASR-recognized text, outputting text feature vectors with contextual semantics. .

[0008] As a preferred embodiment of the media asset retrieval method using multimodal fusion technology described in this invention, the identification of target entities includes: extracting visual anchor points using a semantic anchor point capture algorithm, running a target detection operator, and identifying specific objects in the image; if the identification confidence is greater than a threshold, then recording the type, location coordinates, and frame sequence of the current object, and marking it as a candidate entity; Audio and text anchor points are extracted, and proper nouns of target entities are extracted from ASR text using recognition operators and used as semantic trigger points; If a candidate entity appears in the visual system at the same time and a proper noun of the target entity appears in the text, a composite semantic anchor is generated, and a list of target entities containing entity ID, modality source, and time location is output. Extracted Bind the corresponding composite semantic anchor ID to the timeline and output a structured collaborative awareness tensor.

[0009] As a preferred embodiment of the media asset retrieval method using multimodal fusion technology described in this invention, wherein: mapping the target entity as a semantic anchor to the knowledge graph includes connecting the captured composite semantic anchor to the knowledge graph network; Using the extracted target entity ID, target nodes are retrieved in the knowledge graph through text similarity and contextual features; if entities with the same name appear, semantic ambiguity is eliminated by combining environmental information in the collaborative perception tensor. When the extracted target entity corresponds to multiple target nodes in the knowledge graph, the modal context feature vector in the collaborative perception tensor is extracted, the cosine similarity between the context feature vector and the attribute features of each target node is calculated, and the target node with the highest similarity is selected as the final mapping target node to eliminate entity ambiguity. By using a pre-trained knowledge representation model, successfully mapped knowledge graph nodes are transformed into knowledge vectors in a high-dimensional space; Extract the timestamp information carried by each modality feature, and verify whether the heterogeneous features mapped to the same knowledge graph target node are within the preset time overlap window; if the time offset exceeds the threshold, it is determined to be an irrelevant modality pair, and semantic remapping is performed; Construct a public latent semantic space based on a knowledge graph ontology; transform visual features using linear or nonlinear transformation matrices. Audio features and text features Projected into the public latent semantic space; A contrastive loss function is introduced to minimize the Euclidean distance between different modal vectors describing the same knowledge graph node, thereby performing cross-modal semantic alignment.

[0010] As a preferred embodiment of the media asset retrieval method using multimodal fusion technology described in this invention, the step of performing cross-modal semantic completion on the original features includes real-time detection of features of each modality. , , The confidence score is calculated, and when the confidence score of a certain modality within the target time period is lower than a preset threshold, the composite semantic anchor points of the target entities already identified in other modalities are used to perform [further action] in the knowledge graph. By skipping the associated path retrieval, prior knowledge vectors related to the missing modalities are extracted, and the prior knowledge vectors are injected into the features of the low-confidence modalities through weighted fusion. The injection ratio of the prior knowledge vector is determined by a dynamic compensation factor. Decide; The compensation factor It is inversely proportional to the original confidence score of the missing modality and directly proportional to the logical relevance of the associated paths in the knowledge graph. The intervention intensity is automatically adjusted according to the degree of modality damage to dynamically complete the model.

[0011] As a preferred embodiment of the media asset retrieval method using multimodal fusion technology described in this invention, the step of weight reshaping and deep fusion of the aligned multimodal features includes transforming the activated local knowledge subgraph topology into a mathematical bias to guide attention allocation. Traverse the edge relationships between entities in the subgraph and calculate the semantic association degree between entities using a random walk shortest path algorithm. ; correlation A knowledge bias matrix of the same dimension as the attention graph is constructed by mapping a non-linear activation function to bias coefficients. A cross-attention architecture is adopted, using one modality as the query vector to drive feature reconstruction in other modalities; The aligned visual feature vectors and audio feature vectors are mapped to key vectors respectively. Sum value vector Mapping text semantic vectors to query vectors ; In calculation and When calculating dot product similarity, a knowledge bias matrix is ​​forcibly injected. Cross-attention calculation is performed to increase the weight of features in the final fusion vector based on the correlation strength between features and retrieval instructions in the knowledge graph. Multiple attention heads are used for parallel computation to capture knowledge associations in different dimensions of spatial attributes, action logic, and acoustic emotion. After concatenating the outputs of each head, the heterogeneous features are nonlinearly compressed through a feedforward network (FFN) with residual connections. Adaptive pooling is then performed on the interactive feature sequences to eliminate time length differences, and the final deep fusion feature vector is output.

[0012] As a preferred embodiment of the media asset retrieval method using multimodal fusion technology described in this invention, the step of constructing a semantic index based on the fused feature vector includes: dividing the deep fused feature vector into multiple subspaces using product quantization technology and performing cluster quantization; replacing the original high-dimensional vector with a stored subspace centroid index; and performing... Normalization is performed to map the processed vectors to a unit hypersphere space; a multi-layer navigation index structure is constructed using a hierarchical small world graph algorithm, where the top layer consists of sparse feature representative points and the bottom layer contains full-modality feature vectors; Parse the user's original search command to extract the search entity; retrieve the entity from the pre-built knowledge graph. Jump to logically related extended nodes; perform weighted aggregation of the feature vectors of the retrieved entity and the feature vectors of the extended nodes to generate a generalized query vector that covers the user's potential intent; Based on the contextual information of the search command, the fusion weight between the search entity and the extended nodes is dynamically adjusted; when the search command contains action qualifiers, the weight ratio of action-related path nodes in the knowledge graph is increased, and a generalized query vector including action semantic bias is generated for retrieval.

[0013] Secondly, the present invention provides a media asset retrieval system implemented using multimodal fusion technology, including a multi-path collaborative perception module responsible for extracting multimodal data features from a high-quality media asset dataset; The knowledge mapping module utilizes external knowledge graphs to achieve logical alignment and information repair between modalities; The knowledge bias deep fusion module uses graph logic to guide feature recombination, and after multi-head attention parallel processing and residual network compression, it outputs a deep fusion feature vector that eliminates time differences. The semantic index retrieval module is responsible for dynamically adjusting the graph path weights based on the action qualifiers in the instructions, so as to achieve accurate matching and result output with semantic bias.

[0014] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the media asset retrieval method using multimodal fusion technology as described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the media asset retrieval method using multimodal fusion technology as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By introducing composite semantic anchors and mapping features to a knowledge graph, a leap from surface statistical correlation to deep logical cognition is achieved. The structured priors of the knowledge graph are used to align and verify heterogeneous features, effectively solving the semantic offset and false association problems commonly found in traditional multimodal fusion. A knowledge-driven semantic completion mechanism is implemented through dynamic compensation factors. In modal impairment scenarios such as visual occlusion and audio noise, the system can automatically inject prior features using knowledge paths activated by other modalities, ensuring the stability of retrieval performance under extreme conditions. The graph topology is transformed into a bias parameter injection cross-attention model, giving feature weight allocation "logical navigation" capabilities. The generated fusion feature vector not only contains sensory information but also implies the spatiotemporal correlation logic between entities, significantly improving the discrimination accuracy under complex retrieval tasks. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1This is a flowchart of a media asset retrieval method implemented using multimodal fusion technology.

[0019] Figure 2 This is a schematic diagram illustrating the retrieval process of a media asset retrieval method implemented using multimodal fusion technology.

[0020] Figure 3 This is a diagram of a computer device used for a media asset retrieval method implemented with multimodal fusion technology. Detailed Implementation

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0024] Reference Figures 1-2 This is one embodiment of the present invention, which provides a media asset retrieval method using multimodal fusion technology, comprising the following steps: S1: Obtain the original features of video, audio and text from the high-quality media asset dataset, and identify target entities from them.

[0025] Furthermore, the original features include extracting data features using multi-path parallel sensing, activating three independent sensing operators, and mapping the data in the high-quality media asset dataset to a high-dimensional vector space.

[0026] The visual feature extraction process utilizes a visual Transformer model, dividing video frames into multiple patches for self-attention computation, and outputting visual feature vectors containing spatial object distribution and scene information. .

[0027] Audio feature extraction utilizes an acoustic convolutional neural network to process Mel spectrum, capturing the acoustic features of ambient sounds, intonation, and background music, and outputting an audio feature vector. .

[0028] Text feature extraction utilizes a pre-trained language model to encode the title, tags, and ASR-recognized text, outputting text feature vectors with contextual semantics. .

[0029] The identification of target entities includes: extracting visual anchor points using a semantic anchor point capture algorithm, running a target detection operator, and identifying specific objects in the image; if the identification confidence is greater than a threshold, the type, location coordinates, and frame sequence of the current object are recorded and marked as a candidate entity.

[0030] Audio and text anchor points are extracted, and proper nouns of target entities are extracted from ASR text using recognition operators and used as semantic trigger points.

[0031] If a candidate entity appears in the visual system at the same time, and the target entity's proper noun appears in the text, a composite semantic anchor is generated, and the output includes a list of target entities containing the entity ID, modality source, and time location.

[0032] Extracted Bind the corresponding composite semantic anchor ID to the timeline and output a structured collaborative awareness tensor.

[0033] The perceptual operator is pre-trained using comparative learning on a high-quality dataset containing tens of millions of data points across the converged media industry. This high-quality dataset covers high-definition audio and video materials from vertical fields such as broadcast television news, documentaries, and live sports events, and features professional-grade human semantic annotations. Compared to general-purpose open-source datasets, this industry dataset has been deeply optimized for the specific scenario logic of converged media.

[0034] It should be noted that the knowledge graph construction process includes defining the core semantic framework of the media asset domain and establishing the skeleton of the knowledge graph. Category definition involves pre-setting multi-level entity categories, including but not limited to people, objects, actions, scenes, and events.

[0035] Relationship definition: Establish the logical predicates between classes, such as execute (person-action), be located (object-scene), belong to (person-organization), etc.

[0036] Attribute definition: Configure descriptive features for various entities, such as a person's profession, an object's color and shape, and the timing of events.

[0037] Using a pre-trained Named Entity Recognition (NER) model, specific entity objects are identified from text metadata of historical media assets, Automatic Speech Recognition (ASR) results, and visual tags. Relation extraction operators from deep learning (such as supervised classification models) are employed to analyze the logical relationships between entities from text sentence structures or video scene co-occurrences, forming subject-verb-object triples. Information from structured databases, such as player databases and match statistics tables, is automatically extracted to refine the attribute features of entities.

[0038] S2: Map the target entity as a semantic anchor to the knowledge graph, obtain the logical relationships between nodes, and perform cross-modal semantic completion on the original features.

[0039] The step of mapping the target entity as a semantic anchor to the knowledge graph includes connecting the captured composite semantic anchor to the knowledge graph network.

[0040] Using the extracted target entity ID, target nodes are retrieved in the knowledge graph through text similarity and contextual features; if entities with the same name appear, semantic ambiguity is eliminated by combining environmental information in the collaborative perception tensor.

[0041] When the extracted target entity corresponds to multiple target nodes in the knowledge graph, the modal context feature vector in the collaborative perception tensor is extracted, the cosine similarity between the context feature vector and the attribute features of each target node is calculated, and the target node with the highest similarity is selected as the final mapping target node to eliminate entity ambiguity.

[0042] By using a pre-trained knowledge representation model, successfully mapped knowledge graph nodes are transformed into knowledge vectors in a high-dimensional space.

[0043] Extract the timestamp information carried by each modality feature, and verify whether the heterogeneous features mapped to the same knowledge graph target node are within the preset time overlap window; if the time offset exceeds the threshold, it is determined to be an unrelated modality pair, and semantic remapping is performed.

[0044] Construct a public latent semantic space based on a knowledge graph ontology; transform visual features using linear or nonlinear transformation matrices. Audio features and text features Projected into the public latent semantic space.

[0045] A contrastive loss function is introduced to minimize the Euclidean distance between different modal vectors describing the same knowledge graph node, thereby performing cross-modal semantic alignment.

[0046] The discrete anchor points captured by S1 are integrated into the knowledge network. Using the entity IDs extracted by S1, target nodes are retrieved in the knowledge graph based on text similarity and contextual features. If entities with the same name appear, disambiguation is performed by combining environmental information from the collaborative perception tensor, such as visually identifying a mobile phone or an orchard.

[0047] Using a pre-trained knowledge representation model, successfully mapped graph nodes are transformed into knowledge vectors in a high-dimensional space. .

[0048] Cross-modal semantic consistency alignment: To eliminate the "expressive gap" between visual, audio, and text, the system performs the following alignment operations: Construct a public latent semantic space based on a knowledge graph ontology. Transform visual features using linear or nonlinear transformation matrices. Audio features and text features Projected into that space.

[0049] A contrastive loss function is introduced to force the distance between different modal vectors describing the same knowledge node to be narrowed. For example, this ensures that the visual vector of a goal and the audio vector of a successful shot highly overlap in space.

[0050] A specific object refers to a physical entity that is initially captured by the algorithm within a particular video frame or audio segment. It is concrete and has coordinates and boundaries.

[0051] For example, in the upper left corner of frame 15 of the video, there is an area whose pixel features match those of a human face and is selected by a rectangle.

[0052] Target entity: refers to a logical unit with a unique identifier (ID) in a knowledge base (such as a knowledge graph). It is abstract and has background information.

[0053] For example: in a knowledge graph, the ID is The node represents football player Lionel Messi.

[0054] A specific object is the presentation layer, and different specific objects may point to the same target entity.

[0055] The video showed close-ups of Messi's face, silhouettes of his back, and the commentator mentioning the word "King of Football."

[0056] The target entity is the core layer: all the specific objects mentioned above (images, outlines, sounds, and text) must ultimately be converged and mapped onto the target entity Messi through entity linking technology.

[0057] Specific object Mapped to Target entity.

[0058] The cross-modal semantic completion of the original features includes real-time detection of features in each modality. , , The confidence score is calculated, and when the confidence score of a certain modality within the target time period is lower than a preset threshold, the composite semantic anchor points of the target entities already identified in other modalities are used to perform [further action] in the knowledge graph. By skipping the associated path retrieval, prior knowledge vectors related to the missing modalities are extracted, and the prior knowledge vectors are injected into the feature stream of the low-confidence modal through weighted fusion.

[0059] The injection ratio of the prior knowledge vector is determined by a dynamic compensation factor. Decide.

[0060] The compensation factor It is inversely proportional to the original confidence score of the missing modality and directly proportional to the logical relevance of the associated paths in the knowledge graph. The intervention intensity is automatically adjusted according to the degree of modality damage to dynamically complete the model.

[0061] Knowledge-driven semantic completion of missing modalities utilizes the relational chains of knowledge graphs to repair damaged or missing information. By assessing confidence, identify missing modalities with weak information within the current time series (e.g., video footage only shows spectators in the stands, but the commentator mentions specific player names in the audio). Following the player nodes identified in the audio, retrieve their corresponding information from the graph. Jump to related nodes (such as team, number, habitual action) to retrieve the corresponding prior knowledge vector. .

[0062] Inverse proportional constraint: Original confidence score of missing modality Inversely proportional. That is, the more severe the modality impairment and the lower the confidence level, the more the system tends to rely on the knowledge graph for reconstruction. Directly proportional constraint: Strength of association paths retrieved from the knowledge graph It is directly proportional. That is, the clearer the logical connection, the higher the certainty of completion. The formula is expressed as: in, is the normalization coefficient. This formula ensures that when... and At higher levels, When the value approaches its maximum, the system will implement strong intervention.

[0063] Calculated , vector of prior knowledge With the damaged original feature vector Weighted fusion is performed to obtain the completed modal feature vector. The formula is expressed as: in, This is a cross-modal mapping matrix responsible for mapping the knowledge graph space back to the physical perception feature space. Through the above process, even when visual image information is completely lost, the system can still use entity clues provided by audio to infer the visual semantic features that should be present in that time period using the knowledge graph, thereby generating a deep fusion feature vector with logical integrity.

[0064] The execution in the knowledge graph The jump-related path retrieval involves using the entity ID in the composite semantic anchor point output by S1 as the retrieval starting point, and locating the corresponding seed node in the pre-built media asset knowledge graph through index matching. An initial activation procedure is then initiated, setting the activation status value of the seed node to the maximum weight.

[0065] Starting from the seed node, the hierarchy expands outwards along the attribute relationship edges in the graph. Skip search is implemented using a recursive search algorithm: 1-hop search: retrieves first-order neighbor nodes directly connected to the seed node (e.g., the team to which a person belongs, the location where the event occurred).

[0066] Two-hop and multi-hop retrieval: Starting from a first-order neighbor node, continue to explore second-order and higher-order related nodes (such as the organization to which the team belongs, or the geographical region attribute of the location).

[0067] During the diffusion process, the logical relevance score of each path is calculated based on the type of relation edge, including strongly related subordinate relationships and weakly related relationships. A decay factor is used to correct the semantic strength that decreases with the number of hops, and only paths with scores higher than a preset threshold are retained.

[0068] All activated Jump nodes and their connecting edges are logically aggregated, and isolated branches unrelated to the current media asset context are removed, thereby extracting a local knowledge subgraph highly relevant to the current retrieval task from the full knowledge graph. The subgraph defines the semantic boundary of the current media asset fragment, providing a structured basis for subsequent feature completion and bias matrix calculation.

[0069] It should be noted that the inferred features of related nodes are transformed into prior knowledge vectors and injected into the feature stream of weak modalities. This achieves the effect of listening to and watching; even if the field is not captured in the footage, the system can still learn the underlying motion semantics of the segment through the knowledge graph.

[0070] The aligned features are fused with the completed semantic labels to generate a semantically enhanced tensor. Each unit in this tensor is no longer a simple signal frequency or pixel, but a semantic feature carrying a knowledge node ID and logical attributes, providing logically consistent input for S3 deep fusion.

[0071] S3: The logical relationship is injected into the cross-attention model as a bias parameter, and the weights of the aligned multimodal features are reshaped and deeply fused to generate a fused feature vector.

[0072] The weight reshaping and deep fusion of the aligned multimodal features includes transforming the activated local knowledge subgraph topology into a mathematical bias to guide attention allocation.

[0073] Traverse the edge relationships between entities in the subgraph and calculate the semantic association degree between entities using a random walk shortest path algorithm. .

[0074] correlation A knowledge bias matrix of the same dimension as the attention graph is constructed by mapping a non-linear activation function to bias coefficients. A cross-attention architecture is adopted, using one modality as the query vector to drive feature reconstruction in other modalities.

[0075] To simultaneously consider global connectivity and local paths, an energy diffusion model that combines the ideas of shortest path and random walk is adopted: Shortest path decay, calculating the shortest path decay between two nodes in a knowledge graph network. and Shortest weighted path length between Its semantic contribution decreases exponentially with increasing distance: in The preset distance attenuation factor (e.g., 0.5).

[0076] To capture the topological abundance of graph structures, a random walk probability matrix is ​​introduced. .node For nodes The steady-state transition probability is the correlation score. This score reflects the performance of the graph from... Depart, and arrive via multiple routes. The level of difficulty.

[0077] Final semantic relevance This is a weighted fusion of the two: in It is an adjustment factor used to balance direct logical paths and global topology.

[0078] The aligned visual feature vectors and audio feature vectors are mapped to key vectors respectively. Sum value vector Mapping text semantic vectors to query vectors .

[0079] In calculation and When calculating dot product similarity, a knowledge bias matrix is ​​forcibly injected. Cross-attention calculation is performed to increase the weight of features in the final fusion vector based on the correlation strength between features and retrieval instructions in the knowledge graph.

[0080] Multiple attention heads are used for parallel computation to capture knowledge associations in different dimensions such as spatial attributes, action logic, and acoustic emotion.

[0081] After concatenating the outputs of each head, the heterogeneous features are nonlinearly compressed through a feedforward network (FFN) with residual connections. Adaptive pooling is then performed on the interactive feature sequences to eliminate time length differences, and the final deep fusion feature vector is output.

[0082] A multi-head cross-attention mechanism is employed, using one modality as the query source to drive feature reorganization in other modalities. The aligned text feature vectors are then... Mapped to query vector Visual features With audio features Mapped to key vectors respectively Sum value vector Weighted calculation of injection bias: The bias matrix is ​​superimposed when calculating the similarity score. : Bias term It acts as a semantic navigator, ensuring that even if certain modal signals are weak (such as tiny targets in the background of a picture), as long as they conform to the logical chain of the knowledge graph, they will be assigned higher attention weights. This represents the scaling factor, which is usually the square root of the vector dimension, used to prevent the gradient from vanishing due to excessively large dot product values. The knowledge bias matrix represents the knowledge bias matrix, and its dimensions are the same as those of the attention graph. Consistent.

[0083] and When, it represents an entity. With entity Strong logical connections exist within knowledge graphs, which enhance the weight of feature pairs. When the two entities are logically unrelated, their original weights are suppressed or maintained.

[0084] By employing a multilayer perceptron (MLP) and residual connections, nonlinear fusion and detail preservation of heterogeneous information in a high-dimensional space are achieved. Pooling is then performed on the fused tensor to output the final deep-fused feature vector. This vector not only contains sensory features, but also implies the logical connections defined in the knowledge graph.

[0085] S4: Construct a semantic index based on fused feature vectors, respond to retrieval commands, and output retrieval results after semantic generalization matching.

[0086] The semantic index construction based on the fused feature vector includes: dividing the deep fused feature vector into multiple subspaces using product quantization technology and performing clustering quantization; replacing the original high-dimensional vector with a stored subspace centroid index; and performing... Normalization is performed to map the processed vectors to a unit hypersphere space; a multi-layer navigation index structure is constructed using a hierarchical small world graph algorithm, where the top layer consists of sparse feature representative points and the bottom layer contains full-modality feature vectors.

[0087] like Figure 2 As shown, the original search command input by the user is parsed to extract the search entity; the entity is then searched within a pre-built knowledge graph. Jump to logically related extended nodes; perform weighted aggregation of the feature vectors of the retrieved entity and the feature vectors of the extended nodes to generate a generalized query vector that covers the user's potential intent.

[0088] Based on the contextual information of the search command, the fusion weight between the search entity and the extended nodes is dynamically adjusted; when the search command contains action qualifiers, the weight ratio of action-related path nodes in the knowledge graph is increased, and a generalized query vector including action semantic bias is generated for retrieval.

[0089] The mathematical expression for the generalization and extension of search intent is as follows: in, This represents the original query vector, which is derived from the original search command entered by the user. This represents the generalized query vector, which, after being expanded by the knowledge graph, becomes the final retrieval vector containing the underlying semantic intent. Represents an extended node vector, referring to a node in the knowledge graph that exists alongside the original entity. The first jump association The feature representation of each neighboring node. The intent retention factor is used to balance the weighting of original instructions and extended knowledge, and its value ranges from [value range missing]. . The correlation strength coefficient depends on the extended nodes. The number of steps (hops) from the center node and the edge type weight. Similarity scores are typically calculated using cosine similarity. In the index Distance: To handle media asset data exceeding tens of millions of records, deep fusion of feature vectors is required. Preprocessing is performed first. Normalization eliminates dimensional differences, ensuring that similarity calculation depends only on the angle between vectors. Subsequently, an HNSW index is constructed.

[0090] The HNSW index constructs a multi-layered graph structure, performing a jump-based localization at the sparse top layer during the search, followed by refined neighbor finding at the bottom layer. To reduce memory usage, the system introduces the PQ product quantization algorithm. 3D vector segmentation Each subspace stores only the index value of its cluster center.

[0091] After performing similarity calculations in the vector library, a set of candidate video clips is obtained. To eliminate false positives due to visual similarity but logical irrelevance (e.g., both actions occur within the penalty area, but are actually fouls rather than shots), the system initiates a logical reordering: We trace back to the collaborative perception tensor generated in the first stage of this segment and check whether it contains key semantic anchors such as football and shooting action.

[0092] Calculate the path distance between the real-time label of the candidate fragment and the search intent in the knowledge graph. The closer the distance, the higher the ranking.

[0093] It not only returns the video file, but also shows the user the precise start and end times of the shot (e.g., 01:12-01:15) based on the start and end points of the frame sequence recorded in the tensor, thus completing the closed-loop retrieval.

[0094] This embodiment also provides a media asset retrieval system implemented using multimodal fusion technology, including: The multi-path collaborative perception module is responsible for the underlying digitization of media asset data and preliminary entity discovery.

[0095] The knowledge mapping module utilizes external knowledge graphs to achieve logical alignment and information repair between modalities.

[0096] The knowledge bias deep fusion module uses graph logic to guide feature recombination, and after multi-head attention parallel processing and residual network compression, outputs a deep fusion feature vector that eliminates time differences.

[0097] The semantic index retrieval module is responsible for dynamically adjusting the graph path weights based on the action qualifiers in the instructions, so as to achieve accurate matching and result output with semantic bias.

[0098] This embodiment also provides a computer device, such as... Figure 3 As shown, the method applicable to media asset retrieval using multimodal fusion technology includes: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the media asset retrieval method using multimodal fusion technology as proposed in the above embodiments.

[0099] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0100] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the media asset retrieval method using multimodal fusion technology as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0101] In summary, this invention achieves the following: First, it acquires the original visual, audio, and textual features of media asset data through multi-path parallel perception and identifies target entities from them. Second, it maps the target entities as semantic anchors to a knowledge graph, obtains the logical relationships between nodes, and performs cross-modal semantic completion on the original features. Third, it injects the logical relationships as bias parameters into a cross-attention model, reshapes the weights of the aligned multimodal features, and performs deep fusion to generate a fused feature vector. Fourth, it constructs a semantic index based on the fused feature vector, responds to retrieval commands, and outputs the retrieval results after semantic generalization matching.

[0102] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A media asset retrieval method using multimodal fusion technology, characterized in that: This includes acquiring the original features of video, audio, and text from high-quality media asset datasets and identifying target entities from them; The target entity is used as a semantic anchor point to map to the knowledge graph, the logical relationship between nodes is obtained, and cross-modal semantic completion is performed on the original features; The logical relationship is injected into the cross-attention model as a bias parameter, and the weights of the aligned multimodal features are reshaped and deeply fused to generate a fused feature vector. A semantic index is constructed based on fused feature vectors, which responds to retrieval commands and outputs retrieval results after semantic generalization matching.

2. The media asset retrieval method using multimodal fusion technology as described in claim 1, characterized in that: The original features include extracting data features using multi-path parallel sensing, activating three independent sensing operators, and mapping the data in the high-quality media asset dataset to a high-dimensional vector space. The extraction of visual features utilizes a visual Transformer model, dividing video frames into multiple patches for self-attention computation, and outputting visual feature vectors containing spatial object distribution and scene information. ; Audio feature extraction utilizes an acoustic convolutional neural network to process the Mel spectrum, capturing the acoustic features of ambient sounds, intonation, and background music, and outputting an audio feature vector. ; Text feature extraction utilizes a pre-trained language model to encode the title, tags, and ASR-recognized text, outputting text feature vectors with contextual semantics. .

3. The media asset retrieval method using multimodal fusion technology as described in claim 2, characterized in that: The identification of target entities includes: extracting visual anchor points using a semantic anchor point capture algorithm, running a target detection operator, and identifying specific objects in the image; if the identification confidence is greater than a threshold, the type, location coordinates, and frame sequence of the current object are recorded and marked as a candidate entity. Audio and text anchor points are extracted, and proper nouns of target entities are extracted from ASR text using recognition operators and used as semantic trigger points; If a candidate entity appears in the visual system at the same time and a proper noun of the target entity appears in the text, a composite semantic anchor is generated, and a list of target entities containing the entity ID, modality source, and time location is output. Extracted Bind the corresponding composite semantic anchor ID to the timeline and output a structured collaborative awareness tensor.

4. The media asset retrieval method using multimodal fusion technology as described in claim 3, characterized in that: The step of mapping the target entity as a semantic anchor to the knowledge graph includes connecting the captured composite semantic anchor to the knowledge graph network; Using the extracted target entity ID, target nodes are retrieved from the knowledge graph based on text similarity and contextual features; If entities with the same name appear, semantic ambiguity is eliminated by combining environmental information in the collaborative perception tensor. When the extracted target entity corresponds to multiple target nodes in the knowledge graph, the modal context feature vector in the collaborative perception tensor is extracted, the cosine similarity between the context feature vector and the attribute features of each target node is calculated, and the target node with the highest similarity is selected as the final mapping target node to eliminate entity ambiguity. By using a pre-trained knowledge representation model, successfully mapped knowledge graph nodes are transformed into knowledge vectors in a high-dimensional space; Extract the timestamp information carried by each modality feature, and verify whether the heterogeneous features mapped to the same knowledge graph target node are within the preset time overlap window; if the time offset exceeds the threshold, it is determined to be an irrelevant modality pair, and semantic remapping is performed; Construct a public latent semantic space based on a knowledge graph ontology; transform visual features using linear or nonlinear transformation matrices. Audio features and text features Projected into the public latent semantic space; A contrastive loss function is introduced to minimize the Euclidean distance between different modal vectors describing the same knowledge graph node, thereby performing cross-modal semantic alignment.

5. The media asset retrieval method using multimodal fusion technology as described in claim 4, characterized in that: The cross-modal semantic completion of the original features includes real-time detection of features in each modality. , , The confidence score is calculated, and when the confidence score of a certain modality within the target time period is lower than a preset threshold, the composite semantic anchor points of the target entities already identified in other modalities are used to perform [further action] in the knowledge graph. By skipping the associated path retrieval, prior knowledge vectors related to the missing modalities are extracted, and the prior knowledge vectors are injected into the features of the low-confidence modalities through weighted fusion. The injection ratio of the prior knowledge vector is determined by a dynamic compensation factor. Decide; The compensation factor It is inversely proportional to the original confidence score of the missing modality and directly proportional to the logical relevance of the associated paths in the knowledge graph. The intervention intensity is automatically adjusted according to the degree of modality damage to dynamically complete the model.

6. The media asset retrieval method using multimodal fusion technology as described in claim 5, characterized in that: The weight reshaping and deep fusion of the aligned multimodal features includes transforming the activated local knowledge subgraph topology into a mathematical bias to guide attention allocation; Traverse the edge relationships between entities in the subgraph and calculate the semantic association degree between entities using a random walk shortest path algorithm. ; correlation A knowledge bias matrix of the same dimension as the attention graph is constructed by mapping a non-linear activation function to bias coefficients. A cross-attention architecture is adopted, using one modality as the query vector to drive feature reconstruction in other modalities; The aligned visual feature vectors and audio feature vectors are mapped to key vectors respectively. Sum value vector ; Mapping text semantic vectors to query vectors ; In calculation and When calculating dot product similarity, a knowledge bias matrix is ​​forcibly injected. Cross-attention calculation is performed to increase the weight of features in the final fusion vector based on the correlation strength between features and retrieval instructions in the knowledge graph. Multiple attention heads are used for parallel computation to capture knowledge associations in different dimensions of spatial attributes, action logic, and acoustic emotion. After concatenating the outputs of each head, the heterogeneous features are nonlinearly compressed through a feedforward network (FFN) with residual connections. Adaptive pooling is then performed on the interactive feature sequences to eliminate time length differences, and the final deep fusion feature vector is output.

7. The media asset retrieval method using multimodal fusion technology as described in claim 6, characterized in that: The semantic index construction based on the fused feature vector includes: dividing the deep fused feature vector into multiple subspaces using product quantization and performing clustering quantization; replacing the original high-dimensional vector with a stored subspace centroid index; and performing... Normalization is performed to map the processed vectors to a unit hypersphere space; a multi-layer navigation index structure is constructed using a hierarchical small world graph algorithm, where the top layer consists of sparse feature representative points and the bottom layer contains full-modality feature vectors; Parse the user's original search command to extract the search entity; retrieve the entity from the pre-built knowledge graph. Jump to logically related extended nodes; perform weighted aggregation of the feature vectors of the retrieved entity and the feature vectors of the extended nodes to generate a generalized query vector that covers the user's potential intent; Based on the contextual information of the search command, the fusion weight between the search entity and the extended nodes is dynamically adjusted; when the search command contains action qualifiers, the weight ratio of action-related path nodes in the knowledge graph is increased, and a generalized query vector including action semantic bias is generated for retrieval.

8. A media asset retrieval system using multimodal fusion technology, based on the media asset retrieval method using multimodal fusion technology as described in any one of claims 1 to 7, characterized in that: This includes a multi-path collaborative perception module, responsible for extracting multimodal data features from high-quality media asset datasets; The knowledge mapping module utilizes external knowledge graphs to achieve logical alignment and information repair between modalities; the knowledge bias deep fusion module uses graph logic to guide feature recombination, and outputs a deep fusion feature vector that eliminates time differences through multi-head attention parallel processing and residual network compression. The semantic index retrieval module is responsible for dynamically adjusting the graph path weights based on the action qualifiers in the instructions, so as to achieve accurate matching and result output with semantic bias.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the media asset retrieval method using multimodal fusion technology as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the media asset retrieval method using multimodal fusion technology as described in any one of claims 1 to 7.

Images