An AI-driven media knowledge base retrieval method based on region tagging
By using a multimodal large model to perform semantic segmentation and cross-modal network construction on media files, the problems of high annotation costs and insufficient semantic association in traditional media retrieval are solved, achieving efficient and accurate cross-modal content retrieval and meeting users' personalized needs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING LIUJINSUIYUE TECH CO LTD
- Filing Date
- 2025-11-27
- Publication Date
- 2026-06-02
AI Technical Summary
Traditional media content retrieval methods rely on manually labeled metadata, which is costly, inefficient, and difficult to ensure consistency. They cannot effectively capture detailed information about specific areas of the content, and there is a lack of semantic connections between media content of different modalities, resulting in poor retrieval results.
Multimodal large models are used to perform semantic segmentation on media files, generate structured tag sets, and construct cross-modal semantic association networks. Cross-modal semantic links are established by calculating the semantic similarity of regional tag vectors, and the priority of search results is dynamically adjusted in combination with user intent and interaction behavior.
It achieves deep semantic association across modal content, provides search results that are highly matched with the user's query intent, breaks the limitations of traditional single-modal retrieval, and improves retrieval efficiency and accuracy.
Smart Images
Figure CN121681853B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to an AI-driven media knowledge base retrieval method based on region tagging. Background Technology
[0002] With the rapid development of the internet and digital media technologies, the generation, storage, and dissemination of media content have experienced explosive growth. Massive amounts of unstructured media data, such as videos, images, and audio, contain immense informational value. However, efficiently and accurately retrieving the information users need from this data has become a core challenge for current media management and utilization.
[0003] Traditional media content retrieval methods primarily rely on manually annotated metadata, such as titles, keywords, and descriptions. This approach suffers from high annotation costs and low efficiency. Different annotators may have varying understandings and descriptions of the content, leading to strong subjectivity and difficulty in ensuring consistency. Furthermore, traditional metadata often only describes the overall concept of media content, failing to capture detailed information about specific areas. The lack of effective semantic connections between different modalities of media content prevents users from retrieving related content from one modality using information from another. Summary of the Invention
[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an AI-driven media knowledge base retrieval method based on region tagging to address the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical solution: an AI-driven media knowledge base retrieval method based on region tagging, specifically including:
[0006] S1. Using a multimodal large model, perform semantic segmentation on multimodal media files in the knowledge base, identify the basic units carrying independent semantics in the media files, including spatial regions, temporal segments and text paragraphs, and generate a structured tag set for the identified semantic units. The multimodal media files include images, videos, audio and text files, and the structured tag set includes entity tags, attribute tags and relation tags.
[0007] S2. Based on the structured tag set generated in step S1, a cross-modal semantic association network is constructed. The nodes in the network represent regions of different media files, and the edges represent the semantic similarity relationship between nodes. By calculating the semantic similarity of the region tag vectors, cross-modal semantic links are established between regions of different media to achieve semantic association of cross-modal content.
[0008] S3. Vectorize the user's current search query and record the historical queries and user interaction behaviors in the current session, representing them as user intent context. Based on the user's current search query and user intent context, perform a preliminary search in the cross-modal semantic association network to obtain the first candidate set. Then, use the collaborative re-ranking algorithm to re-rank the first candidate set with regional semantic matching degree, cross-modal association strength, and regional saliency weight as input features to obtain the final result list sorted by comprehensive relevance score.
[0009] S4. Present the user with a list of reordered search results, and highlight the region most relevant to the query in each result. By capturing the user's interaction with the search results, dynamically update the region saliency weights and optimize the edge weights and connections in the cross-modal semantic association network.
[0010] Preferably, as a preferred embodiment of the AI-driven media knowledge base retrieval method based on region labeling described in this invention, it includes, in step S1, using a multimodal large model to perform semantic segmentation on multimodal media files in the knowledge base, identifying basic units carrying independent semantics in the media files, including spatial regions, temporal segments, and text paragraphs, and generating a structured tag set from the identified semantic units. The multimodal media files include images, videos, audio, and text files, and the structured tag set includes entity tags, attribute tags, and relation tags, specifically including the following:
[0011] For image and video files, a deep learning-based visual perception model is used to identify and extract visual semantic entities in the image, determine their spatial regions in the image, and generate a first entity label, a first attribute label, and a first relationship label for each identified visual semantic entity region. The first entity label describes the specific visual object represented by the visual semantic entity region, the first attribute label describes the visual features of the visual object, and the first relationship label describes the visual relationship between the visual semantic entity region and other regions.
[0012] For the audio track portion of audio and video files, a deep learning-based acoustic perception model is used to identify and extract semantic entities in the audio, determine their temporal segments in the audio stream, and generate a second entity label, a second attribute label, and a second relation label for each identified audio semantic entity segment. The second entity label describes the specific acoustic event represented by the segment, the second attribute label describes the attributes of the acoustic event entity, and the second relation label describes the relationship between the audio semantic entity segment and other audio segments.
[0013] For text files, a deep learning-based natural language processing model is used to identify and extract semantic entities in the text, determine their paragraphs in the text, and generate a third entity label, a third attribute label, and a third relation label for each identified text semantic entity paragraph. The third entity label describes the specific text content represented by the paragraph, the third attribute label describes the attribute of the text content, and the third relation label describes the semantic relationship between the text paragraph and other text paragraphs.
[0014] All generated entity tags, attribute tags, and relationship tags are converted into high-dimensional vector representations and stored in a unified multimodal semantic index library.
[0015] Preferably, as a preferred embodiment of the AI-driven media knowledge base retrieval method based on region tagging described in this invention, it includes, in step S2, constructing a cross-modal semantic association network based on the structured tag set generated in step S1. Nodes in the network represent regions of different media files, and edges represent semantic similarity relationships between nodes. By calculating the semantic similarity of region tag vectors, cross-modal semantic links are established between regions of different media to achieve semantic association of cross-modal content. Specifically, this includes the following:
[0016] Each semantic unit generated in S1, including spatial regions of images, temporal segments of audio, and paragraphs of text, is treated as an independent node. Each node, when added to the network, has the following attributes: the corresponding original media file ID, spatiotemporal location information, and a structured tag vector set. All tag vectors from each node are then fused using a weighted average to generate a unified node representation vector. ,in, It is an entity label vector. It is the average of the attribute label vectors. It is the average of the relation label vectors. These are preset weights used to distinguish the importance of different category labels;
[0017] For any two nodes and Calculate its unified node representation vector and The cosine similarity between them is calculated using the following formula: ,in, It is a node With nodes Cosine similarity score between them , They are the unified representation vectors , The k-th component, where n is the dimension of the vector;
[0018] Set a similarity threshold When the cosine similarity between two nodes is greater than a certain threshold, an undirected edge is established between the two nodes. Set the weight of the undirected edge to . The constructed semantic association network is then instantiated as a weighted undirected graph. , where V is the set of nodes and E is the set of edges.
[0019] Preferably, as a preferred embodiment of the AI-driven media knowledge base retrieval method based on regional labeling described in this invention, it includes, in step S3, vectorizing the user's current search query and recording historical queries and user interaction behaviors in the current session, representing them as user intent context. Based on the user's current search query and user intent context, a preliminary search is performed in a cross-modal semantic association network to obtain a first candidate set. Then, a collaborative re-ranking algorithm is used to re-rank the first candidate set using regional semantic matching degree, cross-modal association strength, and regional saliency weight as input features, to obtain a final result list sorted by comprehensive relevance score, specifically including the following:
[0020] The current search query submitted by the user Transformed into query vectors through a pre-trained language model The user intent context is represented as a set of historical query vectors. User interaction logs This includes the media items that the user has clicked and the set of specific semantic unit nodes that are focused. and query vector Cross-modal semantic association networks Unified node representation vector Semantic similarity matching was performed, and the cosine similarity score was obtained. Select similarity score The top K nodes, where V is the set of nodes, E is the set of edges, and the media files associated with each node. Constitute the first candidate set ;
[0021] A collaborative reordering model is used for the first candidate set. Each candidate media The model performs a comprehensive scoring, and its input features include the region semantic matching degree. Cross-modal correlation strength Regional significance weight Further including:
[0022] Region semantic matching degree: Calculate the query vector With candidate media A unified node representation vector for all semantic units within the domain. The highest cosine similarity between them is calculated using the following formula: ,in, Indicates to candidate media The maximum similarity score of all included nodes is taken;
[0023] Cross-modal correlation strength: Calculating candidate media The set of nodes that the user has followed in the context of the user's intent The strongest semantic association strength between them is calculated using the following formula: ,in, It is a connecting node in a semantic association network. and edge weights; when If the set is empty, then the eigenvalue is zero;
[0024] Regional saliency weight: Calculating candidate media The dynamic weight corresponding to the semantic unit that best matches the current query. This weight is updated based on historical user interaction feedback, and the calculation formula is as follows: ,in, It is a saliency weight function that takes a semantic unit as input and outputs the saliency weight value of that semantic unit;
[0025] Using the extracted features as input, a linear weighted model is used to calculate the comprehensive relevance score for each candidate media file. According to the calculation The media files in the first candidate set are sorted in descending order to obtain the final result list sorted by comprehensive relevance score, where A, B, and C are the weight coefficients of the linear weighted model.
[0026] Preferably, as a preferred embodiment of the AI-driven media knowledge base retrieval method based on region labeling described in this invention, it includes, in step S4, presenting the user with a reordered list of search results, highlighting the region most relevant to the query in each result, and dynamically updating the region saliency weights and optimizing the edge weights and connections in the cross-modal semantic association network by capturing user interaction with the search results. Specifically, it includes the following:
[0027] The final result list after reordering in step S3 is presented to the user terminal. For each media file in the list, the semantic unit that best matches the current query vector is highlighted. The semantic unit is calculated in step S3 based on the region semantic matching degree. When, this makes the region corresponding to the node with the highest similarity score;
[0028] Record user interaction behaviors during the interaction with search results, including clicks, long dwell times, region focusing, and negative feedback marking. By capturing user interaction behaviors with search results, dynamically update the region saliency weights and optimize the edge weights and connection relationships in the cross-modal semantic association network, further including:
[0029] Dynamic updates of regional saliency weights: The trigger condition is set when a user clicks on a media result and stays on it for a long time; the update rule is to update the most relevant highlighted node in that media. The significance weights are strengthened, and the updated formula is: ,in, This is the updated weight. It is the weight before the update. It is a fixed positive increment;
[0030] Optimization of cross-modal semantic association networks: Set the trigger condition to occur when a user interacts consecutively with two nodes from different media within a session. and When an interaction occurs, the optimization rule is when the node... and An edge already exists between them. Then, the weight of the edge is increased based on the intensity of user interaction; when the node and There are no edges between them, cosine similarity If the preset threshold is met, construct a new edge and calculate the cosine similarity. As initial weights.
[0031] On the other hand, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements the steps of an AI-driven media knowledge base retrieval method based on region tagging as described above.
[0032] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements an AI-driven media knowledge base retrieval method based on region tagging as described above in the present invention.
[0033] The technical effects and advantages provided by the present invention in the above technical solution are as follows:
[0034] By performing unified semantic segmentation and tag generation on multiple modal files including images, videos, audio, and text, and constructing a cross-modal semantic association network, this invention breaks through the limitations of traditional single-modal retrieval. It can understand the deep semantic relationships between cross-modal content, introduce regional semantic matching degree, cross-modal association strength, and regional saliency weights, and through a collaborative re-ranking algorithm, dynamically adjust the priority of search results according to user intent and the characteristics of the media itself, providing results that better meet user expectations. Compared with traditional keyword-based retrieval methods, this invention can delve into the semantic level of media content and provide results that are highly matched with the user's query intent. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0036] Figure 1 This is a flowchart of an AI-driven media knowledge base retrieval method based on region tagging, according to the present invention.
[0037] Table 1 is a data record table of the simulation experiment of the AI-driven media knowledge base retrieval method based on regional labeling according to the present invention. Detailed Implementation
[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0039] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0040] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.
[0041] Example 1
[0042] This embodiment provides, for example Figure 1 This paper presents an AI-driven media knowledge base retrieval method based on region tagging, which specifically includes:
[0043] S1. Using a multimodal large model, perform semantic segmentation on multimodal media files in the knowledge base, identify the basic units carrying independent semantics in the media files, including spatial regions, temporal segments and text paragraphs, and generate a structured tag set for the identified semantic units. The multimodal media files include images, videos, audio and text files, and the structured tag set includes entity tags, attribute tags and relation tags.
[0044] S2. Based on the structured tag set generated in step S1, a cross-modal semantic association network is constructed. The nodes in the network represent regions of different media files, and the edges represent the semantic similarity relationship between nodes. By calculating the semantic similarity of the region tag vectors, cross-modal semantic links are established between regions of different media to achieve semantic association of cross-modal content.
[0045] S3. Vectorize the user's current search query and record the historical queries and user interaction behaviors in the current session, representing them as user intent context. Based on the user's current search query and user intent context, perform a preliminary search in the cross-modal semantic association network to obtain the first candidate set. Then, use the collaborative re-ranking algorithm to re-rank the first candidate set with regional semantic matching degree, cross-modal association strength, and regional saliency weight as input features to obtain the final result list sorted by comprehensive relevance score.
[0046] S4. Present the user with a list of reordered search results, and highlight the region most relevant to the query in each result. By capturing the user's interaction with the search results, dynamically update the region saliency weights and optimize the edge weights and connections in the cross-modal semantic association network.
[0047] Preferably, in step S1, a multimodal large model is used to perform semantic segmentation on multimodal media files in the knowledge base, identifying basic units carrying independent semantics in the media files, including spatial regions, temporal segments, and text paragraphs. The identified semantic units are then used to generate a structured tag set. The multimodal media files include images, videos, audio, and text files, and the structured tag set includes entity tags, attribute tags, and relation tags, specifically including the following:
[0048] For image and video files, a deep learning-based visual perception model is used to identify and extract visual semantic entities in the image, including people, animals, objects, brand logos, and text information. The spatial regions of these entities in the image are determined, and a first entity label, a first attribute label, and a first relationship label are generated for each identified visual semantic entity region. The first entity label describes the specific visual object represented by the visual semantic entity region, the first attribute label describes the visual features of the visual object, and the first relationship label describes the visual relationship between the visual semantic entity region and other regions.
[0049] For the audio track portion of audio and video files, a deep learning-based acoustic perception model is used to identify and extract semantic entities in the audio, including speech, background sound effects, and music segments, to determine their temporal segments in the audio stream. For each identified audio semantic entity segment, a second entity label, a second attribute label, and a second relation label are generated. The second entity label describes the specific acoustic event represented by the segment, the second attribute label describes the attributes of the acoustic event entity, and the second relation label describes the relationship between the audio semantic entity segment and other audio segments.
[0050] For text files, a deep learning-based natural language processing model is used to identify and extract semantic entities from the text, including names of people, places, organizations, events, and concepts. The paragraphs in the text are determined, and a third entity label, a third attribute label, and a third relation label are generated for each identified text semantic entity paragraph. The third entity label describes the specific text content represented by the paragraph, the third attribute label describes the attribute of the text content, and the third relation label describes the semantic relationship between the text paragraph and other text paragraphs.
[0051] All generated entity tags, attribute tags, and relationship tags are converted into high-dimensional vector representations and stored in a unified multimodal semantic index library.
[0052] Preferably, in step S2, a cross-modal semantic association network is constructed based on the structured tag set generated in step S1. Nodes in the network represent regions of different media files, and edges represent semantic similarity relationships between nodes. By calculating the semantic similarity of region tag vectors, cross-modal semantic links are established between regions of different media to achieve semantic association of cross-modal content. Specifically, this includes the following:
[0053] Each semantic unit generated in S1, including spatial regions of images, temporal segments of audio, and paragraphs of text, is treated as an independent node. Each node, when added to the network, has the following attributes: the corresponding original media file ID, spatiotemporal location information, and a structured tag vector set. All tag vectors from each node are then fused using a weighted average to generate a unified node representation vector. ,in, It is an entity label vector. It is the average of the attribute label vectors. It is the average of the relation label vectors. These are preset weights used to distinguish the importance of different category labels;
[0054] For any two nodes and Calculate its unified node representation vector and The cosine similarity between them is calculated using the following formula: ,in, It is a node With nodes Cosine similarity score between them , They are the unified representation vectors , The k-th component, where n is the dimension of the vector;
[0055] Set a similarity threshold When the cosine similarity between two nodes is greater than a certain threshold, an undirected edge is established between the two nodes. Set the weight of the undirected edge to . The constructed semantic association network is then instantiated as a weighted undirected graph. , where V is the set of nodes and E is the set of edges.
[0056] Preferably, in step S3, the user's current search query is vectorized, and historical queries and user interaction behaviors in the current session are recorded and represented as user intent context. Based on the user's current search query and user intent context, a preliminary search is performed in the cross-modal semantic association network to obtain a first candidate set. Then, a collaborative re-ranking algorithm is used to re-rank the first candidate set using regional semantic matching degree, cross-modal association strength, and regional saliency weight as input features to obtain a final result list sorted by comprehensive relevance score, specifically including the following:
[0057] The current search query submitted by the user Transformed into query vectors through a pre-trained language model The user intent context is represented as a set of historical query vectors. User interaction logs This includes the media items that the user has clicked and the set of specific semantic unit nodes that are focused. and query vector Cross-modal semantic association networks Unified node representation vector Semantic similarity matching was performed, and the cosine similarity score was obtained. Select similarity score The top K nodes, where V is the set of nodes, E is the set of edges, and the media files associated with each node. Constitute the first candidate set ;
[0058] A collaborative reordering model is used for the first candidate set. Each candidate media The model performs a comprehensive scoring, and its input features include the region semantic matching degree. Cross-modal correlation strength Regional significance weight Further including:
[0059] Region semantic matching degree: Calculate the query vector With candidate media A unified node representation vector for all semantic units within the domain. The highest cosine similarity between them is calculated using the following formula: ,in, Indicates to candidate media The maximum similarity score of all included nodes is taken;
[0060] Cross-modal correlation strength: Calculating candidate media The set of nodes that the user has followed in the context of the user's intent The strongest semantic association strength between them is calculated using the following formula: ,in, It is a connecting node in a semantic association network. and edge weights; when If the set is empty, then the eigenvalue is zero;
[0061] Regional saliency weight: Calculating candidate media The dynamic weight corresponding to the semantic unit that best matches the current query. This weight is updated based on historical user interaction feedback, and the calculation formula is as follows: ,in, It is a saliency weight function that takes a semantic unit as input and outputs the saliency weight value of that semantic unit;
[0062] Using the extracted features as input, a linear weighted model is used to calculate the comprehensive relevance score for each candidate media file. According to the calculation The media files in the first candidate set are sorted in descending order to obtain the final result list sorted by comprehensive relevance score, where A, B, and C are the weight coefficients of the linear weighted model.
[0063] Preferably, in step S4, a reordered list of search results is presented to the user, and the region most relevant to the query is highlighted in each result. By capturing the user's interaction with the search results, the saliency weights of the regions are dynamically updated and the edge weights and connections in the cross-modal semantic association network are optimized. Specifically, this includes the following:
[0064] The final result list after reordering in step S3 is presented to the user terminal. For each media file in the list, the semantic unit that best matches the current query vector is highlighted. The semantic unit is calculated in step S3 based on the region semantic matching degree. At that time, the region corresponding to the node with the highest similarity score is drawn with a bounding box on the image, the segment is marked on the video timeline, and the paragraph is highlighted in the text;
[0065] Record user interaction behaviors during the interaction with search results, including clicks, long dwell times, region focusing, and negative feedback marking. By capturing user interaction behaviors with search results, dynamically update the region saliency weights and optimize the edge weights and connection relationships in the cross-modal semantic association network, further including:
[0066] Dynamic updates of regional saliency weights: The trigger condition is set when a user clicks on a media result and stays on it for a long time; the update rule is to update the most relevant highlighted node in that media. The significance weights are strengthened, and the updated formula is: ,in, This is the updated weight. It is the weight before the update. It is a fixed positive increment;
[0067] Optimization of cross-modal semantic association networks: Set the trigger condition to occur when a user interacts consecutively with two nodes from different media within a session. and When an interaction occurs, the optimization rule is when the node... and An edge already exists between them. Then, the weight of the edge is increased based on the intensity of user interaction; when the node and There are no edges between them, cosine similarity If the preset threshold is met, construct a new edge and calculate the cosine similarity. As initial weights.
[0068] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.
[0069] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the steps of implementing an AI-driven media knowledge base retrieval method based on region tagging as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0070] Example 2
[0071] The following is another embodiment of the present invention, which provides an AI-driven media knowledge base retrieval method based on region tagging. In order to verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.
[0072] This experiment aims to verify the effectiveness of the AI-driven media knowledge base retrieval method based on regional labeling. Through multimodal media semantic unit recognition, structured label generation, cross-modal semantic association network construction, user intent-aware collaborative retrieval and reordering, and result presentation and feedback learning, the accuracy, personalization and dynamic adaptability of media knowledge base retrieval are improved.
[0073] The simulation experiment steps are implemented according to the content of the AI-driven media knowledge base retrieval method based on region labeling provided in Example 1. The specific steps include:
[0074] Collect a media knowledge base containing multiple modalities including images, videos, audio, and text, perform semantic segmentation, identify independent semantic units, generate entity labels, attribute labels, and relation labels for each semantic unit, and store them in vector form;
[0075] Based on the generated structured tag set, a cross-modal semantic association network is constructed. Each semantic unit is treated as a node, and the cosine similarity between different nodes is calculated. According to a preset threshold, weighted edges are established between nodes with high semantic similarity to form the initial cross-modal semantic association network.
[0076] Simulate a series of search queries submitted by a user, vectorize the user queries, and construct a user intent context by combining simulated user history queries and interaction behaviors. Perform preliminary retrieval in a cross-modal semantic association network to obtain the first candidate set.
[0077] Calculate the regional semantic matching degree, cross-modal association strength, and regional saliency weight for each candidate media. Initially, the saliency weights of all regions are set to the default values.
[0078] The collaborative re-ranking algorithm is used to calculate the comprehensive relevance score, sort the results, and generate a final result list.
[0079] Present the reordered final results list to the simulated user. For each search result, the simulation highlights the area most relevant to the query, including bounding boxes in images, markers on the video timeline, and highlighted paragraphs in text.
[0080] Simulate user interaction with search results and dynamically update the salience weights of the interacted regions accordingly, while optimizing edge weights and connections in the semantic association network.
[0081] The specific data from the above simulation experiment are as follows:
[0082] Query sequence Query content Top 1 media type returned Region semantic matching degree Cross-modal correlation strength Regional saliency weights (before / after) Overall Relevance Score Is it related to manual judgment? The system took action (after modification). Q1 "Golden Retriever on the Beach" image 0.92 0.00 1.0 / 1.1 0.552 yes Highlight the dog area; update the weight of this area. Q2 Its cry audio 0.88 0.78 1.0 / 1.0 0.762 yes Calculate S(dog graph, dog bark audio) = 0.78 > λ, and construct a new edge. Q3 "Red sports car" video 0.95 0.00 1.0 / 1.0 0.570 yes Highlight vehicle area Q4 "Its engine roaring..." audio 0.82 0.85 1.2 / 1.3 0.823 yes The edge already exists; strengthen its weight. Update the audio node weight. Q5 (Cross-modal) Describe a sailboat at sunset in words. text 0.90 0.15 1.0 / 1.0 0.555 yes Return to relevant poetry paragraphs
[0083] Table 1
[0084] Experimental Analysis:
[0085] By analyzing the retrieval results of the model under different simulated queries and comparing the consistency between the dynamically adjusted comprehensive relevance score and the simulated user interaction behavior, the accuracy of the present invention in identifying user intent and providing personalized and accurate retrieval is verified.
[0086] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.
Claims
1. An AI-driven media knowledge base retrieval method based on region tagging, characterized in that: include: S1. Using a multimodal large model, perform semantic segmentation on multimodal media files in the knowledge base, identify the basic units carrying independent semantics in the media files, including spatial regions, temporal segments and text paragraphs, and generate a structured tag set for the identified semantic units. The multimodal media files include images, videos, audio and text files, and the structured tag set includes entity tags, attribute tags and relation tags. S2. Based on the structured tag set generated in step S1, a cross-modal semantic association network is constructed. The nodes in the network represent regions of different media files, and the edges represent the semantic similarity relationship between nodes. By calculating the semantic similarity of the region tag vectors, cross-modal semantic links are established between regions of different media to achieve semantic association of cross-modal content. S3. Vectorize the user's current search query and record historical queries and user interaction behaviors in the current session, representing them as user intent context. Based on the user's current search query and user intent context, perform a preliminary search in the cross-modal semantic association network to obtain a first candidate set. Then, use a collaborative re-ranking algorithm with regional semantic matching degree, cross-modal association strength, and regional saliency weight as input features to re-rank the first candidate set, obtaining a final result list sorted by comprehensive relevance score, specifically including the following: The current search query submitted by the user Transformed into query vectors through a pre-trained language model The user intent context is represented as a set of historical query vectors. User interaction logs This includes the media items that the user has clicked and the set of specific semantic unit nodes that are focused. and query vector Cross-modal semantic association networks Unified node representation vector Semantic similarity matching was performed, and the cosine similarity score was obtained. Select similarity score The top K nodes, where V is the set of nodes, E is the set of edges, and the media files associated with each node. Constitute the first candidate set ; A collaborative reordering model is used for the first candidate set. Each candidate media The model performs a comprehensive scoring, and its input features include the region semantic matching degree. Cross-modal correlation strength Regional significance weight ; Using the extracted features as input, a linear weighted model is used to calculate the comprehensive relevance score for each candidate media file. According to the calculation The media files in the first candidate set are sorted in descending order to obtain the final result list sorted by comprehensive relevance score, where A, B, and C are the weight coefficients of the linear weighted model. S4. Present the user with a list of reordered search results, and highlight the region most relevant to the query in each result. By capturing the user's interaction with the search results, dynamically update the region saliency weights and optimize the edge weights and connections in the cross-modal semantic association network.
2. The AI-driven media knowledge base retrieval method based on region tagging according to claim 1, characterized in that: In step S1, a multimodal large model is used to perform semantic segmentation on multimodal media files in the knowledge base, identifying basic units carrying independent semantics in the media files, including spatial regions, temporal segments, and text paragraphs. The identified semantic units are then used to generate a structured tag set. The multimodal media files include images, videos, audio, and text files, and the structured tag set includes entity tags, attribute tags, and relation tags, specifically including the following: For image and video files, a deep learning-based visual perception model is used to identify and extract visual semantic entities in the image, determine their spatial regions in the image, and generate a first entity label, a first attribute label, and a first relationship label for each identified visual semantic entity region. The first entity label describes the specific visual object represented by the visual semantic entity region, the first attribute label describes the visual features of the visual object, and the first relationship label describes the visual relationship between the visual semantic entity region and other regions. For the audio track portion of audio and video files, a deep learning-based acoustic perception model is used to identify and extract semantic entities in the audio, determine their temporal segments in the audio stream, and generate a second entity label, a second attribute label, and a second relation label for each identified audio semantic entity segment. The second entity label describes the specific acoustic event represented by the segment, the second attribute label describes the attributes of the acoustic event entity, and the second relation label describes the relationship between the audio semantic entity segment and other audio segments. For text files, a deep learning-based natural language processing model is used to identify and extract semantic entities in the text, determine their paragraphs in the text, and generate a third entity label, a third attribute label, and a third relation label for each identified text semantic entity paragraph. The third entity label describes the specific text content represented by the paragraph, the third attribute label describes the attribute of the text content, and the third relation label describes the semantic relationship between the text paragraph and other text paragraphs. All generated entity tags, attribute tags, and relationship tags are converted into high-dimensional vector representations and stored in a unified multimodal semantic index library.
3. The AI-driven media knowledge base retrieval method based on region tagging according to claim 1, characterized in that: In step S2, based on the structured tag set generated in step S1, a cross-modal semantic association network is constructed. Nodes in the network represent regions of different media files, and edges represent semantic similarity relationships between nodes. By calculating the semantic similarity of region tag vectors, cross-modal semantic links are established between regions of different media to achieve semantic association of cross-modal content. Specifically, this includes the following: Each semantic unit generated in S1, including spatial regions of images, temporal segments of audio, and paragraphs of text, is treated as an independent node. Each node, when added to the network, has the following attributes: the corresponding original media file ID, spatiotemporal location information, and a structured tag vector set. All tag vectors from each node are then fused using a weighted average to generate a unified node representation vector. ,in, It is an entity label vector. It is the average of the attribute label vectors. It is the average of the relation label vectors. These are preset weights; For any two nodes and Calculate its unified node representation vector and The cosine similarity between them is calculated using the following formula: ,in, It is a node With nodes Cosine similarity score between them , They are the unified representation vectors , The k-th component, where n is the dimension of the vector; Set a similarity threshold When the cosine similarity between two nodes is greater than a certain threshold, an undirected edge is established between the two nodes. Set the weight of the undirected edge to . The constructed semantic association network is then instantiated as a weighted undirected graph. , where V is the set of nodes and E is the set of edges.
4. The AI-driven media knowledge base retrieval method based on region tagging according to claim 1, characterized in that: The first candidate set is processed using a collaborative reordering model. Each candidate media The model performs a comprehensive scoring, and its input features include the region semantic matching degree. Cross-modal correlation strength Regional significance weight Further including: Region semantic matching degree: Calculate the query vector With candidate media A unified node representation vector for all semantic units within the domain. The highest cosine similarity between them is calculated using the following formula: ,in, Indicates to candidate media The maximum similarity score of all included nodes is taken; Cross-modal correlation strength: Calculating candidate media The set of nodes that the user has followed in the context of the user's intent The strongest semantic association strength between them is calculated using the following formula: ,in, It is a connecting node in a semantic association network. and edge weights; when If it is an empty set, then the eigenvalue is zero; Regional saliency weight: Calculating candidate media The dynamic weight corresponding to the semantic unit that best matches the current query. This weight is updated based on historical user interaction feedback, and the calculation formula is as follows: ,in, It is a saliency weight function that takes a semantic unit as input and outputs the saliency weight value of that semantic unit.
5. The AI-driven media knowledge base retrieval method based on region tagging according to claim 1, characterized in that: In step S4, a reordered list of search results is presented to the user, and the region most relevant to the query is highlighted in each result. By capturing the user's interaction with the search results, the salience weights of the regions are dynamically updated and the edge weights and connections in the cross-modal semantic association network are optimized. Specifically, this includes the following: The final result list after reordering in step S3 is presented to the user terminal. For each media file in the list, the semantic unit that best matches the current query vector is highlighted. The semantic unit is calculated in step S3 based on the region semantic matching degree. When, this makes the region corresponding to the node with the highest similarity score; Record user interaction behaviors during the interaction with search results, including clicks, long dwell times, region focusing, and negative feedback marking. By capturing user interaction behaviors with search results, dynamically update the region saliency weights and optimize the edge weights and connection relationships in the cross-modal semantic association network.
6. The AI-driven media knowledge base retrieval method based on region tagging according to claim 5, characterized in that: The recording of user interaction behaviors during the interaction with search results, including clicks, long dwell times, region focusing, and negative feedback marking, and the dynamic updating of the region saliency weights and optimization of edge weights and connection relationships in the cross-modal semantic association network by capturing user interaction behaviors with search results, further includes: Dynamic updates of regional saliency weights: The trigger condition is set when a user clicks on a media result and stays on it for a long time; the update rule is to update the most relevant highlighted node in that media. The significance weights are strengthened, and the updated formula is: ,in, This is the updated weight. It is the weight before the update. It is a fixed positive increment; Optimization of cross-modal semantic association networks: Set the trigger condition to occur when a user interacts consecutively with two nodes from different media within a session. and When an interaction occurs, the optimization rule is when the node... and An edge already exists between them. Then, the weight of the edge is increased based on the intensity of user interaction; when the node and There are no edges between them, cosine similarity If the preset threshold is met, construct a new edge and calculate the cosine similarity. As initial weights.