A video big data retrieval method and system based on semantic tags

Abstract

This invention relates to the field of video data processing and multimedia information retrieval technology, specifically a video big data retrieval method and system based on semantic tags. The method includes: a data mapping step: mapping a video stream into a sequence of feature vectors using a multimodal encoder; a difference calculation step: calculating the rate of change of differences between adjacent feature vectors to generate a semantic velocity vector; an anchor point extraction step: extracting semantic abrupt change event anchor points based on a comparison of the velocity vector magnitude with a threshold; an index construction step: constructing a differential manifold index with semantic transitions as nodes and time spans as edges; a dynamic dimensionality reduction step: constructing a dynamic mask matrix based on the retrieval request, projecting nodes to a low-dimensional subspace; and a matching output step: calculating the matching degree in the subspace and outputting the result. This invention significantly improves retrieval speed and signal-to-noise ratio, and substantially reduces storage volume through index event anchor points and dynamic subspace folding.

Description

Technical Field

[0001] This invention relates to the field of video data processing and multimedia information retrieval technology, specifically to a video big data retrieval method and system based on semantic tags. Background Technology

[0002] In video big data processing applications, retrieval systems rely on feature extraction and index construction of massive unstructured video streams to ensure the accuracy and response speed of content queries. This usually requires the use of multimodal coding technology to map video images into high-dimensional feature vectors for semantic alignment.

[0003] For video content retrieval, existing solutions generally adopt full frame scanning or static keyframe indexing architecture, that is, to extract and store features indiscriminately for each frame or fixed interval frames in the video stream, build a dense static feature library, and calculate the global similarity between the query vector and all stored features during retrieval.

[0004] Although this solution is feasible in a small-scale or simple context, it ignores the evolution of video content over time and lacks a filtering mechanism for redundant information. When faced with massive amounts of monitoring data, the system has to store a large number of invalid indexes that represent static backgrounds or repetitive images, resulting in an extreme waste of storage resources.

[0005] Furthermore, performing full matching calculations in a high-dimensional feature space is prone to the curse of dimensionality, causing the retrieval time to increase exponentially. It also cannot effectively shield against environmental noise such as rain, snow, or background clutter, resulting in low signal-to-noise ratio and unstable recall of the retrieval results, making it difficult to support millisecond-level real-time semantic analysis and logical reasoning of large-scale video data.

[0006] Therefore, how to establish an indexing mechanism with semantic dynamic perception capabilities, effectively eliminate static redundancy and environmental noise, and improve the real-time performance and robustness of retrieval through dimensionality reduction processing, has become an urgent technical problem to be solved. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention discloses a video big data retrieval method and system based on semantic tags. Specifically, the technical solution of this invention is as follows:

[0008] A video big data retrieval method based on semantic tags includes:

[0009] The video data stream to be retrieved is acquired, and a multimodal encoder is used to map the video data stream into a continuous sequence of feature vectors.

[0010] Initialize and establish storage space for storing temporal semantic features;

[0011] In response to receiving a retrieval request for a video data stream, a differential retrieval process based on semantic change rate is triggered, including:

[0012] Step 1: Based on the feature vector sequence, calculate the rate of change of the difference between feature vectors at adjacent time points to generate a semantic velocity vector;

[0013] Step 2: Compare the magnitude of the semantic velocity vector with the preset mutation threshold, and extract the time point where the semantics undergoes a significant mutation as the event anchor point based on the comparison result;

[0014] Step 3: Construct a differential manifold index based on event anchors, wherein the differential manifold index is configured as a directed acyclic graph data structure with semantic state transitions as nodes and time spans as edges;

[0015] Step 4: Based on the query vector generated by the retrieval request, construct a dynamic mask matrix and project the node data in the differential manifold index to the low-dimensional subspace indicated by the non-zero elements in the dynamic mask matrix;

[0016] Step 5: Calculate the matching degree between the query vector and the projected nodes in the low-dimensional subspace, and output the retrieval results.

[0017] Preferably, based on the feature vector sequence, the rate of change of the difference between feature vectors at adjacent time points is calculated to generate a semantic velocity vector, configured as follows:

[0018] Obtain the feature vector at the current time step and the feature vector at the previous time step;

[0019] Calculate the difference vector obtained by subtracting the feature vector from the feature vector at the previous time step from the feature vector at the current time step;

[0020] The difference vector is used as a semantic velocity vector to characterize the rate of change of semantic features on the time axis, where the semantic velocity vector is used to characterize the evolution trend of actions or events in the video content.

[0021] Preferably, based on the comparison result of the semantic velocity vector magnitude and the preset mutation threshold, the time point where a significant semantic mutation occurs is extracted as the event anchor point, configured as follows:

[0022] The Euclidean norm of the semantic velocity vector is calculated as the semantic change intensity value;

[0023] If the semantic change intensity value is greater than the preset mutation threshold, it is determined that a semantic state transition has occurred at the current moment. The semantic velocity vector corresponding to the moment of transition is retained and marked as the event anchor point.

[0024] If the semantic change intensity value is less than or equal to the preset mutation threshold, the current moment is determined to be a static redundant segment, and the corresponding feature vector is discarded.

[0025] Preferably, constructing a differential manifold index based on event anchors includes:

[0026] Using the independent component analysis algorithm, event anchors are decoupled into orthogonal semantic basis vectors;

[0027] Based on the temporal causal relationship, the decoupled event anchor points are connected to form a directed acyclic graph structure;

[0028] The semantic state changes are stored in the nodes of the directed acyclic graph (DAG), and the time span of the semantic state changes is stored in the edges of the DAG, generating a differential manifold index.

[0029] Preferably, a dynamic mask matrix is ​​constructed based on the query vector generated from the retrieval request, and configured as follows:

[0030] Parse the search request and extract key semantic terms from it;

[0031] Determine the dimension index corresponding to the key semantic words in the feature space, and mark the dimension index as the activation dimension;

[0032] Generate a dynamic mask matrix, where the values ​​in the active dimension are set to retain coefficients, and the values ​​in the inactive dimension are set to zero;

[0033] Dimensionality reduction is achieved by using a dynamic mask matrix and a differential manifold index for matrix operations, thereby preserving semantic dimension data relevant to the retrieval request.

[0034] Preferably, the matching degree between the query vector and the projected nodes is calculated in the low-dimensional subspace, and the retrieval results are output, configured as follows:

[0035] In a low-dimensional subspace defined by a dynamic mask matrix, calculate the cosine similarity between the projected query vector and the projected node vector.

[0036] Based on the cosine similarity value, the path with the highest cumulative similarity is searched in the directed acyclic graph of the differential manifold index as the maximum matching path.

[0037] The video segment corresponding to the maximum matching path is output as the search result that meets the search request.

[0038] A video big data retrieval system based on semantic tags, comprising:

[0039] The data mapping module is configured to acquire the video data stream to be retrieved and map the video data stream into a continuous sequence of feature vectors using a multimodal encoder;

[0040] The retrieval triggering module is configured to trigger a differential retrieval process based on semantic change rate in response to receiving a retrieval request for a video data stream.

[0041] The differential manifold retrieval module includes:

[0042] The velocity calculation unit is configured to calculate the rate of difference in feature vectors between adjacent time steps based on the feature vector sequence, and generate a semantic velocity vector.

[0043] The anchor point extraction unit is configured to compare the magnitude of the semantic velocity vector with a preset mutation threshold, and extract the time point where a significant semantic mutation occurs as the event anchor point based on the comparison result.

[0044] The index building unit is configured to build a differential manifold index based on event anchors, wherein the differential manifold index is configured as a directed acyclic graph data structure with semantic state transitions as nodes and time spans as edges;

[0045] The spatial folding unit is configured to construct a dynamic mask matrix based on the query vector generated by the retrieval request, and project the node data in the differential manifold index to the low-dimensional subspace indicated by the non-zero elements in the dynamic mask matrix;

[0046] The matching output unit is configured to calculate the matching degree between the projected query vector and the projected node vector in the low-dimensional subspace and output the retrieval results.

[0047] Preferably, the index building unit is further configured to:

[0048] Using the independent component analysis algorithm, event anchors are orthogonally decoupled to separate the semantic components of objects, actions, and environment;

[0049] Only semantic components with non-zero semantic speed modulus are stored in the differential manifold index to compress the index size.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. This invention effectively solves the problem of static redundant information occupying storage resources in massive video data by using semantic velocity vector calculation and adaptive threshold determination mechanism. Unlike traditional full frame storage, this scheme calculates the differential change rate of feature vectors at adjacent time points and uses a threshold strategy based on statistical mean and standard deviation to extract significant mutation points, thereby automatically filtering out long-standing static semantic segments in the background. This achieves semantic-level deduplication and non-uniform discretization sampling of video content, maximizing the utilization of storage space while ensuring that key events are not lost.

[0052] 2. This invention introduces query-driven dynamic mask matrix and subspace folding technology, which greatly improves the response speed and computational efficiency of the retrieval system. By parsing the key semantic words in the retrieval request and constructing a diagonal mask matrix, this method can project high-dimensional feature data into a low-dimensional subspace that is only related to the query at the moment of retrieval. This forces the shielding of noise interference from irrelevant dimensions during the calculation process, effectively avoids the curse of dimensionality in high-dimensional space, and achieves millisecond-level real-time retrieval response.

[0053] 3. This invention employs an independent component analysis decoupling and differential manifold indexing strategy, which significantly enhances the system's anti-interference capability and semantic matching signal-to-noise ratio in complex environments. By using the independent component analysis algorithm to separate the mixed video signal into orthogonal semantic basis vectors of objects, actions, and environment, and combining them with semantic labeling logic to establish a mapping, the system can accurately distinguish dynamic behavior from environmental backgrounds such as rain and snow, thus maintaining extremely high retrieval accuracy and robustness even under adverse weather conditions or background clutter interference.

[0054] 4. This invention establishes a maximum matching path search and time-weighted logic based on directed acyclic graphs, giving the retrieval results rigorous causality and logical depth. The system transforms the linear time axis into a topological structure that includes branches and convergences, and introduces a conditional exponential decay mechanism based on time span during path search. This not only restores the fine evolution process of events, but also verifies the continuity and temporal rhythm of actions through mathematical means, effectively preventing logical error recall caused by accidental feature matching. Attached Figure Description

[0055] The present invention will be further explained below with reference to the accompanying drawings and embodiments:

[0056] Figure 1 This is a flowchart of the method of the present invention.

[0057] Figure 2 This is a system structure diagram of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0059] Example 1:

[0060] Please see Figure 1 A video big data retrieval method based on semantic tags includes: acquiring the video data stream to be retrieved; mapping the video data stream into a continuous sequence of feature vectors using a multimodal encoder; initializing and establishing a storage space for storing temporal semantic features; and, in response to receiving a retrieval request for the video data stream, triggering a differential retrieval process based on semantic change rate, including:

[0061] Step 1: Based on the feature vector sequence, calculate the rate of change of the difference between feature vectors at adjacent time points to generate a semantic velocity vector;

[0062] Step 2: Compare the magnitude of the semantic velocity vector with the preset mutation threshold, and extract the time point where the semantics undergoes a significant mutation as the event anchor point based on the comparison result;

[0063] Step 3: Construct a differential manifold index based on event anchors, wherein the differential manifold index is configured as a directed acyclic graph data structure with semantic state transitions as nodes and time spans as edges;

[0064] Step 4: Based on the query vector generated by the retrieval request, construct a dynamic mask matrix and project the node data in the differential manifold index to the low-dimensional subspace indicated by the non-zero elements in the dynamic mask matrix;

[0065] Step 5: Calculate the matching degree between the query vector and the projected nodes in the low-dimensional subspace, and output the retrieval results;

[0066] This embodiment discloses a video big data retrieval method based on semantic tags; the system executes the function of the data mapping module, obtains the video data stream to be retrieved (i.e., the unprocessed raw video signal) through a multimodal encoder, and maps it into a continuous sequence of feature vectors. Specifically, the multimodal encoder adopts a CLIP-based dual-stream architecture, in which the visual encoder uses the ViT-L / 14 structure, containing 24 Transformer modules with a hidden layer dimension of 1024; the text encoder uses a 12-layer Transformer structure.

[0067] By conducting comparative learning training on the LAION-2B large-scale dataset, the two achieved the alignment of video frame features and text features in a unified high-dimensional spherical space, thus providing a unified metric basis for subsequent semantic mapping.

[0068] Simultaneously, a graph structure storage space for storing temporal semantic features is initialized. This storage space is the underlying container for the differential manifold index built in subsequent steps. Both point to the same object in physical storage and logical definition, ensuring the consistency of storage structure names. The system is in a listening state and, in response to receiving a retrieval request for the video data stream, triggers a differential retrieval process based on semantic change rate. This process first calculates the differential change rate of feature vectors at adjacent time points based on the feature vector sequence to generate a semantic velocity vector, which aims to characterize the trend of video content evolution. The magnitude of the semantic velocity vector is compared with a preset mutation threshold. Based on the comparison result, the time point where a significant semantic mutation occurs is extracted as the event anchor point, thereby realizing non-uniform discretization sampling of the continuous video stream.

[0069] Based on this, a differential manifold index is constructed using the extracted event anchors. This index adopts a directed acyclic graph structure with semantic state transitions as nodes and time spans as edges. Nodes store semantic changes while edges store time intervals. A dynamic mask matrix is ​​constructed based on the query vector generated from the retrieval request. This matrix is ​​used to project the node data in the differential manifold index onto a low-dimensional subspace, aiming to shield noise from irrelevant dimensions during retrieval. The matching degree between the query vector and the projected nodes is calculated in the low-dimensional subspace defined by the dynamic mask matrix, and the retrieval results are output.

[0070] This embodiment no longer indexes massive amounts of redundant static frames, but only indexes event anchors whose semantics have changed; through dynamic subspace folding, only dimensions related to the query are calculated during retrieval, thus avoiding the curse of dimensionality in high-dimensional space;

[0071] To verify the technical effectiveness of this embodiment, a comparative experiment was conducted on a test set containing 10,000 hours of surveillance video. The experimental results show that compared with the traditional full-frame scanning retrieval method, the average retrieval time of this method decreased from 450ms to 12ms, demonstrating a significant improvement in retrieval speed. It should be noted that the experimental data of 12ms is based on a test scenario with a pre-built index. That is, for the 10,000 hours of existing video, the system pre-executed the differential manifold index construction work in steps 1 to 3, converting it into a structured graph for storage. When a retrieval request is received, the system only needs to execute steps 4 and 5, namely spatial folding and path search.

[0072] This offline indexing-online retrieval processing logic eliminates the time-consuming feature extraction of massive amounts of data, thereby achieving millisecond-level response speeds and verifying the efficiency of this method in processing large-scale video data. At the same time, while maintaining a high recall rate of 94.5%, the index storage volume is reduced by 88%. Furthermore, the semantic matching signal-to-noise ratio (SNR) is improved by 4.2 dB in complex environments such as rain and snow, strongly supporting the assertion that this method improves retrieval speed, ensures high recall, and enhances the SNR.

[0073] Example 2:

[0074] Based on the feature vector sequence, the rate of change of the difference between the feature vectors at adjacent time points is calculated to generate a semantic velocity vector. The configuration is as follows: obtain the feature vector at the current time point and the feature vector at the previous time point; calculate the difference vector obtained by subtracting the feature vector at the previous time point from the feature vector at the current time point; use the difference vector as a semantic velocity vector to represent the rate of change of semantic features on the time axis, where the semantic velocity vector is used to represent the evolution trend of actions or events in the video content.

[0075] This embodiment specifies the steps for generating the semantic velocity vector; the system retrieves the current time from the cache. eigenvectors Compared to the previous moment eigenvectors The difference vector obtained by subtracting the feature vector from the feature vector at the previous time step is calculated using the following formula:

[0076] ;

[0077] in, : The semantic velocity vector at any given time is derived from differential calculation, and its physical meaning is to represent the direction of semantic change. : The original semantic feature vector at each time step is derived from the output of the multimodal encoder; The original semantic feature vector from the previous time step;

[0078] Based on this, the system will use the difference vector As a semantic velocity vector representing the rate of change of semantic features on the time axis, it is used to capture the evolution trend of actions or events, such as accurately capturing the downward movement semantic component in the process of going from standing to sitting.

[0079] This embodiment can automatically filter out static semantics that have been present in the background for a long time by calculating the difference vector. That is, for static objects, the difference vector approaches zero. This allows the indexing system to automatically focus on dynamic behaviors and events in the video, improving the system's sensitivity and robustness to action-related searches.

[0080] Example 3:

[0081] Based on the comparison between the magnitude of the semantic velocity vector and the preset mutation threshold, the time point where a significant semantic mutation occurs is extracted as the event anchor point. The configuration is as follows: calculate the Euclidean norm of the semantic velocity vector as the semantic change intensity value; if the semantic change intensity value is greater than the preset mutation threshold, it is determined that a semantic state transition has occurred at the current moment, the semantic velocity vector corresponding to the moment of transition is retained, and it is marked as the event anchor point; if the semantic change intensity value is less than or equal to the preset mutation threshold, it is determined that the current moment is a static redundant segment, and the corresponding feature vector is discarded.

[0082] This embodiment specifies the steps for extracting event anchor points; it also calculates the semantic velocity vector. The Euclidean norm as a value of semantic change intensity The calculation formula is as follows:

[0083] ;

[0084] in, : The intensity of semantic change at any given time is derived from norm calculation, and its physical meaning is the degree of drasticness of semantic change; Semantic velocity vector;

[0085] Execute threshold determination and classification processing logic; respond to semantic change intensity value Greater than the preset mutation threshold The system determines that a semantic state transition has occurred at the current moment, such as a screen change or a burst of action, and retains the semantic velocity vector corresponding to the moment of the transition, marking it as an event anchor point. To ensure the determinism of parameters and the reproducibility of the technical solution, a mutation threshold is preset. The calculation method is defined as follows: Select a historical benchmark video data with a duration of not less than 24 hours during the system initialization phase, and calculate the average semantic velocity vector magnitude at all times. and standard deviation and set This threshold setting strategy is based on statistical methods. The rule ensures that only extreme points deviating from the mean by more than three standard deviations—that is, significant abrupt changes with a statistical probability of less than 0.3%—are used as event anchors; conversely, events are anchored based on the intensity of semantic changes. Less than or equal to the preset mutation threshold The system determines that the current time segment is a static redundant segment and directly discards the corresponding feature vector without writing it into the index;

[0086] This embodiment achieves semantic-level deduplication and adaptive sampling of video content; it retains more anchor points in segments with violent motion, while retaining almost no anchor points in segments with still images; this mechanism maximizes the utilization of storage space and effectively reduces the storage cost of the system while ensuring that no critical events are lost.

[0087] Example 4:

[0088] According to a video big data retrieval method based on semantic tags in Embodiment 3, a differential manifold index is constructed based on event anchors, including: using the independent component analysis algorithm to decouple the event anchors into orthogonal semantic basis vectors; connecting the decoupled event anchors according to temporal causality to form a directed acyclic graph structure; storing the change in semantic state in the nodes of the directed acyclic graph, storing the time span experienced by the change in semantic state in the edges of the directed acyclic graph, and generating a differential manifold index;

[0089] This embodiment details the steps for constructing the differential manifold index; it utilizes the FastICA algorithm as the independent component analysis algorithm and uses negative entropy as a measure of non-Gaussianity to anchor events. Decoupling results in orthogonal semantic basis vectors; specifically, this is done to compute the unmixing matrix. The system selects the 24-hour historical benchmark video data mentioned in Example 3 for calculating the threshold, extracts all event anchors generated within this time period to construct a training set, and runs the FastICA algorithm on this training set to obtain a globally universal unmixing matrix. and the corresponding independent component space; aiming to separate mixed semantic signals into statistically independent source signals;

[0090] To address the issue of unclear component semantics after ICA decoupling, the system performs a semantic labeling step: pre-constructing a calibration dataset containing four standard labels: objects such as vehicles and pedestrians, attributes such as red and bright, actions such as running and falling, and environments such as rain and dimness.

[0091] Input the calibration data into the system and calculate the Pearson correlation coefficient between the activation intensity sequence of each independent component and the occurrence sequence of the standard label; if the... If the correlation coefficient between an independent component and the action label sequence is greater than 0.85, then the first independent component is considered to be an action label sequence. The dimension is labeled as the action semantic dimension; if the correlation coefficient between an independent component and any standard label sequence is lower than a preset threshold, such as 0.5, it is labeled as an unclassified dynamic dimension to prevent the loss of semantic changes of unknown types; and so on, a mapping table containing the correspondence between dimension index and semantic category is constructed and stored in the system metadata; according to the temporal causal relationship, the decoupled event anchors are connected to form a directed acyclic graph structure.

[0092] To overcome the limitation that temporal concatenation of a single video stream can only form a linear linked list, thus causing graph search algorithms to fail, this step specifically adopts a multi-scale temporal window concatenation strategy: setting a temporal look-ahead window. For example, 5 seconds, for any anchor point Not only connecting to its next adjacent anchor point It also establishes anchor points pointing to subsequent points within the window. The jump connection edges; this topology introduces a large number of branching and merging paths in the graph, representing the fine evolution of events and the fast summarization manifold, respectively, thus giving the DAG structure a substantial path selection space;

[0093] Based on this, the system stores the changes in semantic state, i.e., the decoupled semantic velocity vector, in the nodes of the directed acyclic graph; at the same time, it stores the time span experienced by the semantic state changes in the edges of the directed acyclic graph. And connection type marking; specifically, the system distinguishes connection attributes when constructing edges: for time-adjacent anchor points directly connected due to the discarding of static redundant segments in Embodiment 3, i.e., spanning discarded static frames, their connection type is marked as a static crossing edge; for skip connections between non-adjacent anchor points generated by the multi-scale temporal window strategy in this embodiment, their connection type is marked as a skip connection edge; wherein, time span The calculation formula is:

[0094] ;

[0095] in, : The timestamp corresponding to the current event anchor point; : The timestamp of the previous node directly connected to the current node in a directed acyclic graph;

[0096] This time span parameter is specifically used to quantify the duration between semantic state transitions, thereby enabling the restoration of the temporal rhythm of events during retrieval.

[0097] This embodiment transforms video data on a linear time axis into a structured topological graph through independent component analysis decoupling and directed acyclic graph construction. This structure not only records the magnitude of semantic changes but also fully preserves the causal order and duration of the changes, enabling the system to support complex retrieval tasks such as temporal logic retrieval and significantly improving the logical depth of the retrieval.

[0098] Example 5:

[0099] According to a semantic tag-based video big data retrieval method in Embodiment 4, a dynamic mask matrix is ​​constructed based on the query vector generated from the retrieval request. The configuration is as follows: parsing the retrieval request and extracting key semantic words from the retrieval request; determining the dimension index corresponding to the key semantic words in the feature space and marking the dimension index as the active dimension; generating a dynamic mask matrix, wherein the values ​​of the dynamic mask matrix on the active dimension are set as retention coefficients and the values ​​on the inactive dimension are set to zero; and performing matrix operations using the dynamic mask matrix and the differential manifold index to achieve dimensionality reduction, thereby retaining the semantic dimension data related to the retrieval request.

[0100] This embodiment details the steps for constructing a dynamic mask matrix; it parses the retrieval request and uses a natural language processing model to extract key semantic words; the natural language processing model specifically adopts a BERT-Base pre-trained model and connects it to a CRF conditional random field layer for named entity recognition and part-of-speech tagging fine-tuning;

[0101] Determine the dimensional indices of key semantic words in the feature space, and mark these corresponding dimensional indices as activation dimensions. Specifically, to correct the logical defects caused by relying solely on part-of-speech mapping—namely, the misclassification of environmental nouns such as rain as objects, and the lack of adjective mapping rules—the system executes hierarchical semantic mapping logic: First, keywords are matched against a pre-set environmental semantic dictionary, which covers meteorological environmental terms such as rain, snow, fog, and sunlight. If a match is found, it is directly marked as an environmental category. Second, if no match is found, part-of-speech mapping is performed, mapping verbs to actions and nouns to objects. Third, for adjectives or adverbs, such as "red," dependency parsing is used to identify the core word they modify, but instead of simply inheriting the core word's category, it is mapped to an independent attribute category.

[0102] The system queries the dimension index mapping table constructed in Example 4, extracts independent component dimensions highly correlated with attribute labels such as red, and marks these dimensions as activation dimensions. Through this orthogonal mapping, red cars and blue cars activate different attribute dimension combinations in the mask matrix, thereby achieving fine-grained feature differentiation. The system also queries the dimension index-semantic category mapping table generated in step 4, extracting all dimension indices under the semantic category as activation dimensions. For example, when the search term is "running in the rain," the system maps rain to the environment through first-level logic and running to the action through second-level logic, thereby activating all feature dimensions marked as environment and action in the mapping table, such as dimensions 12, 45, and 89, thus establishing a precise mapping between natural language keywords and feature vector dimensions. Furthermore, the system executes empty set verification logic.

[0103] If, after the above hierarchical semantic mapping, the set of activation dimensions... If the value is empty, meaning the search term does not match any preset semantic category, a global rollback mechanism is automatically triggered, marking all dimensions as active dimensions. Including a full-dimensional index, the dynamic mask matrix degenerates into an identity matrix or scalar matrix to perform fuzzy retrieval based on full features, avoiding empty search results; generating the dynamic mask matrix. This matrix is ​​a matrix with dimension 1. a diagonal matrix, where This represents the total number of feature dimensions in the feature vector sequence, and its element assignment rules are as follows:

[0104] For any dimension index ( ):

[0105] when At that time, if , ;like ;

[0106] when hour, ;

[0107] in, : Dynamic mask matrix Line number Column elements; The set of activation dimensions is derived from the semantic parsing results; : Retention factor, usually set to 1; The residual background coefficient, set to a small non-zero value, aims to preserve weak background semantic information and prevent key features from being completely truncated due to classification errors in the natural language processing model, thereby improving the system's fault tolerance. In scenarios with extremely high computational speed requirements or extremely noisy backgrounds, this coefficient... It can also be set as ;

[0108] Dimensionality reduction is achieved by performing matrix operations using the dynamic mask matrix and the node vectors in the differential manifold index. The calculation formula is as follows:

[0109] ;

[0110] in, Refers to the node vector in the differential manifold index. This is a diagonal mask matrix dynamically generated based on the retrieval request. Subscripts are introduced here to clearly distinguish it from the original semantic feature vector defined in Example 2. This preserves semantic dimension data relevant to the retrieval request while forcing other irrelevant dimensions to zero.

[0111] This embodiment employs a query-driven dynamic dimensionality reduction mechanism. Unlike traditional global static dimensionality reduction, this mechanism only folds away irrelevant dimensions based on the user's intent at the moment of retrieval. This results in an exponential decrease in retrieval computation, and because the noise interference of irrelevant dimensions is removed, the retrieval accuracy still maintains extremely high robustness in complex contexts.

[0112] Example 6:

[0113] According to a semantic tag-based video big data retrieval method in Embodiment 5, the matching degree between the query vector and the projected nodes is calculated in a low-dimensional subspace, and the retrieval results are output. The configuration is as follows: In the low-dimensional subspace defined by the dynamic mask matrix, the cosine similarity value between the projected query vector and the projected node vector is calculated; based on the cosine similarity value, a temporal decay coefficient is introduced, and the similarity between nodes is weighted and decayed according to the time span of the edges in the path; the path with the highest cumulative weighted similarity is traversed and searched in the directed acyclic graph of the differential manifold index as the maximum matching path; the video segment corresponding to the maximum matching path is output as the retrieval result that meets the retrieval request.

[0114] This embodiment specifies the steps for calculating and outputting the matching degree; in order to solve the problem of dimensional mismatch in the feature space, i.e., the query vector The system resides in the original high-dimensional spherical space, while the node vectors reside in the feature space after ICA decoupling. The system performs a space alignment operation: reading the ICA demixing matrix generated during the index construction phase. Given that the mathematical premise of the ICA algorithm is data centralization, and the query vector... It is a static text feature vector output by a multimodal encoder, that is, it represents the positional meaning in the feature space. If we directly subtract the mean vector of all semantic velocity vectors in the training set... That is, to represent the meaning of velocity, it will lead to confusion of the physical meaning of position and velocity and dimensional errors;

[0115] Therefore, the system corrected the alignment logic here: it reads the mean vector of the static feature vector training set. Calculate the centralized query vector This operation ensures data centralization while maintaining the consistency of physical dimensions, i.e., position minus position. It should be noted that, based on the shared embedding space characteristics of the multimodal encoder, it is assumed that the feature distribution centers of the text modality and the video modality are statistically close. By subtracting the mean of the video static features, the discrete text query vector is mapped to a relative change space aligned with the video features.

[0116] To address the basis mismatch problem between the static feature space and the differential velocity feature space—that is, static text / image feature vectors cannot be directly trained using velocity data—[further details needed]. The matrix is ​​correctly parsed; the system incorporates a pre-trained static-dynamic mapping matrix. The matrix It is a linear transformation operator trained by minimizing the projection error between the static feature vector and the corresponding semantic velocity vector on a large-scale dataset; the mapping matrix Based on statistical regularities, the correlation between static features and high-frequency associated actions is learned, thereby predicting the most likely direction of semantic evolution in the feature space;

[0117] In order to construct an effective data pair for training this matrix The system executes the following alignment strategy: for any segment marked as a text description The video clips are input into a text encoder to obtain static feature vectors. As input Simultaneously, calculate the set of semantic velocity vectors corresponding to all frames within the video segment. The maximum pooling strategy is used to extract the vector with the largest modulus within the specified time period as the representative semantic velocity vector. ,Right now:

[0118] ;

[0119] Take it as the target Solve using the least squares method To minimize the loss function.

[0120] ;

[0121] Set training convergence threshold When the calculated loss function value The system determines convergence and stops iteration, thus ensuring that static features can be mapped to the direction of the most drastic semantic change. The system then performs a spatial alignment step: mapping the centered query vector to the semantic velocity space, calculated using the following formula:

[0122] ;

[0123] The mapped fitted velocity vector Projected onto the independent component space, the global unmixing matrix generated in Example 4 is explicitly referenced here. To ensure a consistent spatial base, the calculation formula is revised as follows:

[0124] ;

[0125] Using dynamic mask matrix For the mapped query vector Dimensionality reduction is performed to obtain the projected query vector. ,Right now Simultaneously, the node vectors in the differential manifold index are projected using a dynamic mask matrix to obtain the projected node vectors. ,Right now ;

[0126] In the low-dimensional subspace defined by the dynamic mask matrix, compute the projected query vector. With the projected node vector Cosine similarity values ​​between To address the potential risk of division-by-zero anomalies under sparse storage strategies, specifically when the semantic components stored by a node are completely orthogonal to the query intent, the projection vector... Since the modulus is zero, the system introduces numerical stability verification logic: calculate the denominator term. ;like ,in To preset the minimum value, for example Then directly determine the matching degree of the node. This avoids circuit breaking during the process; if Then, the following standard cosine similarity calculation formula is applied:

[0127] ;

[0128] in, Cosine similarity is a numerical value, and its physical meaning is the closeness of the angle between semantic vectors. The query vector after ICA spatial mapping and dynamic mask dimensionality reduction; : The node vector after dynamic masking;

[0129] Based on cosine similarity values, a dynamic programming algorithm is used to calculate and search for the path with the highest cumulative weighted similarity in the directed acyclic graph of the differential manifold index, which is then used as the maximum matching path. This is done to activate the time span parameter stored on the edges in the preceding steps. To address the dead parameter problem, the system restructured the path score calculation logic and introduced temporal continuity weights; the corrected path score... The calculation formula is as follows:

[0130] ;

[0131] in, The total number of nodes contained in the path is introduced into the denominator to normalize the length, prevent the score accumulation bias caused by the path growth alone, and ensure that the search results focus on the segment with the highest matching density within a unit time step. For the first on the path The cosine similarity values ​​of each node; The preset time decay coefficient is used to control the weight of the time span on the path score, and its preferred value range is... to For example, take ;in The unit is set to the reciprocal of the second. ), to ensure the index term The results are dimensionless numerical values; where Normalization is required, with the unit defined as the standard time step or seconds, to ensure that the exponential decay term is calculated in a dimensionless numerical space. For effective time series span function;

[0132] To address the large span that may result from the static redundant segment discarding strategy in Example 3. This will lead to The value approaches 0, thus incorrectly truncating valid paths. The function is defined as:

[0133] ;

[0134] Specifically, for static cross-edges, the determination is that they are not substantial time elapsed, and therefore... That is, no temporal decay penalty is applied; if the edge is marked as a normal sequence edge, then:

[0135] ;

[0136] That is, the correlation decay due to the passage of time is fully accounted for; if the edge is marked as a skip connection edge, then:

[0137] ;

[0138] in, The preset logical jump cost constant has the physical dimension of time (seconds), which is significantly smaller than the physical time span. This is intended to prevent the exponential term from being affected by the large time lag caused by rapid summarization. Approaching zero, thus ensuring that logical paths with long time spans remain competitive in retrieval;

[0139] This formula, through a conditional exponential decay mechanism, perfectly complies with the system's compression strategy while penalizing abnormally loose semantic transitions. It selects the path with the highest path score as the maximum matching path, locates the start and end time points in the original video stream based on the node timestamps in the path, and extracts the corresponding video segments as the search results that meet the search request.

[0140] This embodiment achieves accurate matching of the complete event evolution process by finding the maximum matching path on the directed acyclic graph and combining it with time span parameters for temporal weighting. This not only avoids the numerical risks of sparse vector computation, but also verifies the continuity and causality of action occurrence through mathematical means, thereby improving the robustness of complex action retrieval.

[0141] Example 7:

[0142] Please see Figure 2A video big data retrieval system based on semantic tags includes: a data mapping module configured to acquire the video data stream to be retrieved and map the video data stream into a continuous sequence of feature vectors using a multimodal encoder; a retrieval triggering module configured to trigger a differential retrieval process based on semantic change rate in response to a retrieval request for the video data stream; and a differential manifold retrieval module including: a velocity calculation unit configured to calculate the differential change rate of feature vectors at adjacent time points based on the feature vector sequence to generate a semantic velocity vector; and an anchor point extraction unit configured to compare the magnitude of the semantic velocity vector with a preset abrupt change threshold and, based on the comparison result... The system extracts time points where significant semantic changes occur as event anchors; an index construction unit is configured to build a differential manifold index based on the event anchors, wherein the differential manifold index is configured as a directed acyclic graph data structure with semantic state transitions as nodes and time spans as edges; a spatial folding unit is configured to construct a dynamic mask matrix based on the query vector generated by the retrieval request, and project the node data in the differential manifold index onto a low-dimensional subspace indicated by the non-zero elements in the dynamic mask matrix; and a matching output unit is configured to calculate the matching degree between the projected query vector and the projected node vector in the low-dimensional subspace, and output the retrieval results.

[0143] This embodiment provides a video big data retrieval system based on semantic tags. Specifically, the system includes: a data mapping module, physically corresponding to a computing unit equipped with an accelerator card, configured to acquire the video data stream to be retrieved and map the video data stream into a continuous sequence of feature vectors using a multimodal encoder; a retrieval triggering module, corresponding to the system's interactive interface, configured to trigger a differential retrieval process based on semantic change rate in response to a received retrieval request for the video data stream; and a core differential manifold retrieval module comprising multiple units: a velocity calculation unit configured to perform differential calculations to generate semantic velocity vectors; and anchor point extraction. The unit is configured to compare the magnitude of the semantic velocity vector with a preset mutation threshold to extract the time point where a significant semantic mutation occurs as the event anchor point; the index building unit is configured to build a differential manifold index based on the event anchor point and maintain a directed acyclic graph data structure with semantic state transitions as nodes and time spans as edges; the spatial folding unit is configured to build a dynamic mask matrix based on the query vector generated by the retrieval request and project the node data in the differential manifold index to a low-dimensional subspace; the matching output unit is configured to calculate the matching degree between the projected query vector and the projected node vector in the low-dimensional subspace and output the retrieval result.

[0144] The system architecture of this embodiment separates computation from storage and decouples feature extraction from index construction. The modules work together to support the real-time retrieval needs of large-scale video data. Especially in high-concurrency query scenarios, the spatial folding unit significantly reduces the computational load, demonstrating excellent system throughput performance.

[0145] Example 8:

[0146] The index building unit is also used to: orthogonally decouple event anchors using independent component analysis algorithms to separate the semantic components of objects, actions, and environment; and store only semantic components with non-zero semantic velocity moduli into the differential manifold index to compress the index size.

[0147] This embodiment further optimizes the configuration of the index building unit. This unit is also configured to perform the following operations: Utilizing Independent Component Analysis (ICA) algorithms, orthogonally decouple event anchors to separate the semantic components of objects, actions, and environment. For example, a complex scene can be decoupled into three orthogonal vectors: object, running action, and rain environment. To eliminate semantic uncertainty after ICA decoupling, this unit further executes semantic labeling logic: loading a calibration dataset containing labels for objects, actions, and environment; calculating the Pearson correlation coefficient between each independent component and the standard label sequence; associating dimensions with Pearson correlation coefficients higher than a preset threshold, such as 0.85, with their corresponding semantic categories, generating and storing a dimension index-semantic category mapping table, thereby providing accurate dimension selection criteria for subsequent spatial folding units; and implementing a sparse storage strategy, storing only semantic components with non-zero semantic velocity moduli in the differential manifold index. For example, in response to a situation where only the action changes while the object and environment remain unchanged, only the action component is updated in the index, and other components do not occupy storage space.

[0148] This embodiment further compresses the index size by decoupling semantic components and using sparse storage; more importantly, it supports fine-grained combinatorial retrieval, which greatly improves the flexibility and generalization ability of the retrieval system, enabling the system to cope with complex semantic queries across scenarios and objects.

[0149] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A video big data retrieval method based on semantic tags, characterized in that, include: The video data stream to be retrieved is acquired, and a multimodal encoder is used to map the video data stream into a continuous sequence of feature vectors. Initialize and establish storage space for storing temporal semantic features; In response to receiving a retrieval request for a video data stream, a differential retrieval process based on semantic change rate is triggered, including: Step 1: Based on the feature vector sequence, calculate the rate of change of the difference between feature vectors at adjacent time points to generate a semantic velocity vector; Step 2: Compare the magnitude of the semantic velocity vector with the preset mutation threshold, and extract the time point where the semantics undergoes a significant mutation as the event anchor point based on the comparison result; Step 3: Construct a differential manifold index based on event anchors, wherein the differential manifold index is configured as a directed acyclic graph data structure with semantic state transitions as nodes and time spans as edges; Step 4: Based on the query vector generated by the retrieval request, construct a dynamic mask matrix and project the node data in the differential manifold index to the low-dimensional subspace indicated by the non-zero elements in the dynamic mask matrix; Step 5: Calculate the matching degree between the query vector and the projected nodes in the low-dimensional subspace, and output the retrieval results; Based on the comparison between the magnitude of the semantic velocity vector and the preset mutation threshold, the time points where significant semantic mutations occur are extracted as event anchor points and configured as follows: The Euclidean norm of the semantic velocity vector is calculated as the semantic change intensity value; If the semantic change intensity value is greater than the preset mutation threshold, it is determined that a semantic state transition has occurred at the current moment. The semantic velocity vector corresponding to the moment of transition is retained and marked as the event anchor point. If the semantic change intensity value is less than or equal to the preset mutation threshold, the current time is determined to be a static redundant segment, and the corresponding feature vector is discarded; Differential manifold indexes are constructed based on event anchors, including: Using the independent component analysis algorithm, event anchors are decoupled into orthogonal semantic basis vectors; Based on the temporal causal relationship, the decoupled event anchor points are connected to form a directed acyclic graph structure; The semantic state changes are stored in the nodes of the directed acyclic graph (DAG), and the time span of the semantic state changes is stored in the edges of the DAG, generating a differential manifold index. Based on the query vector generated from the retrieval request, a dynamic mask matrix is ​​constructed and configured as follows: Parse the search request and extract key semantic terms from it; Determine the dimension index corresponding to the key semantic words in the feature space, and mark the dimension index as the activation dimension; Generate a dynamic mask matrix, where the values ​​in the active dimension are set to retain coefficients, and the values ​​in the inactive dimension are set to zero; Dimensionality reduction is achieved by using a dynamic mask matrix and a differential manifold index for matrix operations, thereby preserving semantic dimension data relevant to the retrieval request.

2. The video big data retrieval method based on semantic tags according to claim 1, characterized in that, Based on the feature vector sequence, the rate of change of the difference between feature vectors at adjacent time points is calculated to generate a semantic velocity vector, configured as follows: Obtain the feature vector at the current time step and the feature vector at the previous time step; Calculate the difference vector obtained by subtracting the feature vector from the feature vector at the previous time step from the feature vector at the current time step; The difference vector is used as a semantic velocity vector to characterize the rate of change of semantic features on the time axis, where the semantic velocity vector is used to characterize the evolution trend of actions or events in the video content.

3. The video big data retrieval method based on semantic tags according to claim 1, characterized in that, Calculate the matching degree between the query vector and the projected nodes in the low-dimensional subspace, and output the retrieval results, configured as follows: In a low-dimensional subspace defined by a dynamic mask matrix, calculate the cosine similarity between the projected query vector and the projected node vector. Based on the cosine similarity value, the path with the highest cumulative similarity is searched in the directed acyclic graph of the differential manifold index as the maximum matching path. The video segment corresponding to the maximum matching path is output as the search result that meets the search request.

4. A video big data retrieval system based on semantic tags, applied to the video big data retrieval method based on semantic tags as described in any one of claims 1-3, characterized in that, include: The data mapping module is configured to acquire the video data stream to be retrieved and map the video data stream into a continuous sequence of feature vectors using a multimodal encoder; The retrieval triggering module is configured to trigger a differential retrieval process based on semantic change rate in response to receiving a retrieval request for a video data stream. The differential manifold retrieval module includes: The velocity calculation unit is configured to calculate the rate of change of the difference between feature vectors at adjacent time points based on the feature vector sequence, and generate a semantic velocity vector. The anchor point extraction unit is configured to compare the magnitude of the semantic velocity vector with a preset mutation threshold, and extract the time point where a significant semantic mutation occurs as the event anchor point based on the comparison result. The index building unit is configured to build a differential manifold index based on event anchors, wherein the differential manifold index is configured as a directed acyclic graph data structure with semantic state transitions as nodes and time spans as edges; The spatial folding unit is configured to construct a dynamic mask matrix based on the query vector generated by the retrieval request, and project the node data in the differential manifold index to the low-dimensional subspace indicated by the non-zero elements in the dynamic mask matrix; The matching output unit is configured to calculate the matching degree between the projected query vector and the projected node vector in the low-dimensional subspace and output the retrieval results.

5. A video big data retrieval system based on semantic tags according to claim 4, characterized in that, The index building unit is also used for: Using the independent component analysis algorithm, event anchors are orthogonally decoupled to separate the semantic components of objects, actions, and environment; Only semantic components with non-zero semantic speed modulus are stored in the differential manifold index to compress the index size.

Images