A Video-Aware Feature Extraction Method Based on Semantic Database and Temporal Similarity Modeling

Abstract

This invention discloses a video perception feature extraction method based on semantic sentence database and similarity temporal modeling. The method utilizes a video language model and a large language model to obtain semantic sentence databases for different perception dimensions of video segments. The similarity of the semantic sentence databases for different perception dimensions with the video segments is compared to obtain a similarity matrix between each video frame and each semantic sentence under each perception dimension. This similarity matrix enables the interpretability of the final video perception feature prediction values ​​for each perception dimension, and it can directly characterize the temporal fluctuations of perception features. Furthermore, training the model based on this similarity matrix can more accurately obtain the mapping relationship of the predicted values ​​of video perception features under different perception dimensions of the current video segment.

Description

Technical Field

[0001] This invention belongs to the field of video feature extraction, specifically involving a video-perceptual feature extraction method based on semantic corpus and similarity temporal modeling. Background Technology

[0002] As film and television works play an increasingly important role in social communication and cultural expression, human understanding and analysis of audiovisual arts are gradually shifting from subjective experience to computational and intelligent methods. As a comprehensive art form highly coupled with visual and auditory elements, the aesthetic experience of film and television works depends not only on visual elements such as camera angles, color, and composition, but is also significantly influenced by auditory cues such as music, sound effects, and rhythm. In recent years, the development of artificial intelligence technology has driven progress in areas such as video understanding, sentiment analysis, and music generation. However, in the film and television field, how to model the complex "audiovisual perception experience" of humans from the perspective of machine perception remains a challenge that has not yet been fully resolved.

[0003] Current research in computer vision and multimodal analysis largely focuses on low-level signal features (such as color, brightness, motion vectors, and spectral features) or semantic features (such as emotion recognition, scene classification, and behavior recognition). However, the visual experience in works of art ultimately undergoes human perceptual processing and is translated into higher-level features (such as audience emotions, cultural resonance, and aesthetic taste). Due to the significant information differences between low-level and high-level features, computers struggle to map the complex input of low-level features to high-level feature representations. Therefore, this invention proposes a "perceptual layer" as an intermediate representation between low-level and high-level features, encompassing the comprehensive experience of viewers' complex feelings such as character tension, character affinity, scene tension, action energy, narrative rhythm, and the richness of narrative content. Existing research lacks a systematic modeling method for this layer, causing film and television works to remain at the level of "physical analysis" or "semantic label classification" in machine understanding, making it difficult to further capture the true human perception and aesthetic experience.

[0004] Patent application CN111930999A discloses a frame-by-frame cross-modal similarity association method for locating video segments based on text queries. It utilizes the similarity or distance between cross-modal features based on deep neural networks to maximize the correlation between the video and the text query. By modeling the text query frame-by-frame using attention information from video semantic perception, it enhances the expressive power of the text query features. Furthermore, by extracting and fusing frames within a video segment, it maximizes the information contained within cross-modal features, improving the accuracy of video segment location. This patent application provides a novel model integrating an attention mechanism and a cross-modal similarity / distance function to explore the interaction between video frames and text query features. By extracting and fusing frames within a video segment, it maximizes the use of information contained within cross-modal features, improving the accuracy of video segment location and achieving precise video segment localization. However, this patent application cannot construct a mapping relationship between perceptual features of the video perception dimension and the video segment.

[0005] Several key challenges remain in video perception feature modeling: (1) Lack of effective semantic bridging mechanisms. Traditional deep learning models rely on end-to-end labeled data, but it is difficult to establish direct feature mappings for abstract perception features (such as "scene tension" or "image affinity"); (2) Insufficient modeling of the temporal dimension. The perception of film and television images often evolves dynamically over time, while most existing image or video feature extraction methods are mainly based on frame-level static analysis, which makes it difficult to characterize the temporal fluctuations of perception features; (3) Lack of interpretability. Although deep models can predict emotion or perception scores, their internal mechanisms make it difficult to explain which semantic cues in the images lead to specific perception judgments, which is insufficient to meet the interpretability needs in art research or creative assistance. Summary of the Invention

[0006] This invention provides a video perception feature extraction method based on semantic database and similarity time-series modeling. It can obtain video perception feature values ​​of each perception dimension of a video segment with relatively accurate results, as well as time-series feature matrices of each perception dimension that can explain the video perception feature values, thereby achieving interpretability.

[0007] A specific embodiment of the present invention provides a video-aware feature extraction method based on a semantic database and similarity temporal modeling, including:

[0008] By inputting perceptual dimension cue words and video clips into the video language model, natural language descriptions under different perceptual dimensions are obtained;

[0009] By splitting the natural language description under each perceptual dimension using a large language model, we obtain the semantic sentence library under each perceptual dimension. Then, we encode the semantic sentence library under each perceptual dimension to obtain the text vector set under each perceptual dimension.

[0010] The similarity between the encoded video segment and the text vector set of each dimension is calculated to obtain the time series feature matrix of each dimension of the video segment;

[0011] A training model is constructed by inputting the time series feature matrix of each perceptual dimension into the training model to obtain the video perceptual feature prediction value for each perceptual dimension of the video segment. Based on the video perceptual feature prediction value of each perceptual dimension and the manually labeled value, the training model is trained through a loss function to obtain a score prediction model. The time series feature matrix of each perceptual dimension of the current video segment is input into the score prediction model to obtain the final video perceptual feature prediction value for each perceptual dimension of the current video segment.

[0012] Preferably, the video sequence frame vectors obtained by encoding the video segment are compared with the text vector sets of each dimension to calculate the similarity, resulting in a time-series feature matrix for each dimension of the video segment, including:

[0013] The video segment is encoded to obtain a video sequence frame vector;

[0014] The similarity between each frame vector in the video sequence and each text vector in each dimension of the text vector set is calculated to obtain the similarity between each frame vector and each text vector in each dimension.

[0015] A time-series feature matrix for each dimension of the video clip is constructed based on the similarity between each frame vector and each text vector in each dimension.

[0016] Preferably, the video segment is decomposed into consecutive frame images in chronological order, and the consecutive frame images are sequentially encoded by an image encoder using a CLIP model to obtain a video sequence frame vector.

[0017] Preferably, the first i frame vectors With the j Text vectors similarity for: in, It is an L2 norm.

[0018] Preferably, perceptual dimension prompts are obtained by setting instructional questions based on each perceptual dimension, wherein the perceptual dimensions include image features, action features, structural features, and color features.

[0019] Preferably, the natural language description under each perception dimension is split into multiple short sentences by a large language model, and the multiple short sentences obtained are used to construct a semantic sentence library under each perception dimension.

[0020] Preferably, the semantic sentence library under each perceptual dimension is encoded to obtain a set of text vectors under each perceptual dimension, including:

[0021] After embedding the semantic sentence library into a text vector space that can perform distance calculations using CLIP's text encoder, clustering is performed. Representative sentences are selected from each cluster set to obtain a set of text vectors for each perceptual dimension.

[0022] Preferably, the training model includes a Transformer encoder, an attention-weighted pooling mechanism, and a linear regression layer. The time-series feature matrix of each perceptual dimension is sequentially passed through the Transformer encoder, the attention-weighted pooling mechanism, and the linear regression layer to obtain the predicted video perceptual feature value of each perceptual dimension.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] This invention utilizes video language models and large language models to obtain semantic sentence libraries for different perceptual dimensions of video segments. The similarity of these semantic sentence libraries with the video segments is compared to obtain a similarity matrix between each video frame and each semantic sentence in each perceptual dimension. This similarity matrix enables the interpretability of the final video perceptual feature prediction values ​​for each perceptual dimension. Furthermore, this similarity matrix can directly characterize the temporal fluctuations of perceptual features. Training the model based on this similarity matrix also allows for a more accurate mapping relationship between the predicted video perceptual features of the current video segment in different perceptual dimensions. Attached Figure Description

[0025] Figure 1 The flowchart illustrates a video-aware feature extraction method based on semantic database and temporal similarity modeling, provided for a specific embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0027] In order to obtain the numerical values ​​of the video perceptual features of each video segment and to achieve the interpretability of these values, this invention provides a video perceptual feature extraction method based on a semantic database and temporal similarity modeling. Figure 1 As shown, it includes:

[0028] (1) Input the perceptual dimension prompt words and video clips from the film and television library into the video language model (VLM), such as VideoLLaMA3, to obtain natural language descriptions under different perceptual dimensions: In this embodiment, the video clips in the dataset and the semantic questions for a certain perceptual feature dimension are used to form perceptual dimension prompt words. The video language model can generate natural language descriptions under different perceptual dimensions based on the input video clips and perceptual dimension prompt words.

[0029] In one specific embodiment, this invention, through systematic literature review, constructs a representation paradigm based on video perception features. This paradigm integrates film and television aesthetics theory with big data analysis technology, forming a three-dimensional video perception feature system comprising four main dimensions and nine fine-grained indicators, providing a framework for cross-modal perception analysis of film and television works. Among the nine fine-grained features, three are rule-driven features extracted directly from the visual content of film and television works, relying on predefined rules and algorithms; six are data-driven features extracted using the machine learning technology of this invention after forming a dataset through extensive data collection, analysis, and manual annotation.

[0030] For the six data-driven perceptual features, a manually labeled dataset has been built, containing scores of more than a thousand film and television clips on these six perceptual features, with scores on a 100-point scale.

[0031] Table 1 provides a detailed definition of the perceptual features calculated in this embodiment.

[0032]

[0033] It is understandable that the data-driven approach provided in the specific embodiments of the present invention is as follows: the perceptual feature dimension cannot be calculated by designing specific rules, so it is necessary to calculate the score by constructing a dataset and designing a machine learning model for training the dataset.

[0034] The theoretical driving force provided by the specific embodiments of the present invention is that the perceptual feature dimension can find corresponding explicit computable features (such as the warmth or coolness of color and saturation) from its theoretical definition, so the calculation rules can be directly designed to obtain the feature score.

[0035] In one embodiment, the system targets several video perception feature dimensions (such as motion energy, narrative rhythm, scene tension, character tension, etc.), and inputs instructional questions to the video language model for each dimension. For example: "Analyze the behavior or movement in this scene, identify the main physical activities, describe their speed, rhythm, force, etc., and explain their overall impact on the motion energy of the scene. Use film and television terminology as much as possible in your answers." The video language model will then output a textual description containing rich semantic information, serving as a natural language description (video content description) for the motion energy perception dimension.

[0036] (2) The natural language description under each perception dimension is split by the large language model to obtain the semantic sentence library under each perception dimension, and the semantic sentence library under each perception dimension is encoded to obtain the text vector set under each perception dimension.

[0037] Specifically, in this embodiment of the invention, the natural language description under each perceptual dimension is input into a large language model (LLM) for semantic refinement and decomposition. The language model provided in this embodiment (such as Qwen3-8B) is used to automatically decompose long descriptions into several structured, independent short sentences, so that each short sentence can independently express a single semantic information related to that perceptual dimension.

[0038] In one embodiment, within the motion energy perception dimension, the natural language description under the motion energy perception dimension is: "The video opens with a close-up of a man's face, his expression focused and determined. He is wearing a sleeveless shirt and a hat, suggesting a physically demanding environment. The camera then pulls back, showing a group of men working in a field, wielding shovels. Their movements are methodical and synchronized, highlighting a shared goal. The camera captures their sweating profusely, emphasizing the arduousness of the task. As the scene progresses, the focus shifts to a close-up of the man's hands digging in the soil, emphasizing the physically demanding nature of the work. The entire image conveys determination and..." The atmosphere of perseverance, the rhythmic swing of the shovels, and the workers' resolute expressions all contribute to highlighting this spiritual strength. After decomposition using a large language model, the following series of short sentences can be obtained: "A photo of workers digging with coordinated movements," "A scene of men working together to dig," "A shot of hands pushing soil forcefully," "A scene of rhythmic digging movements," "A scene of physical exertion and synchronized labor," "A photo of a worker idly moving without any obvious action," "A scene of slow and uncoordinated digging," "A shot of a hand barely touching the soil," and "A scene showing little movement and a lack of rhythm." This process, by breaking down complex long sentences into simpler, semantically concise short sentences, effectively improves semantic granularity, making the generated sentence database more suitable for subsequent similarity calculation tasks. All the reorganized short sentences together form the original semantic sentence database.

[0039] In a specific embodiment of the present invention, before subsequent similarity calculation, semantic deduplication, format standardization, and quality screening can be performed on the original sentence database. Specifically, this embodiment introduces a sentence merging mechanism based on semantic similarity clustering: after embedding the original sentence database into a text vector space that can perform distance calculation using a text encoder (such as CLIP's text encoder), clustering is performed. Representative sentences are selected from each cluster set, thereby merging semantically repetitive or highly similar sentences into a single expression to improve the coverage and diversity of the sentence database.

[0040] Through the above steps, this module realizes the automated mapping process from film and television clips to a semantic short sentence library, and establishes a high-quality semantic sentence library for specific perceptual feature dimensions.

[0041] (3) In this embodiment, the similarity calculation is performed between the video sequence frame vector obtained by encoding the video segment and the text vector set of each dimension to obtain the time series feature matrix of each dimension of the video segment.

[0042] In one specific embodiment, this embodiment calculates the similarity between the video sequence frame vector obtained after encoding the video segment and the text vector set of each dimension to obtain the time series feature matrix of each dimension of the video segment, including:

[0043] The video segment is encoded to obtain a video sequence frame vector;

[0044] The similarity between each frame vector in the video sequence and each text vector in each dimension of the text vector set is calculated to obtain the similarity between each frame vector and each text vector in each dimension.

[0045] A time-series feature matrix for each video segment is constructed based on the similarity between each frame vector and each text vector in each dimension.

[0046] In a specific embodiment of the present invention, the video segment is decomposed into consecutive frame images in chronological order, and the consecutive frame images are sequentially encoded by an image encoder of the CLIP model to obtain a video sequence frame vector.

[0047] In step (2), in this specific embodiment of the invention, the text encoder of CLIP is used to vectorize all text descriptions in the sentence database to obtain the text vector corresponding to each description, that is, each short sentence. The text encoder embeds sentences into a pre-trained multimodal alignment space, making semantically similar texts closer in the vector space.

[0048] Meanwhile, this embodiment decomposes the video segment into consecutive frame images in chronological order, and inputs them sequentially into the CLIP model's image encoder to extract the image feature vector of each frame. This encoder is typically based on a Visual Transformer (ViT) structure. Through this step, video and text are uniformly represented in a comparable vector form.

[0049] After obtaining the video sequence frame vectors obtained by encoding the video segment and the text vector set for each dimension, pairwise similarity calculation is performed between the two. That is, the similarity between each text vector in the sentence library and the image vector of each frame in the video is calculated. This invention adopts the standard similarity measurement method in the OpenAI CLIP model, namely normalized cosine similarity.

[0050] No. i frame vectors With the j Text vectors similarity for:

[0051]

[0052] in, The L2 norm (Euclidean norm) is a common method for measuring the size or length of a vector. It is calculated by taking the square root of the sum of the squares of the vector's elements. After normalization, this similarity value can take values ​​in the range [0,1], and is used to quantify the consistency between frame-level images and semantic descriptions.

[0053] In a specific embodiment of the present invention, by performing the above-described similarity calculation on the entire video frame sequence, a temporal similarity matrix can be obtained. Each row of this matrix represents the similarity change curve between the video frame sequence and a certain sentence in the sentence database on the time axis, and each column corresponds to the set of similarity values ​​between the current video frame image and each semantic description in the sentence database.

[0054] Through the above steps, the similarity calculation module achieves fine-grained alignment between the video and the semantic database in the time dimension, providing an interpretable multimodal input signal basis for subsequent perceptual feature modeling and prediction.

[0055] The CLIP (Contrastive Language-Image Pre-training) model provided in this invention constructs a shared multimodal representation space through large-scale image-text contrastive learning. Its core mechanism is to bring matching "image-text" pairs closer together in the vector space during the training phase, while pushing mismatched pairs further apart. Through this contrastive objective function, the model can learn a consistent semantic structure across modalities, ensuring that the vector representations of the same semantic concept in both image and text modalities have directional consistency and amplitude similarity. Therefore, regardless of whether the input comes from an image encoder or a text encoder, content with the same or similar semantics will automatically cluster in the same high-dimensional neighborhood, thus achieving cross-modal comparability. Based on this, the distance or angle between multimodal features can directly reflect their semantic similarity, and cosine similarity, due to its sensitivity to vector direction and minimal scale influence, becomes the most suitable similarity metric.

[0056] (4) Construct a training model, input the time series feature matrix of each perception dimension into the training model to obtain the video perception feature prediction value for each perception dimension of the video segment, and train the training model through the loss function based on the video perception feature prediction value and the manually labeled value of each perception dimension to obtain the score prediction model, and input the time series feature matrix of each perception dimension of the current video segment into the score prediction model to obtain the final video perception feature prediction value for each perception dimension of the current video segment.

[0057] Specifically, the constructed training model includes a Transformer encoder, an attention-weighted pooling mechanism, and a linear regression layer. It performs unified alignment and normalization on the frame features of each video segment arranged in a time series, ensuring that the time series feature matrix of each perceptual dimension of each video segment is consistent. Satisfy timing length Size of the semantic database The requirements, among which Indicates the video number Frame and the Similarity of semantic descriptions.

[0058] Subsequently, the time-series feature matrix is ​​input into a multi-layer Transformer encoder for temporal modeling. The multi-layer Transformer encoder calculates the temporal dependencies between frames through a multi-head self-attention mechanism, obtaining the results for each time step. Hidden vectors on The formula is expressed as:

[0059]

[0060] in Position encoding is used to preserve the temporal order information of the sequence. In this way, the model can capture global information at different time steps in the video and the importance of keyframes. To aggregate time-series features into a global representation, this module further employs an attention-weighted pooling mechanism:

[0061]

[0062] in Indicates the first Frame importance weights The summation index is used to iterate through all time steps in the denominator, and the weight vector... This can be learned through training, enabling the model to automatically focus on keyframes that contribute most to the prediction of perceptual features.

[0063] Finally, the aggregated global feature vectors The input is fed into a linear regression layer, and the output is a predicted value of the perceptual features of a certain perceptual dimension of the video clip. :

[0064]

[0065] During training, W represents the weights, b represents the bias term, and the final video perception feature prediction value is obtained through a certain perceptual dimension. (Right now Figure 1 In , , ) and manually labeled values ​​in the dataset (Right now Figure 1 The mean square error between a, b, and c in the given information is:

[0066]

[0067] Where i is the index of the video segment and N is the number of video segments, the model can effectively learn the dynamic change pattern of video frame features over time and achieve quantitative prediction of perceptual features such as motion energy through the above method.

[0068] This invention provides specific embodiments to interpret and present the output of a video perceptual feature prediction model, thereby enhancing the model's interpretability and verifiability. Specifically, by demonstrating the temporal distribution of similarity changes between video frames and the semantic database, this invention intuitively reflects the contribution of keyframes in the video to the perceptual feature score.

[0069] Specifically, this refers to the time-series feature matrix of each perceptual dimension. As input, where For video frame rate, The number of semantic descriptions. Elements of the matrix. Indicates the first Frame and the The degree of similarity in semantic descriptions. Then, the similarity matrix... The time series is constructed and visualized using heatmaps, making the contribution of each frame to the final prediction clearly visible.

[0070] The visualization module can display overall trends, providing an explanatory output of the model's decision-making process. In this way, users can intuitively understand how the model matches video frames with a semantic database and generates the final perceptual feature score, enhancing the system's transparency and credibility.

Claims

1. A video-aware feature extraction method based on semantic corpus and similarity temporal modeling. Its distinctive features include: By inputting perceptual dimension cue words and video clips into the video language model, natural language descriptions under different perceptual dimensions are obtained; By splitting the natural language description under each perceptual dimension using a large language model, we obtain the semantic sentence library under each perceptual dimension. Then, we encode the semantic sentence library under each perceptual dimension to obtain the text vector set under each perceptual dimension. The similarity between the encoded video segment and the text vector set of each dimension is calculated to obtain the time series feature matrix of each dimension of the video segment; A training model is constructed. The time series feature matrix of each perception dimension is input into the training model to obtain the video perception feature prediction value for each perception dimension of the video segment. Based on the video perception feature prediction value of each perception dimension and the manually labeled value, the training model is trained through a loss function to obtain a score prediction model. The time series feature matrix of each perception dimension of the current video segment is input into the score prediction model to obtain the final video perception feature prediction value for each perception dimension of the current video segment. Based on setting instructional questions for each perceptual dimension, prompt words for the perceptual dimension are obtained. The perceptual dimensions include image features, action features, structural features, and color features. Encoding the semantic sentence library under each perceptual dimension yields a set of text vectors for each perceptual dimension, including: After embedding the semantic sentence library into a text vector space that can perform distance calculation through CLIP's text encoder, clustering is performed. Representative sentences are selected from each cluster set to obtain a set of text vectors under each perceptual dimension. The training model includes a Transformer encoder, an attention-weighted pooling mechanism, and a linear regression layer. The time-series feature matrix of each perceptual dimension is sequentially passed through the Transformer encoder, the attention-weighted pooling mechanism, and the linear regression layer to obtain the predicted video perceptual feature value for each perceptual dimension.

2. The video-aware feature extraction method based on semantic database and similarity temporal modeling according to claim 1, is characterized in that, The video sequence frame vectors obtained after encoding the video segment are compared with the text vector sets of each dimension to calculate the similarity, resulting in a time-series feature matrix for each dimension of the video segment, including: The video segment is encoded to obtain a video sequence frame vector; The similarity between each frame vector in the video sequence and each text vector in each dimension of the text vector set is calculated to obtain the similarity between each frame vector and each text vector in each dimension. A time-series feature matrix for each dimension of the video clip is constructed based on the similarity between each frame vector and each text vector in each dimension.

3. The video-perceptual feature extraction method based on semantic database and similarity temporal modeling according to claim 2, is characterized in that, The video segment is decomposed into consecutive frame images in chronological order, and the consecutive frame images are encoded sequentially by an image encoder using the CLIP model to obtain a video sequence frame vector.

4. The video-aware feature extraction method based on semantic database and similarity temporal modeling according to claim 2, characterized in that, No. i frame vectors With the j Text vectors similarity for: in, It is an L2 norm.

5. The video-aware feature extraction method based on semantic database and similarity temporal modeling according to claim 1, characterized in that, By splitting the natural language description under each perceptual dimension using a large language model, multiple short sentences are obtained, and the resulting short sentences are used to construct a semantic sentence library for each perceptual dimension.

Images