A video semantic understanding model based on fine-grained hierarchical event features and a generation method thereof

Abstract

The application comprises a visual encoder, an image feature compressor, a pre-trained BERT model, an event feature encoding layer and a large language model. The visual encoder encodes video frames into frame features; the event feature encoding layer extracts and filters event features; the event feature extractor extracts coupled fine-grained event features from the video through an adaptive sliding window, and the event feature perceiver filters relevant event features according to the user input question. The application uses hierarchically placed event feature extractors to fully consider the correlation between video frames, encodes video information into fine-grained hierarchical event features, solves the coarse-grained feature extraction problem existing in the prior art model, uses the event feature perceiver to select the most relevant features from the event features output by each layer of event feature extractor according to the user query, greatly reduces the number of feature tokens of the video, and solves the long video understanding difficulty problem with the aid of a similar frame fusion module and an image feature compressor.

Description

Technical Field

[0001] This invention relates to the fields of video understanding and deep learning technology, and in particular to a video semantic understanding model and generation method based on fine-grained hierarchical event features. Background Technology

[0002] The 21st century is the era of multimodality. With the rapid development of social media, the Internet of Things, and other digital technologies, the presentation of modern information has become increasingly diverse, encompassing multiple data modalities such as images, text, and video. Video, as a crucial component of multimodal data, transmits information through dynamic, continuous image streams, possessing not only significant temporal sequence but also rich contextual and emotional interaction features. Video semantic understanding aims to enable models to accurately identify and understand key elements such as objects, scenes, and behaviors in videos, and to infer the deeper meaning expressed by the video, thereby achieving a comprehensive interpretation of complex multimodal information. In today's rapidly evolving digital media technology landscape, video has become an important medium for people to share information, express opinions, and record their lives. Therefore, video semantic understanding is particularly crucial in the current context. Mature video semantic understanding technology can not only optimize users' media experience but also play a key role in multiple fields such as education, security, and entertainment. A well-developed video semantic understanding model can significantly improve users' productivity, save manpower and time costs, provide strong support for building an intelligent society, and demonstrate profound research and application value.

[0003] Traditional video semantic understanding technologies typically focus on optimizing specific video analysis tasks, lacking the ability to comprehensively understand and flexibly respond to the broad information within videos. While they have demonstrated exceptional capabilities in specific tasks such as video question answering and video content retrieval, they often fall short when dealing with complex real-world scenarios.

[0004] In recent years, large language models have achieved remarkable results in various fields such as text generation, summarization, translation, and question answering. They not only demonstrate strong versatility but also support user interaction, understanding and following specific user instructions. Researchers have combined the language processing capabilities of large language models with the visual recognition capabilities of visual foundational models to design several video semantic understanding models based on large language models. These models primarily employ end-to-end training methods or utilize pre-trained components to transform video content into a form that large language models can effectively understand. These models can complete various video semantic understanding tasks in the form of dialogues, but some problems still exist when processing video information:

[0005] The first problem is coarse-grained feature extraction. Most models employ coarse-grained video processing techniques, simply aligning video information extracted from the visual base model into the semantic space of the large language model using simple modules, resulting in low utilization of video information. For example, Video-ChatGPT proposed by Mazz et al. performs spatial and temporal pooling on the video separately, then maps the extracted features to the feature space of the large language model through a simple linear layer; Valley proposed by Luo et al. uses a temporal modeling module composed of a single-layer Transformer and an average pooling layer to obtain temporal features, and aligns different modalities through a simple projection layer. These models fail to deeply consider the interrelationships between video frames, making it difficult for the large language model to understand the detailed information in the video.

[0006] The second problem is the difficulty in understanding long videos. Existing models typically represent video content by uniformly sampling a small number of frames from the video; for example, Video-LLaMA proposed by Zhang et al. samples only 8 frames from the video. This method is relatively effective when processing short videos of a few seconds, but when processing longer videos of minutes, it often loses key information, affecting the accuracy and completeness of the results. To reduce information loss, increasing the number of sampled frames seems like an intuitive solution, but this would lead to an excessive number of visual tokens input to a large language model, increasing the computational load on the model and raising the requirements for computing resources. Summary of the Invention

[0007] To address the challenges of coarse-grained feature extraction and long video understanding in existing video semantic understanding models based on large language models, this invention provides a video semantic understanding model and generation method based on fine-grained hierarchical event features.

[0008] To achieve the above objectives, according to a first aspect of the present invention, a video semantic understanding model based on fine-grained hierarchical event features is proposed, comprising a visual encoder, a pre-trained BERT model, an event feature encoding layer, and a large language model, wherein,

[0009] The pre-trained BERT model is used to encode questions in text form, and the encoded questions are called user queries;

[0010] A visual encoder is used to encode video frames into frame features;

[0011] The event feature encoding layer is used to extract and filter event features; it includes an event feature extractor and an event feature perceptron. The event feature extractor extracts coupled fine-grained event features from the video through an adaptive sliding window, and the event feature perceptron filters relevant event features based on user-input questions to improve the information density of video feature representation.

[0012] Furthermore, it also includes a similar frame fusion module, which is used to fuse similar frames, reduce the total number of frame features, and reduce the computational burden.

[0013] Furthermore, in the similar frame fusion module, adjacent frame features are merged by determining whether the cosine similarity between features of adjacent frames exceeds a set similarity threshold.

[0014] Furthermore, it also includes an image feature compressor, which is based on a multi-head cross-attention mechanism to compress the number of features in video frames and adds temporal coding.

[0015] Furthermore, the image feature compressor includes a first multi-head cross-attention unit and a first feedforward unit connected in sequence. The first multi-head cross-attention unit is composed of a multi-head cross-attention layer, a residual connection, and a layer normalization connected in sequence. The first feedforward unit is composed of a feedforward network, a residual connection, and a layer normalization connected in sequence.

[0016] Furthermore, the event feature extractor includes a second multi-head self-attention unit, a second multi-head cross-attention unit, and a second feedforward unit connected in sequence. The second multi-head self-attention unit consists of a multi-head self-attention layer, a residual connection, and a layer normalization connected in sequence; the second multi-head cross-attention unit consists of a multi-head cross-attention layer, a residual connection, and a layer normalization connected in sequence; the second feedforward unit consists of a feedforward network, a residual connection, and a layer normalization connected in sequence. The event feature extractor uses an adaptive sliding window to segment the video and divides the video into coupled event features by fusing features within the window unit. Preferably, when the event feature extractor is multi-level, the input of the first-level event feature extractor is the video frame features output by the image feature compressor, and starting from the second-level event feature extractor, its input is the output of the previous-level event feature extractor.

[0017] Furthermore, the event feature perceiver includes a third multi-head self-attention unit, a third multi-head cross-attention unit, a differentiable Top-K module, and a third feedforward unit connected in sequence, which filter the event features output by the event feature extractor based on the user's input question; the third multi-head self-attention unit is composed of a multi-head self-attention layer, residual connection, and layer normalization connected in sequence; the third multi-head cross-attention unit is composed of a multi-head cross-attention layer, residual connection, and layer normalization connected in sequence; the third feedforward unit is composed of a feedforward network, residual connection, and layer normalization connected in sequence.

[0018] According to another aspect of the present invention, a method for generating a video semantic understanding model is proposed, characterized by comprising:

[0019] S1. Pre-train the video semantic understanding model based on fine-grained hierarchical event features;

[0020] S2. Fine-tune the video semantic understanding model based on fine-grained hierarchical event features.

[0021] Furthermore, both pre-training and fine-tuning include:

[0022] S11. Perform uniform sampling on the video to obtain video frames; preferably, after uniform sampling, if the number of video frames is large, resample the video frames according to a preset sampling threshold.

[0023] S12. Encode video frames using a visual encoder to obtain feature representations of the video frames; preferably, then fuse similar video frame features using a similar frame fusion module.

[0024] S13. Compress video frame features using an image feature compressor, and add temporal coding to each video frame feature.

[0025] S14. Extract the first layer of event features from the video features output in S13 using the event feature extractor;

[0026] S15. The event feature perceptron filters the event features most relevant to the user query from the first layer of event features and injects the user query information into these event features; the user query is obtained by encoding questions about the video content in text form through a pre-trained BERT model.

[0027] S16. The event features output from the event feature encoding layer are concatenated with the embedded representation of questions about the video content through a linear layer and then input into the large language model. The model generates answers to the questions and adjusts the overall parameters of the model accordingly.

[0028] Preferably, the parameters are adjusted using the backpropagation algorithm. In S1, the parameters of the image feature compressor, the event feature encoding layer, and the linear layer are adjusted; in S2, the parameters of the image feature compressor, the BERT model, the event feature encoding layer, and the linear layer are adjusted, and the parameters of the large language model are adjusted using low-rank adaptation fine-tuning technology.

[0029] Preferably, the event feature encoding layer has n layers, and S16 further includes:

[0030] (1) Take the output of the event feature extractor in the previous event feature encoding layer as the input of the current event feature encoding layer to obtain the event features output by the current layer;

[0031] (2) The event features output by the multi-layer event feature encoding layer are concatenated and then concatenated with the embedded representation of questions about video content through a linear layer and input into the large language model. The model generates answers to the questions and adjusts the overall parameters of the model accordingly.

[0032] Furthermore, in S11, when the input is an image, the image is treated as a video frame.

[0033] The present invention provides a video semantic understanding model and generation method based on fine-grained hierarchical event features, which has the following advantages compared with existing video semantic understanding models and generation methods:

[0034] (1) To address the problem of coarse-grained feature extraction in existing models, this invention proposes an event feature extractor that fully considers the correlation between video frames and encodes video information into fine-grained event features. At the same time, the hierarchical placement of the event feature extractor enables the event features output by different layers to capture event information spanning different time spans.

[0035] (2) To address the difficulty of understanding long videos in existing models, this invention proposes an event feature perceptron to filter the event features most relevant to the user query, significantly reducing the number of feature tokens input to the large language model. Simultaneously, this invention also proposes a similar frame fusion module and an image feature compressor, reducing the number and dimensionality of video frame features and further alleviating the computational burden. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the structure of a video semantic understanding model based on fine-grained hierarchical event features according to an embodiment of the present invention;

[0038] Figure 2 This is a flowchart illustrating an image feature compressor according to an embodiment of the present invention;

[0039] Figure 3 This is a flowchart illustrating an event feature extractor according to an embodiment of the present invention;

[0040] Figure 4 This is a flowchart illustrating an event feature sensor according to an embodiment of the present invention.

[0041] Figure 5 This is a schematic diagram of instruction fine-tuning data according to an embodiment of the present invention;

[0042] Figure 6 This is a flowchart illustrating a method for generating a video semantic understanding model based on fine-grained hierarchical event features, according to an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] This invention provides a video semantic understanding model based on fine-grained hierarchical event features, the overall structure of which is as follows: Figure 1 As shown, it includes a visual encoder, a pre-trained BERT model, an event feature encoding layer, and a large language model.

[0045] The pre-trained BERT model is used to encode text-based questions about video content. The encoded questions are called user queries and are represented as follows: .

[0046] A visual encoder is used to encode video frames into frame features. In one embodiment, the Vision Transformer part of the EVA-CLIP model proposed by Fang et al. can be used, assuming that a certain video frame is... The frame features obtained after encoding are represented as follows: In addition, Vision Transformer models trained by other researchers can be used as visual encoders.

[0047] In one embodiment, given the potential redundancy among the encoded video frame features, this invention provides a similar frame fusion module to fuse similar frames, thereby reducing the total number of frame features and lowering the computational burden. For example, in one embodiment, the similar frame fusion module employs a greedy strategy, determining whether the cosine similarity between adjacent frame features exceeds a set similarity threshold. This involves merging features from adjacent frames. Merging refers to averaging the features of two frames. (Adjacent frame features) and The cosine similarity between them is defined as:

[0048]

[0049] After processing video frames through the similar frame fusion module, the number of features for each video frame is usually still quite large. For example, according to the model parameter settings of the visual encoder used above, the number of video frame features is 257, which puts a significant burden on subsequent calculations. To address this issue, this invention also provides an image feature compressor based on a multi-head cross-attention mechanism to reduce the number of features for each video frame, thereby further reducing the computational burden. The image feature compressor includes a first multi-head cross-attention unit and a first feedforward unit connected in sequence. The first multi-head cross-attention unit consists of a multi-head cross-attention unit, a residual connection, and a layer normalization layer connected in sequence; the first feedforward unit consists of a feedforward network, a residual connection, and a layer normalization layer connected in sequence. Preferably, the feedforward network has two layers. Assuming the input video frame features are... The image feature compressor compresses it and adds temporal coding to obtain the output. Specifically, the processing flow of the image feature compressor is as follows: Figure 2 As shown, it includes the following steps:

[0050] (1) Use a set of learnable query vectors The input video frame features are compressed using the first multi-head cross-attention unit. , get output In the input of the first multi-head cross-attention unit, Used for calculating queries in the attention mechanism. Used to calculate keys and values. and Represent the number and dimensions of learnable queries, respectively, satisfying and , and They represent The number of features and dimensions, the number of attention heads in the first multi-head cross-attention unit is... The construction of a multi-head cross-attention layer is existing technology and will not be described in detail here. Its parameters include a matrix for linearly transforming the input and a linear transformation matrix for combining the outputs of each attention head.

[0051] (2) Processing using the first feedforward unit Then add timing coding , get output .

[0052] The processing flow of the image feature compressor is represented as follows:

[0053]

[0054]

[0055] Here, MultiHead Cross Attention represents a multi-head cross attention layer, LN represents layer normalization, and Feed Forward represents a feedforward network.

[0056] Based on the view that video content consists of a series of long-term or short-term events, this invention provides an event feature encoding layer to extract and filter event features over a specific time span. The event feature encoding layer comprises two parts: an event feature extractor and an event feature perceptron. The event feature extractor extracts coupled fine-grained event features from the video using an adaptive sliding window, while the event feature perceptron filters relevant event features based on the input question to improve the information density of the video feature representation. In one embodiment, the video semantic understanding model uses a multi-layer event feature encoding layer to extract event features across different time spans. By using a hierarchical event feature encoding layer, fine-grained encoding and filtering of video information are achieved, resulting in a smaller number of event features with higher information density. This enables detailed understanding of both long and short videos with low computational requirements. The following explanation uses a three-layer event feature encoding layer as an example.

[0057] The event feature extractor extracts higher-level event features based on multi-head self-attention and multi-head cross-attention mechanisms. The first-layer event feature extractor takes video frame features output from the image feature compressor as input, while the second-layer extractor and subsequent layers take the output of the previous layer as input. The three-layer event feature extractor utilizes a hierarchical adaptive sliding window to segment the video into window units spanning different time lengths. By fusing features within these window units, the video is further subdivided into coupled event features. Compared to coarse-grained methods that directly fuse features from all frames, the event feature extractor fuses similar video frames in a fine-grained manner, capturing richer details. The event feature extractor comprises a second multi-head self-attention unit, a second multi-head cross-attention unit, and a second feedforward unit connected sequentially. The second multi-head self-attention unit consists of sequentially connected multi-head self-attention layers, residual connections, and layer normalization; the second multi-head cross-attention unit consists of sequentially connected multi-head cross-attention units, residual connections, and layer normalization; the second feedforward unit consists of sequentially connected feedforward networks, residual connections, and layer normalization, with the feedforward network preferably consisting of two layers. The processing flow of the event feature extractor is as follows: Figure 3 As shown, it includes the following steps:

[0058] (1) Input features are segmented using a sliding window, and the window size and sliding step size of each event feature encoding layer are set as follows:

[0059]

[0060]

[0061]

[0062] in, and They represent the first The window size and sliding step of the layer, Indicates the first The total number of input features in each layer. The size of the window in each layer is dynamically determined by the number of input features in that layer; generally, the window size decreases with each layer. In the th... In the layer event feature extractor, the video frame features or event features after sliding window segmentation are represented as follows: , . Indicates the first The number of windows in the layer, and They represent the first The number and dimensions of each input feature in the layer. Representing the set of real numbers is a commonly used mathematical representation.

[0063] (2) For each window unit Flatten its dimensions into Then, the output is obtained through a second multi-head self-attention unit. The input to the second multi-head self-attention unit only includes... Used to calculate the query, key, and value in the attention mechanism, with a number of attention heads. The construction of multi-head self-attention layers is existing technology and will not be described in detail here. Its parameters include a matrix used to perform linear transformation on the input and a linear transformation matrix used to combine the outputs of each attention head.

[0064] (3) Use a set of learnable query vectors From each window unit via the second multi-head cross-attention unit Extract event features to obtain output. In the input of the second multi-head cross-attention unit, Used for calculating queries in the attention mechanism. Used to calculate keys and values, with an attention head count of [number]. .

[0065] (4) Process using the second feedforward unit , get output , which will be represented as . No. The output of the layer event feature extractor is represented as .in, Indicates the first The number of event features at each layer is also the number of events at the first layer. The number of sliding windows in a layer.

[0066] The processing flow of the event feature extractor for each window unit is represented as follows:

[0067]

[0068]

[0069]

[0070] Among them, MultiHead Self Attention represents a multi-head self-attention layer, MultiHead CrossAttention represents a multi-head cross-attention layer, Fla represents dimension flattening, LN represents layer normalization, and Feed Forward represents a feedforward network.

[0071] The event feature perceptron comprises a third multi-head self-attention unit, a third multi-head cross-attention unit, a differentiable Top-K module, and a third feedforward unit, connected sequentially. The event feature perceptron filters the event features output by the event feature extractor based on the user-input question, ensuring that the extracted event features are highly relevant to the question. This significantly reduces the number of feature tokens in the video, making the model suitable for long video understanding scenarios. Simultaneously, the event feature perceptron also injects user queries into the event features. Information guides the user to focus on contextual information. The third multi-head self-attention unit consists of sequentially connected multi-head self-attention layers, residual connections, and layer normalization; the third multi-head cross-attention unit consists of sequentially connected multi-head cross-attention units, residual connections, and layer normalization; the third feedforward unit consists of sequentially connected feedforward networks, residual connections, and layer normalization, preferably with two layers of feedforward networks. The processing flow of the event feature perceiver is as follows: Figure 4 As shown, it includes the following steps:

[0072] (1) For the first Output of layer event feature extractor Flatten its dimensions into Then, the input is fed into the third multi-head self-attention unit to obtain the output. The input to the third multi-head self-attention unit only includes the flattened... Used to calculate the query, key, and value in the attention mechanism, with a number of attention heads. .

[0073] (2) The third multi-head cross-attention layer is used to obtain the output from the user query input. Attention score In the input of the third multi-head cross-attention layer, The query used in the attention mechanism is used to compute the key and value; the number of attention heads is [number missing]. .

[0074] (3) To Perform residual connections and layer normalization in the third multi-head cross-attention unit, and use the differentiable Top-K algorithm according to... Filtering the most relevant results to the user's query Each event characteristic yields the output. .

[0075] (4) Process using the third feedforward unit , get output .

[0076] The processing flow of the event feature sensor is represented as follows:

[0077]

[0078]

[0079]

[0080]

[0081] Where Top-K represents the differentiable Top-K algorithm, MultiHead Self Attention represents a multi-head self-attention layer, MultiHead Cross Attention represents a multi-head cross attention layer, Fla represents dimension flattening, LN represents layer normalization, and Feed Forward represents a feedforward network.

[0082] The large language model uses the Vicuna-7B model, whose input consists of two parts: event features output from multiple event feature encoding layers and a question input by the user. Specifically, the filtered event features output from the multiple event feature encoding layers are concatenated, projected into the input space of the large language model through a linear layer, and then concatenated with the embedding representation of the question about the video content, serving as the input to the large language model. The output of the large language model is the answer to the question.

[0083] According to another aspect of the present invention, a method for generating the above-described video semantic understanding model is also provided, such as... Figure 3 As shown, it includes:

[0084] S1. Pre-train the video semantic understanding model;

[0085] S2. Fine-tune the video semantic understanding model using instructions.

[0086] A video semantic understanding model based on fine-grained hierarchical event features undergoes a two-stage training process: pre-training and instruction fine-tuning. The main goal of the pre-training stage is to improve the model's ability to automatically extract event features from videos and map them to a large language model feature space. During pre-training, the model's input is an image or video, and the target output is the title of the image or video. The instruction fine-tuning stage aims to align the visual and linguistic spaces, improving the model's ability to follow human instructions to complete various video-based tasks. During instruction fine-tuning, the model's input is a video and instructions in question form, and the target output is the corresponding question answer. During training, pre-training is performed first, followed by instruction fine-tuning. Examples of instruction fine-tuning data are shown below. Figure 5 As shown in the diagram, questions such as "What is the blonde woman doing?" are the input instructions, while answers such as "Flying a kite" are the target outputs of the model.

[0087] The pre-training dataset includes image-text pairs and video-text pairs, where the text is the title of the image or video; the instruction fine-tuning dataset includes video instruction data, where the instructions are questions about the video content. Steps S11-S16 are performed sequentially on the respective datasets in both phases to complete the training.

[0088] S11. Perform uniform sampling on the video to obtain... Video frames. After uniform sampling, the number of video frames is large. For example, sampling at a sampling rate of 1 frame per second results in a large number of video frames. To reduce the computational burden, a preset sampling threshold is used. The video frames are then resampled further. After resampling, the video is represented as a collection of frames. ,in This represents the number of video frames after sampling. It's important to note that during the pre-training phase, part of the input is an image. In this case, the image is directly treated as a video frame without any sampling operation; the image is represented as... .

[0089] S12. Encode each video frame using a visual encoder. Obtain video frame features Preferably, if there are still many video frames, the features of similar video frames are fused using a similar frame fusion module. The fused video features are represented as follows: .in, This indicates the number of frame features after fusion.

[0090] S13. Compress the features of each video frame using an image feature compressor. Simultaneously, temporal coding is added to the features of each video frame to obtain the output. The compressed video features are represented as follows: .

[0091] S14, From the event feature extractor Extracting the first layer of event features, denoted as... .in, This indicates the number of event features in the first layer.

[0092] S15. Using the event feature sensor, from the first layer of event features Screening and Most relevant event features Furthermore, user query information is injected into these event features, enabling the event features to focus more on information relevant to user queries. Among these, This represents the number of event features after filtering. A pre-trained BERT model encodes a text-based question about the video content; this encoded question is called the user query, and is represented as... And shared across all layers. User query , This indicates the number of tokens the user queried. This represents the feature dimension of the BERT model.

[0093] S16. The event features output by the event feature encoding layer The video content is passed through a linear layer and concatenated with an embedding representation of a question about the video content before being fed into a large language model. The model generates answers to the questions and adjusts the overall parameters of the model accordingly.

[0094] In one embodiment, the event feature encoding layer can be multiple layers, then S16 further includes:

[0095] (1) Take the output of the event feature extractor in the previous event feature encoding layer as the input of the current event feature encoding layer to obtain the event features output by the current layer;

[0096] (2) The event features output from the multi-layer event feature encoding layer are concatenated and then fed into the large language model after being concatenated with the embedding representation of questions about video content through a linear layer. The model generates answers to the questions and adjusts the overall parameters of the model accordingly.

[0097] For example, when the event feature encoding layer has three layers, S16 also includes:

[0098] (1) The output of the event feature extractor in the first event feature encoding layer The event features are used as input to the second event feature encoding layer to obtain the event features output by the second layer. The output of the event feature extractor in the second event feature encoding layer. As input to the third event feature encoding layer, the event features output by the third layer are obtained. .

[0099] (2) Concatenate the event features output from the first, second, and third event feature encoding layers. , and The data is passed through a linear layer and concatenated with embedded representations of questions about the video content before being input into a large language model. The model generates answers to the questions and adjusts the overall parameters of the model accordingly.

[0100] In step S11, the resampling strategy can be expressed as:

[0101]

[0102] The sampled video is represented as , . and These represent the height and width of the video frame, respectively, and 3 represents the number of channels (this value is for illustrative purposes only and is not a limitation).

[0103] In step S12, video frame features , This indicates the number of image patches in the Vision Transformer. Representing feature dimension, Indicates the size of the image patch. Features of each video frame after similar frame fusion. .

[0104] In step S16, the large language model generates an answer to the question, calculates the loss value between the answer and the standard answer, and then adjusts the parameters using the backpropagation algorithm. Preferably, the parameters are adjusted using the backpropagation algorithm. In S1, the parameters of the image feature compressor, the event feature encoding layer, and the linear layer are adjusted; in S2, the parameters of the image feature compressor, the BERT model, the event feature encoding layer, and the linear layer are adjusted, and the parameters of the large language model are adjusted using low-rank adaptation fine-tuning techniques.

[0105] The video semantic understanding model based on fine-grained hierarchical event features provided by this invention is used as follows: input video and text in the form of questions about the video content, and obtain the answers to the questions through the video semantic understanding model.

[0106] Experimental verification:

[0107] After training the entire model using the steps described above, the model was evaluated using three open-source zero-shot video semantic understanding benchmarks: MSVD-QA, MSRVTT-QA, and ActivityNet-QA. The model was also compared with nine baseline models, with evaluation metrics including accuracy and score. The experimental results are shown in the table below. Bold text indicates the model with the highest score for that metric, and underlined text indicates the model with the second highest score. It can be seen that the video semantic understanding model based on fine-grained hierarchical event features provided by this invention outperforms existing state-of-the-art models on all three benchmarks, improving accuracy by 2.2%, 6.7%, and 2.4%, respectively, and improving scores by 0.2, 0.3, and 0.1, respectively.

[0108]

Claims

1. A method for generating a video semantic understanding model based on fine-grained hierarchical event features, characterized in that, include: S1. Pre-train the video semantic understanding model based on fine-grained hierarchical event features; S2. Fine-tune the video semantic understanding model based on fine-grained hierarchical event features using instructions; Video semantic understanding models based on fine-grained hierarchical event features include a visual encoder, a pre-trained BERT model, an event feature encoding layer, and a large language model. The pre-trained BERT model is used to encode questions in text form, and the encoded questions are called user queries; A visual encoder is used to encode video frames into frame features; The event feature encoding layer is used to extract and filter event features, including an event feature extractor and an event feature perceptron. The event feature extractor extracts coupled fine-grained event features from the video through an adaptive sliding window, and the event feature perceptron filters relevant event features based on user-input questions to improve the information density of video feature representation. When the event feature extractor is multi-level, the input of the first-level event feature extractor is the video frame features output by the image feature compressor. Starting from the second-level event feature extractor, its input is the output of the previous-level event feature extractor.

2. The generation method according to claim 1, characterized in that, The video semantic understanding model based on fine-grained hierarchical event features also includes a similar frame fusion module, which is used to fuse similar frames, reduce the total number of frame features, and reduce the computational burden.

3. The generation method according to claim 2, characterized in that, In the similar frame fusion module, adjacent frame features are merged by determining whether the cosine similarity between features of adjacent frames exceeds a set similarity threshold.

4. The generation method according to claim 1, characterized in that, The video semantic understanding model based on fine-grained hierarchical event features also includes an image feature compressor. The image feature compressor is based on a multi-head cross-attention mechanism to compress the number of features in video frames and add temporal coding.

5. The generation method according to claim 4, characterized in that, The image feature compressor includes a first multi-head cross-attention unit and a first feedforward unit connected in sequence. The first multi-head cross-attention unit is composed of a multi-head cross-attention layer, a residual connection, and a layer normalization connected in sequence. The first feedforward unit is composed of two feedforward layers, a residual connection, and a layer normalization connected in sequence.

6. The generation method according to claim 1, characterized in that, The event feature extractor includes a second multi-head self-attention unit, a second multi-head cross-attention unit, and a second feedforward unit connected in sequence. The second multi-head self-attention unit consists of a multi-head self-attention layer, a residual connection, and a layer normalization connected in sequence. The second multi-head cross-attention unit consists of a multi-head cross-attention layer, a residual connection, and a layer normalization connected in sequence. The second feedforward unit consists of two feedforward layers, a residual connection, and a layer normalization connected in sequence. The event feature extractor uses an adaptive sliding window to segment the video and divides the video into coupled event features by fusing the features within the window unit.

7. The generation method according to claim 1, characterized in that, The event feature perceiver includes a third multi-head self-attention unit, a third multi-head cross-attention unit, a differentiable Top-K module, and a third feedforward unit connected in sequence, which filter the event features output by the event feature extractor based on the question input by the user; The third multi-head self-attention unit consists of a multi-head self-attention layer, residual connection, and layer normalization connected in sequence; the third multi-head cross-attention unit consists of a multi-head cross-attention layer, residual connection, and layer normalization connected in sequence; the third feedforward unit consists of two feedforward networks, residual connection, and layer normalization connected in sequence.

8. The generation method according to claim 1, characterized in that, Both pre-training and fine-tuning include: S11. Perform uniform sampling on the video to obtain video frames; S12. Encode video frames using a visual encoder to obtain feature representations of the video frames; S13. Compress video frame features using an image feature compressor, and add temporal coding to each video frame feature. S14. Extract the first layer of event features from the video features output in S13 using the event feature extractor; S15. The event feature perceptron filters the event features most relevant to the user query from the first layer of event features and injects the user query information into these event features; the user query is obtained by encoding questions about the video content in text form through a pre-trained BERT model. S16. The event features output from the event feature encoding layer are concatenated with the embedded representation of questions about the video content through a linear layer and then input into the large language model. The model generates answers to the questions and adjusts the overall parameters of the model accordingly.

9. The generation method according to claim 8, characterized in that, In S11, after uniform sampling, if the number of video frames is large, the video frames are resampled according to the preset sampling threshold.

10. The generation method according to claim 8, characterized in that, In S12, similar video frame features are then fused through the similar frame fusion module.

11. The generation method according to claim 8, characterized in that, In S16, the backpropagation algorithm is used to adjust the parameters. In S1, the parameters of the image feature compressor, the event feature encoding layer, and the linear layer are adjusted. In S2, the parameters of the image feature compressor, the BERT model, the event feature encoding layer, and the linear layer are adjusted, and the parameters of the large language model are adjusted through low-rank adaptation fine-tuning technology.

12. The generation method according to claim 8, characterized in that, The event feature encoding layer has n layers, and S16 also includes: (1) Take the output of the event feature extractor in the previous event feature encoding layer as the input of the current event feature encoding layer to obtain the event features output by the current layer; (2) The event features output by the multi-layer event feature encoding layer are concatenated and then concatenated with the embedded representation of questions about video content through a linear layer and input into the large language model. The model generates answers to the questions and adjusts the overall parameters of the model accordingly.

13. The generation method according to claim 8, characterized in that, In S11, when the input is an image, the image is treated as a video frame.

Images