A dense video description generation method based on event boundary processing and multi-feature fusion

Abstract

The application discloses a dense video description generation method and system based on event boundary processing and multi-feature fusion, which realizes accurate positioning of events in a video by enhancing the difference of event boundary features, fully excavates the internal correlation between visual and text modal features through multi-feature cross splicing and fusion, realizes collaborative improvement of the two tasks through joint optimization of event positioning and description generation, and finally generates more accurate, smooth and complete dense video description, and the application calculates the similarity distribution at different time steps by using inter-frame cosine similarity, and judges the existence of the boundary based on the similarity distribution. By comparing the subtle feature differences between adjacent frames, the key event frames and ordinary frames can be more effectively distinguished, so that the accuracy and generation quality of the subsequent event positioning and description generation module are improved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and natural language processing. Specifically, it relates to a method and system for generating dense video descriptions based on event boundary processing and multi-feature fusion. This method and system can be widely applied to various scenarios such as video review, visual assistance, and subtitle generation, providing core technical support for intelligent video content analysis and understanding. Background Technology

[0002] In today's digital age, video data is exploding in size. How to efficiently understand video content and convert it into natural language descriptions has become a research hotspot in the field of computer vision. Dense video description generation, as a key task, requires accurately locating each event in the video and generating a corresponding natural language description for each event. However, existing methods often suffer from problems such as blurred event boundaries, insufficient fusion of multimodal features, and poor coordination between event localization and description generation tasks when processing videos.

[0003] Traditional video description generation methods mostly treat the video as a whole for feature extraction, ignoring the boundary differences between different events in the video. This leads to low event localization accuracy, and the generated descriptions cannot accurately correspond to specific segments in the video. Furthermore, in terms of multimodal feature fusion, existing methods typically employ simple concatenation or weighted summation, failing to fully explore the intrinsic relationship between visual features and textual semantic features, thus limiting the expressive power of the fused features. In addition, event localization and description generation tasks are usually processed separately, lacking an effective collaborative optimization mechanism, making it difficult to achieve mutual promotion and performance improvement between the two tasks. Therefore, there is an urgent need for a dense video description generation method that can accurately identify event boundaries, fully integrate multimodal features, and achieve collaborative optimization of event localization and description generation.

[0004] The typical architecture of existing end-to-end dense video description methods is as follows: Figure 1 As shown, the typical methods for end-to-end dense video description currently available mainly include the following three points.

[0005] The first is the parallel decoding end-to-end framework (PDVC), which mainly treats dense video description as a set prediction task. It overlays an event counter on top of the Transformer decoder and inputs the enhanced representation of event queries into the localization head and description head in parallel, reducing the need for manually designed components. However, it does not enhance the accuracy of event boundaries, and the multimodal feature fusion is only a shallow stitching, failing to solve the semantic difference problem. The second is the pre-trained framework based on pseudo-boundary generation (DIBS), which mainly uses a large language model to generate pseudo-event boundaries and descriptions and improves boundary quality through online optimization. However, it relies on a large amount of unlabeled data and does not achieve deep fusion of multimodal features, thus having limited improvement in boundary localization accuracy in complex scenarios. Thirdly, the method based on multimodal memory knowledge mainly involves constructing a multimodal memory knowledge base, extracting multimodal features from videos, and combining them with external knowledge to generate descriptions. However, event boundary localization relies on traditional detection methods and does not include boundary enhancement design, which can easily lead to boundary ambiguity issues.

[0006] Dense Video Captioning (DVC) involves locating events in unedited video and generating descriptive captions for each located event. Traditional methods employ a two-stage approach: first, events are located in the video, and then captions are generated for each event. This "locate first, caption later" strategy treats the two subtasks independently. However, the two-stage approach fails to consider the interaction between the event location and caption generation tasks, and cannot dynamically establish the interaction relationship between the two subtasks. Therefore, some researchers have attempted to explore the interaction between the two subtasks. For example, using the Transformer architecture for DVC abandons the original fixed sentence pattern filling generation mode and adopts an Encoder-Decoder generation mode, integrating different modal information to aid the output. However, the event proposal generation part is ambiguous, and using Transformer solely for temporal modeling is inefficient when processing long videos.

[0007] Some researchers treat dense video description as a set prediction problem, and achieve deep correlation and mutual promotion between the two tasks by feeding enhanced event query representations in parallel to the location header and title header.

[0008] Some researchers have also proposed a parallel path-dense video description framework that decomposes the "localization-then-description" sequential approach, which can simultaneously locate and describe events.

[0009] Some researchers have also proposed a visual-semantic embedding (ViSE) framework, which models the word (phrase)-context distribution properties across the entire semantic space.

[0010] Other researchers have proposed a multi-concept recurrent learning network, which establishes a bidirectional guidance mechanism between semantic perception and event localization, improving the quality of localization and description through recurrent collaborative learning. This model utilizes multiple concepts at the frame level to enhance video features and provide temporal event cues, but it does not fully consider the auxiliary role of event boundary features in localization.

[0011] Other researchers have designed a universal encoder and decoder with visual and text cross-attention modules by using external memory to fuse prior knowledge and employing a memory retrieval method for cross-modal video-to-text matching.

[0012] Another researcher has developed a novel memory module based on clustered incoming labels to process videos of arbitrary length, and proposed a streaming decoding algorithm that enables the model to make predictions before the video is processed.

[0013] Other researchers have proposed Event Equal Dense Video Description (E2DVC) to overcome time bias and treat all possible events equally.

[0014] Clearly, existing work has not studied the learning of event boundary information during the event detection process. Encoding boundary frames as inappropriate coarse-grained visual features will cause the decoder to mislocalize events, reduce the overall learning and expressive ability of the model, and thus affect the model performance, making it difficult to generate satisfactory natural descriptions. Summary of the Invention

[0015] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for generating dense video descriptions based on event boundary processing and multi-feature fusion. By enhancing the differences in event boundary features, accurate event localization in videos is achieved; by cross-splitting and fusing multiple features, the inherent correlation between visual and textual modal features is fully explored; and by jointly optimizing event localization and description generation, the synergistic improvement of the two tasks is achieved, ultimately generating more accurate, fluent, and complete dense video descriptions.

[0016] The dense video description generation method provided by this invention mainly includes the following steps: Step 1: Data Preprocessing: Obtain the raw video data and its corresponding labeled text data; perform frame extraction and feature encoding on the video data to obtain the video visual feature sequence; perform word segmentation and embedding on the labeled text data to obtain the text semantic feature sequence. During frame extraction, a uniform sampling method is used to extract keyframes from the video at preset time intervals to ensure coverage of the main content of the video. Feature encoding employs pre-trained convolutional neural network models, such as ResNet and ViT, to encode the extracted keyframes, converting high-dimensional image data into fixed-dimensional video visual features and preserving the core visual information of the keyframes. When performing word segmentation on labeled text data, rule-based or statistical word segmentation algorithms, such as Jieba segmentation and HanLP segmentation, are used to segment continuous text into words with independent semantics. Embedding processing uses pre-trained language models, such as BERT and GPT, to convert the segmented text into fixed-dimensional text semantic features, capturing the semantic information of the text.

[0017] Step 2: Boundary Differentiation Enhancement: The cosine similarity algorithm is used to calculate the similarity between features of adjacent frames in the video visual feature sequence, and an initial event boundary candidate set is determined based on the similarity threshold. When the similarity between features of adjacent frames is lower than the threshold, it is considered that there may be an event boundary between these two frames; The boundary features in the initial event boundary candidate set are enhanced by introducing a boundary attention mechanism to highlight the differences between boundary features and non-boundary features. This mechanism calculates the attention weight between each feature location and the boundary feature location, weighting the non-boundary features to significantly improve the discriminability of boundary features in the feature sequence, resulting in enhanced event boundary features.

[0018] Step 3: Multi-feature cross-connection and fusion: A dual-branch architecture is designed, where one branch further extracts and encodes the enhanced event boundary features to obtain the boundary fusion features. This branch adopts a Transformer encoder structure, using a multi-head self-attention mechanism and a feedforward neural network to deeply mine the boundary features and extract more abstract and representative boundary fusion features. Another branch performs cross-modal interactive encoding on video visual feature sequences and text semantic feature sequences to obtain cross-modal fusion features. It also employs a Transformer encoder structure and uses a multi-head self-attention mechanism to achieve interaction between visual features and text semantic features, fully exploring the intrinsic relationship between the two to generate cross-modal fusion features that simultaneously reflect video content and text semantics.

[0019] The boundary fusion features and cross-modal fusion features are cross-stitched together to obtain the final multimodal fusion features. The cross-stitching adopts a dimensional stitching method, merging the two features along the feature dimension, so that the final fusion features contain both key information of the event boundary and comprehensive semantic information of visual and textual modalities.

[0020] Step 4: Joint optimization of event localization and description generation: Construct an end-to-end multi-task collaborative learning framework, using the final multimodal fusion features as input, to train both the event localization and description generation tasks simultaneously.

[0021] The event localization task aims to accurately locate events in a video by predicting the start and end frames of the event. This task employs a smooth L1 loss function to measure the difference between the predicted event boundaries and the true boundaries, guiding the model to learn accurate event localization capabilities. The description generation task generates corresponding natural language descriptions based on event features obtained from localization. This task uses the cross-entropy loss function to measure the difference between the generated description text and the real labeled text, prompting the model to generate more accurate and fluent natural language descriptions. The model parameters of the two tasks are jointly optimized by a multi-task loss function, which is a weighted sum of the event localization loss and the description generation loss. By adjusting the weight coefficients, the learning priorities of the two tasks are balanced, thereby achieving a synergistic improvement in event localization and description generation.

[0022] During model training, the ActivityNet Captions dataset was used for both training and validation. Ablation experiments were conducted to verify the effectiveness of the boundary difference enhancement module, the multi-feature cross-splitting and fusion module, and the joint optimization framework. PyTorch or TensorFlow deep learning frameworks were used to build and train the model, ensuring its efficient operation and scalability.

[0023] The model's performance is evaluated using automatic evaluation metrics such as BLEU, METEOR, ROUGE, and CIDEr, while also incorporating human evaluation to comprehensively assess the accuracy, fluency, and completeness of the generated descriptive text, thus fully measuring the model's performance.

[0024] This invention also provides a dense video description generation system based on event boundary processing and multi-feature fusion, including a data preprocessing module, a boundary difference enhancement module, a multi-feature cross-stitching and fusion module, and a joint optimization module. The functions of each module are as follows: (1) Data preprocessing module: This module is used to acquire the original video data and its corresponding labeled text data, extract frames and encode features from the video data to obtain the video visual feature sequence; and perform word segmentation and embedding on the labeled text data to obtain the text semantic feature sequence. This module provides high-quality input data for subsequent processing steps to ensure that the model can learn accurate feature information.

[0025] (2) Boundary Differentiation Enhancement Module: This module uses the cosine similarity algorithm to calculate the similarity between features of adjacent frames in the video visual feature sequence, and determines the initial event boundary candidate set based on the similarity threshold. It then performs differentiation enhancement processing on the boundary features in the initial event boundary candidate set, introducing a boundary attention mechanism to highlight the difference between boundary features and non-boundary features, thus obtaining the enhanced event boundary features. This module effectively solves the problem of ambiguous event boundaries and provides an accurate basis for event localization.

[0026] (3) Multi-feature cross-concatenation and fusion module: This module is used to design a dual-branch architecture. One branch performs further feature extraction and encoding on the enhanced event boundary features to obtain boundary fusion features; the other branch performs cross-modal interactive encoding on the video visual feature sequence and the text semantic feature sequence to obtain cross-modal fusion features; the boundary fusion features and the cross-modal fusion features are cross-concatenated to obtain the final multi-modal fusion features. This module fully explores the intrinsic relationship between multi-modal features and improves the expressive power of the fusion features.

[0027] (4) Joint Optimization Module: This module is used to construct an end-to-end multi-task collaborative learning framework. It takes the final multimodal fusion features as input and simultaneously trains the event localization and description generation tasks. The event localization task accurately locates events in the video by predicting the start and end frames of the events. The description generation task generates corresponding natural language descriptions based on the event features obtained from localization. The module jointly optimizes the model parameters of the two tasks through a multi-task loss function, achieving a synergistic improvement in event localization and description generation. This module implements the synergistic optimization of the two tasks, effectively improving the overall performance of the model.

[0028] After adopting the above-mentioned technical improvements, the present invention has the following advantages: 1. Enhanced Boundary Differentiation: The similarity distribution at different time steps is calculated using inter-frame cosine similarity, and the existence of boundaries is determined by the difference between adjacent frames. By comparing subtle feature differences between adjacent frames, key event frames can be more effectively distinguished from ordinary frames, thereby improving the accuracy and generation quality of subsequent event localization and description modules. 2. Multi-feature cross-splitting and fusion: The dual-branch structure extracts the salient features and latent semantic features of the boundary frames respectively, fully considering information at different levels, making the video representation more hierarchical and semantically rich; 3. Joint optimization of event localization and description generation: This project adopts an end-to-end multi-task collaborative learning framework to organically combine event localization and description generation tasks, give full play to the complementary advantages of the two, and realize bidirectional interaction between localization information and text generation information. Attached Figure Description

[0029] Figure 1 The diagram shows a typical architecture of the existing end-to-end dense video description method in this invention. Figure 2 The diagram illustrates the calculation process of the boundary difference enhancement algorithm of this invention. Figure 3 The algorithm shown is a multi-feature cross-splitting and fusion algorithm of the present invention; Figure 4 The model shown is an end-to-end dense video description generation model with event boundary enhancement and multi-feature fusion according to the present invention. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0031] This invention first provides a method for generating dense video descriptions, which mainly includes the following steps: Step 1: Data Preprocessing: Obtain the raw video data and its corresponding labeled text data; perform frame extraction and feature encoding on the video data to obtain the video visual feature sequence; perform word segmentation and embedding on the labeled text data to obtain the text semantic feature sequence. During frame extraction, a uniform sampling method is used to extract keyframes from the video at preset time intervals to ensure coverage of the main content of the video. Feature encoding employs pre-trained convolutional neural network models, such as ResNet and ViT, to encode the extracted keyframes, converting high-dimensional image data into fixed-dimensional video visual features and preserving the core visual information of the keyframes. When performing word segmentation on labeled text data, rule-based or statistical word segmentation algorithms, such as Jieba segmentation and HanLP segmentation, are used to segment continuous text into words with independent semantics. Embedding processing uses pre-trained language models, such as BERT and GPT, to convert the segmented text into fixed-dimensional text semantic features, capturing the semantic information of the text.

[0032] Step Two: Boundary Differentiation Enhancement: The cosine similarity algorithm is used to calculate the similarity between features of adjacent frames in the video visual feature sequence. An initial event boundary candidate set is determined based on the similarity threshold. When the similarity between features of adjacent frames is below the threshold, it is considered that an event boundary may exist between these two frames (see details...). Figure 2 ); The boundary features in the initial event boundary candidate set are enhanced by introducing a boundary attention mechanism to highlight the differences between boundary features and non-boundary features. This mechanism calculates the attention weight between each feature location and the boundary feature location, weighting the non-boundary features to significantly improve the discriminability of boundary features in the feature sequence, resulting in enhanced event boundary features.

[0033] Step 3: Multi-feature cross-concatenation and fusion: A dual-branch architecture is designed. One branch further extracts and encodes the enhanced event boundary features to obtain the boundary fusion features. This branch adopts a Transformer encoder structure, using a multi-head self-attention mechanism and a feedforward neural network to deeply mine the boundary features and extract more abstract and representative boundary fusion features. For the multi-feature cross-concatenation and fusion algorithm, please refer to [link to algorithm details]. Figure 3 ; Another branch performs cross-modal interactive encoding on video visual feature sequences and text semantic feature sequences to obtain cross-modal fusion features. It also employs a Transformer encoder structure and uses a multi-head self-attention mechanism to achieve interaction between visual features and text semantic features, fully exploring the intrinsic relationship between the two to generate cross-modal fusion features that simultaneously reflect video content and text semantics.

[0034] The boundary fusion features and cross-modal fusion features are cross-stitched together to obtain the final multimodal fusion features. The cross-stitching adopts a dimensional stitching method, merging the two features along the feature dimension, so that the final fusion features contain both key information of the event boundary and comprehensive semantic information of visual and textual modalities.

[0035] Step 4: Joint optimization of event localization and description generation: Construct an end-to-end multi-task collaborative learning framework, using the final multimodal fusion features as input, to train both the event localization and description generation tasks simultaneously.

[0036] The event localization task aims to accurately locate events in a video by predicting the start and end frames of the event. This task employs a smooth L1 loss function to measure the difference between the predicted event boundaries and the true boundaries, guiding the model to learn accurate event localization capabilities. The description generation task generates corresponding natural language descriptions based on event features obtained from localization. This task uses the cross-entropy loss function to measure the difference between the generated description text and the real labeled text, prompting the model to generate more accurate and fluent natural language descriptions. The model parameters of the two tasks are jointly optimized by a multi-task loss function, which is a weighted sum of the event localization loss and the description generation loss. By adjusting the weight coefficients, the learning priorities of the two tasks are balanced, thereby achieving a synergistic improvement in event localization and description generation.

[0037] During model training, the ActivityNet Captions dataset was used for both training and validation. Ablation experiments were conducted to verify the effectiveness of the boundary difference enhancement module, the multi-feature cross-splitting and fusion module, and the joint optimization framework. PyTorch or TensorFlow deep learning frameworks were used to build and train the model, ensuring its efficient operation and scalability.

[0038] The model's performance is evaluated using automatic evaluation metrics such as BLEU, METEOR, ROUGE, and CIDEr, while also incorporating human evaluation to comprehensively assess the accuracy, fluency, and completeness of the generated descriptive text, thus fully measuring the model's performance.

[0039] Combination Figure 4 The present invention also provides a dense video description generation system based on event boundary processing and multi-feature fusion, including a data preprocessing module, a boundary difference enhancement module, a multi-feature cross-stitching and fusion module, and a joint optimization module. The functions of each module are as follows: (1)Data preprocessing module: It is used to obtain the original video data and its corresponding annotated text data, extract frames from the video data and perform feature encoding to obtain the video visual feature sequence; perform word segmentation and embedding processing on the annotated text data to obtain the text semantic feature sequence. This module provides high-quality input data for subsequent processing steps to ensure that the model can learn accurate feature information.

[0040] The data preprocessing module is mainly responsible for processing the original video data and annotated text data, and providing high-quality feature sequences for subsequent modules.

[0041] In terms of video data processing, key frames are extracted by uniform sampling. Suppose the total duration of the video is T seconds and the preset time interval is Δt seconds, then the number of key frames extracted N = T / Δt. For example, for a video with a duration of 300 seconds, if Δt is set to 2 seconds, then a total of 150 key frames are extracted. Feature encoding uses the pre-trained ResNet-50 model, which is pre-trained on the ImageNet dataset and can effectively extract the high-level semantic features of images. The extracted key frames are input into the ResNet-50 model. After forward propagation, video visual features with a fixed dimension of 2048 are obtained in the average pooling layer, and finally a video visual feature sequence V = [v1, v2, ..., v] is formed, where n is the number of key frames, and each v is a 2048-dimensional vector.

[0042] In terms of text data processing, the Jieba word segmentation algorithm is used to perform word segmentation on the annotated text. For example, for the annotated text "A man is running in the park", the word segmentation result is ["A", "man", "is", "running", "in", "the", "park"]. Embedding processing uses the pre-trained BERT-base model, which is pre-trained on a large-scale corpus and can convert words into vectors with semantic information. The segmented words are input into the BERT-base model to obtain text semantic features with a fixed dimension of 768, forming a text semantic feature sequence T = [t1, t2, ..., t], where m is the number of segmented words, and each t is a 768-dimensional vector.

[0043] (2)Boundary difference enhancement module: It is used to calculate the similarity between adjacent frame features in the video visual feature sequence using the cosine similarity algorithm, determine the initial event boundary candidate set according to the similarity threshold; perform difference enhancement processing on the boundary features in the initial event boundary candidate set, and by introducing the boundary attention mechanism, highlight the difference between the boundary features and non-boundary features to obtain the enhanced event boundary features. This module effectively solves the problem of blurred event boundaries and provides an accurate basis for event localization.

[0044] The core of the boundary difference enhancement module is to accurately identify event boundaries and enhance their feature differences.

[0045] First, calculate the cosine similarity between features of adjacent frames in the video visual feature sequence. For adjacent visual features v and v +1 cosine similarity Sim(v) , v +1 The formula for calculating ) is: [ Sim(v_i, v_{i+1}) = \frac{v_i \cdot v_{i+1}}{|v_i| |v_{i+1}|} ] where "·" represents the vector dot product, |v | represents vector v The L2 norm. Set the similarity threshold to θ, for example, θ = 0.8, when Sim(v , v +1 When θ < , it is assumed that there is an event boundary between the i-th frame and the (i+1)-th frame. The i-th frame and the (i+1)-th frame are added to the initial event boundary candidate set B = [b1, b2, ..., b], where k is the number of boundary candidate frames.

[0046] Then, a boundary attention mechanism is introduced to enhance the discriminative power of boundary features. For each boundary feature b ∈ B, its difference from each feature v in the video visual feature sequence is calculated. Attention weight α The calculation formula is: [ \alpha_{ij} = \frac{\exp(\text{MLP}([v_i; b_j]))}{\sum_{p=1}^{n} \exp(\text{MLP}([v_p; b_j]))}], where [;] represents vector concatenation. MLP is a two-layer perceptron; the first hidden layer has a dimension of 1024 and uses ReLU activation. The second output layer has a dimension of 1. After obtaining the attention weights, the non-boundary features are weighted and adjusted. The adjusted non-boundary features v' For the boundary feature b, keep it unchanged, and finally obtain the enhanced event boundary feature sequence B' = [b1, b2, ..., b] and the adjusted video visual feature sequence V' = [v'1, v'2, ..., v'].

[0047] (3) Multi-feature cross-concatenation and fusion module: This module is used to design a dual-branch architecture. One branch performs further feature extraction and encoding on the enhanced event boundary features to obtain boundary fusion features; the other branch performs cross-modal interactive encoding on the video visual feature sequence and the text semantic feature sequence to obtain cross-modal fusion features; the boundary fusion features and the cross-modal fusion features are cross-concatenated to obtain the final multi-modal fusion features. This module fully explores the intrinsic relationship between multi-modal features and improves the expressive power of the fusion features.

[0048] The multi-feature cross-splitting and fusion module adopts a dual-branch architecture, which processes boundary features and cross-modal features separately, and then fuses them.

[0049] The first branch further extracts and encodes features from the enhanced event boundary feature sequence B'. A Transformer encoder structure is used, consisting of six encoder layers, each containing a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism has eight attention heads, and the feedforward neural network has a hidden layer dimension of 4096. B' is input into the Transformer encoder, and after forward propagation, the boundary fusion feature F_b is obtained, with the same dimension as the input feature, i.e., 2048 dimensions.

[0050] The second branch performs cross-modal interactive encoding on the video visual feature sequence V' and the text semantic feature sequence T. It also employs a Transformer encoder structure, and in the multi-head self-attention mechanism, it calculates not only self-attention between visual features and between text features, but also cross-attention between visual and text features. Specifically, for each visual feature v'... Calculate the attention weight of each text feature t with all text features t, and then perform a weighted summation of the text features to obtain the visual features that incorporate the semantic information of the text; similarly, for each text feature t, calculate its attention weight with all visual features v'. The attention weights are used to perform a weighted summation of the visual features, resulting in text features that incorporate visual information. After processing by the Transformer encoder, the cross-modal fusion feature F_c is obtained, with a dimension of 2048 + 768 = 2816.

[0051] Finally, the boundary fusion feature F_b and the cross-modal fusion feature F_c are concatenated to obtain the final multimodal fusion feature F = [F_b; F_c], with a dimension of 2048 + 2816 = 4864.

[0052] (4) Joint Optimization Module: This module constructs an end-to-end multi-task collaborative learning framework. It uses the final multimodal fusion features as input to simultaneously train the event localization and description generation tasks. The event localization task accurately locates events in the video by predicting the start and end frames. The description generation task generates corresponding natural language descriptions based on the event features obtained from localization. The module jointly optimizes the model parameters of both tasks through a multi-task loss function, achieving a synergistic improvement in event localization and description generation. This module achieves synergistic optimization of the two tasks, effectively improving the overall performance of the model. For an end-to-end dense video description generation model with event boundary enhancement and multi-feature fusion, see [link to relevant documentation]. Figure 4 .

[0053] The joint optimization module constructs an end-to-end multi-task collaborative learning framework, simultaneously training the event localization and description generation tasks.

[0054] The goal of the event localization task is to predict the start and end frames of an event. Using multimodal fusion features F as input, a two-layer fully connected architecture is employed for prediction. The first hidden layer has a dimension of 2048 and uses ReLU activation. The second output layer has a dimension of 2, corresponding to the predicted values ​​for the start and end frames of the event, respectively. A smoothed L1 loss function, L_loc, is used to measure the difference between the predicted and true values.

[0055] Example 1: Generation of video descriptions for sports events This embodiment generates dense video descriptions for sports event videos, taking a basketball game video as an example.

[0056] In the data preprocessing stage, keyframes were extracted from the basketball game video at 2-second intervals. Assuming the video length was 1800 seconds, a total of 900 keyframes were extracted. The ResNet-50 model was used to encode the keyframes, resulting in a 2048-dimensional video visual feature sequence. The labeled text consisted of event descriptions from the basketball game, such as "Player A makes a shot" and "Player B grabs a rebound," and Jieba word segmentation and the BERT-base model were used to obtain a 768-dimensional text semantic feature sequence.

[0057] In the boundary difference enhancement stage, the cosine similarity of features between adjacent frames is calculated, with a threshold of 0.75. When the similarity of features between adjacent frames is lower than this threshold, an event boundary is considered to exist. For example, when a player transitions from a dribbling state to a shooting state, the visual feature differences between adjacent frames are significant, and this is identified as an event boundary. By enhancing the difference in boundary features through the boundary attention mechanism, the model can more accurately capture the start and end of events.

[0058] In the multi-feature cross-splitting and fusion stage, the dual-branch architecture processes boundary features and cross-modal features separately. Boundary fusion features highlight the boundary information of the event, while cross-modal fusion features integrate video visual information and text semantic information. The final multi-modal fusion feature contains both the boundary features of the event and rich semantic information.

[0059] During the joint optimization phase, the event localization task accurately predicts the start and end frames of each basketball event. For example, the start frame of a shooting event is the moment the player prepares to shoot, and the end frame is the moment the basketball enters the basket. The description generation task generates corresponding natural language descriptions based on the event features obtained from localization, such as "Player A made a shot between 120 and 125 seconds." Through joint optimization of the multi-task loss function, the model's performance on both event localization and description generation tasks is significantly improved. Tested on a subset of sports events in the ActivityNet Captions dataset, the mean absolute error (MAE) for event localization is reduced to 5.2 seconds, and the CIDEr metric for description generation is improved to 1.25.

[0060] Example 2: Generation of Surveillance Video Event Descriptions This embodiment generates dense video descriptions for surveillance videos, taking shopping mall surveillance videos as an example.

[0061] In the data preprocessing stage, keyframes were extracted from the shopping mall surveillance video at 1-second intervals. For a 3600-second video, a total of 3600 keyframes were extracted. The ViT-base model was used to encode the keyframes, resulting in a 768-dimensional video visual feature sequence. The labeled text consisted of event descriptions from the surveillance video, such as "a customer is selecting goods in front of the shelf" or "a thief stole items from the shelf." HanLP word segmentation and the RoBERTa model were used to obtain a 768-dimensional text semantic feature sequence.

[0062] In the boundary difference enhancement stage, a cosine similarity threshold of 0.8 is set. When a change in human behavior occurs in the monitoring footage, such as a customer changing from standing to walking, the similarity of features in adjacent frames will fall below the threshold, and this is identified as an event boundary. Enhancing the difference of these boundary features through a boundary attention mechanism helps the model accurately classify different events.

[0063] In the multi-feature cross-splitting and fusion stage, a dual-branch architecture processes boundary features and cross-modal features separately. Boundary fusion features accurately reflect the boundary information of the event, while cross-modal fusion features integrate visual information from the monitoring footage and semantic information from the labeled text, providing strong support for subsequent event localization and description generation.

[0064] During the joint optimization phase, the event localization task can accurately locate abnormal events in surveillance videos. For example, in a theft incident, the start frame is the moment the thief approaches the shelf, and the end frame is the moment the thief leaves the shelf. The description generation task, based on the event features obtained from localization, generates detailed natural language descriptions, such as "A thief dressed in black stole a mobile phone from the shelf between 1800 and 1810 seconds." Tested on a custom surveillance video dataset, the event localization F1-score reached 0.92, and the description generation BLEU-4 metric improved to 0.48.

[0065] Example 3: Generation of Educational Video Content Descriptions This embodiment focuses on generating dense video descriptions for educational videos, taking a math teaching video as an example.

[0066] In the data preprocessing stage, keyframes were extracted from the math teaching videos at 3-second intervals. For a 1200-second video, a total of 400 keyframes were extracted. The ResNet-101 model was used to encode the keyframes, resulting in a 2048-dimensional video visual feature sequence. The labeled text consisted of descriptions of knowledge points from the teaching videos, such as "the teacher explains the quadratic formula" or "a student demonstrates the solution process." Jieba word segmentation and the GPT-2 model were used to obtain a 768-dimensional text semantic feature sequence.

[0067] In the boundary difference enhancement stage, a cosine similarity threshold of 0.7 is set. When the teaching content changes in the instructional video, such as from explaining theoretical knowledge to demonstrating a problem-solving process, the similarity of features between adjacent frames will fall below the threshold and be identified as an event boundary. By enhancing the difference of these boundary features through a boundary attention mechanism, the model can accurately distinguish different teaching events.

[0068] In the multi-feature cross-concatenation and fusion stage, the dual-branch architecture processes boundary features and cross-modal features separately. Boundary fusion features highlight the boundary information of teaching events, while cross-modal fusion features integrate the visual information of the teaching video and the semantic information of the labeled text, providing rich feature representations for event localization and description generation.

[0069] During the joint optimization phase, the event localization task accurately pinpoints the explanation periods of various knowledge points in the teaching video. For example, the starting frame for explaining the quadratic equation's root formula is the moment the teacher begins writing the formula, and the ending frame is the moment the teacher finishes explaining its application. The description generation task, based on the event features obtained from localization, generates accurate natural language descriptions, such as "The teacher explained the quadratic equation's root formula in detail between seconds 300 and 360." Tested on an educational video dataset, the event localization's Intersection over Union (IoU) reached 0.88, and the description generation's METEOR metric improved to 0.35.

[0070] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating dense video descriptions based on event boundary processing and multi-feature fusion, characterized in that, Includes the following steps: Step 1: Data preprocessing: Obtain the raw video data and its corresponding labeled text data, perform frame extraction and feature encoding on the video data to obtain the video visual feature sequence; The annotated text data is segmented and embedded to obtain a sequence of text semantic features. Step 2: Boundary Differentiation Enhancement: The cosine similarity algorithm is used to calculate the similarity between features of adjacent frames in the video visual feature sequence, and the initial event boundary candidate set is determined based on the similarity threshold; the boundary features in the initial event boundary candidate set are subjected to differentiation enhancement processing. By introducing a boundary attention mechanism, the difference between boundary features and non-boundary features is highlighted, and the enhanced event boundary features are obtained. Step 3: Multi-feature cross-concatenation and fusion: Design a dual-branch architecture. One branch performs further feature extraction and encoding on the enhanced event boundary features to obtain boundary fusion features; the other branch performs cross-modal interactive encoding on the video visual feature sequence and the text semantic feature sequence to obtain cross-modal fusion features; cross-concatenate the boundary fusion features and the cross-modal fusion features to obtain the final multi-modal fusion features. Step 4: Joint optimization of event localization and description generation: Construct an end-to-end multi-task collaborative learning framework, using the final multimodal fusion features as input, to train both the event localization task and the description generation task simultaneously; The event localization task achieves accurate localization of events in videos by predicting the start and end frames of the events; the description generation task generates corresponding natural language descriptions based on the event features obtained from localization; and the model parameters of the two tasks are jointly optimized by a multi-task loss function to achieve synergistic improvement of event localization and description generation.

2. The dense video description generation method based on event boundary processing and multi-feature fusion according to claim 1, characterized in that, In step one, when extracting frames from video data, a uniform sampling method is used to extract key frames from the video at preset time intervals; feature encoding uses a pre-trained convolutional neural network model, such as ResNet or ViT, to encode the extracted key frames and obtain fixed-dimensional video visual features.

3. The dense video description generation method based on event boundary processing and multi-feature fusion according to claim 1, characterized in that, In step one, when segmenting the labeled text data, rule-based or statistical segmentation algorithms, such as Jieba segmentation and HanLP segmentation, are used; the embedding process uses pre-trained language models, such as BERT and GPT, to convert the segmented text into fixed-dimensional text semantic features.

4. The dense video description generation method based on event boundary processing and multi-feature fusion according to claim 1, characterized in that, In step two, the boundary attention mechanism calculates the attention weight between each feature position and the boundary feature position, and performs weighted adjustment on the non-boundary features, thereby significantly improving the recognizability of the boundary features in the feature sequence.

5. The dense video description generation method based on event boundary processing and multi-feature fusion according to claim 1, characterized in that, In step three, both branches in the dual-branch architecture adopt the Transformer encoder structure, which encodes features through a multi-head self-attention mechanism and a feedforward neural network; the cross-splitting of boundary fusion features and cross-modal fusion features adopts a dimensional splicing method, merging the two features in the feature dimension.

6. The dense video description generation method based on event boundary processing and multi-feature fusion according to claim 1, characterized in that, In step four, the end-to-end multi-task collaborative learning framework adopts a structure of shared low-level features and task-specific top-level outputs. The loss function for the event localization task adopts smooth L1 loss, and the loss function for the description generation task adopts cross-entropy loss. The multi-task loss function is a weighted sum of the event localization loss and the description generation loss, and the learning priority of the two tasks is balanced by adjusting the weight coefficients.

7. The dense video description generation method based on event boundary processing and multi-feature fusion according to claim 1, characterized in that, During model training, the ActivityNet Captions dataset was used for training and validation. Ablation experiments were conducted to verify the effectiveness of the boundary difference enhancement module, the multi-feature cross-splitting and fusion module, and the joint optimization framework. The PyTorch or TensorFlow deep learning framework was used to build and train the model.

8. The dense video description generation method based on event boundary processing and multi-feature fusion according to claim 1, characterized in that, The performance evaluation of the model uses automatic evaluation metrics such as BLEU, METEOR, ROUGE, and CIDEr, while also combining manual evaluation to comprehensively assess the accuracy, fluency, and completeness of the generated descriptive text.

9. A dense video description generation system based on event boundary processing and multi-feature fusion, characterized in that, It includes a data preprocessing module, a boundary difference enhancement module, a multi-feature cross-splitting and fusion module, and a joint optimization module; (1) Data preprocessing module: used to obtain raw video data and its corresponding labeled text data, extract frames and encode features from the video data to obtain video visual feature sequences; The annotated text data is segmented and embedded to obtain a sequence of text semantic features; (2) Boundary difference enhancement module: It is used to calculate the similarity between adjacent frame features in the video visual feature sequence using the cosine similarity algorithm, and determine the initial event boundary candidate set according to the similarity threshold; perform difference enhancement processing on the boundary features in the initial event boundary candidate set, and highlight the difference between boundary features and non-boundary features by introducing a boundary attention mechanism to obtain the enhanced event boundary features; (3) Multi-feature cross-splitting and fusion module: used to design a dual-branch architecture. One branch performs further feature extraction and encoding on the enhanced event boundary features to obtain boundary fusion features; the other branch performs cross-modal interactive encoding on the video visual feature sequence and the text semantic feature sequence to obtain cross-modal fusion features; the boundary fusion features and cross-modal fusion features are cross-splitting to obtain the final multi-modal fusion features. (4) Joint optimization module: used to build an end-to-end multi-task collaborative learning framework, taking the final multimodal fusion features as input, and simultaneously training the event localization task and the description generation task; The event localization task achieves accurate localization of events in videos by predicting the start and end frames of the events; the description generation task generates corresponding natural language descriptions based on the event features obtained from localization; and the model parameters of the two tasks are jointly optimized by a multi-task loss function to achieve synergistic improvement of event localization and description generation.

10. A dense video description generation system based on event boundary processing and multi-feature fusion according to claim 9, characterized in that, The system can be applied to fields such as video review, visual assistance, and subtitle generation. By accurately locating and describing events in videos, it provides intelligent video content analysis and understanding capabilities for relevant application scenarios.

Images