A video description method based on advanced semantic information feature coding

Abstract

This invention provides a video description method based on advanced semantic information feature encoding. The invention includes: predicting multiple independent events contained in an input video sequence using an artificial intelligence network, and outputting the start and end times of each event along with corresponding textual descriptions; extracting the spatial and temporal features of the original video in a decoupled manner, discarding the traditional backbone network, and retaining more complete semantic information; refining video features through an advanced semantic information encoder and a feature fusion module, decoding the advanced semantic information to obtain video description features, achieving better description results; and processing the video description features through a timestamp prediction module and a description content prediction module to obtain the final description result. This invention is mainly aimed at videos containing multiple events, and features low parameter count, high accuracy, and convenient video manipulation; the generated description information can be used for tasks such as classification, retrieval, and audiovisual assistance depending on the application scenario.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, computer vision, and video understanding, and particularly to a method for advanced semantic information extraction and generation of long video descriptions. It is applicable to information and event retrieval in everyday videos and to assisting people with disabilities in viewing and listening. Background Technology

[0002] With the development of multimedia and internet technologies in recent years, the scale of video data has exploded. Faced with the massive volume of video data, the chaotic distribution of data features, and the length and complexity of videos, network information supervision presents numerous challenges. While artificial intelligence can handle many problems that are difficult for humans to solve, its inability to directly perceive information within videos limits network operations on videos, and in most cases, the classification of video information still requires human intervention. Similarly, the data used in related tasks currently still requires manual tagging. Continuous daily management and operation of long videos consumes significant time and manpower.

[0003] The Vision Transformer (ViT) network model, as a neural network for processing sequential structures, possesses a more flexible receptive field compared to temporal convolutional models. Its dynamic multi-head attention mechanism also enables the network to generalize more effectively in feature learning, significantly improving prediction accuracy. The ViT model divides the image into blocks, allowing the network to learn the relationship matrix between blocks through sub-attention mechanisms during training, ultimately acquiring the semantic information of the entire image. However, because multi-head attention involves computation for each pixel, the model's parameters increase exponentially with the size of the input image. Currently, many models apply the ViT architecture to every frame when learning video representations, increasing the number of parameters the network needs to process to (number of video frames × image patch width). 2 × Number of image patches 2 ) 2 .

[0004] In addition, current video description networks still use traditional temporal convolutions as their backbone. Because convolutions and optical flow-extracted video features cannot be completely eliminated, while the video description network model after backbone intervention improves prediction performance, it still cannot achieve a deeper understanding of the video during training. Compared to traditional video features, semantic information can more concisely describe scenes, and introducing it into video description can deepen the network's understanding of the video. The concept of semantic information originates from natural language processing, using vectors to represent the meaning of each word to represent the relationships between words. If the pixel features of the video input are elevated to high-level semantic information and linked with sentences describing events, the video description process can be made more direct, improving the description effect. However, high-level semantic information has not yet been applied to video-related fields, preventing a fundamental improvement in video description research. This invention replaces pixel feature extraction with high-level semantic information extraction, obtaining high-level semantic information from the video. This reduces the overall number of network parameters while learning better representations in the video, achieving better results in video description tasks. Summary of the Invention

[0005] This invention proposes a video description method based on high-level semantic information feature encoding. This method generates text descriptions corresponding to events in a video, and also obtains the start and end events of each event in the video. The output provides convenience for subsequent operations in video processing. The results of this invention can be applied to assist people with disabilities in understanding video content, information retrieval, and other scenarios.

[0006] This invention addresses existing problems by designing a video description method based on high-level semantic information feature encoding. It employs an encoder-decoder structure, fusing decoupled spatial and temporal encoders to learn representations for long videos. First, the proposed high-level semantic information encoder acquires video sequence features. Then, the decoder decodes the representations. Finally, a multilayer perceptron network is linked for timestamp prediction, and a traditional LSTM network is combined to generate video descriptions. Unlike traditional methods that use convolutional neural networks as the backbone and encoder, this invention introduces high-level semantic information into the encoder for video representation learning through a multi-head self-attention mechanism, thereby enhancing the network's understanding of the video and removing redundant pixel information. The overall network structure is proposed as follows: Figure 1 As shown.

[0007] This invention fully considers the parameter growth problem that occurs when the ViT model is used for temporal feature learning. It employs different decoupled feature learning methods, proposing two separate encoders to learn spatial and temporal features respectively. An attention mechanism is then used to fuse these two types of features, extracting high-level temporal and spatial semantic information in parallel, thereby reducing the number of parameters. The proposed network extracts spatial features from high-resolution frames of video in the spatial stream to obtain detailed high-level semantic information such as objects and scenes. Simultaneously, it extracts temporal features from low-resolution continuous video frames in the temporal stream to obtain motion features in the video. Furthermore, this invention proposes a fusion module based on multi-head self-attention to further refine the spatiotemporal features and obtain even higher-level semantic information. This decoupled structure reduces the number of network parameters to (the width of each frame). 2 +Video frame rate 2 )×Image patch width 2 . Figure 2 This is a detailed structural diagram of the advanced semantic information encoder proposed in this invention.

[0008] The technical solution of this invention is implemented as follows:

[0009] A temporal description method based on high-level semantic information feature encoding includes:

[0010] (1) A video description method based on advanced semantic information feature encoding, consisting of a preprocessing module, an encoder, a decoder, and a prediction head;

[0011] (2) As a component of feature (1), a native video frame extraction module is constructed to preprocess the video input. While extracting the most representative key frames in the video, the module performs pixel-level and frame-level downsampling on the image to compress and obtain a video frame sequence with equal intervals. The key frame KEY is used to preserve the complete spatial information of the scene and ensure the robustness of the description content, while the compressed video sequence SEQ is used to enhance the dynamic perception of events, reduce redundant spatial information, and improve the global temporal modeling effect.

[0012] (3) As a component of feature (1), the network constructs a high-level semantic information encoder based on transformer to encode pixel features. As the backbone network of the model, it extracts high-level semantic information features from the video in parallel from the spatial and temporal dimensions. These features come from the multi-head self-attention mechanism of pixel features. Among them, the excellent modeling ability of self-attention for global information and the innovative encoding structure in this invention can significantly improve the accuracy of video description while reducing the number of parameters in the backbone network;

[0013] (4) As a component of feature (3), a spatial feature encoder is constructed. It is used to encode the high-level spatial semantic information within the keyframe KEY proposed in (2) and to complete the network spatial flow modeling;

[0014] (5) As a component of feature (3), a temporal feature encoder is constructed to encode the high-level temporal semantic information of the video sequence SEQ proposed in (2) and obtain global information such as the number of events and action relationships in the video.

[0015] (6) As a component of feature (3), a fusion module based on multi-head self-attention mechanism is constructed to perform feature fusion on the temporal and spatial semantic information obtained in (4) and (5), further refine the spatial and temporal semantic information, remove redundant features and reduce the number of parameters.

[0016] (7) As a component of feature (1), a decoder based on cross-attention mechanism is constructed to decode the latent features containing high-level semantic information into event description and start and end time features. The LSTM network constitutes an event description generator, which is used to predict the event description content through the generative model using the features obtained by the decoder.

[0017] (8) As a component of feature (1), an event start and end time predictor based on high-level semantic information is used to fuse the event description features obtained in (7) and the temporal feature encoding obtained in (5) to predict the start and end times of each event in the video.

[0018] The execution logic of the video description method based on high-level semantic information feature encoding in this invention is as follows:

[0019] S1 processes the video sequence into a single-frame sequence F = {f0, f1} with intervals s and length T. S ,f 2S ,…,f T};

[0020] S2 selects the intermediate frame f of the sequence obtained in S2. T / 2 As a video keyframe, it is stored as f k S3 compresses each frame of F to a size of h*w pixels; S3 uses a spatial feature encoder to process the f obtained in S2. k Encoding as feature emb s =(S cls ,S patch );

[0021] S4 uses a temporal feature encoder to process F = {f0, f...} obtained in S2. S ,f 2S ,…,f T Encoded as emb t =(T cls ,T patch );

[0022] S5 uses a multi-head self-attention fusion module to integrate the emb from S3.S and emb in S4 T The fusion process yields the emb. HS ;

[0023] S6 encodes the high-level semantic information obtained from S5 into emb. HS The input decoder obtains the video description features F enc ;

[0024] S7 inputs the decoded feature C generated by S6 into the time prediction module and the description generation module respectively to obtain the description start time Ct and the description content Cl;

[0025] S8 combines the description Cl from S7 with the timestamp Ct to generate the final prediction result.

[0026] Beneficial effects

[0027] This invention proposes a video description method based on high-level semantic information feature encoding, which can generate video descriptions for videos containing dense events, improving the accuracy of event start and end time prediction and description content generation. The proposed high-level semantic information encoding module effectively improves encoding efficiency, including reducing the number of model parameters through decoupling methods, reducing redundant information through spatiotemporal high-level semantic information learning and fusion, and refining semantic information through a self-attention fusion module to improve prediction accuracy. Simultaneously, a cross-attention decoding module fuses image and language multimodal information, decoding high-level semantic information features into video description features, with parallel prediction heads corresponding to event start and end times and content descriptions respectively. The start and end time prediction head, incorporating video temporal coding, further improves time prediction accuracy. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall network structure of the present invention;

[0029] Figure 2 This is a detailed structural diagram of the advanced semantic information encoder of the present invention;

[0030] Figure 3 This is a detailed structural diagram of the spatial and temporal coding module of the present invention;

[0031] Figure 4 This is a schematic diagram of the video preprocessing module of the present invention;

[0032] Figure 5 This is a rendering of the spatiotemporal feature fusion module of the present invention. Detailed Implementation

[0033] Specific embodiments of the present invention are given below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0034] A video description method based on high-level semantic information feature encoding according to an embodiment of the present invention can be applied in practical applications as follows: Figure 1 As shown, the network structure of the present invention is deployed, including:

[0035] A native video processor for extracting high-definition spatial feature keyframes and compressing temporal feature frame sequences from an input network;

[0036] A spatial feature encoder for encoding high-level spatial semantic information in keyframes of a video sequence;

[0037] A temporal feature encoder for encoding temporal high-level semantic information in compressed video sequences;

[0038] A feature fusion module based on a multi-head self-attention mechanism is used to extract high-level semantic information from encoded spatiotemporal features, reduce redundant parameters, and generate latent features.

[0039] A decoder module for obtaining video description features through hidden features;

[0040] A description generation module decodes latent features into language sequences using an LSTM network to obtain the final text description;

[0041] A timestamp prediction module takes as input latent features and high-level semantic information features from temporal networks, and obtains the start and end timestamps of events through a multilayer perceptron network.

[0042] To facilitate understanding of the above technical solutions of the present invention, the following detailed description is provided through actual deployment and application. To complete the video description task, the network needs to ultimately generate two types of labels: one for the start and end times of the description, and the other for the specific description content. During training, this embodiment uses multiple datasets to train the network model, namely Kinetic400, SomethingSomethingV2, and ActivityNet Caption. These datasets are characterized in that all input data contains start and end time annotations for one or more events, as well as human textual descriptions of each event.

[0043] This invention proposes a video description method that typically employs an encoder-decoder structure. The encoder extracts latent features from video pixel information through a backbone network, the decoder decodes these latent features into the information needed to generate the video description, and the prediction head module predicts the final description result. The entire network structure includes feature extraction, feature fusion, feature alignment, generation, and classification tasks. The framework proposed in this invention is as follows: Figure 1 As shown, the network input is a raw video sequence. The preprocessing module decomposes the video sequence into a frame sequence and keyframes. The resulting frames are encoded with high-level semantic information by two decoupled encoder modules. The separated decoupled information is then refined into high-level semantic information and redundant parameters are reduced by a self-attention fusion module to obtain latent features. The latent features are then decoded to obtain a mapping between video features and descriptive features. Finally, the descriptive features are combined with temporal features to obtain the start and end events of the events, which are then decoded by an LSTM network to obtain the descriptive content.

[0044] Early network architectures required pre-trained networks to process videos independently. These pre-trained models were typically based on convolutional neural networks and contained parameters for tasks like image classification and action classification, which couldn't fully address video description tasks. The next stage of the network processed information that had already been transformed into representations, inevitably leading to information loss. In contrast, the network in this invention takes a complete, original video sequence as input, thus avoiding feature loss that occurs in pre-trained networks. It obtains higher-level semantic information end-to-end based on video description to perfectly correspond to the task. Current video description methods utilize the Transformer structure, increasing the network's interpretability through a more flexible receptive field, but also increasing the number of network parameters. For example... Figure 4 As shown, the decoupled video processing method designed in this invention uses only one high-resolution keyframe and a compressed low-resolution frame sequence. The input video sequence T frames are extracted at intervals of k to obtain... Frame image, take the first The original 256×256 pixels are retained for keyframes as spatially decoupled images, while the frame sequence T′ is compressed to 32×32 pixels as temporally decoupled images. In comparison, for processing a video of length T, the number of parameters is reduced from P=(n*T*p) to P'=(n 2 +T' 2 )*p 2 , where n and p are the number of image patches and the number of pixels in each patch, respectively.

[0045] The goal of video description is to convert redundant pixel information into refined linguistic information. This necessitates aligning input and output information at the hidden layer feature stage during the encoding and decoding process of the video description architecture. Addressing the current limitations of networks in efficiently aligning cross-modal features, this invention proposes a backbone network designed for feature extraction in video description, resulting in more complete and refined encoding. For example... Figure 2 As shown, the advanced semantic information encoder proposed in this invention includes the following steps: inputting the obtained decoupled images into a spatial advanced semantic information encoding module and a temporal advanced semantic information encoding module, respectively. The temporal encoding module is an innovation of this invention; it employs a multi-head self-attention mechanism to acquire the action features of each frame of the video in temporal sequence. Its innovation lies in using each frame of the video as a complete input to the multi-head self-attention mechanism, causing it to ignore spatial features and focus on temporal features. The final feature encoding is {f}. cls ,f0,f1,f2,…,f n}, where f cls The first element is a classifier used for integrating temporal coding; the rest are individual temporal codes for each frame. The structure of the temporal coding module and its differences from traditional spatial encoders are shown in... Figure 3 This invention's advanced semantic information encoder also includes a spatial advanced semantic information encoder to extract more detailed spatial information to obtain features such as objects and scenes in the description. This encoder pre-loads advanced semantic information image classification weights trained in the iBoT network to improve the network's accuracy and generalization in video description. The advanced semantic information fusion module included in the advanced semantic information encoder is an innovation of this invention. Its function is to fuse decoupled spatial and temporal semantic information to refine the comprehensive advanced semantic information. The fusion module uses spatially encoded classifiers and temporally encoded frame sequences for fusion to obtain the overall advanced semantic information classifier and frame codes. This module employs a multi-head self-attention mechanism, starting from spatial advanced semantics and processing temporal frame features through self-attention weights. Its visualization results are provided by [example code missing]. Figure 5 As shown. The formula for the advanced semantic information encoder of this invention is expressed as:

[0046] HS emb =σ(attn((Vit(f) k ) CLS ,[Act({f0,f s ,f 2s ,…,f T}) CLS )))

[0047] Where Vit is a spatial encoder incorporating high-level semantic information, Act is a temporal encoder incorporating high-level semantic information, and attn is a high-level semantic information fusion module based on a multi-head self-attention mechanism. σ represents the linear layer and the normalization function.

[0048] The video description network achieves feature alignment between video and linguistic information through an encoding-decoding structure. This invention uses a cross-self-attention mechanism as the decoder, querying high-level semantic information encoding by inputting randomly initialized text features to obtain the implicit features of the video description. To avoid the impact of using a pre-trained action segmentation network for event proposal on overall description accuracy, this invention combines implicit features with temporal encoding, using a time prediction head and a description generation head to process features simultaneously, obtaining end-to-end video description prediction results. The description content prediction head consists of an LSTM network, pre-loaded with the video description network weights. The event start-end time prediction head is an innovation of this invention; it fuses the implicit description features and temporal features from each frame as input, using a multilayer perceptron network for final prediction. To improve prediction accuracy, this invention uses an IoU loss function in the temporal dimension for training, represented as follows:

[0049]

[0050] in These represent the start and end times of the predicted result and the labeled true value for the i-th event, respectively. The final event description result is formed by combining the results of the two prediction heads, and its formula is:

[0051]

[0052] Where [T] n start ,T n end [] represents the start and end timestamps of the nth event, caption n The video description content is predicted by the network, dec is the decoder module based on a multi-head cross-attention mechanism, LSTM is the video description prediction head module, ReLU is the activation function of the time start and end time prediction module, and σ represents the normalization function after each module. The video description method proposed in this invention is trained end-to-end, and maintains two loss functions, IoU and edit distance, for the corresponding output results.

[0053] In summary, by performing a series of operations on video, including feature encoding, implicit feature decoding, start and end time determination, and description generation, all events can be described in text form. This allows for a clear expression of the events that occur in the video without the need for manual editing, thus overcoming the shortcomings of existing video processing technologies.

Claims

1. A video description method based on high-level semantic information feature encoding, characterized in that, It consists of a preprocessing module, encoder, decoder, and prediction head; specifically, it includes the following steps: (1) Construct a video frame extraction module for preprocessing video input; the module extracts the most representative key frames in the video and performs pixel-level and frame-level downsampling on the image so as to compress the video frame sequence with equal intervals; the key frame KEY is used to preserve the complete spatial information of the scene and ensure the robustness of the description content, while the compressed video sequence SEQ is used to enhance the dynamic perception of events, reduce redundant spatial information, and improve the global temporal modeling effect. (2) A high-level semantic information encoder based on transformer is constructed to encode pixel features; this encoder uses a multi-head self-attention mechanism to extract high-level semantic information features from the video in parallel from the spatial and temporal dimensions respectively; (3) Construct a spatial feature encoder to encode the high-level spatial semantic information within the key frame KEY and complete network spatial flow modeling; (4) Construct a temporal feature encoder to encode the temporal high-level semantic information within the video sequence SEQ, and obtain global information of the video from it, including the number of events and the relationship between actions; (5) Construct a fusion module based on a multi-head self-attention mechanism to perform feature fusion on the aforementioned spatial high-level semantic information and temporal high-level semantic information, further refine the spatial and temporal semantic information, remove redundant features and reduce the number of parameters; (6) Construct a decoder based on cross-attention mechanism to decode latent features containing high-level semantic information into event description and start and end time features. The LSTM network constitutes an event description generator, which is used to predict the event description content through the generative model using the features obtained from the decoder. (7) Construct an event start and end time predictor based on high-level semantic information to predict the start and end times of each event in the video by fusing event description features and global information.

2. The video description method based on high-level semantic information feature encoding according to claim 1, characterized in that, The video frame extraction module in step (1) is divided into two branches: extracting a video keyframe for network coding spatial features, and extracting a frame sequence on the time axis for network extraction of video temporal features; The specific implementation method is as follows: Given a video sequence of length T, frames are extracted at intervals of k to obtain T' = T / k frames. The T' / 2th frame is taken as the keyframe, retaining its original 256x256 pixels as a spatially decoupled image. Simultaneously, the sequence... The frames in the image are downsampled and compressed to 32x32 pixels to serve as a temporally decoupled image.

3. The video description method based on high-level semantic information feature encoding according to claim 1, characterized in that, The advanced semantic information encoder described in step (2) also includes a spatial encoder to extract more detailed spatial information to obtain the object and scene features in the description; A time-series encoder is used to extract coarser global temporal information to obtain action features in the description; Its formula is expressed as ； in For spatial encoders, TMT is a time encoder. This is a high-level semantic information fusion module based on a multi-head self-attention mechanism; σ represents the linear layer and the normalization function; the two encoder modules are constructed through pre-trained weights to enhance their perception of high-level semantic information.

4. The video description method based on high-level semantic information feature encoding according to claim 1, characterized in that, The spatial feature encoder described in step (3) has a Vision Transformer network module structure and loads pre-trained model parameters. This structure segments a single frame image into image patches based on pixel spatial locations. The image patches are processed by N multi-head self-attention modules to obtain high-level spatial semantic information. The final output feature encoding is {S}. cls , S0, S1, S2,…, S n }, where S cls One is the classifier used for integrated spatial coding, and the rest are the individual spatial codes for each image patch.

5. The video description method based on high-level semantic information feature encoding according to claim 1, characterized in that, The temporal feature encoder described in step (4) is defined as a Temporal Motion Transformer, which compresses each frame of the video sequence and uses it as the overall input; the sequence input module has an M-layer multi-head self-attention mechanism to obtain high-level temporal semantic information, and finally outputs a frame sequence feature encoding of length n as {f cls , f0, f1, f2, …, f n }, where f cls One is the classifier used for integrating timing codes; the rest are individual timing codes for each frame.

6. The video description method based on high-level semantic information feature encoding according to claim 1, characterized in that, The fusion module described in step (5) performs feature fusion and refinement on the decoupled temporal and spatial high-level semantic information; it first obtains the corresponding spatial code S. cls Dimension S dim and timing coding T cls Dimension T dim The two are aligned in dimension through a feature alignment module; this module is basically composed of a linear layer, a GeLu activation function, and a Dropout layer, and the data is looped twice in this module; T cls Encoding input to the same module makes its dimension S dim = T dim Secondly, a feature fusion module based on multi-head self-attention is established to fuse decoupled spatial and temporal semantic information, while also extracting features. The fusion module uses spatial coding and frame sequence feature coding for fusion, and its input is represented as {S}. cls , S0, S1, S2, …, S n The module outputs a classifier for the overall high-level semantic information and the encoding of each frame. The multi-head self-attention mechanism in this module starts from the spatial high-level semantics and dynamically processes the temporal features of each frame through self-attention weights. The output classifier represents the overall high-level semantic information of the video, and the weight sequence of each frame represents the temporal features combined with spatial information.

7. The video description method based on high-level semantic information feature encoding according to claim 1, characterized in that, The decoder described in step (6) achieves feature alignment between video and language information; by combining latent features with temporal coding, multi-head parallel prediction is performed to obtain the video description results end-to-end; a cross-attention mechanism is used as the decoder, and the high-level semantic information coding is queried by inputting randomly initialized text coding to obtain the latent features F of the video description. hidden The decoder takes as input the latent features (values), keys, and a random vector (query) of equal length as the initial description content; the subsequent description generation module consists of an LSTM network, which preloads the video description network parameters, takes as input the latent features, and outputs a text description.

8. The video description method based on high-level semantic information feature encoding according to claim 1, characterized in that, The event start and end time predictor described in step (7) uses the fusion of implicit features and temporal features of each frame of the time series as input {F}. hidden , f0, f1, f2, …, f n The final prediction is made using a multilayer perceptron (MLP) network. To improve prediction accuracy, an IoU loss function in the time dimension is used for training, and its representation is as follows. ； in , ) represent the start and end times of the network prediction and the actual true value of the i-th event, respectively; the prediction results of these start and end times are fused with the textual description results to generate the final prediction.

Images