A video description method and electronic device based on spatiotemporal super-resolution

Abstract

The present invention relates to a video description method and electronic device based on spatiotemporal super-resolution. The method is implemented based on a video description model and includes the following steps: obtaining an input video, sampling the input video to obtain a video frame sequence containing several compressed size frames; performing multimodal feature extraction and feature encoding on the video frame sequence through the video description model, dynamically fusing the encoded multimodal features, and gradually decoding to generate a video description sentence; wherein, when the video description model is trained, the frames of the original resolution and the intermediate missing frames between adjacent sampled frames are reconstructed from the two dimensions of space and time, and a loss function is constructed with the reconstruction error and the decoding prediction error to implement model training. Compared with the prior art, the present invention has the advantages of rich and accurate description, strong generalization ability, and low computational overhead.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and natural language, and in particular to a video description method and electronic equipment based on spatiotemporal super-resolution. Background Art

[0002] In recent years, with the popularization of 5G networks, video has become a medium for information interaction and has been widely used in people's daily lives. At the same time, it has also brought various new challenges, such as automatic classification, retrieval, action and event detection of large-scale videos and other video understanding tasks. As one of the key tasks of video understanding, video description aims to automatically generate a natural language description for a given video clip. It has very broad application prospects in the fields of human-computer interaction, infant education, and visual impairment assistance. Due to the richness and complex temporal nature of video scenes, it is difficult to model video information. Compared with static image description tasks, video description tasks still have room for development.

[0003] With the major breakthroughs of deep learning in the field of computer science, video description methods based on deep models have achieved excellent results one after another, such as the video description generation method based on deep learning and probabilistic graph models disclosed in patent CN106845411B. Most of the existing methods sample the video frames, compress them to a smaller size and transmit them to the pre-trained convolutional neural network for feature extraction, and then use the encoder-decoder structure to process the extracted features to generate a natural language description. However, this approach ignores the information loss caused by frame sampling and image compression. At the same time, if frame sampling is not performed and the high resolution of the original image is maintained for feature extraction, a large amount of computing cost will be introduced, and the model running speed will be significantly reduced. Therefore, how to design an efficient video description method that captures the rich visual features of the video is one of the research problems that need to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a video description method and electronic device based on spatiotemporal super-resolution with rich and accurate description, strong generalization ability and low computational overhead.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A video description method based on spatiotemporal super-resolution is implemented based on a video description model and includes the following steps:

[0007] An input video is obtained, and sampling is performed on the input video to obtain a video frame sequence including a plurality of compressed size frames;

[0008] Through the video description model, multimodal feature extraction and feature encoding are performed on the video frame sequence, the encoded multimodal features are dynamically integrated, and video description sentences are gradually decoded;

[0009] When the video description model is trained, the frames of the original resolution and the intermediate missing frames between adjacent sampling frames are reconstructed from the two dimensions of space and time, and the loss function is constructed with the reconstruction error and the decoding prediction error to realize the model training.

[0010] Furthermore, the multimodal features include image features, action features and local features.

[0011] The image features are extracted from each video frame through a two-dimensional convolutional neural network, and the temporal correlation of the extracted image features is modeled using a Transformer encoder to achieve feature encoding.

[0012] Furthermore, the motion features of 16 consecutive frames centered on each video frame are extracted through a three-dimensional convolutional neural network, and the temporal correlation of the extracted motion features is modeled using the Transformer encoder to achieve feature encoding.

[0013] Furthermore, the local features are generated through a scene graph detection network, and the local features include objects and semantic associations between objects. A graph neural network is used to model the objects and the semantic associations between objects to achieve feature encoding.

[0014] Furthermore, the image features and local features are fused through a super-resolution decoder to reconstruct frames of original resolution, and the image features and motion features are fused through a super-resolution decoder to restore lost intermediate frames between adjacent sampling frames.

[0015] Furthermore, the super-resolution decoder includes a position-aware attention layer, a feed-forward network layer, and layer normalization, wherein the position-aware attention layer includes a learnable relative position matrix and a mask matrix.

[0016] Furthermore, the step-by-step decoding to generate the video description sentence is specifically:

[0017] At each time step, we first use word embedding to vectorize the words of the previous time step, then dynamically fuse the multimodal features, and use softmax to calculate the probability distribution of the words at the current moment using the vector output by the final fully connected layer.

[0018] Furthermore, the constructed loss function is expressed as:

[0019] L=L lang +λ s L ssr +λt L tsr

[0020] Where L lang represents the language loss describing the decoding prediction error, L ssr represents the super-resolution loss in the spatial dimension, L tsr represents the super-resolution loss in the temporal dimension, λ s and λ t Represents the hyperparameter used to balance the weights of temporal and spatial super-resolution losses.

[0021] The present invention also provides an electronic device, comprising:

[0022] one or more processors;

[0023] Memory;

[0024] One or more programs stored in a memory, the one or more programs comprising instructions for executing the video description method based on spatiotemporal super-resolution as described above.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. Rich and accurate description: The present invention effectively solves the information loss caused by frame sampling and image compression process, enhances the expressive power of each encoder through spatiotemporal super-resolution, captures key object detail information and makes frames more coherent, thereby enriching the visual features of the encoder output. At the same time, the super-resolution decoder can perceive the relative temporal position between frames and focus attention around the current frame to better guide the learning of the encoder. Therefore, the enhanced encoded visual features allow the Transformer decoder to decode and generate sentences with richer semantic information and more accurate descriptions.

[0027] 2. Strong generalization ability: The present invention is no longer sensitive to the video frame sampling frequency and compression size. The spatiotemporal super-resolution guides the encoder to learn richer and more robust visual features and avoids the computational cost of high-definition images. It has strong universality and generalization.

[0028] 3. Low computational overhead: The present invention does not need to increase the sampling frequency and retain the original high-definition resolution. It enriches visual features through spatiotemporal super-resolution. The present invention has low computational cost and efficient operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a video description method based on spatiotemporal super-resolution according to the present invention;

[0030] Figure 2 is a schematic diagram of the structure of a super-resolution decoder and its variants;

[0031] Figure 3 Example of a video description generated for the present invention. DETAILED DESCRIPTION

[0032] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0033] This embodiment proposes a video description method based on spatiotemporal super-resolution. Figure 1 As shown, the method is implemented based on a video description model, and includes the following steps: obtaining an input video, sampling the input video to obtain a video frame sequence containing several compressed size frames; performing multimodal feature extraction and feature encoding on the video frame sequence through the video description model, dynamically fusing the encoded multimodal features, and gradually decoding to generate a video description sentence. When the video description model is trained, the frames of the original resolution and the intermediate missing frames between adjacent sampled frames are reconstructed from the two dimensions of space and time, and a loss function is constructed with the reconstruction error and the decoding prediction error to implement model training.

[0034] 1) Video sampling

[0035] This embodiment performs video sampling on the input video to obtain a video frame sequence including a plurality of compressed size frames, and the frame sequence includes T video frames with a size of 224×224.

[0036] 2) Feature extraction

[0037] The multimodal features considered in this embodiment include image features, action features and local features. Specifically, a two-dimensional convolutional neural network is used to extract image features for each frame in a frame sequence, a scene graph detection network is used to detect possible objects in each frame, and RoI Align is used to obtain local features and establish semantic connections between objects, and a three-dimensional convolutional neural network is used to extract action features from 16 consecutive frame segments centered on each frame.

[0038] 3) Feature Encoding

[0039] For the extracted image features and action features, since they are temporal, the Transformer encoder is used to model their temporal sequence. The Transformer encoder has 4 layers, each of which contains a self-attention layer, a feedforward network layer, and a layer normalization. The model dimension is 512 dimensions, and the output is the encoded image features and action features. For local features, due to their natural graph structure, a graph neural network is used to model the semantic connections between objects. The graph neural network has 2 layers, each of which contains a fully connected layer and a ReLU activation function. The model dimension is 512 dimensions, and the output is the encoded local features.

[0040] 4) Loss Function

[0041] When extracting features, for convenience and to reduce computational overhead, the present invention, like most existing methods, uses frame sampling and image compression, but there is a certain degree of information loss in this process. Therefore, when training the video description model and optimizing its model parameters, the present invention uses a video super-resolution step to reconstruct the original resolution frames and the intermediate missing frames between adjacent sampled frames in both spatial and temporal dimensions to improve information reliability.

[0042] The video super-resolution steps include:

[0043] (1) Spatial super-resolution. In the spatial dimension, image features lose a lot of detail information after compression, while local features happen to contain rich object information and semantic information. Therefore, the present invention uses a super-resolution decoder to reconstruct the frame features of the original resolution based on the encoded image features combined with the local features, and uses the mean square error loss function to measure the reconstructed features, which is calculated as follows:

[0044]

[0045] in Represents the original resolution frame image features reconstructed by the super-resolution decoder, Represents the image features at the original resolution without image compression.

[0046] (2) Temporal super-resolution. In the temporal dimension, the temporal correlation between image features is weakened due to the frame sampling process, while the motion features just capture the rich temporal dynamic changes. Therefore, the present invention uses a super-resolution decoder to restore the lost intermediate frame features of adjacent sampled frames based on the encoded image features combined with the motion features. The mean square error is used to measure the difference between the predicted frame features and the real frame features, and the calculation method is as follows:

[0047]

[0048] in represents the intermediate frame image features restored by the super-resolution decoder, The image features are extracted from the middle frame images representing adjacent sampling frames.

[0049] In the process of restoring the super-resolution frame, for the super-resolution frame at time t, focusing on the surrounding frames with a radius of r around it can effectively obtain context information, and further obtaining the relative temporal position relationship between frames can better restore the super-resolution frame features. The present invention designs a super-resolution decoder to implement the above video super-resolution steps. Figure 2 As shown in Figure 1, the super-resolution decoder includes a position-aware attention layer and a feedforward network layer. Assuming that the length of the frame sequence is T, and the relative time position ranges from [-T+1, T-1], the relative position within this range can be parameterized to obtain a perceptible position matrix, whose i-th row and j-th column means the relative position of the i-th frame relative to the j-th frame. In addition, the mask matrix is ​​used to limit each frame to only focus on the range with a maximum distance of no more than r from the center. After obtaining these two matrices, the perceptible position attention layer is expressed as follows:

[0050]

[0051] Where Q, K, V represent the matrix after the linear transformation of the previous layer input, d represents the dimension of the model, and b pa represents the perceptible position matrix, b mask represents the mask matrix. Next, the feedforward network layer can be expressed as follows:

[0052] feed-forward(x)=ReLU(xW1+b1)W2+b2

[0053] Where x represents the input vector, W1, W2, b1, b2 represent learnable parameters.

[0054] The present invention uses the Transformer decoder to gradually decode and generate sentences, using special words at the initial moment <bos>As the beginning of the sentence, in each of the following time steps, the word output in the previous time step is vectorized using word embedding, and then the features F, M, and O of the three modes are dynamically fused. The vector output by the final fully connected layer is used to calculate the probability distribution of the word at the current moment through softmax:

[0055] P(w t |w ＜t ,F,M,O)=softmax(W3H t +b3)

[0056] where w ＜t represents the words generated before time t, W3 and b3 represent learnable parameters, H t Represents the hidden state of the decoder output at time t. Assuming that the entire description has L words, the cross entropy loss function is used to measure the decoder prediction ability:

[0057]

[0058] During training, the total loss function L is composed of the language loss L lang , spatial super-resolution loss L ssr and the temporal super-resolution loss L tsr The three are composed of the following formula:

[0059] L=L lang +λ s L ssr +λ t L tsr

[0060] where λ s ,λ t Represents the hyperparameter used to balance the weights of temporal and spatial super-resolution losses.

[0061] The above video super-resolution steps are only performed during model training, and will be removed during the testing phase or actual application phase, leaving only the work line for description generation to improve operational efficiency.

[0062] like Figure 3 The example shown is a video description using the video description model constructed by the present invention. From the results, it can be seen that the present invention can obtain a video description that is closer to manual annotation.

[0063] In order to verify the performance of the above method, the following experiments were designed.

[0064] This method is experimentally verified on two benchmark datasets, MSVD and MSR-VTT. Among them, the MSVD dataset contains 1970 videos, all of which are collected from the YouTube website and contain various activities. Each video has about 40 corresponding English descriptions. This method divides this dataset into a training set (1200 videos), a validation set (100 videos) and a test set (670 videos). The MSR-VTT dataset contains 10,000 videos from 20 themes including music, sports, cooking, etc. Each video has a corresponding 20 English descriptions. This method divides this dataset into a training set (6513 videos), a validation set (497 videos) and a test set (2990 videos).

[0065] The objective evaluation indicators used in the experiment include BLEU@4, METEOR, ROUGE-L and CIDEr. Among them, BLEU@4 measures the accuracy of word blocks between real descriptions and generated descriptions, while METEOR measures not only the accuracy but also the recall rate, and extends the matching synonyms. ROUGE-L calculates the harmonic mean of the longest common subsequence between the real description and the generated description. Finally, CIDEr is a consensus-based evaluation indicator specially proposed for image description. It generates an evaluation indicator by measuring the similarity between the generated description sentence and the real description sentence. This is also the objective evaluation indicator that is most consistent with manual evaluation.

[0066] In order to verify the effectiveness of the proposed method, the results of the proposed method are compared with the results of the models in the existing research, and the results are shown in Table 1. The specific details of the 10 methods compared with the proposed method are as follows: (1) TVT: Transformer structure is used to dynamically fuse multimodal information; (2) MARN: memory structure is used to model the compatibility between synonyms; (3) ORG-TRL: external corpus is used to guide the generation of word probability distribution of the decoder; (4) ST-Graph: spatiotemporal graph is used to model the interaction of objects in the video; (5) POS-CG: global grammatical part-of-speech information is used to assist description generation; (6) STAT: spatial attention mechanism is used; (7) SGN: semantic grouping network is used to capture the most discriminative phrases; (8) RecNet: autoencoder is used to reconstruct image features; (9) Multi-Task: video description and video prediction and synonym generation tasks are trained together; (10) SibNet: object branch and semantic branch are used to reconstruct image features and semantic features respectively. In each column of the table, the optimal indicator value is bolded, and the suboptimal indicator is underlined.

[0067] Table 1 Performance comparison of the proposed method with existing methods on MSVD and MSR-VTT datasets

[0068]

[0069] As can be seen from Table 1, on MSVD, the method proposed in the present invention achieved the best level in three indicators, especially in CIDEr, which was significantly improved by 2.6%. On MSR-VTT, the method proposed in the present invention achieved the best CIDEr score, which was improved by 0.9%, and achieved suboptimal results in other indicators. The reason may be that MSR-VTT is a large-scale dataset, which benefits more from the external corpus used in ORG-TRL. CIDEr is specially designed for description tasks and is more in line with human judgment. Therefore, in general, the present invention has achieved excellent results compared with most existing methods.

[0070] In order to evaluate the effects of the proposed temporal and spatial super-resolution, this method designed an ablation experiment for comparison. The results are shown in Table 2.

[0071] Table 2 Performance comparison of the present invention with different types of super-resolution

[0072]

[0073] As can be seen from Table 2, whether using temporal super-resolution or spatial super-resolution, the four evaluation indicators on both datasets exceed the baseline model, indicating that temporal or spatial super-resolution can effectively improve the learning ability of the encoder and obtain rich and coherent visual features. In addition, the use of spatiotemporal super-resolution has further improvement compared to using temporal or spatial super-resolution alone, indicating that the super-resolution of the two dimensions of time and space is complementary to the improvement of the encoder. The temporal dimension focuses on improving the coherence of visual features, while the spatial dimension focuses on learning visual features containing rich detail information.

[0074] In order to verify the performance of the super-resolution decoder proposed in the present invention, the present invention designs an ablation experiment to compare the super-resolution decoder and its variants (such as Figure 2 The performance comparison is shown in Table 3, where the unimodal decoder only uses the image features of a single modality to restore the super-resolution features, and the multimodal decoder uses the traditional multi-head attention layer. The super-resolution decoder of the present invention can not only fuse multimodal information, but also has an attention layer that can perceive the position. The performance comparison results are shown in Table 3.

[0075] Table 3 Performance comparison of the present invention with different types of super-resolution decoders

[0076]

[0077] It can be seen from Table 3 that the worst result was achieved by using a single-modal decoder, the second place was achieved by using a multi-modal decoder, and the best result was achieved by using the super-resolution decoder proposed in the present invention. This shows that it is insufficient to use only a single-modal information for super-resolution decoding, and it is unable to fully guide the encoding capability of the multi-modal encoder. On the other hand, adding a position-aware attention layer can more efficiently aggregate the information of the surrounding frames, thereby more effectively guiding the encoding capability of the multi-modal encoder.

[0078] Experiments have shown that this method has achieved excellent results on the two benchmark datasets, MSVD and MSR-VTT, and can outperform existing methods in most indicators. It is easy to implement and does not require additional computational overhead. Among the currently known video description generation methods, this method has strong advantages and application prospects.

[0079] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0080] In another embodiment, an electronic device is provided, comprising one or more processors, a memory, and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the video description method based on spatiotemporal super-resolution as described above.

[0081] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.< / bos>

Claims

1. A video description method based on spatiotemporal super-resolution, characterized in that: The method is implemented based on a video description model and includes the following steps: An input video is obtained, and sampling is performed on the input video to obtain a video frame sequence including a plurality of compressed size frames; Through the video description model, multimodal feature extraction and feature encoding are performed on the video frame sequence, the encoded multimodal features are dynamically integrated, and video description sentences are gradually decoded to generate the video description sentences, wherein the multimodal features include image features, action features and local features; When the video description model is trained, the video super-resolution step is used to reconstruct the frames of the original resolution and the intermediate missing frames between adjacent sampling frames in both spatial and temporal dimensions, and a loss function is constructed with the reconstruction error and the decoding prediction error to implement model training; The video super-resolution step comprises: (1) Spatial super-resolution: In the spatial dimension, a super-resolution decoder is used to reconstruct the frame features of the original resolution based on the encoded image features combined with the local features, and the mean square error loss function is used to measure the reconstructed features. The calculation method is as follows: in Represents the original resolution frame image features reconstructed by the super-resolution decoder, Represents the image features of the original resolution without image compression; (2) Temporal super-resolution: In the temporal dimension, a super-resolution decoder is used to restore the lost intermediate frame features of adjacent sampling frames based on the encoded image features combined with the motion features. The mean square error is used to measure the difference between the predicted frame features and the real frame features. The calculation method is as follows: in represents the intermediate frame image features restored by the super-resolution decoder, The image features extracted from the intermediate frame images representing adjacent sampling frames; The super-resolution decoder includes a position-aware attention layer and a feed-forward network layer, wherein the position-aware attention layer is represented by the following formula: Where Q, K, V represent the matrix after the linear transformation of the previous layer input, d represents the dimension of the model, and b pa represents the perceptible position matrix, b mask represents a mask matrix. Assuming that the length of the frame sequence is T, the relative time position range is [-T+1, T-1]. The relative position within this range is parameterized to obtain the perceptible position matrix, whose i-th row and j-th column means the relative position of the i-th frame relative to the j-th frame. The mask matrix is ​​used to limit each frame to focus on the range with the farthest distance from it not exceeding r; The feed-forward network layer is represented by the following formula: feed-forward(x)=ReLU(xW1+b1)W2+b2 Where x represents the input vector, W1, W2, b1, b2 represent learnable parameters; The image features are extracted from each video frame through a two-dimensional convolutional neural network, and the temporal correlation of the extracted image features is modeled using a Transformer encoder to achieve feature encoding. Specifically: Use the Transformer decoder to gradually decode the generated sentence, using special words at the initial moment <bos> As the beginning of the sentence, in each of the following time steps, the word output in the previous time step is vectorized using word embedding, and then the features F, M, and O of the three modes are dynamically fused. The vector output by the final fully connected layer is used to calculate the probability distribution of the word at the current moment through softmax:< / bos> P(w t |w <t ,F,M,O)=softmax(W3H t +b3) where w <t represents the words generated before time t, W3 and b3 represent learnable parameters, H t Represents the hidden state of the decoder output at time t; Assuming that the entire description has L words, the cross entropy loss function is used to measure the decoder prediction ability: The constructed loss function is expressed as: L=L lang +λ s L ssr +λ t L tsr Where L lang represents the language loss describing the decoding prediction error, L ssr represents the super-resolution loss in the spatial dimension, L tsr represents the super-resolution loss in the temporal dimension, λ s and λ t represents the hyperparameter used to balance the weight of temporal and spatial super-resolution loss; The above video super-resolution step is only performed during model training and is removed during the testing phase or actual application phase, leaving only the work line for description generation.

2. The video description method based on spatiotemporal super-resolution according to claim 1, characterized in that: The motion features of 16 consecutive frames centered on each video frame are extracted through a three-dimensional convolutional neural network, and the temporal correlation of the extracted motion features is modeled using the Transformer encoder to achieve feature encoding.

3. The video description method based on spatiotemporal super-resolution according to claim 1, characterized in that: The local features are generated through a scene graph detection network, and the local features include objects and semantic associations between objects. Objects and semantic connections between objects are modeled using a graph neural network to achieve feature encoding.

4. The video description method based on spatiotemporal super-resolution according to claim 1, characterized in that: The super-resolution decoder also includes layer normalization.

5. The video description method based on spatiotemporal super-resolution according to claim 1, characterized in that: The step-by-step decoding to generate the video description sentence is specifically: At each time step, we first use word embedding to vectorize the words of the previous time step, then dynamically fuse the multimodal features, and use softmax to calculate the probability distribution of the words at the current moment using the vector output by the final fully connected layer.

6. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs stored in a memory, wherein the one or more programs include instructions for executing the video description method based on spatiotemporal super-resolution as claimed in any one of claims 1-5.

Images