A video description method based on motion guidance

Abstract

This invention discloses a video description method based on action guidance, comprising the following steps: S1: Extraction of global and local video features. Global and local visual features of the video are extracted through convolutional neural networks and sampling regression. Specifically, the steps are as follows: Extracting global features of the video: obtaining image sequences by sampling each original video, processing the image sequences using a 2D convolutional neural network, using the output of the terminal pooling layer as static features, segmenting the original video to obtain segments, processing the segment sequences using a 3D convolutional neural network to obtain dynamic features, and using them together with the static features as global features of the video; Extracting local features of the video: extracting the middle frames of the segments, using Faster-RCNN as an object detector, extracting the target object region and coordinates of the video, and fusing the two using a self-attention mechanism as local features of the video; S2: Semantic generation of subject object and predicate action, based on the global and local visual features extracted in S1.

Description

Technical Field

[0001] This invention relates to the field of video description technology, and in particular to a video description method based on motion guidance. Background Technology

[0002] Video captioning, a complex interdisciplinary problem combining computer vision and natural language processing, requires automatically generating a complete natural language description summarizing a given video clip. This task has garnered increasing attention. Early captioning tasks focused on images, and models have achieved relative maturity in terms of accuracy, richness, and stylization when dealing with static visual information. However, video, as dynamic visual information, possesses far greater content complexity than images, emphasizing the accurate capture of the subjects and relationships between them. Exciting progress has been made in areas such as object recognition and human-computer question answering. Video captioning, as a core task, plays a crucial role in areas such as assisting blind individuals with cognition and building multimedia platforms, attracting significant scholarly attention.

[0003] In existing technologies, most exchange models rely on co-occurrence for visual content judgment. That is, when a person and a bicycle appear simultaneously in a video, the model recognizes the simultaneous existence of the two objects and generates a corresponding action. However, this action is not based on the specific interaction process between the two objects, so it may generate multiple actions such as riding or washing, which often leads to errors. Summary of the Invention

[0004] Given the existing technical issues, the exchange model's judgment of visual content is mostly based on co-occurrence, that is, when a person and a bicycle appear simultaneously in a video, the model recognizes the simultaneous existence of the two objects and generates a corresponding action. However, this action is not based on the specific interaction process between the two objects, so it may generate multiple actions such as riding or washing, which often leads to technical problems. This invention proposes a video description method based on action guidance.

[0005] This invention proposes a video description method based on motion guidance, comprising the following steps:

[0006] S1: Global and local feature extraction from video;

[0007] The global and local visual features of the video are extracted using convolutional neural networks and sampling regression. The specific steps are as follows:

[0008] Extracting global features from the video: Sample each original video to obtain an image sequence, process the image sequence using a 2D convolutional neural network, use the output of the terminal pooling layer as static features, segment the original video to obtain segments, process the segment sequence using a 3D convolutional neural network to obtain dynamic features, and use them together with the static features as global features of the video.

[0009] Extracting local features from the video: Extract the middle frame of the segment, use Faster-RCNN as the object detector to extract the target object region and coordinates of the video, and use a self-attention mechanism to fuse the two as local features of the video;

[0010] S2: Semantic generation of the subject and predicate action;

[0011] Based on the global and local visual features extracted by S1, the global dependencies between visual information are captured through a multi-head attention mechanism and a multilayer perceptron. Subject word vectors are generated based on local visual features, and predicate word vectors are generated based on the subject word vectors and dynamic global features in the global features.

[0012] S3: Description generation based on motion guidance;

[0013] The global visual features extracted in S1 are used as the input to the description generation model. The action word vectors generated in S2 are used as a guide to filter the local visual features to obtain the local object features corresponding to the current target generated word. The output of the generation model and its corresponding local object features are fused according to weight using a gating mechanism to generate the corresponding sentence description.

[0014] In the S3 description generation process, the local features generated in S1 and S2 by the hidden layer output of the recurrent neural network are repeatedly interacted and fused with the action word vectors through a gating unit. The process is as follows: First, the global visual features extracted in S1 and the generated description word vectors are concatenated and input into the recurrent neural network. The output hidden layer variables and the action word vectors generated in S2 are mapped and embedded into the same 1024-dimensional variable through a linear layer. The multi-head attention mechanism uses this variable as Q and the local visual features extracted in S1 as K and V to calculate the local object information corresponding to the current video theme action. The local object information and the hidden layer variables are mapped to the same feature space through a linear layer and then subjected to GLU gating operation with the hidden layer variables. The process is shown in the formula:

[0015]

[0016] in, This represents the hidden layer variables generated by decoding the global features at time t using a long short-term memory network. This represents the local features at time t after weighted averaging of action semantics and hidden variable queries.

[0017] Under the control of gating networks, by... and Element-wise multiplication generates weights, which are then added together to output a probability distribution. The word with the highest probability is selected as the result. j The word representing the word generated in the previous moment, y tThis represents the description generated by the current output.

[0018] Preferably, in steps S1 and S2, the global features of the video are extracted as follows: An InceptionResnet model pre-trained on the ImageNet dataset is used to sample 28 frames at fixed intervals to obtain an image sequence, which is then fed into the InceptionResnet model. The 2048-dimensional vector output of the final pooling layer is extracted as the feature vector of the current frame, and the 28 feature vectors are merged as the static global features of the video. A C3D model pre-trained on the Sport 1M dataset is used to segment the video into segments of 16 frames with 8 overlapping frames. The results are input into the C3D model, and the 4096-dimensional vector output of the fc6 layer is extracted as the dynamic global features of the video.

[0019] Feature extraction of local features from the video: A pre-trained Faster-RCNN model is used to extract intermediate frames from the original video and input them into the model. The number of target regions and corresponding locations output is limited to 10. Any excess is filtered based on coordinates. The output results in 1024-dimensional visual information of the target object and 4-dimensional (coordinates and region width and height) location information. These two are concatenated as input and processed using a multi-head attention mechanism. The process is as follows: The 10 input feature matrices are mapped to intermediate variables through linear layers, multiplied together, and their inner products are calculated to generate weight matrices. Element-wise multiplication is then performed on each feature matrix according to its weight. The calculation process is as follows:

[0020]

[0021] Where Q represents the local feature multidimensional matrix after mapping, K and V are the same as Q, and dk represents the dimension of the input.

[0022] Preferably, in the S2 action predicate generation, the process of capturing global dependencies of visual information and generating word vectors is as follows:

[0023] Subject word vector generation: First, based on the global and local visual features extracted in S1, the multi-head attention calculation method, where Q is the global visual feature and KV is the local visual feature, performs weight-based matrix multiplication to output a 1024-dimensional intermediate variable, which is then mapped to a 512-dimensional word vector through a linear layer to obtain the subject.

[0024] Predicate action word vector generation: First, based on the dynamic global visual features extracted in S1 and the subject word vector generated in S2, the two feature vectors are concatenated and input into a fully connected layer, and then mapped to a 512-dimensional word vector through a linear layer to obtain the predicate action. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method;

[0026] Figure 2 This is a framework diagram of the method. Detailed Implementation

[0027] The present invention will be further explained below with reference to specific embodiments.

[0028] Example

[0029] refer to Figure 1-2 This embodiment proposes a video description method based on motion guidance, including the following steps:

[0030] S1: Global and local feature extraction from video;

[0031] The global and local visual features of the video are extracted using convolutional neural networks and sampling regression. The specific steps are as follows:

[0032] Extracting global features from the video: Sample each original video to obtain an image sequence, process the image sequence using a 2D convolutional neural network, use the output of the terminal pooling layer as static features, segment the original video to obtain segments, process the segment sequence using a 3D convolutional neural network to obtain dynamic features, and use them together with the static features as global features of the video.

[0033] Extracting local features from the video: Extract the middle frame of the segment, use Faster-RCNN as the object detector to extract the target object region and coordinates of the video, and use a self-attention mechanism to fuse the two as local features of the video;

[0034] S2: Semantic generation of the subject and predicate action;

[0035] Based on the global and local visual features extracted by S1, the global dependencies between visual information are captured through a multi-head attention mechanism and a multilayer perceptron. Subject word vectors are generated based on local visual features, and predicate word vectors are generated based on the subject word vectors and dynamic global features in the global features.

[0036] S3: Description generation based on motion guidance;

[0037] The global visual features extracted in S1 are used as the input to the description generation model. The action word vectors generated in S2 are used as a guide to filter the local visual features to obtain the local object features corresponding to the current target generated word. The output of the generation model and its corresponding local object features are fused according to weight using a gating mechanism to generate the corresponding sentence description.

[0038] In the S3 description generation process, the local features generated in S1 and S2 by the hidden layer output of the recurrent neural network are repeatedly interacted and fused with the action word vectors through gating units. The process is as follows: First, the global visual features extracted by S1 and the generated description word vectors are concatenated and input into the recurrent neural network. The output hidden layer variables and the action word vectors generated by S2 are mapped and embedded into the same 1024-dimensional variable through a linear layer. The multi-head attention mechanism uses this variable as Q and the local visual features extracted by S1 as K and V to calculate the local object information corresponding to the current video theme action. The local object information and the hidden layer variables are mapped to the same feature space through a linear layer and then subjected to GLU gating operations with the hidden layer variables. The process is shown in the formula:

[0039]

[0040] in, This represents the hidden layer variables generated by decoding the global features at time t using a long short-term memory network. This represents the local features at time t after weighted averaging of action semantics and hidden variable queries.

[0041] Under the control of gating networks, by... and Element-wise multiplication generates weights, which are then added together to output a probability distribution. The word with the highest probability is selected as the result. j The word representing the word generated in the previous moment, y t This represents the description generated by the current output.

[0042] In steps S1 and S2, the global features of the video are extracted as follows: An InceptionResNet model pre-trained on the ImageNet dataset is used to sample 28 frames at fixed intervals to obtain an image sequence, which is then fed into the InceptionResNet model. The 2048-dimensional vector output of the final pooling layer is extracted as the feature vector of the current frame, and the 28 feature vectors are merged as the static global features of the video. A C3D model pre-trained on the Sport1M dataset is used to segment the video into 16-frame segments with 8 overlapping frames. The results are input into the C3D model, and the 4096-dimensional vector output of the fc6 layer is extracted as the dynamic global features of the video.

[0043] Feature extraction of local features from the video: A pre-trained Faster-RCNN model is used to extract intermediate frames from the original video and input them into the model. The number of target regions and corresponding locations output is limited to 10. Any excess is filtered based on coordinates. The output results in 1024-dimensional visual information of the target object and 4-dimensional (coordinates and region width and height) location information. These two are concatenated as input and processed using a multi-head attention mechanism. The process is as follows: The 10 input feature matrices are mapped to intermediate variables through linear layers, multiplied together, and their inner products are calculated to generate weight matrices. Element-wise multiplication is then performed on each feature matrix according to its weight. The calculation process is as follows:

[0044]

[0045] Where Q represents the local feature multidimensional matrix after mapping, K and V are the same as Q, and dk represents the dimension of the input.

[0046] In the S2 action predicate generation, the process of capturing global dependencies of visual information and generating word vectors is described as follows:

[0047] Subject word vector generation: First, based on the global and local visual features extracted in S1, the multi-head attention calculation method, where Q is the global visual feature and KV is the local visual feature, performs weight-based matrix multiplication to output a 1024-dimensional intermediate variable, which is then mapped to a 512-dimensional word vector through a linear layer to obtain the subject.

[0048] Predicate action word vector generation: First, based on the dynamic global visual features extracted in S1 and the subject word vector generated in S2, the two feature vectors are concatenated and input into a fully connected layer, and then mapped to a 512-dimensional word vector through a linear layer to obtain the predicate action.

[0049] Working principle: 1. Global and local feature extraction from video:

[0050] (1) Global feature extraction of video

[0051] To extract global features from the video, we first sample the original video. For each video, we sample 28 frames based on the video's frame length, maintaining a fixed interval between each frame. Each frame is an image. The generated image sequence is fed into an InceptionResNet model pre-trained on the ImageNet dataset for image recognition tasks. Subsequently, during training, the model is iteratively fine-tuned with a small learning rate. The 2048-dimensional vector output of the model's final pooling layer is extracted as the feature vector of the current frame. These 28 feature vectors are merged to form the video's static global feature V. rThe original video was then segmented into 16-frame segments with an 8-frame overlap. This segment sequence was fed into a C3D model pre-trained on the Sport 1M dataset for action recognition. We extracted the output of the 4096-dimensional vector from the fc6 layer as the dynamic global feature V of the video. m ;

[0052] (2) Local feature extraction of video

[0053] For extracting local features from the video, we use a pre-trained Faster-RCNN model. We extract intermediate frames from the original video and input them into the model. The model outputs the detected targets and the size and coordinates of their corresponding regions. The number of output target regions and their corresponding locations is limited to 10; any exceeding this limit is filtered based on coordinates. The output results in 1024-dimensional visual information of the target object and 4-dimensional (coordinates and region width and height) positional information. These two sets are concatenated as input and processed using a multi-head attention mechanism. The process is as follows: The 10 input feature matrices are mapped to intermediate variables through linear layers, multiplied together, and their inner products are calculated to generate weight matrices. Element-wise multiplication is then performed on each feature matrix according to its weight. The calculation process is as follows:

[0054]

[0055] Where Q represents the mapped local feature matrix vector, K and V are the same as Q, dk represents the dimension of the input features, multiple target objects and their location information and the degree of correlation between them are different, important targets are assigned higher weights, significant features are enhanced, and the output result is used as the local feature V' of the video. b ;

[0056] 2. Generation of object and action semantics

[0057] (1) Generation of object semantics

[0058] Video descriptions typically consist of a subject, verb, object, and related modifiers. The subject, verb, and object, as the main body, respectively refer to key information about objects and actions within the video content. We first predict the semantics of the subject object. We then use a multi-head attention mechanism to model the correlation between global and local features. Based on the multi-head attention formula above, we use the global feature V... r As query Q, the local feature is query V′. b The main semantics of the content predicted by K and V are as follows:

[0059] p θ (s|V' b V r = softmax(W) s fatt(W v V r,V' b ,V' b ))

[0060] Where s represents the generated target subject word vector, W is the parameter matrix, each local feature is reassigned weights according to its degree of correlation with the global feature to form a new intermediate layer feature, which is then fed into the softmax layer under the mapping of the linear layer to generate a probability distribution, and the word vector with the highest probability is selected as the subject object semantics.

[0061] (2) Generation of action semantics

[0062] With the help of a multilayer perceptron, we use the subject and the feature V extracted by a 3D convolutional network to represent dynamic global information. m The predicate is decoded as follows:

[0063] p θ (a|s,V m = softmax(W) a ReLU[s,W m V m ])

[0064] Where a represents the predicted predicate word, W is the parameter matrix, the dynamic global features and the subject semantic word vector are concatenated and then fed into the fully connected layer through the ReLU activation function. The two different distributions of features achieve non-linear calculation under the action of the parameter matrix and output the probability through the softmax layer. The word with the highest probability is the action semantic.

[0065] 3. Description generation based on motion guidance

[0066] (1) Preliminary Decoding Based on Global Features

[0067] We first concatenate the global features of the video and the previously predicted descriptive words and feed them into an LSTM for computation, as shown in the following formula.

[0068]

[0069] Using t as the time marker, the LSTM iterates cyclically according to time, outputting a result each time it runs, where h t-1 With y t-1 H represents the hidden layer variables output by the LSTM at the previous time step and the word vectors generated for the description, respectively. In the first description generation, h... t-1 With y t-1 Each variable is initialized separately; the global features of the video are initially decoded to generate intermediate variables. The output of the hidden layer variables of the LSTM represents a preliminary decoding result, including the vocabulary to be generated at the current time step and the corresponding visual information.

[0070] (2) Feature fusion and description generation of motion instruction

[0071] To enable the decoder to capture inter-object relationship information in the video content when needed, we use the action semantics extracted above to determine specific visual local features, and use a gated fusion mechanism to implement feature input and fusion, ultimately obtaining the distribution prediction described at the current time step, as shown in the following formula.

[0072]

[0073] Action semantics a and the hidden layer output of LSTM At time t, the two parameter matrices are mapped and added to the same dimension, serving as Q in the multi-head attention mechanism described above. Local features V are used as K, and V is weighted and recombined. Under the influence of action semantics, local features directly related to the video content are filtered out. Subsequently, under the GLU gating mechanism, they are further fused with hidden variables, as shown in the following formula.

[0074]

[0075] Where t represents the current time step, the filtered local features and hidden variables are mapped together, and weights are generated based on this mapping. The hidden variables are then compared with... The sums are used as the output to generate a probability distribution, and the word with the highest probability is used as the currently generated description.

[0076] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A video description method based on motion guidance, characterized in that, Includes the following steps: S1: Global and local feature extraction from video; The global and local visual features of the video are extracted using convolutional neural networks and sampling regression. The specific steps are as follows: Extracting global features from the video: Sample each original video to obtain an image sequence, process the image sequence using a 2D convolutional neural network, use the output of the terminal pooling layer as static features, segment the original video to obtain segments, process the segment sequence using a 3D convolutional neural network to obtain dynamic features, and use them together with the static features as global features of the video. Extracting local features from the video: Extract the middle frame of the segment, use Faster-RCNN as the object detector to extract the target object region and coordinates of the video, and use a self-attention mechanism to fuse the two as local features of the video; S2: Semantic generation of the subject and predicate action; Based on the global and local visual features extracted by S1, the global dependencies between visual information are captured through a multi-head attention mechanism and a multilayer perceptron. Subject word vectors are generated based on local visual features, and predicate word vectors are generated based on the subject word vectors and dynamic global features in the global features. S3: Description generation based on motion guidance; The global visual features extracted in S1 are used as the input to the description generation model. The action word vectors generated in S2 are used as a guide to filter the local visual features to obtain the local object features corresponding to the current target generated word. The output of the generation model and its corresponding local object features are fused according to weight using a gating mechanism to generate the corresponding sentence description. In the S3 description generation process, the local features generated in S1 and S2 by the hidden layer output of the recurrent neural network are repeatedly interacted and fused with the action word vectors through gating units. The process is as follows: First, the global visual features extracted by S1 and the generated description word vectors are concatenated and input into the recurrent neural network. The output hidden layer variables and the action word vectors generated by S2 are mapped and embedded into the same 1024-dimensional variable through a linear layer. The multi-head attention mechanism uses this variable as Q and the local visual features extracted by S1 as K and V to calculate the local object information corresponding to the current video theme action. The local object information and the hidden layer variables are mapped to the same feature space through a linear layer and then subjected to GLU gating operation with the hidden layer variables. The process is shown in the formula: in, This represents the hidden layer variables generated by decoding the global features at time t using a long short-term memory network. This represents the local features at time t after weighted averaging of action semantics and hidden variable queries. Under the control of gating networks, by... and Element-wise multiplication generates weights, which are then added together to output a probability distribution. The word with the highest probability is selected as the result. j The word representing the word generated in the previous moment, y t This represents the description generated by the current output.

2. The video description method based on motion guidance according to claim 1, characterized in that, In steps S1 and S2, the global features of the video are extracted: the InceptionResnet model pre-trained on the ImageNet dataset is used, and the image sequence is obtained by sampling 28 frames at fixed intervals and input into the InceptionResnet model. The 2048-dimensional vector output of the end pooling layer is extracted as the feature vector of the current frame, and the 28 feature vectors are merged as the static global features of the video. A C3D model pre-trained on the Sport 1M dataset is used to segment the video into segments of 16 frames with 8 overlapping frames. The results are input into the C3D model, and the output of the 4096-dimensional vector at the fc6 layer is extracted as the dynamic global vector of the video. Feature extraction of local features from the video: A pre-trained Faster-RCNN model is used to extract intermediate frames from the original video and input them into the model. The number of target regions and corresponding locations in the output is limited to 10. Any excess is filtered based on coordinates. The output results in 1024-dimensional visual information and 4-dimensional location information of the target object. These two are concatenated as input and processed using a multi-head attention mechanism. The process is as follows: The 10 input feature matrices are mapped to intermediate variables through linear layers, multiplied together, and their inner product is calculated to generate a weight matrix. Element-wise multiplication is then performed on each feature matrix according to its weight. The calculation process is as follows: Where Q represents the local feature multidimensional matrix after mapping, K and V are the same as Q, and dk represents the dimension of the input.

3. A video description method based on motion guidance according to claim 1 or 2, characterized in that, In the S2 action predicate generation, the process of capturing global dependencies of visual information and generating word vectors is described as follows: Subject word vector generation: First, based on the global and local visual features extracted in S1, the multi-head attention calculation method, where Q is the global visual feature and KV is the local visual feature, performs weight-based matrix multiplication to output a 1024-dimensional intermediate variable, which is then mapped to a 512-dimensional word vector through a linear layer to obtain the subject. Predicate action word vector generation: First, based on the dynamic global visual features extracted in S1 and the subject word vector generated in S2, the two feature vectors are concatenated and input into a fully connected layer, and then mapped to a 512-dimensional word vector through a linear layer to obtain the predicate action.

Images