Unsupervised video description method based on self-constrained dynamic text features

Abstract

The present invention discloses an unsupervised video description method based on self-constrained dynamic text features. The method of the present invention first uniformly samples the video to extract its video appearance and motion feature vector, target feature vector and target category tag, obtains the one-hot encoding vector of the random description sentence, and inputs it and the video feature vector into the description sentence generation module; obtains the generated sentence feature matrix and inputs it into the pseudo-tag generation module to obtain the pseudo-text feature matrix; replaces the random description sentence feature matrix with the pseudo-text feature matrix, and re-enters the description sentence generation module to obtain the final generated sentence feature matrix; finally, the corresponding description sentence is generated by a greedy search algorithm. The method of the present invention does not require manual labeling, and matches visual information and language knowledge through constrained activation units to reduce semantic differences, and generates pseudo-tags using a dynamic optimization method of text features, which is conducive to obtaining natural sentences that describe video content more fluently and accurately.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, in particular to the field of visual understanding and analysis, and relates to an unsupervised video description method based on self-constrained dynamic text features. Background Art

[0002] In the 5G era, video, as the content carrier that can best restore the real world, has become the main form of content expression on the Internet. How to accurately understand video content has become an urgent challenge in practical application fields such as video title generation, video retrieval, visually impaired assistance, and human-computer interaction. How to describe video content in natural language has become an important research direction for computer vision semantic understanding. In daily life, due to the lack or incompleteness of visual channels, it is difficult for visually impaired people to effectively transmit external multimedia information to the brain, which makes this group's environmental perception ability weak. For example, it is impossible to achieve synchronous understanding of the picture, voice and text of film and television content like people with normal vision; for the visually impaired, video description technology can be used to convert video content into text description, and then voice conversion technology can be used to convert text into voice broadcast, helping the visually impaired to understand the content of film and television pictures through sound, thereby meeting the daily needs of this group.

[0003] The video description task aims to extract features from a given video and convert it into a natural sentence or paragraph that conforms to correct grammar. Compared with the image description task that only needs to capture static information, the video description task requires the analysis, processing and re-expression of the diverse and complex spatiotemporal features and semantic features in the video data, which is more challenging.

[0004] At present, the mainstream video description method adopts the encoder-decoder framework. Among them, the encoder generally uses the residual network (ResNet) and the convolutional 3D neural network (C3D) to obtain the appearance features and motion features of the video respectively; the decoder generally uses the long short-term memory network (LSTM) to decode the video features and generate the corresponding description sentences. The current video description method relies on manual text tagging, which is a pair relationship between the text tag and the video, and the model is trained in a fully supervised way. In order to reduce the dependence on text tags, it is necessary to explore unsupervised video description models. In recent years, large-scale pre-trained language models have developed rapidly, containing a large amount of language knowledge, such as grammatical logic, part of speech information, and tense information, which can significantly improve the semantic coherence and accuracy of generated sentences, and can be used to help unsupervised models generate natural sentences that accurately reflect the content of the video. In addition, in the field of natural language processing, unsupervised sentence generation tasks generally use reinforcement learning, and by designing a certain learning strategy, the reward of the model in the process of interacting with the environment is maximized.

[0005] The above methods have the following main shortcomings: (1) Existing video description models rely on a large number of video-text description pairs, but text descriptions need to be manually labeled, resulting in high labor costs and even labeling errors, which greatly limits the application scope of video description models in practical scenarios; (2) In the field of video description, when using large pre-trained language models, since the pre-trained language models are not designed for visual tasks, there is a mismatch between the pre-trained models and the video description tasks, which will lead to semantic errors in the generated sentences, such as generating words that are irrelevant to the video content; (3) In the field of natural language processing, unsupervised generation tasks use reinforcement learning to fine-tune large pre-trained language models, which will bring high computational overhead and is difficult to meet the actual tasks with huge data volumes. Based on the above considerations, there is an urgent need for an unsupervised video description method that can balance visual information and language knowledge and automatically generate supervised text. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the prior art and propose an unsupervised video description method based on self-constrained dynamic text features, design a constrained activation unit, balance visual input and language knowledge, and solve the mismatch problem between the pre-trained language model and the video description task; use the text feature dynamic optimization method to generate pseudo-labels, solve the problem of lack of supervisory information in unsupervised tasks, and finally generate natural and fluent video description sentences.

[0007] After the method of the present invention obtains the video data set, the following operations are performed in sequence:

[0008] Step (1) uniformly sampling the video to obtain a frame sequence, using a two-dimensional and a three-dimensional convolutional network to obtain appearance features and motion feature vectors respectively, and obtaining a target category label and a target feature vector through a faster regional convolutional neural network;

[0009] Step (2) constructing a description sentence generation module using a constraint activation unit, the constraint activation unit is composed of a visual constraint gate and a language constraint gate, the input is appearance features, motion features and target feature vectors, and the output is a generated sentence feature matrix;

[0010] Step (3) constructing a pseudo-label generation module, the input of which is the generated sentence feature matrix and the target category label, and obtaining the pseudo-text feature matrix by dynamically updating the generated sentence feature matrix;

[0011] Step (4) re-inputting the pseudo text feature matrix into the sentence generation module as a pseudo tag, and optimizing the description sentence generation module and the pseudo tag generation module using a stochastic gradient descent algorithm;

[0012] Step (5) obtains the probability distribution of the generated sentence for the new video through (1) to (4) in sequence, and obtains the corresponding description sentence using the greedy search algorithm.

[0013] Further, step (1) specifically comprises:

[0014] (1-1) Uniformly sample N video frames to obtain a frame sequence set 3D tensor x i is the i-th frame image, C′, H′, W′ are the number of channels, height and width of the video frame, 3≤N≤10, is the real number domain; the set of one-hot encoding vectors of random description sentences is T is the length of the description sentence, 1≤T≤20, y t is the unique hot vector of the tth word in the vocabulary of the description sentence, n represents the number of words in the vocabulary, where the description sentence and the video are not in correspondence, that is, any random natural sentence can be used as the description sentence of the video;

[0015] (1-2) Using residual network to extract the appearance feature vector set of the video represents the appearance feature vector of the i-th frame, d a Represents the dimension size of the appearance feature vector, and the superscript and subscript a represent appearance; the Convolutional 3DNeural Network (C3D) is used to extract the motion feature vector set of the video represents the motion feature vector extracted from the i-1th frame to the i+1th frame, d m Represents the dimension of the motion feature vector, and the superscript and subscript m represent motion;

[0016] (1-3) For video frame x i , use Faster R-CNN (Fast Region-based Convolutional Neural Network) to detect targets and obtain the target feature vector set of the i-th frame and the object category labels contained in the entire video o i,j represents the feature vector of the jth target in the i-th frame, Q represents the number of set video frame targets, 1≤Q≤5, d o Indicates the dimension of the target feature vector, subscript o indicates the target (objective), c k Represents the category label vector of the kth target, K is the number of target categories, K ≥ 1, d c Indicates the dimension size of the target category, and the subscript c represents the category (class);

[0017] (1-4) Perform the maximum pooling operation on the target feature vector corresponding to each video frame to obtain the pooled target feature vector set of the video in Maxpool(·) represents the maximum pooling operation in the spatial dimension, which transforms the appearance feature vector Motion feature vector And the pooled target feature vector Splice them one by one to get a set of video feature vectors where v i Represented as the video feature vector of the i-th frame, the video feature vector dimension is d v =d a +d m +d o , the subscript v represents video.

[0018] Furthermore, step (2) is specifically:

[0019] (2-1) Constructing a description sentence generation module: The description sentence generation module consists of the second-generation generative pre-training transformer GPT-2 (Generative Pre-training Transformer-2) and a constraint activation unit, where GPT-2 consists of L layers of transformers (Transformer), L ≥ 1, GPT-2 is a unidirectional language pre-training model for generative tasks, GPT-2 has been pre-trained on a large-scale dataset, and the initial parameters of the model are the pre-training results; the constraint activation unit consists of a visual constraint gate and a language constraint gate, which are used to balance visual information and language knowledge;

[0020] (2-2) The video feature vector v i and the one-hot encoding vector y of the random description sentence t , mapped to the same dimension size d through the fully connected layer, and the mapped video feature matrix is ​​obtained and the random description sentence feature matrix [·] T Represents mathematical transpose, matrix The i-th row matrix The tth row of and is the learnable parameter matrix, and the subscript y represents the sentence;

[0021] (2-3) Calculate the position encoding matrix of video features Postion(·) is the function for calculating the position encoding. The specific operation is: Represents the matrix PE v The value of the ωth column in the i-th row, mod means the remainder operation; similarly, the sentence position encoding matrix is ​​obtained The video position encoding matrix PE v The i-th row in and the video feature vector set The i-th vector in Add, sentence position encoding matrix PE y The tth row in and a set of random description sentence feature vectors The tth vector in Add them together to get the video feature matrix with position encoding and the random description sentence feature matrix matrix The i-th row matrix The tth row of It means element-by-element addition;

[0022] (2-4) Add the video feature matrix with position encoding and the random description sentence feature matrix In the attention submodule of the first layer of the converter in the one-way language pre-training model GPT-2, the second layer starts to convert the output matrix of the previous layer of converter Video feature matrix for alternative position encoding and the random description sentence feature matrix Input the attention submodule of the current layer converter in the unidirectional language pre-training model GPT-2; the attention submodule consists of a multi-head attention layer and a linear layer, and calculates the intermediate state matrix of each converter LayerNorm(·) represents the layer normalization operation, MultiHead(·) represents the multi-head attention operation;

[0023] Concat(·) represents the channel concatenation operation. is the learnable parameter matrix corresponding to the lth layer, is the h-th head attention vector, h=1,2,…,H, H represents the total number of attention heads, H≥1, Attention(·) represents the attention operation, in are the learnable parameter matrices corresponding to the lth layer and the hth head;

[0024] (2-5) The intermediate state matrix Input constraint activation unit to get vision-language output matrix The constraint activation unit consists of a visual constraint gate and a language constraint gate. Represents element-wise multiplication;

[0025] First, get the visual constraint gate matrix and the language constraint gate matrix The superscript vis stands for vision, and the superscript lan stands for language; the matrix B vis The value of the tth row and ωth column Matrix B lan The value of the tth row and ωth column σ(·) represents the Sigmoid activation function, the threshold is 0<τ<1, ζ(·) represents the discriminant function, if the statement in the function meets the condition, it returns 1, otherwise it returns 0;

[0026] Then the visual constraint gate matrix B vis Perform standardization operation to obtain the standardized visual constraint gate matrix matrix The value of the tth row and ωth column For the language constraint gate matrix B lan Perform standardization operations to obtain the standardized language constraint gate matrix matrix The value of the tth row and ωth column

[0027] (2-6) Output the visual-language matrix Input a feedforward network layer consisting of two fully connected layers to obtain the output matrix of the converter in GPT-2 Where FFN(·) represents the feedforward operation, max(·) represents the maximum value, are learnable parameter matrices and vectors;

[0028] (2-7) After all layer converters are processed, the final output matrix is ​​obtained Get the generated sentence feature matrix through this output matrix Y′=H L W′, where is the learnable parameter matrix.

[0029] Furthermore, step (3) is specifically:

[0030] (3-1) The pseudo-label generation module consists of a robustly optimized BERT pre-training model RoBERTa (A Robustly Optimized BERT Pretraining Approach, BERT is a bidirectional encoder representation converter (Bidirectional Encoder Representation from Transformers)), a target attribute judgment submodule, a text segment selection submodule, and a text segment optimization submodule; RoBERTa consists of M layers of converters, M ≥ 1, RoBERTa is a bidirectional language pre-training model for text word prediction and modification, and RoBERTa has been pre-trained on a large-scale dataset, and the initial parameters of the model are the pre-training results; the target category judgment submodule consists of multiple linear layers, which is used to judge whether the previously obtained generated sentence contains the detected target category tag; the text segment selection submodule consists of multiple linear layers, which is used to dynamically select the text segment that needs to be modified in the generated sentence; the text segment optimization submodule consists of multiple linear layers, which is used to dynamically optimize the text segment obtained by the text segment selection submodule;

[0031] (3-2) An all-zero vector Splice to the first row of the generated sentence feature matrix Y′ to obtain a generated sentence feature matrix with target category attributes

[0032] (3-3) Generate sentence feature matrix with target category attributes In the attention submodule of the first layer of the input bidirectional language pre-training model, the second layer starts to convert the output matrix of the previous layer of the converter Alternative generation sentence feature matrix As input; the attention submodule consists of a multi-head attention layer, a linear layer, and a feedforward network layer to obtain the output matrix of the converter

[0033]

[0034] is the intermediate state matrix of the mth layer, are learnable parameter matrices and vectors;

[0035] (3-4) After all layer converters are processed, the final output matrix is ​​obtained Take the first row of each layer output matrix to form the target prediction category feature vector set in Represents the first row vector of the output matrix of the mth layer;

[0036] (3-5) Set the target prediction category feature vector Input the target category judgment submodule to obtain the target prediction category probability vector set k-th target prediction category probability Where softmax(·) is a normalized exponential function, represents the first row vector of the output matrix of the kth layer, is the learnable parameter matrix;

[0037] (3-6) The target attribute probability a of the query set k and the true target category label vector c k , the target category loss is calculated by the cross entropy loss function

[0038] (3-7) Use the stochastic gradient descent method to update the bidirectional language pre-training model and obtain the back-propagation gradient matrix set of each layer Represents the gradient matrix of the mth layer;

[0039] (3-8) The back propagation gradient matrix of the first layer In the input text fragment selection submodule, Remove the first row of , and then calculate the bi-norm of the channel dimension to obtain the target attribute score set corresponding to each word The tth word |·||2 indicates l2-norm calculation, is the gradient matrix of the first layer The tth row; determine the selected segment length to be υ, υ ≥ 1, calculate the segment score, and add the scores of the tth to t+υth words. Select the segment with the lowest score Indicates the position of the first word in the segment with the lowest score;

[0040] (3-9) by minimum segment score Select the first word position The first Replace the rows with all-one vectors to obtain the generated sentence feature matrix after replacement Then the matrix is ​​input into the bidirectional language pre-training model, and the operation is the same as (3-3) and (3-4), and finally the output matrix corresponding to the generated sentence after replacement of the K-th layer converter is obtained.

[0041] (3-10) Select the corresponding output matrix after replacement No. Rows, get part of the output matrix Will Input the text fragment optimization submodule to obtain the dynamically optimized text fragment feature matrix in is the learnable parameter matrix;

[0042] (3-11) will generate the sentence feature matrix Y′ Rows are replaced with text snippet feature matrix Finally, the pseudo text feature matrix is ​​obtained

[0043] Furthermore, step (4) is specifically:

[0044] (4-1) The pseudo text feature matrix Input description statement generation module, used to replace the random description statement feature matrix Execute steps (2-3) to (2-7) again to obtain the final generated sentence feature matrix

[0045] (4-2) Use the fully connected layer and softmax function to calculate the probability matrix of the final generated sentence in Represents the learnable parameter matrix in the first fully connected layer; then calculates the probability matrix of pseudo-labeling in represents the learnable parameter matrix in the second fully connected layer;

[0046] (4-3) Probability matrix for the final generated sentence Pseudo-labeled probability matrix Calculate the cross entropy loss between the two in is the probability matrix for generating sentences The tth row of is the probability matrix of pseudo-labeling The tth row of .

[0047] Furthermore, step (5) is specifically:

[0048] (5-1) optimizing the video description model by minimizing the cross entropy loss function using a stochastic gradient descent method until convergence, wherein the video description model comprises a description sentence generation module and a pseudo-label generation module;

[0049] (5-2) Input the new video and uniformly sample N video frames to get At the same time, get the one-hot encoding vector of the new random description sentence First, go through steps (1) to (4) to get the new probability matrix of the final generated sentence For each row of this matrix, a greedy search algorithm is used to select the word with the highest probability corresponding index from the vocabulary as the generated word y. t ′, and finally obtain the description statement {y1′,...,y t ′,...,y′ T′}, is the tth word, and T′ is the length of the generated sentence.

[0050] The present invention proposes an unsupervised video description method based on self-constrained dynamic text features, which has the following characteristics: (1) It is the first time to propose an unsupervised video description model, which does not require manually labeled description sentences, but only uses random description sentences, thereby reducing the cost of manual labeling; (2) It balances the visual input information and the existing language knowledge in the large-scale language pre-training model to avoid generating words that are irrelevant to the video content; (3) It generates pseudo-labels to provide supervision information for the model, thereby eliminating the need to use methods with high computational overhead such as reinforcement learning, thereby reducing computational costs.

[0051] The present invention is applicable to video description tasks without manually labeled sentences, and has the following beneficial effects: (1) no manual labeling is required, which greatly reduces labor costs and greatly increases the possibility of applying the video description model in actual scenarios; (2) a constraint activation unit is used to achieve a balance between visual input information and language knowledge, thereby narrowing the semantic difference between generated sentences and video content; (3) a text feature dynamic optimization method is used to generate pseudo-labels, and the pseudo-labels are required to contain target category information in the video, so that the pseudo-label content matches the video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings.

[0054] like Figure 1 , an unsupervised video description method based on self-constrained dynamic text features, firstly, uniformly sample the video to extract its video appearance and motion features, target features and target category labels, and obtain the one-hot encoding vector of the random description sentence; then, input the video feature vector and the one-hot encoding vector of the random description sentence into the description sentence generation module to obtain the generated sentence feature matrix; then, input the generated sentence feature matrix into the pseudo-label generation module to obtain the pseudo-text feature matrix; replace the random description sentence feature matrix with the pseudo-text feature matrix, and re-input it into the description sentence generation module to obtain the final generated sentence feature matrix; and then generate the corresponding description sentence through a greedy search algorithm. The method proposed in the present invention utilizes a constrained activation unit to achieve a balance between visual input information and language knowledge, thereby generating a description sentence that conforms to the video content, and then uses a text feature dynamic optimization method to generate a pseudo-label text, providing supervision information for the model, which is conducive to narrowing the semantic gap between visual content and text. After obtaining a video data set, the method performs the following operations in sequence:

[0055] Step (1) uniformly samples the video to obtain a frame sequence, uses two-dimensional and three-dimensional convolutional networks to obtain appearance features and motion feature vectors respectively, and obtains target category labels and target feature vectors through a faster regional convolutional neural network; specifically:

[0056] (1-1) Uniformly sample N video frames to obtain a frame sequence set 3D tensor x i is the i-th frame image, C′, H′, W′ are the number of channels, height and width of the video frame, 3≤N≤10, is the real number domain; the set of one-hot encoding vectors of random description sentences is T is the length of the description sentence, 1≤T≤20, yt is the unique hot vector of the tth word in the vocabulary of the description sentence, n represents the number of words in the vocabulary, where the description sentence and the video are not in correspondence, that is, any random natural sentence can be used as the description sentence of the video;

[0057] (1-2) Using residual network to extract the appearance feature vector set of the video represents the appearance feature vector of the i-th frame, d a Represents the dimension size of the appearance feature vector, and the superscript and subscript a represent appearance; the Convolutional 3DNeural Network (C3D) is used to extract the motion feature vector set of the video represents the motion feature vector extracted from the i-1th frame to the i+1th frame, d m Represents the dimension of the motion feature vector, and the superscript and subscript m represent motion;

[0058] (1-3) For video frame x i , use Faster R-CNN (Fast Region-based Convolutional Neural Network) to detect targets and obtain the target feature vector set of the i-th frame and the object category labels contained in the entire video o i,j represents the feature vector of the jth target in the i-th frame, Q represents the number of set video frame targets, 1≤Q≤5, d o Indicates the dimension of the target feature vector, subscript o indicates the target (objective), c k Represents the category label vector of the kth target, K is the number of target categories, K ≥ 1, d c Indicates the dimension size of the target category, and the subscript c represents the category (class);

[0059] (1-4) Perform the maximum pooling operation on the target feature vector corresponding to each video frame to obtain the pooled target feature vector set of the video in Maxpool(·) represents the maximum pooling operation in the spatial dimension, which transforms the appearance feature vector Motion feature vector And the pooled target feature vector Splice them one by one to get a set of video feature vectors where v i Represented as the video feature vector of the i-th frame, the video feature vector dimension is dv =d a +d m +d o , the subscript v represents video.

[0060] Step (2) uses the constraint activation unit to construct a description sentence generation module. The constraint activation unit consists of a visual constraint gate and a language constraint gate. The input is the appearance feature, motion feature and target feature vector, and the output is the generated sentence feature matrix; specifically:

[0061] (2-1) Constructing a description sentence generation module: The description sentence generation module consists of the second-generation generative pre-training transformer GPT-2 (Generative Pre-training Transformer-2) and a constraint activation unit, where GPT-2 consists of L layers of transformers (Transformer), L ≥ 1, GPT-2 is a unidirectional language pre-training model for generative tasks, GPT-2 has been pre-trained on a large-scale dataset, and the initial parameters of the model are the pre-training results; the constraint activation unit consists of a visual constraint gate and a language constraint gate, which are used to balance visual information and language knowledge;

[0062] (2-2) The video feature vector v i and the one-hot encoding vector y of the random description sentence t , mapped to the same dimension size d through the fully connected layer, and the mapped video feature matrix is ​​obtained and the random description sentence feature matrix [·] T Represents mathematical transpose, matrix The i-th row matrix The tth row of and is the learnable parameter matrix, and the subscript y represents the sentence;

[0063] (2-3) Calculate the position encoding matrix of video features Postion(·) is the function for calculating the position encoding. The specific operation is: Represents the matrix PE v The value of the ωth column in the i-th row, mod means the remainder operation; similarly, the sentence position encoding matrix is ​​obtained The video position encoding matrix PE v The i-th row in and the video feature vector set The i-th vector in Add, sentence position encoding matrix PE y The tth row in and a set of random description sentence feature vectors The tth vector in Add them together to get the video feature matrix with position encoding and the random description sentence feature matrix matrix The i-th row matrix The tth row of It means element-by-element addition;

[0064] (2-4) Add the video feature matrix with position encoding and the random description sentence feature matrix In the attention submodule of the first layer of the converter in the one-way language pre-training model GPT-2, the second layer starts to convert the output matrix of the previous layer of converter Video feature matrix for alternative position encoding and the random description sentence feature matrix Input the attention submodule of the current layer converter in the unidirectional language pre-training model GPT-2; the attention submodule consists of a multi-head attention layer and a linear layer, and calculates the intermediate state matrix of each converter LayerNorm(·) represents the layer normalization operation, MultiHead(·) represents the multi-head attention operation;

[0065] Concat(·) represents the channel concatenation operation. is the learnable parameter matrix corresponding to the lth layer, is the h-th head attention vector, h=1,2,…,H, H represents the total number of attention heads, H≥1, Attention(·) represents the attention operation, in are the learnable parameter matrices corresponding to the lth layer and the hth head;

[0066] (2-5) The intermediate state matrix Input constraint activation unit to get vision-language output matrix The constraint activation unit consists of a visual constraint gate and a language constraint gate. Represents element-wise multiplication;

[0067] First, get the visual constraint gate matrix and the language constraint gate matrix The superscript vis stands for vision, and the superscript lan stands for language; the matrix B vis The value of the tth row and ωth column Matrix B lan The value of the tth row and ωth column σ(·) represents the Sigmoid activation function, the threshold is 0<τ<1, ζ(·) represents the discriminant function, if the statement in the function meets the condition, it returns 1, otherwise it returns 0;

[0068] Then the visual constraint gate matrix B vis Perform standardization operation to obtain the standardized visual constraint gate matrix matrix The value of the tth row and ωth column For the language constraint gate matrix B lan Perform standardization operations to obtain the standardized language constraint gate matrix matrix The value of the tth row and ωth column

[0069] (2-6) Output the visual-language matrix Input a feedforward network layer consisting of two fully connected layers to obtain the output matrix of the converter in GPT-2 Where FFN(·) represents the feedforward operation, max(·) represents the maximum value, are learnable parameter matrices and vectors;

[0070] (2-7) After all layer converters are processed, the final output matrix is ​​obtained Get the generated sentence feature matrix through this output matrix Y′=H L W′, where is the learnable parameter matrix.

[0071] Step (3) constructs a pseudo-label generation module, the input of which is the generated sentence feature matrix and the target category label, and obtains the pseudo-text feature matrix by dynamically updating the generated sentence feature matrix; specifically:

[0072] (3-1) The pseudo-label generation module consists of a robustly optimized BERT pre-training model RoBERTa (A Robustly Optimized BERT Pretraining Approach, BERT is a bidirectional encoder representation converter (Bidirectional Encoder Representation from Transformers)), a target attribute judgment submodule, a text segment selection submodule, and a text segment optimization submodule; RoBERTa consists of M layers of converters, M ≥ 1, RoBERTa is a bidirectional language pre-training model for text word prediction and modification, and RoBERTa has been pre-trained on a large-scale dataset, and the initial parameters of the model are the pre-training results; the target category judgment submodule consists of multiple linear layers, which is used to judge whether the previously obtained generated sentence contains the detected target category tag; the text segment selection submodule consists of multiple linear layers, which is used to dynamically select the text segment that needs to be modified in the generated sentence; the text segment optimization submodule consists of multiple linear layers, which is used to dynamically optimize the text segment obtained by the text segment selection submodule;

[0073] (3-2) An all-zero vector Splice to the first row of the generated sentence feature matrix Y′ to obtain a generated sentence feature matrix with target category attributes

[0074] (3-3) Generate sentence feature matrix with target category attributes In the attention submodule of the first layer of the input bidirectional language pre-training model, the second layer starts to convert the output matrix of the previous layer of the converter Alternative generation sentence feature matrix As input; the attention submodule consists of a multi-head attention layer, a linear layer, and a feedforward network layer to obtain the output matrix of the converter

[0075] is the intermediate state matrix of the mth layer, are learnable parameter matrices and vectors;

[0076] (3-4) After all layer converters are processed, the final output matrix is ​​obtained Take the first row of each layer output matrix to form the target prediction category feature vector set in Represents the first row vector of the output matrix of the mth layer;

[0077] (3-5) Set the target prediction category feature vector Input the target category judgment submodule to obtain the target prediction category probability vector set k-th target prediction category probability Where softmax(·) is a normalized exponential function, represents the first row vector of the output matrix of the kth layer, is the learnable parameter matrix;

[0078] (3-6) The target attribute probability a of the query set k and the true target category label vector c k , the target category loss is calculated by the cross entropy loss function

[0079] (3-7) Use the stochastic gradient descent method to update the bidirectional language pre-training model and obtain the back-propagation gradient matrix set of each layer Represents the gradient matrix of the mth layer;

[0080] (3-8) The back propagation gradient matrix of the first layer In the input text fragment selection submodule, Remove the first row of , and then calculate the bi-norm of the channel dimension to obtain the target attribute score set corresponding to each word The tth word ||·||2 indicates l2-norm calculation, is the gradient matrix of the first layer The tth row; determine the selected segment length to be υ, υ ≥ 1, calculate the segment score, and add the scores of the tth to t+υth words. Select the segment with the lowest score Indicates the position of the first word in the segment with the lowest score;

[0081] (3-9) by minimum segment score Select the first word position The first Replace the rows with all-one vectors to obtain the generated sentence feature matrix after replacement Then the matrix is ​​input into the bidirectional language pre-training model, and the operation is the same as (3-3) and (3-4), and finally the output matrix corresponding to the generated sentence after replacement of the K-th layer converter is obtained.

[0082] (3-10) Select the corresponding output matrix after replacement No. Rows, get part of the output matrix Will Input the text fragment optimization submodule to obtain the dynamically optimized text fragment feature matrix in is the learnable parameter matrix;

[0083] (3-11) will generate the sentence feature matrix Y′ Rows are replaced with text snippet feature matrix Finally, the pseudo text feature matrix is ​​obtained

[0084] Step (4) re-inputs the pseudo text feature matrix into the sentence generation module as a pseudo tag, and uses the stochastic gradient descent algorithm to optimize the description sentence generation module and the pseudo tag generation module; specifically:

[0085] (4-1) The pseudo text feature matrix Input description statement generation module, used to replace the random description statement feature matrix Execute steps (2-3) to (2-7) again to obtain the final generated sentence feature matrix

[0086] (4-2) Use the fully connected layer and softmax function to calculate the probability matrix of the final generated sentence in Represents the learnable parameter matrix in the first fully connected layer; then calculates the probability matrix of pseudo-labeling in represents the learnable parameter matrix in the second fully connected layer;

[0087] (4-3) Probability matrix for the final generated sentence Pseudo-labeled probability matrix Calculate the cross entropy loss between the two in is the probability matrix for generating sentences The tth row of is the probability matrix of pseudo-labeling The tth row of .

[0088] Step (5) obtains the probability distribution of the generated sentence for the new video through (1) to (4) in sequence, and obtains the corresponding description sentence using the greedy search algorithm; specifically:

[0089] (5-1) optimizing the video description model by minimizing the cross entropy loss function using a stochastic gradient descent method until convergence, wherein the video description model comprises a description sentence generation module and a pseudo-label generation module;

[0090] (5-2) Input the new video and uniformly sample N video frames to get At the same time, get the one-hot encoding vector of the new random description sentence First, go through steps (1) to (4) to get the new probability matrix of the final generated sentence For each row of this matrix, a greedy search algorithm is used to select the word with the highest probability corresponding index from the vocabulary as the generated word y. t ′, and finally obtain the description statement {y1′,...,y t ′,...,y′ T′}, is the tth word, and T′ is the length of the generated sentence.

[0091] The contents described in this embodiment are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as being limited to the specific forms described in the embodiment. The protection scope of the present invention also extends to the equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An unsupervised video description method based on self-constrained dynamic text features, characterized in that: First, obtain the video data set, and then perform the following operations: Step (1) uniformly sampling the video to obtain a frame sequence, using a two-dimensional and a three-dimensional convolutional network to obtain appearance features and motion feature vectors respectively, and obtaining a target category label and a target feature vector through a faster regional convolutional neural network; Step (2) constructing a description sentence generation module using a constraint activation unit, the constraint activation unit is composed of a visual constraint gate and a language constraint gate, the input is appearance features, motion features and target feature vectors, and the output is a generated sentence feature matrix; Step (3) constructing a pseudo-label generation module, the input of which is the generated sentence feature matrix and the target category label, and obtaining the pseudo-text feature matrix by dynamically updating the generated sentence feature matrix; Step (4) re-inputting the pseudo text feature matrix into the sentence generation module as a pseudo tag, and optimizing the description sentence generation module and the pseudo tag generation module using a stochastic gradient descent algorithm; Step (5) obtains the probability distribution of the generated sentence for the new video through (1) to (4) in sequence, and obtains the corresponding description sentence using the greedy search algorithm.

2. The unsupervised video description method based on self-constrained dynamic text features as claimed in claim 1, characterized in that: Step (1) specifically includes: (1-1) Uniformly sample N video frames to obtain a frame sequence set 3D tensor x i is the i-th frame image, C′, H′, W′ are the number of channels, height and width of the video frame, 3≤N≤10, is the real number domain; the set of one-hot encoding vectors of random description sentences is T is the length of the description sentence, 1≤T≤20, y t is the unique hot vector of the tth word in the vocabulary of the description sentence, n represents the number of words in the vocabulary, where the description sentence and the video are not in correspondence, that is, any random natural sentence can be used as the description sentence of the video; (1-2) Using residual network to extract the appearance feature vector set of the video represents the appearance feature vector of the i-th frame, d a Represents the dimension size of the appearance feature vector; uses a convolutional 3D network to extract the motion feature vector set of the video represents the motion feature vector extracted from the i-1th frame to the i+1th frame, d m Represents the dimension of the motion feature vector; (1-3) For video frame x i , use a faster regional convolutional neural network for target detection and obtain the target feature vector set of the i-th frame and the object category labels contained in the entire video o i,j represents the feature vector of the jth target in the i-th frame, Q represents the number of set video frame targets, 1≤Q≤5, d o Indicates the dimension size of the target feature vector, c k Represents the category label vector of the kth target, K is the number of target categories, K ≥ 1, d c Indicates the dimension size of the target category; (1-4) Perform the maximum pooling operation on the target feature vector corresponding to each video frame to obtain the pooled target feature vector set of the video in Maxpool(·) represents the maximum pooling operation in the spatial dimension, which transforms the appearance feature vector Motion feature vector And the pooled target feature vector Splice them one by one to get a set of video feature vectors where v i Represented as the video feature vector of the i-th frame, the video feature vector dimension is d v =d a +d m +d o .

3. The unsupervised video description method based on self-constrained dynamic text features as claimed in claim 2, characterized in that: Step (2) is specifically: (2-1) Constructing a description sentence generation module: The description sentence generation module consists of the second-generation generative pre-trained transformer GPT-2 and a constraint activation unit, where GPT-2 consists of L-layer transformers, L ≥ 1, and GPT-2 is a unidirectional language pre-training model for generative tasks. GPT-2 has been pre-trained on a large-scale dataset, and the initial parameters of the model are the pre-training results; the constraint activation unit consists of a visual constraint gate and a language constraint gate, which are used to balance visual information and language knowledge; (2-2) The video feature vector v i and the one-hot encoding vector y of the random description sentence t , mapped to the same dimension size d through the fully connected layer, and the mapped video feature matrix is ​​obtained and the random description sentence feature matrix [·] T Represents mathematical transpose, matrix The i-th row matrix The tth row of and is the learnable parameter matrix; (2-3) Calculate the position encoding matrix of video features Postion(·) is the function for calculating the position encoding. The specific operation is: 1≤ω≤d, Represents the matrix PE v The value of the ωth column in the i-th row, mod means the remainder operation; similarly, the sentence position encoding matrix is ​​obtained The video position encoding matrix PE v The i-th row in and the video feature vector set The i-th vector in Add, sentence position encoding matrix PE y The tth row in and a set of random description sentence feature vectors The tth vector in Add them together to get the video feature matrix with position encoding and the random description sentence feature matrix matrix The i-th row matrix The tth row of It means element-by-element addition; (2-4) Add the video feature matrix with position encoding and the random description sentence feature matrix In the attention submodule of the first layer of the converter in the one-way language pre-training model GPT-2, the second layer starts to convert the output matrix of the previous layer of converter Video feature matrix for alternative position encoding and the random description sentence feature matrix Input the attention submodule of the current layer converter in the one-way language pre-training model GPT-2; The attention submodule consists of a multi-head attention layer and a linear layer to calculate the intermediate state matrix of each converter LayerNorm(·) represents the layer normalization operation, MultiHead(·) represents the multi-head attention operation; Concat(·) represents the channel concatenation operation. is the learnable parameter matrix corresponding to the lth layer, is the h-th head attention vector, h=1,2,…,H, H represents the total number of attention heads, H≥1, Attention(·) represents the attention operation, in are the learnable parameter matrices corresponding to the lth layer and the hth head; (2-5) The intermediate state matrix Input constraint activation unit to get vision-language output matrix The constraint activation unit consists of a visual constraint gate and a language constraint gate. Represents element-wise multiplication; First, get the visual constraint gate matrix and the language constraint gate matrix Matrix B vis The value of the tth row and ωth column Matrix B lan The value of the tth row and ωth column σ(·) represents the Sigmoid activation function, the threshold is 0<τ<1, ζ(·) represents the discriminant function, if the statement in the function meets the condition, it returns 1, otherwise it returns 0; then the visual constraint gate matrix B vis Perform standardization operation to obtain the standardized visual constraint gate matrix matrix The value of the tth row and ωth column For the language constraint gate matrix B lan Perform standardization operations to obtain the standardized language constraint gate matrix matrix The value of the tth row and ωth column (2-6) Output the visual-language matrix Input a feedforward network layer consisting of two fully connected layers to obtain the output matrix of the converter in GPT-2 Where FFN(·) represents the feedforward operation, max(·) represents the maximum value, are learnable parameter matrices and vectors; (2-7) After all layer converters are processed, the final output matrix is ​​obtained Get the generated sentence feature matrix through this output matrix Y=H L W, where is the learnable parameter matrix.

4. The unsupervised video description method based on self-constrained dynamic text features as claimed in claim 3, characterized in that: Step (3) is specifically: (3-1) The pseudo-label generation module consists of a robustly optimized BERT pre-trained model RoBERTa, a target attribute judgment submodule, a text segment selection submodule, and a text segment optimization submodule; RoBERTa consists of M layers of transformers, M ≥ 1, and RoBERTa is a bidirectional language pre-training model for text word prediction and modification. At the same time, RoBERTa has been pre-trained on a large-scale dataset, and the initial parameters of the model are the pre-training results; The target category judgment submodule consists of multiple linear layers, which is used to judge whether the previously generated sentence contains the detected target category tag; the text segment selection submodule consists of multiple linear layers, which is used to dynamically select the text segment that needs to be modified in the generated sentence; the text segment optimization submodule consists of multiple linear layers, which is used to dynamically optimize the text segment obtained by the text segment selection submodule; (3-2) An all-zero vector Splice to the first row of the generated sentence feature matrix Y′ to obtain a generated sentence feature matrix with target category attributes (3-3) Generate sentence feature matrix with target category attributes In the attention submodule of the first layer of the input bidirectional language pre-training model, the second layer starts to convert the output matrix of the previous layer of the converter Alternative generation sentence feature matrix As input; The attention submodule consists of a multi-head attention layer, a linear layer, and a feedforward network layer to obtain the output matrix of the converter is the intermediate state matrix of the mth layer, are learnable parameter matrices and vectors; (3-4) After all layer converters are processed, the final output matrix is ​​obtained Take the first row of each layer output matrix to form the target prediction category feature vector set in Represents the first row vector of the output matrix of the mth layer; (3-5) Set the target prediction category feature vector Input the target category judgment submodule to obtain the target prediction category probability vector set k-th target prediction category probability Where softmax(·) is a normalized exponential function, represents the first row vector of the output matrix of the kth layer, is the learnable parameter matrix; (3-6) The target attribute probability a of the query set k and the true target category label vector c k , the target category loss is calculated by the cross entropy loss function (3-7) Use the stochastic gradient descent method to update the bidirectional language pre-training model and obtain the back-propagation gradient matrix set of each layer Represents the gradient matrix of the mth layer; (3-8) The back propagation gradient matrix of the first layer In the input text fragment selection submodule, Remove the first row of , and then calculate the bi-norm of the channel dimension to obtain the target attribute score set corresponding to each word The tth word ||·||2 indicates l2-norm calculation, is the gradient matrix of the first layer The tth row; determine the selected segment length to be υ, υ ≥ 1, calculate the segment score, and add the scores of the tth to t+υth words. Select the segment with the lowest score Indicates the position of the first word in the segment with the lowest score; (3-9) by minimum segment score Select the first word position The first Replace the rows with all-one vectors to obtain the generated sentence feature matrix after replacement Then the matrix is ​​input into the bidirectional language pre-training model, and the operation is the same as (3-3) and (3-4), and finally the output matrix corresponding to the generated sentence after replacement of the K-th layer converter is obtained. (3-10) Select the corresponding output matrix after replacement No. Rows, get part of the output matrix Will Input the text segment optimization submodule to obtain the dynamically optimized text segment feature matrix in is the learnable parameter matrix; (3-11) will generate the sentence feature matrix Y′ Rows are replaced with text snippet feature matrix Finally, the pseudo text feature matrix is ​​obtained 5. The unsupervised video description method based on self-constrained dynamic text features as claimed in claim 4, characterized in that: Step (4) is specifically: (4-1) The pseudo text feature matrix Input description statement generation module, used to replace the random description statement feature matrix Execute steps (2-3) to (2-7) again to obtain the final generated sentence feature matrix (4-2) Use the fully connected layer and softmax function to calculate the probability matrix of the final generated sentence in Represents the learnable parameter matrix in the first fully connected layer; then calculates the probability matrix of pseudo-labeling in represents the learnable parameter matrix in the second fully connected layer; (4-3) Probability matrix for the final generated sentence Pseudo-labeled probability matrix Calculate the cross entropy loss between the two in is the probability matrix for generating sentences The tth row of is the probability matrix of pseudo-labeling The tth row of .

6. The unsupervised video description method based on self-constrained dynamic text features as claimed in claim 5, characterized in that: Step (5) is specifically: (5-1) optimizing the video description model by minimizing the cross entropy loss function using a stochastic gradient descent method until convergence, wherein the video description model comprises a description sentence generation module and a pseudo-label generation module; (5-2) Input the new video and uniformly sample N video frames to get At the same time, get the one-hot encoding vector of the new random description sentence First, we go through steps (1) to (4) to get the probability matrix of the new final generated sentence. For each row of this matrix, a greedy search algorithm is used to select the word with the highest probability corresponding index from the vocabulary as the generated word y. t ′, and finally obtain the description statement {y1′,...,y t ′,...,y′ T′ }, is the tth word, and T′ is the length of the generated sentence.

Images