An end-to-end video text generation method based on feature change

Abstract

An end-to-end video text generation method based on feature change comprises the following steps: S1, the original video is first frame extracted through preprocessing to form a frame sequence; S2, then the visual features are obtained through the encoder; S3, the visual features are subjected to the feature change operation of adding the model through the feature change module FVE to obtain the changed features; S4, the changed features are subjected to the decoder to obtain the finally generated text. The application introduces the feature change in the video text generation task to improve the model performance and generalization ability.

Description

TECHNICAL FIELD

[0001] The present application relates to a video text generation technology, and in particular to an end-to-end video text generation method based on feature changes. BACKGROUND

[0002] With the development of the Internet and network communication technology, more and more video data is generated. According to statistics from Cisco, 80% of the data on the Internet is video data. Compared with traditional text and images, video data is larger in size and more complex in content, and therefore more difficult to classify and search. In deep learning, methods such as Clip4clip [1] are often used to search videos across modalities, and video text generation requires a deeper understanding of videos, which means that better video features need to be extracted.

[0003] In the task of video text generation, an encoder-decoder structure is usually used. Video data is first processed by an encoder and then by a decoder to obtain the final generated text. Existing technical solutions focus on improving the performance of the encoder and the decoder. For the encoder, from the early 3D convolution [2] to the later Transformer [3] and Video Swin Transformer [4] based on general transformer-based encoder optimization, there has been a great improvement in obtaining better visual features. Similarly, for the decoder, from the early RNN [5] to the later BERT [6] large model. In addition to this, the optimization of video text generation or video understanding mainly focuses on how to better understand the temporal logic of the video. In this regard, three-dimensional convolution has been well tried, and the Transformer-based model naturally has the ability to model long sequences. For the understanding of video content, the general method is to fuse multi-modal features (including visual, text, audio, etc.) [7] , or use a target detector [8] to detect the video content and use it as guidance information for generating content. In addition, CLIP [9] such a model pre-trained on a large-scale image-text dataset [10-11] is used for video text generation. By leveraging the powerful generalization ability of CLIP, very good results can be achieved. SwinBERT

[12] attempts to solve the problem of video redundancy by using a dynamically learnable mask, and also achieves good results.

[0004] The traditional video text generation method has the following problems:

[0005] 1. None of them systematically study the core problem of video text generation task;

[0006] 2. The generalization performance is not enough;

[0007] 3. The value of ambiguity understanding for video is not considered, and the video content is not better understood.

[0008] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information which does not constitute prior art known to those of ordinary skill in the art.

[0009] References:

[0010] [1] LUO H, JI L, ZHONG M, et al. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning [J]. Neuro-computing, 2022, 508: 293-304.

[0011] [2] TRAN D, BOURLDEV L, FERGUS R, et al. Learning spatiotem-poral features with 3d convolutional networks [C] / / Proceedings of the IEEE international conference on computer vision. 2015: 4489-4497.

[0012] [3] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale [A]. 2020.

[0013] [4] LIU Z, LIN Y, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: for image captioning [A]. 2021. 10012-10022.

[0014] [5] VENUGOPALAN S, XU H, DONAHUE J, et al. Translating videos to natural language using deep recurrent neural networks [A]. 2014.

[0015] [6] DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding [A]. 2018.

[0016] [7] WANG T, ZHANG R, LU Z, et al. End-to-end dense video captioning with parallel decoding [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 6847-6857.

[0017] [8] MOTWANI T S, MOONEY R J. Improving video activity recognition using object recognition and text mining [M] / / ECAI 2012. IOS Press, 2012: 600-605.

[0018] [9] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision [C] / / International conference on machine learning. PMLR, 2021: 8748-8763.

[0019]

[10] MOKADY R, HERTZ A, BERMANO A H. Clipcap: Clip prefix for image captioning [A]. 2021.

[0020]

[11] TEWEL Y, SHALEV Y, SCHWARTZ I, et al. Zero-shot image-to-text generation for visual-semantic arithmetic [A]. 2021.

[0021]

[12] LIN K, LI L, LIN C C, et al. Swinbert: End-to-end transformers with sparse attention for video captioning [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 17949- 17958.

[0022]

[13] LI X, YIN X, LI C, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks [C] / / ECCV. 2020.

[0023]

[14] HU X, YIN X, LIN K, et al. Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training [C] / / AAAI. 2021.

[0024]

[15] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.

[0025]

[16] CHEN Y, WANG S, ZHANG W, et al. Less is more: Picking informative frames for video captioning[C] / / ECCV. 2018.

[0026]

[17] ZHANG J, PENG Y. Object-aware aggregation with bidirectional temporal graph for video captioning[C] / / CVPR. 2019.

[0027]

[18] AAFAQ N, AKHTAR N, LIU W, et al. Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning[C] / / CVPR. 2019.

[0028]

[19] CHEN S, JIANG Y G. Motion guided spatial attention for video captioning[C] / / AAAI. 2019.

[0029]

[20] WANG B, MA L, ZHANG W, et al. Controllable video captioning with possequence guidance based on gated fusion network[C] / / ICCV. 2019.

[0030]

[21] HOU J, WU X, ZHAO W, et al. Joint syntax representation learning and visual cue translation for video captioning[C] / / ICCV. 2019.

[0031]

[22] ZHENG Q, WANG C, TAO D. Syntax-aware action targeting for video captioning [C] / / CVPR. 2020.

[0032]

[23] PAN B, CAI H, HUANG D A, et al. Spatio-temporal graph for video captioning with knowledge distillation [C] / / CVPR. 2020.

[0033]

[24] CHEN S, JIANG W, LIU W, et al. Learning modality interaction for temporal sentence localization and event captioning in videos [C] / / ECCV. 2020.

[0034]

[25] ZHANG Z, SHI Y, YUAN C, et al. Object relational graph with teacher-recommended learning for video captioning [C] / / CVPR. 2020.

[0035]

[26] ZHANG Z, QI Z, YUAN C, et al. Open-book video captioning with retrieve-copy-generate network [C] / / CVPR. 2021.

[0036]

[27] Liu S, Ren Z, Yuan J. Sibnet: Sibling convolutional encoder for video captioning [C] / / Proceedings of the 26th ACM international conference on Multimedia. 2018: 1425-1434. SUMMARY

[0037] The main purpose of the present application is to overcome the defects of the background art, and provide an end-to-end video text generation method based on feature change.

[0038] To achieve the above object, the application adopts the following technical solutions:

[0039] An end-to-end video text generation method based on feature change, comprising the following steps:

[0040] S1, the original video is first frame extracted by preprocessing to form a frame sequence;

[0041] S2, then get the visual features through the encoder;

[0042] S3, the visual features are operated by the feature change module FVE to increase the model feature change, and the changed features are obtained;

[0043] S4, the changed features are obtained through the decoder to get the final generated text.

[0044] Further:

[0045] In step S1, the frame extraction method is average sampling, and the frame number is fixed to 32 frames to form a frame sequence.

[0046] In step S2, the encoder uses Video Swin Transformer coding.

[0047] In step S3, the feature change module FVE is composed of one layer of Transformer, has multiple self-attention heads, each attention head has the same QKV structure, respectively pays attention to part of the input feature dimension, and finally summarizes; the input feature first passes through the first forward propagation network FF, then enters the first branch and the second branch respectively, the feature output by the first forward propagation network FF in the first branch passes through a QKV module and a forward propagation network FF in turn, and is added to the input feature, and finally the mean vector is obtained; the feature output by the first forward propagation network FF in the second branch passes through a QKV module and a forward propagation network FF in turn, and finally the variance vector is obtained; the changed visual features are obtained by using the mean vector and the variance vector; wherein, the QKV module is a module used to realize self-attention function in Transformer, which includes Q (Query), K (Key) and V (Value) three matrices.

[0048] The changed visual features are obtained by formula (1)

[0049]

[0050] Wherein, μ is the mean vector after adding processing, σ is the variance vector, and ∈ is the random noise sampled from the standard Gaussian.

[0051] The training process uses a loss function that sets a lower limit for the variance vector according to the dimension level, and the value of the variance is not less than the lower limit, which ensures that the variance of the FVE module in the model during the training process will not collapse to zero.

[0052] The decoder adopts BERT decoding; preferably, in the decoding process of the decoder, the 12-layer BERT is reduced to 6 layers, and a label smoothing method is used.

[0053] The video text generation uses a Beam Search method.

[0054] In the process of text generation, an autoregressive seq2seq generation method is used, and each newly generated word is related to the entire sentence generated before.

[0055] A computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the end-to-end video text generation method based on feature change.

[0056] The embodiments of the application have the following beneficial effects:

[0057] The embodiments of the application propose an end-to-end video text generation method based on feature change, which increases the change of visual features. This method adds a feature change module to the two-stage video text generation model, increases the feature change of the model, and finally obtains a video text generation model with better performance and stronger generalization ability through the methods of label smoothing and reducing the decoder. The application first proposes to introduce feature change in the video text generation task to improve the model performance and generalization ability.

[0058] Specific advantages include:

[0059] 1. For the video text generation task, the introduction of visual feature change improves the model's understanding of visual features and improves the model's generalization ability;

[0060] 2. For the video text generation task, the existing model structure is optimized, and the important role of label smoothing in the video text task is introduced;

[0061] 3. Through visual feature change, the learning ability of the model is further improved, the model can benefit from a larger data set, and the Zero-shoting ability of the model is improved. At the same time, the technical method is plug and play, and is very good deployment.

[0062] The end-to-end video text generation method based on feature change proposed in the application can improve the understanding ability of the model to video data, greatly improve the ability of the model to generate video text, and indicate a valuable research direction for subsequent research. The specific performance of the model can be referred to Table 1.

[0063] In experimental use, the current method can stably run in an advertising service and has good generalization ability when using CLIP as an encoder, and can achieve better performance than existing technical solutions in zero-shoting. Therefore, it can be said that the application has very broad prospects in practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The overall flowchart of the end-to-end video text generation method based on feature change of an embodiment of the application is shown.

[0065] Figure 2 The detailed structure diagram of the attention head of the FVE module in an embodiment of the application is shown.

[0066] Figure 3 The multi-head attention in an embodiment of the application is shown. DETAILED DESCRIPTION

[0067] The embodiments of the application are described in detail below. It should be emphasized that the following description is merely exemplary, not intended to limit the scope of the application and its applications.

[0068] It should be noted that when an element is referred to as being "fixed to" or "set to" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be for fixing or for coupling or communicating.

[0069] It should be understood that the terms "length", "width", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the application.

[0070] In addition, the terms "first", "second", etc. are used only for descriptive purposes and do not connote or imply relative importance or an ordering between or among the indicated technical features. Thus, a feature defined with "first", "second", etc. can include one or more of the features implicitly or explicitly. In the description of embodiments of the present application, the meaning of "a plurality" is two or more, unless otherwise expressly specified.

[0071] The current technical solution optimizes and solves the video text generation problem from the encoder, the decoder and the video itself, but still ignores the core of video text generation, that is, the existing solution is to regard video text generation as a subtask of a certain large task (such as cross-modal retrieval or even a new encoder), and ignores the fact that video text generation is essentially a very complex dimension reduction task. This dimension reduction task is different from the cross-modal retrieval task, and has higher requirements for the quality of the visual features after dimension reduction, requiring the visual features to have better ability to summarize the original video and better inclusive ability, because generally speaking, for the same video, different titles can often make the video content represent completely opposite content, so for the same video, the visual features obtained from this video should tolerate more text expressions, that is, tolerate expressions. Generally, the output text is faithful to the input visual features, so through the change of the visual features in a small range, more diverse text output can be obtained. This small range of changes can improve the generalization ability of the model on the one hand, and on the other hand, it can also improve the diversity of the text output by the model. And most fundamentally, it allows the generated text to represent a class of videos rather than just one video, enhancing the understanding ability of the video (which is also in line with common sense. It is very unreasonable to forcibly distinguish the expression of many video contents because of the different texts matched)

[0072] Referring to Figure 1 The embodiment of the present application provides an end-to-end video text generation method based on feature change, comprising the following steps:

[0073] S1, the original video is first frame extracted by pre-processing to form a frame sequence;

[0074] S2, then obtain the visual features through the encoder;

[0075] S3, the feature change module FVE is used to perform the operation of increasing the feature change (similar to reparameterization) of the model on the visual features to obtain the changed features;

[0076] S4, the changed features are input into the decoder to obtain the final generated text.

[0077] The specific embodiments of the present application are described below.

[0078] In some embodiments of the method, the original video is first pre-processed by frame extraction, the frame extraction method is average sampling, the number of frames is fixed at 32 frames, a frame sequence is formed, and then the encoder Video Swin Transformer [4] , the visual features are obtained by the encoder, then the visual features are subjected to similar reparameterization operation by the FVE module of the application, the changed features are obtained, and then the features are subjected to decoding by the decoder BERT [6] , to obtain the finally generated text.

[0079] The encoder is used in the same way as the encoder of the prior art, and in the decoding process of the decoder, compared with the prior art, the method reduces the size of the decoder (reduces the BERT from 12 layers to 6 layers), and uses label smoothing, because label smoothing can improve the effect of the Beam Search method used in video text generation, and is more conducive to the FVE module to play a role. In the process of text generation, the autoregressive seq2seq generation method is adopted, and each newly generated word is related to the entire sentence generated before, for specific techniques, please refer to [13-14] .

[0080] Now the FVE module will be described in detail. The detailed structure of FVE is as follows: Figure 2 .

[0081] Figure 2 The detailed structure of FVE: FF in the figure is feedforward network, i.e. forward propagation network, QKV is a module used to realize self-attention function in transformer

[15] , including Q (Query), K (Key), and V (Value) three matrices, which are the core modules of the transformer model to realize self-attention mechanism.

[0082] FVE is composed of one layer of transformer

[15] , and the structure is as follows: Figure 2 For the input visual feature X, let its shape be (B, C, D), where B is the number of current batch data (Batch), C is the number of channels of the visual feature at this time, and D is the number of dimensions of the feature. After passing through the FF network, the shape obtained is (3, B, H, C, D / H), where 3 represents that Q, K, and V three matrices are to be generated, H represents the number of attention heads, and multi-head attention is as follows: Figure 3As shown, each attention head has the same QKV structure, respectively paying attention to the input feature part dimension, and finally summarizing, in this paper, each attention head pays attention to the visual feature of the D / H size area, and the value of H is an adjustable parameter, we follow the setting of the Transformer, and set H to 8. For the 8 attention heads, we set the generation target of four of them to the mean vector, and the other four to the variance vector, so when performing self-attention calculation, the results of the last multiple attention heads are summarized into two parts instead of one part before, at this time the shape is (B, C, 2, D / 2), at this time the mean and variance shapes are (B, C, 1, D / 2), both of which are finally obtained by FF (B, C, D) which is consistent with the size of the input visual feature, and considering the decisive role of the mean, the input visual feature introduced by the model shortcut module is added to the mean branch. Finally, the changed visual feature X^ is obtained through formula (1).

[0083]

[0084] In formula (1), μ is the mean vector after addition processing, σ is the variance vector, and ∈ is the random noise sampled from the standard Gaussian. In order to prevent the collapse of the model's change vector (including the mean vector and the variance vector) during the training process, the present application also designs a loss function, which sets a lower limit for the variance vector according to the dimension level, and the value of the variance cannot be less than this lower limit, as shown in formulas (2) and (3).

[0085]

[0086] Loss=max(thres-h(N(μ,σ 2 )),0), (3)

[0087] Where D is the feature dimension, thres is the threshold, set to 1 / D, h(N(μ,σ 2 )) represents the cross entropy of the current Gaussian distribution, N(μ,σ 2 ) represents the Gaussian distribution with mean μ and variance σ 2 represents the sum of the current mean according to the dimension.

[0088] In the preferred embodiment, the FVE module is further optimized by label smoothing and reducing the encoder.

[0089] ​In the classification problem, label smoothing can ensure that the negative samples have a certain distance while ensuring that the model's prediction results are as close as possible. In the video text generation task of an embodiment, the algorithm for generating text is Beam Search. Beam Search is a greedy algorithm that reduces the size of the search space by retaining a certain number of candidate solutions, while continuing to explore possible better solutions while retaining these candidate solutions. Specifically, the Beam Search algorithm sorts all feasible solutions in the search space according to a certain evaluation index, and retains the top k best solutions (k is an algorithm parameter, usually small, such as 5 or 10). Then, it takes these retained solutions as candidate solutions, continues to search for all possible solutions in the next layer, and again retains the top k best solutions as candidate solutions for the next round. This is repeated continuously until a certain termination condition is reached (such as finding the optimal solution or reaching the maximum search depth). In this embodiment, label smoothing can separate samples more widely, so it can have a greater gain for the video text generation task. At the same time, the FVE-based method causes the features of the video to fluctuate in a subspace of high-dimensional space, so label smoothing can have a more obvious gain.

[0090] The decoder used by the model is BERT. BERT is a model for processing natural language, and the BERT in this embodiment is trained directly from the initial state. Generally, text data is naturally refined, while for the video generation process, the decoder needs to parse visual features extracted from the video, and video features are naturally redundant. It is undoubtedly very difficult to train existing models using video features. At the same time, BERT itself is very sparse, even when pre-trained on a large corpus, it is also not fully trained. In this embodiment, BERT is reduced from the standard 12 layers to 6 layers. For a three-layer BERT, the performance is similar to that of a six-layer BERT, and a smaller decoder may have a problem of insufficient fitting ability.

[0091] Table 1 Comparison with prior art methods on the MSRVTT dataset

[0092]

[0093] Advantages of the present application compared to prior art:

[0094] 1. The method is plug and play, very convenient to deploy in models based on encoder-decoder structure;

[0095] 2. The method has better generalization performance and can also generate more diverse samples;

[0096] 3. The method truly tries to solve the problems encountered in video text generation from the nature, and the prior art scheme often considers video text generation as a subtask of a large task.

[0097] The specific application scenarios of the application are:

[0098] 1. It can be used in the content generation of conventional videos as a brief introduction of the video, facilitating the management of large-scale video data;

[0099] 2. It is used for video creation, by translating small video clips into text, the text forms a story, and the matched video can also form a long video, forming video creation;

[0100] 3. In academia, it can generate expressions of expressionless video data and clean up the labels of labeled videos.

[0101] The application can train a model with stronger generalization ability through a larger data set, and then use the plug-and-play characteristics to apply it in the existing model on a large scale.

[0102] Those skilled in the art will understand that embodiments of the application can be provided as methods, systems, or computer program products. Therefore, the application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0103] The application is described with reference to flowcharts and / or block diagrams according to the method, device (system), and computer program product of the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one block or multiple blocks.

[0104] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1the function specified in the one or more blocks.

[0105] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable devices provide a process for implementing the flow Figure 1 the flow or flows and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0106] The background section of this patent document can contain background information pertaining to the problems or environments of the present application, and is not necessarily an admission of prior art by the inventor or assignee.

[0107] The above further describes the present application in conjunction with specific / preferred embodiments, and cannot be deemed as limiting the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, they can make several substitutions or modifications to the described embodiments, and these substitutions or modifications shall be deemed as falling within the protection scope of the present application. In the description of the present application, the description of the terms "an embodiment", "some embodiments", "preferred embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. Those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples, without contradiction. Although the embodiments of the present application and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made in the present application without departing from the scope of the patent application.

Claims

1. An end-to-end video text generation method based on feature changes, characterized in that, Includes the following steps: S1. The original video is first preprocessed to extract frames, forming a frame sequence; S2, and then obtain the visual features through the encoder; S3. The visual features are augmented using the Feature Transformation Module (FVE) to obtain the transformed features. The FVE is a single-layer Transformer with multiple self-attention heads, each with the same QKV structure. Each attention head focuses on a dimension of the input features and then sums them up. The input features first pass through the first forward propagation network (FF), then enter the first and second branches. The features output by the first forward propagation network (FF) pass through a QKV module and another forward propagation network (FF) in the first branch, and are then added to the input features to obtain the mean vector. The features output by the first forward propagation network (FF) pass through a QKV module and another forward propagation network (FF) in the second branch to obtain the variance vector. The transformed visual features are obtained using the mean vector and the variance vector. The QKV module is a module in the Transformer used to implement self-attention, and it includes three matrices: Q (Query), K (Key), and V (Value). S4. The modified features are decoded to obtain the final generated text.

2. The end-to-end video text generation method based on feature changes as described in claim 1, characterized in that, In step S1, the frame extraction method is average sampling, and the number of frames is fixed at 32, forming a frame sequence.

3. The end-to-end video text generation method based on feature changes as described in claim 1 or 2, characterized in that, In step S2, the encoder uses Video Swin Transformer encoding.

4. The end-to-end video text generation method based on feature changes as described in claim 1 or 2, characterized in that, The altered visual features are obtained using formula (1). Where μ is the mean vector after addition, σ is the variance vector, and ∈ is the random noise sampled from the standard Gaussian.

5. The end-to-end video text generation method based on feature changes as described in claim 1 or 2, characterized in that, The training process uses a loss function that sets a lower bound for the variance vector according to the dimension level. The variance value is not less than this lower bound so that the variance of the FVE module will not collapse to zero during training.

6. The end-to-end video text generation method based on feature changes as described in any one of claims 1 to 2, characterized in that, The decoder uses BERT decoding.

7. The end-to-end video text generation method based on feature changes as described in claim 6, characterized in that, During the decoding process of the decoder, the 12-layer BERT was reduced to 6 layers, and the label smoothing method was used.

8. The end-to-end video text generation method based on feature changes as described in any one of claims 1 to 2, characterized in that, Video text generation uses the Beam Search method.

9. The end-to-end video text generation method based on feature changes as described in any one of claims 1 to 2, characterized in that, During the text generation process, an autoregressive seq2seq generation method is used, where each newly generated word is related to the entire sentence generated previously.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the end-to-end video text generation method based on feature changes as described in any one of claims 1 to 9.

Images