Method and device of training a captioning model, computer equipment and storage medium

Abstract

A method of training a captioning model used to perform automatic video captioning of an input video, including initializing a plurality of long short-term memory (LSTM) units included in the captioning model using cross-entropy loss; training the LSTM units using reinforcement learning; training the LSTM units and a plurality of convolutional neural networks (CNNs) included in the captioning model using multitask training; and generating a video caption corresponding to the input video using the captioning model.

Description

[0001] priority information [0002] This application claims priority to U.S. Application No. 16 / 396,924, entitled "End-to-End Video Captioning with Multi-Task Reinforcement Learning," filed April 29, 2019, the entire contents of which are incorporated by reference In this application. technical field [0003] This application relates to video subtitle technology. Specifically, the present application relates to a method and device for training a subtitle model, a computer device and a storage medium. Background technique [0004] Video subtitles are crucial for many downstream applications such as video retrieval, indexing, browsing, etc. Existing video captioning methods are trained component by component, and the quality of the overall system is affected by the performance of each individual component. [0005] End-to-end (E2E) training of related techniques is often hampered by hardware constraints (e.g., graphics processing unit (GPU) memory) and is prone to overfit...

AI Technical Summary

Problems solved by technology

[0006] Embodiments of the present application provide a method and device for training subtitle models, computer equipment and storage media, aiming to solve the problem that existing subtitle model training methods consume both memory and data, and the training is difficult and the training quality is not high The problem

Images