Video scene detecting and labeling method and system

Abstract

The invention discloses a video scene detection labeling method and system, and the method comprises the steps: obtaining the modal features of a video, an audio and a text through a pre-training model according to a modal information source embedded by the input video, the audio and the text, carrying out the alignment and fusion of the obtained modal features of the video, the audio and the text, and forming a window basic cross-modal representation, according to the multi-temporal attention and the difference between the adjacent windows, the basic cross-modal representation of the windows is evolved into self-adaptive context sensing representation, the scene is detected according to the obtained self-adaptive context sensing representation, and the attributes of the windows are determined through a window attribute classifier; obtaining an accurate position of a scene boundary in the window through a position offset regression device; and based on the obtained scene boundaries, specifying a plurality of labels for each scene to realize scene labeling, attributing scene detection into window attribute classification and position offset regression, and solving the multi-label labeling problem through integrated learning of two-stage classifiers. The problems of error propagation and huge calculation cost are solved through a unified network of cross-modal clues; scene detection is attributed to window attribute classification and position offset regression, and the multi-label labeling problem is solved through ensemble learning of two-stage classifiers.

Description

technical field [0001] The invention belongs to the field of video processing, and in particular relates to a video scene detection and labeling method and system. The invention is based on the video scene detection and labeling of a multi-modal adaptive context network. Background technique [0002] With the rapid development of 5G technology, video advertising has seen a huge growth in short video applications. In the creative, distribution, and strategy of the advertising ecosystem, a deep understanding of its content is becoming more and more important and demanding. As a key step in the semantic understanding of video advertisements, scene detection and annotation aims to temporally parse a video into different scenes and predict each scene to be annotated in different dimensions, such as presentation form, style, and location. Various potential applications have emerged, including video ad insertion, video summarization, video indexing and retrieval, etc. [0003] In...

AI Technical Summary

Problems solved by technology

But this will face the following three problems: 1) error propagation

2) Huge amount of calculation

If the scene detection is directly based on the frame features, it will consume a lot of computation during the training process

3) Poor generalization ability

Matrix inference requires a fixed length in the time dimension, which is not suitable for annotating variable-length durations of video scenes

Images