Cross-modal video time sequence action positioning method and system based on time sequence-space diagram

Abstract

The invention belongs to the technical field of data representation, and provides a cross-modal video time sequence action positioning method and system based on a time sequence-space diagram, and the method comprises the steps: receiving video data and natural language query information; determining a natural language query feature representation based on the natural language query information; determining candidate video time sequence action segment feature representations based on the video data; on the basis of the candidate video time sequence action segment feature representations and the natural language query feature representation, predicting the time sequence offset of the candidate video time sequence action segments and the correlation between the candidate video time sequence action segments and natural language query information; and performing offset correction on the candidate video time sequence action segment with the highest correlation score to obtain a final target video time sequence action segment positioning result. According to the method and the system, the time sequence offset and the correlation score of the corresponding candidate video time sequence action segment are predicted through the two feature representations, and the accuracy of video time sequence action positioning is greatly improved.

Description

technical field [0001] The invention belongs to the technical field of data representation, and in particular relates to a time-series action positioning method and system for cross-modal video based on a time-spatial diagram. Background technique [0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art. [0003] The rapid development of Internet technology and the increasing popularity of image acquisition equipment have led to an exponential increase in the number of videos and have become a mainstream media form in today's society. Faced with such large-scale video data, video timing action location has become a hot research issue in the field of video analysis, which aims to locate the start and end moments of all actions from a given video, and at the same time classify these actions predict. However, current video temporal action localization methods can only detect and ...

AI Technical Summary

Problems solved by technology

Most of the existing cross-modal video temporal action location work uses sliding windows, multi-scale sampling of anchor points, etc. to generate candidate sets of temporal action segments, resulting in a single length of candidate temporal action segments and low coverage with target temporal action segments. The accuracy of the final positioning result is low

In addition, most existing methods use global representations (such as C3D or I3D) when representing video clips, ignoring the interaction information of objects within or between frames, resulting in insufficient understanding of the video content, which in turn affects the target. Localization accuracy of temporal action clips

Images