Video natural language text retrieval method based on space time sequence characteristics

Abstract

The invention relates to a video text retrieval method based on spatial time sequence characteristics, which comprises the following steps of: performing hierarchical fine-grained comprehensive video unified representation on spatial time sequence semantic information of a video by utilizing three different types of neural networks; and constructing a video text public semantic embedding network to fit a semantic gap of cross-modal data, and training the network by using a comparison sorting loss function. The method can be used for mutual retrieval of video natural language texts, the layered feature extraction method fully excavates more discriminative complex space-time semantic information of video modal data, and the video text public semantic embedding network effectively learns public space feature representation with different modal heterogeneous data semantic feature identical distribution. The semantic association between the high-order features of the video and the natural language text is accurately measured through public space feature representation, and the retrieval precision of the video natural language text is improved.

Description

technical field [0001] The invention belongs to the technical field of physics, and further relates to a video natural language text retrieval method based on spatial and temporal features in the technical field of image and data processing. The invention can be used for semantic information mutual retrieval of large-scale video modal and natural language text modal data emerging from the Internet and social media, video theme detection and content recommendation of video applications. Background technique [0002] The emergence of a large number of user-generated videos on the Internet has increased the demand for video retrieval systems based on natural language text descriptions, and users' requirements for retrieval accuracy have also brought unprecedented challenges to the precise retrieval of video content. Traditional methods mainly support concept-based retrieval for simple natural language text queries, which are ineffective for complex long natural language text qu...

AI Technical Summary

Problems solved by technology

[0007] The purpose of the present invention is to address the deficiencies in the above-mentioned prior art, and propose a video natural language text retrieval method based on spatial and temporal features, aiming at solving the difficulty of accurately modeling the complex temporal and spatial semantic features of video, and the semantic features of different modal data There is a problem that the heterogeneous underlying manifold structure distribution and different semantic gaps affect the accuracy of video natural language text retrieval

Images