A text-to-video object segmentation method based on memory information transmission

Abstract

A text-to-video object segmentation method based on memory information transmission is suitable for the cross field of computer vision and natural language processing. The method takes the previous frame as memory information and applies the memory information to the object inference of the current frame. In order to propagate the memory information, a pseudo mask of the current frame is generated using the memory information to guide the inference of the current frame. In this way, the connection between frames can be established and the temporal continuity can be supplemented without introducing additional confusing information. In order to obtain accurate segmentation results under the input of single-frame visual information, spatial modeling needs to be completed while time modeling is completed. The method combines the propagation of memory information with text-visual feature interaction, filters out visual features related to language through text-guided feature interaction, and completes spatial modeling. By introducing memory information, the method supplements the lack of temporal continuity of the model, enhances the connection between instances of reference between frames, and improves the segmentation performance of the model.

Images