A Real-time Semantic Video Segmentation Method

Abstract

The invention discloses a real-time semantic video segmentation method, which is used for greatly accelerating the video semantic segmentation algorithm. Specifically, it includes the following steps: 1) Obtain multiple sets of data sets for training semantic segmentation, and define algorithm goals; 2) Train a lightweight CNN model for image semantic segmentation; 3) Decode the original video to obtain a residual image, Motion vector and RGB image; 4) If the current frame is an I frame, send it to the segmentation model obtained in 2) to obtain a complete segmentation result; 5) If the current frame is a P frame, use the motion vector to The segmentation result of one frame is passed to the current one, and the sub-block of the current frame is selected for correction using the residual image; 6) Repeat steps 3)~5) until all video frames are segmented. The invention makes full use of the correlation of adjacent frames in the video, and the accelerated processing based on the compressed domain information can quickly complete complex segmentation tasks while maintaining a high accuracy rate, and the efficiency is more than ten times higher than that of common segmentation methods.

Description

technical field [0001] The invention belongs to the field of computer vision, in particular to a real-time semantic video segmentation method. Background technique [0002] Semantic video segmentation is a computer vision task that assigns each pixel of each frame of a video a semantic category. Real-time semantic video segmentation requires certain requirements on the segmentation speed, generally above 24 frames per second. The current advanced semantic video segmentation methods are all based on convolutional neural network (CNN) machine learning methods, which can be roughly divided into two categories based on continuous image frames and directly based on video. The first type of method regards video as a sequence of image frames. They sacrifice a little segmentation accuracy in exchange for real-time semantic segmentation performance by reducing the scale of the input data or cutting the network. Such methods do not exploit the inter-frame coherence inherent in video...

AI Technical Summary

Problems solved by technology

The first type of method regards video as a sequence of image frames. They sacrifice a little segmentation accuracy in exchange for real-time semantic segmentation performance by reducing the scale of the input data or cutting the network.

Such methods do not exploit the inter-frame coherence inherent in videos

The second type of method extracts coherent features between frames through optical flow, 3DCNN, RNN and other technologies on the video, but these technologies themselves are time-consuming, and they themselves will become the bottleneck of semantic video segmentation

However, the inter-frame coherent information provided by compressed video is noisy compared with techniques such as optical flow. How to use compressed information while ensuring accurate segmentation has become a key problem to be solved by this method.

Images