A method for high-resolution video virtual character matting based on deep spatiotemporal learning

Abstract

The present invention discloses a method for matting virtual characters in high-resolution videos based on deep spatiotemporal learning. The method comprises: inputting video data into a trained matting network model to extract a virtual character image mask; obtaining first output data based on the video data through a spatiotemporal encoding and decoding model; obtaining second output data based on the first output data through a detail extraction model; and outputting a virtual character mask image of the video data through a semantic information fusion model based on the second output data; wherein the first output data includes a high-resolution mask image, a low-resolution mask image, high-resolution coding features, and low-resolution coding features; and the second output data includes a high-resolution detail mask image and a low-resolution detail mask image. The present invention addresses the problems of existing methods, such as poor performance in extracting virtual characters from videos, difficulty in processing minute details, inability to achieve real-time processing when processing high-resolution videos, and the occurrence of flickering artifacts and bad frames.

Description

Technical Field

[0001] The present invention relates to the technical field of human portrait extraction, and in particular to a method for matting virtual characters in high-resolution videos based on deep spatiotemporal learning. Background Art

[0002] With intelligent short-form video becoming a major focus of information broadcasting on the contemporary mobile internet, real-time synthesized video images using virtual augmented reality technology have become the result of a fusion of multiple elements, including 3D virtual characters, virtual pre-rendered stages, real people, and real backgrounds. Separating the virtual characters from the background in high-resolution synthesized images to create clear, pure portraits can be used directly as fusion material for multiple videos, avoiding the waste of computing power caused by repeatedly rendering the same 3D model.

[0003] Existing portrait extraction methods can be divided into two categories:

[0004] Manually extracting people from images: Existing photo editing software, such as Adobe Photoshop, allows users to use the lasso tool to draw a rough outline. The software then performs edge matching to extract the person in the photo, allowing for further operations such as background replacement. However, this method is difficult to apply to videos.

[0005] Automatic extraction of portraits in videos: Traditional methods often use edge filtering operators or pixel distribution priors to determine portrait edges, but this is time-consuming and has low accuracy for high-resolution images. Machine learning-based methods often regard portrait extraction as a subtask of instance segmentation, and perform iterative training using datasets containing a large number of portraits and portrait masks. However, research has shown that existing portrait extraction models are basically designed for real people, with few designed for virtual characters. In addition, the computational efficiency of processing high-resolution videos is relatively low, and frame-by-frame processing of portraits in videos can lead to incoherent estimation of cutouts, causing flickering and artifacts, resulting in a poor viewing experience.

[0006] Existing portrait extraction algorithms can be divided into traditional methods and deep learning-based methods: Traditional cutout methods can be divided into manual cutout and computer traditional cutout. Figure 1 Generally, the contour is extracted based on manual drawing; the traditional computer Figure 1Typically, a Bayesian prior is used for image cutouts, assuming that pixels surrounding a pixel and those of similar colors have similar masks. This infers the probability distribution of background pixels, allowing each pixel to be individually identified as belonging to the background, ultimately resulting in a complete cutout of the portrait. Deep learning-based methods modify existing instance segmentation network models, using convolutional neural networks or recurrent neural networks, and auxiliary mask images or trimaps to aid recognition, thereby predicting the portrait's outline. Mean square error and mean absolute distance are commonly used to evaluate image cutout performance.

[0007] The existing method is a multi-channel convolutional neural network portrait extraction model based on the instance segmentation model. The model steps are as follows:

[0008] The input is a combination of an RGB color image or video and a trimap, which is preprocessed by convolution, transformer pre-weighting, or standard normalization. The resulting image (video) is fed into a convolutional neural network module, which extracts feature maps and ROIs of different scales in multiple layers, and outputs candidate regression boxes where the portrait is located. The multi-scale feature maps are feature-encoded using a spatiotemporal attention mechanism, a channel attention mechanism, or a multi-layer perceptron to obtain the final encoded output. The encoded result of the multi-channel convolutional network module is fed into a decoder, which decodes the encoded output to obtain the extracted portrait output, and locates the decoded output based on the candidate regression box. Finally, the pixel loss between the decoded output and the true mask is calculated, and the gradient backpropagation of the loss function is performed to iteratively optimize the network model parameters.

[0009] Existing portrait cutout technologies are all for real people and cannot effectively cut out virtual characters in synthetic videos. In terms of image portrait extraction algorithms:

[0010] Traditional image cutout methods: Methods based on manually drawn bounding boxes are highly accurate, but extremely inefficient and cannot be applied to videos. Edge detection-based methods, such as green screen video cutouts, and Bayesian prior estimation-based discriminant methods have limited applicability, are not good for extracting characters from portraits in complex background environments, and have low accuracy.

[0011] Methods based on deep learning: Methods based on convolutional neural networks can extract the contours of human figures in videos. Through sufficient training on large data sets, the accuracy of the matting can be ensured. However, the more complex the convolutional network, the lower the operating efficiency. It is difficult to simultaneously meet the requirements of complex convolutional structures and lightweight real-time operation.

[0012] Video portrait extraction:

[0013] Method based on a modified instance segmentation model: Existing instance segmentation models improve processing efficiency on high-resolution videos through image processing methods such as mosaic. By adding mechanisms such as channel attention, spatial attention, and domain attention, or using transformers instead of convolutional layers, the effect of the codec can be effectively improved. However, frame-by-frame extraction of videos will result in a lack of coherence between frames, and the pixel information association between different frames is not utilized, resulting in the appearance of bad frames such as flickering artifacts.

[0014] Methods based on recurrent neural networks: Recurrent neural networks learn information associations at different moments in a temporal sequence by introducing repeated chain-connected modules into the neural network. Incorporating time-varying, effective features such as optical flow tracking during keying can improve keying accuracy while ensuring the continuity of portrait extraction results between frames. However, recurrent neural networks struggle to extract subtle pixel details in high-resolution videos, and virtual composite videos often have translucent effects. Optical flow tracking in complex light-cut areas is difficult, resulting in failures in temporal information tracking and, in turn, reduced portrait extraction accuracy. Summary of the Invention

[0015] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0016] To this end, the purpose of the present invention is to propose a high-resolution video virtual character matting method based on deep spatiotemporal learning, which can effectively restore the edge details of the virtual portrait, overcome the problem of insufficient learning ability of traditional recurrent neural networks for spatial information, enhance the realism and detail of the final extracted mask, and complete the overall fusion of detail information and semantic information extracted from the virtual portrait mask, ensuring the accuracy, efficiency and robustness of the mask extraction.

[0017] Another object of the present invention is to provide a high-resolution video virtual character matting device based on deep spatiotemporal learning.

[0018] To achieve the above objectives, the present invention proposes a method for high-resolution video virtual character matting based on deep spatiotemporal learning, comprising:

[0019] Collecting video data in real time; wherein the video data includes a mask image of a virtual character to be extracted;

[0020] Inputting the video data into a trained matting network model to extract the virtual character image mask, wherein the matting network model includes a spatiotemporal encoding and decoding model, a detail extraction model, and a semantic information fusion model;

[0021] Based on the video data, first output data is obtained through the spatiotemporal encoding and decoding model, based on the first output data, second output data is obtained through the detail extraction model, and based on the second output data, a virtual character mask image of the video data is obtained through the semantic information fusion model output; wherein, the first output data includes a high-resolution mask image, a low-resolution mask image, high-resolution coding features and low-resolution coding features; the second output data includes a high-resolution detail mask image and a low-resolution detail mask image.

[0022] In addition, the high-resolution video virtual character matting method based on deep spatiotemporal learning according to the above embodiment of the present invention may also have the following additional technical features:

[0023] Furthermore, in one embodiment of the present invention, before inputting the video data into the trained matting network model, it also includes: performing data preprocessing on the video data; and intra-frame grouping of the preprocessed video data according to the similarity of pixel distribution between frames to split the video data into multiple original frame images.

[0024] Furthermore, in one embodiment of the present invention, the spatiotemporal encoding and decoding model includes a spatiotemporal encoder, a spatiotemporal decoder and an ST-ConvLSTM encoder and decoder, and the first output data is obtained through the spatiotemporal encoding and decoding model based on the video data, including: inputting the original frame image into the spatiotemporal encoder for feature extraction after two-way downsampling to obtain semantic extraction features; passing the semantic extraction features through the ST-ConvLSTM encoder and decoder convolution operation to predict time series tensor features; inputting the time series tensor features into the spatiotemporal decoder, and through the first depth-guided filtering operation on the output data of the spatiotemporal decoder, outputting high-resolution mask images and high-resolution coding features in one way, and outputting low-resolution mask images and low-resolution coding features in the other way.

[0025] Furthermore, in one embodiment of the present invention, the second output data is obtained through the detail extraction model based on the first output data, including: performing channel feature splicing on the two downsampled images of the original frame image with the high-resolution encoding features and the low-resolution encoding features through the detail extraction model, and then inputting the spatiotemporal encoder to obtain a first splicing feature and a second splicing feature; performing channel feature splicing on the first splicing feature and the high-resolution mask, and on the second splicing feature and the low-resolution mask image, and then inputting the spatiotemporal decoder to output a third splicing feature and a fourth splicing feature; through a second depth-guided filtering operation on the third splicing feature and the fourth splicing feature, outputting the high-resolution detail mask image through one path and the low-resolution detail mask image through the other path through the detail extraction model.

[0026] Furthermore, in one embodiment of the present invention, the output of the virtual character mask image of the video data based on the second output data through the semantic information fusion model includes: upsampling the high-resolution mask image through the semantic information fusion model, performing channel feature splicing with the high-resolution coding feature, and then inputting the image into the first convolutional neural network to output a first overall high-resolution mask image; upsampling the low-resolution mask image, performing channel feature splicing with the low-resolution coding feature, inputting the image into the second convolutional neural network, and performing channel feature splicing with the overall high-resolution mask image to obtain a second overall high-resolution mask image; inputting the second overall high-resolution mask image into the third convolutional neural network, and performing a third depth-guided filtering operation on the output data of the third convolutional neural network, and outputting the virtual character mask image of the video data through the semantic information fusion model.

[0027] To achieve the above objectives, the present invention further proposes a fast millimeter wave and terahertz wave security inspection imaging device based on deep learning, comprising:

[0028] A data acquisition module, configured to acquire video data in real time; wherein the video data includes a mask image of a virtual character to be extracted;

[0029] A mask extraction module is used to input the video data into a trained matting network model to extract the mask of the virtual character image, wherein the matting network model includes a spatiotemporal encoding and decoding model, a detail extraction model and a semantic information fusion model;

[0030] A data output module is used to obtain first output data based on the video data through the spatiotemporal coding model, obtain second output data based on the first output data through the detail extraction model, and obtain a virtual character mask image of the video data through the semantic information fusion model output based on the second output data; wherein, the first output data includes a high-resolution mask image, a low-resolution mask image, high-resolution coding features and low-resolution coding features; and the second output data includes a high-resolution detail mask image and a low-resolution detail mask image.

[0031] The high-resolution video virtual character matting method and device based on deep spatiotemporal learning in the embodiments of the present invention can effectively restore the edge details of the virtual portrait, overcome the problem of insufficient learning ability of traditional recurrent neural networks for spatial information, enhance the realism and detail of the final extracted mask, and complete the overall fusion of the detail information and semantic information extracted from the virtual portrait mask, ensuring the accuracy, efficiency and robustness of the mask extraction.

[0032] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 Flowchart of a method for high-resolution video virtual character matting based on deep spatiotemporal learning according to an embodiment of the present invention;

[0035] Figure 2 is a flow chart of a first output data calculation method according to an embodiment of the present invention;

[0036] Figure 3 is a flow chart of a second output data calculation method according to an embodiment of the present invention;

[0037] Figure 4 Flowchart of a method for extracting a mask image of a virtual character according to an embodiment of the present invention;

[0038] Figure 5 2. A schematic diagram of the architecture of a method for matting virtual characters in high-resolution videos based on deep spatiotemporal learning according to an embodiment of the present invention;

[0039] Figure 6 Schematic diagram of the structure of a high-resolution video virtual character matting device based on deep spatiotemporal learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0042] The following describes a method and apparatus for high-resolution video virtual character matting based on deep spatiotemporal learning according to an embodiment of the present invention with reference to the accompanying drawings.

[0043] Figure 1 This is a flowchart of a method for high-resolution video virtual character matting based on deep spatiotemporal learning according to an embodiment of the present invention.

[0044] like Figure 1 As shown, the method includes but is not limited to the following steps:

[0045] S1, real-time acquisition of video data; wherein the video data includes a mask image of a virtual character to be extracted;

[0046] S2, inputting the video data into the trained matting network model to extract the virtual character image mask, wherein the matting network model includes a spatiotemporal encoding and decoding model, a detail extraction model, and a semantic information fusion model;

[0047] S3, based on the video data, obtains first output data through a spatiotemporal encoding and decoding model, based on the first output data, obtains second output data through a detail extraction model, and based on the second output data, obtains a virtual character mask image of the video data through a semantic information fusion model output; wherein, the first output data includes a high-resolution mask image, a low-resolution mask image, high-resolution coding features and low-resolution coding features; the second output data includes a high-resolution detail mask image and a low-resolution detail mask image.

[0048] The embodiments of the present invention are specifically described in detail below with reference to the accompanying drawings.

[0049] Specifically, it is necessary to first collect video data and select and process the data set. The model of the present invention relies on a deep learning network framework and requires the use of a large amount of video data sets for training and iterative learning.

[0050] The data processing method of the embodiment of the present invention includes a synthetic self-supervision strategy and pre-training preprocessing. The self-supervision of the synthetic image is carried out by introducing the target consistency constraint, that is, for the model, the same complete virtual character should be able to be successfully cut out for any different backgrounds, and during training, the virtual character and its background can be randomly combined to perform self-supervision training, thereby improving the generalization ability of the model. The pre-training preprocessing operations can be divided into random rotation, translation, occlusion and other operations on the entire video image, and the pixel distribution of each frame in the video is calculated frame by frame using a statistical method. The similarity of the pixel distribution between frames is used to determine whether the background environment has changed, and based on this, the frames are grouped together instead of inputting all the video frames into the cutout model at the same time.

[0051] Furthermore, the spatiotemporal encoding and decoding module is operated based on the ST-ConvLSTM module.

[0052] As an example, a semantic information codec based on a lightweight convolutional network and a codec based on the Atrous Spatial Convolutional Pooling Pyramid (ASPP) and ST-ConvLSTM are proposed. ASPP introduces atrous convolutions based on the spatial convolutional pooling pyramid, which can capture contextual features of image semantic information at multiple scales. The spatial convolutional pooling pyramid uses multiple pooling layers of different scales to extract fixed-dimensional features for each image, ensuring a fixed-size output for input images of different sizes, allowing training using the same fully connected network. Atrous convolutions at different sampling rates can effectively expand the receptive field of the convolution kernel, improving the overall effectiveness of multi-scale feature extraction. The ST-ConvLSTM module introduces convolution operations based on the traditional LSTM module for processing 1D temporal sequences. This allows the module to predict 2D (3D) temporal tensors from 2D (3D) temporal tensors, thereby learning the temporal dependencies between different frames in a video. Each ST-ConvLSTM module has three inputs and two outputs. The specific calculation process can be summarized by the following formula:

[0053] ST(·)=Spatial_Temporal_Attention_Function(·)

[0054] i(t)=σ(W xi *ST(x(t))+W hi *h(t)+W ci c(t-1)+bias i )

[0055] f(t)=σ(W xf *ST(x(t))+W hf *h(t-1)+W cf c(t-1)+bias f )

[0056] c(t)=f(t)·c(t-1)+i(t)·tanh(W xc *ST(x(t))+W hc *h(t-1)+b c )

[0057] o(t)=σ(W xo *ST(x(t))+W ho *h(t-1)+W co c(t)+bias o )

[0058] h(t)=o(t)·tanh(c(t))

[0059] The output semantics of the spatiotemporal codec operation will be passed through the deep guided filtering module, which uses the original input frame as the guide map to perform guided filtering, thereby calculating the high-resolution mask and low-resolution mask of the input image. The whole step process is as follows Figure 2 shown.

[0060] In an embodiment of the present invention, a model is constructed using a spatiotemporal codec constructed based on ST-ConvLSTM. The ST-ConvLSTM module can effectively extract and memorize feature association information between continuous multi-frame video images with a temporal relationship. By grouping and dividing the complete video according to a specified number of frames and repeatedly chaining the spatiotemporal codec according to the number of divided groups, information in both spatial and temporal dimensions can be simultaneously utilized. The feature extraction of spatial information is mainly achieved by convolutional coding and channel attention mechanism, which can effectively ensure the accuracy of portrait extraction; the feature extraction of temporal information is achieved by predicting the long-term and short-term dependencies between different frames of the video by the temporal codec, which can effectively combat bad frames in the video, greatly enhance the continuity of the model's matting in high dynamic environments, and overcome problems such as flickering artifacts.

[0061] In an embodiment of the present invention, a channel attention mechanism and a light field depth-guided filter are added to a temporal recurrent network, which can further enhance the network model's ability to extract portrait edges. The channel attention mechanism can effectively extract multiple semantic information from each frame, compensating for the weak feature extraction capabilities of the temporal recurrent network. The light field depth-guided filter can restore high-resolution details through downsampling of high-resolution video and depth-guided filtering, improving the model's computational efficiency on high-resolution videos and the processing effect of virtual portrait edges, such as fine hair, translucent clothing, and sharp accessories, which are commonly synthesized components in virtual videos.

[0062] Furthermore, two detail extraction modules are used to extract low-resolution details and high-resolution details of the original image respectively. In the high-resolution detail extraction module, the downsampled image of the original video frame and the intermediate high-resolution features output by the spatiotemporal semantic encoder are spliced ​​into the tensor to be encoded and input into the spatiotemporal encoder, and then spliced ​​with the high-resolution mask output by the spatiotemporal semantic extraction module and input into the spatiotemporal decoder. The spliced ​​tensor is then used as a guide map and the output of the spatiotemporal decoder is subjected to deep guided filtering to obtain the high-resolution detail mask of the original frame. Similarly, in the low-resolution detail extraction module, the intermediate low-resolution features output by the input spatiotemporal semantic encoder and the low-resolution mask output by the spatiotemporal semantic extraction module are used to calculate the low-resolution detail mask of the original frame, thereby completing the detail extraction in the video frame. The specific process is as follows. Figure 3 shown.

[0063] Specifically, the embodiment of the present invention extracts details from virtual portraits in videos using two modules: a low-resolution detail extraction module and a high-resolution detail extraction module. The low-resolution detail extraction module combines low-resolution intermediate features extracted by the spatiotemporal codec with instance semantics to calculate low-resolution details in the video. The high-resolution detail extraction module combines high-resolution intermediate features with low-resolution details and instance semantics to calculate high-resolution details in the video. This allows for the extraction of a more precise virtual portrait mask, alleviating the problem of blurred dynamic details.

[0064] Specifically, in the method of processing detail information: by first fusing the decoded semantic output of the light field filter with the extracted low-resolution details, the detail semantic fusion result is used as the intermediate input, and then filtered and fused with the high-resolution details again frame by frame, thereby effectively consolidating the high-resolution edge details in the extracted portrait and further improving the portrait extraction effect.

[0065] Furthermore, the low-resolution details, high-resolution details and semantic information fusion module (output module): the high-resolution mask output by the spatiotemporal semantic extraction module is upsampled, and then spliced ​​with the high-resolution detail mask, and the output after inputting a lightweight CNN network is used as the overall high-resolution mask, and the low-resolution mask output by the spatiotemporal semantic extraction module is upsampled and spliced ​​with the low-resolution detail mask, and then upsampled again and spliced ​​with the overall high-resolution mask, and then inputted into a lightweight CNN network as the final mask extraction frame, and the original frame is used as the guide image for depth guided filtering, thereby outputting the final complete mask image. At this point, the present invention completes the extraction of the virtual portrait mask video in the video. The specific process is as follows Figure 4 shown.

[0066] The embodiment of the present invention uses the original video of the virtual character mask to be extracted as the model input, performs random rotation, translation, occlusion and other preprocessing, and then performs intra-frame small batch grouping according to the similarity of pixel distribution between frames. The grouping splits the original video and inputs it into the model. The input video will enter the spatiotemporal semantic extraction codec frame by frame after two-way downsampling. The spatiotemporal semantic extraction codec will output the extracted high-resolution mask, low-resolution mask, high-resolution coding features and low-resolution coding features respectively. The high / low resolution coding features will be channel-spliced ​​with the downsampled input, and the high / low resolution masks will be channel-spliced ​​after time series encoding again. After decoding and depth-guided filtering, two outputs of high-resolution extracted details and low-resolution extracted details will be obtained. Finally, the high-resolution mask, low-resolution mask, high-resolution details and low-resolution details are used as the input of the spatiotemporal semantic fusion module and depth-guided filtering is performed to obtain the final virtual portrait mask extraction video. The process of the overall framework is as follows. Figure 5 shown.

[0067] The beneficial effects of the present invention are:

[0068] 1) A spatiotemporal encoding and decoding module based on the ST-ConvLSTM module, a light field depth guidance module, and a multi-channel detail extraction module were added, and the virtual portrait mask extraction problem was considered from the perspective of the fusion of semantic information and detail features. The spatiotemporal encoding and decoding module based on the ST-ConvLSTM module can simultaneously utilize the convolutional network's ability to extract image information and the temporal neural network's ability to extract temporal relationships between frames, achieving effective information utilization in both time and space dimensions and maximizing the input information of the original video. The light field depth guidance module replaces the traditional upsampling module and can effectively perform edge-smoothing linear filtering on the mask image based on the guidance map gradient, balancing filtering efficiency and detail perception. The multi-channel detail extraction module can effectively extract details of different resolutions from each frame of high-resolution video, preventing the model from forgetting overly fine details and preventing the erroneous supplementation of details by mistaking noise for high-resolution details. The semantic information and detail feature fusion module spatially integrates the outputs of all modules at different scales, effectively utilizing all the model's computational results. Using only semantic information will lose detail features and only produce a mask with blurred edges, while detail information alone will not be able to complete guided filtering. Therefore, through this module, we have achieved the overall fusion of detail information and semantic information extracted from virtual portrait masks, ensuring the accuracy, efficiency, and robustness of mask extraction.

[0069] 2) Existing methods mostly focus on real-life image mask extraction, and pay insufficient attention to video image mask extraction of virtual portraits. The existing method of manually extracting frame by frame is too inefficient for video processing. The present invention does not rely on traditional priors, and uses a timing module to extract temporal correlations between frames, which can focus more on temporal correlations between frames than convolutional neural networks based on instance segmentation; at the same time, the present invention introduces a multi-path detail extraction module, which can effectively restore the edge details of virtual portraits, and overcome the problem of insufficient learning ability of traditional recurrent neural networks for spatial information, thereby enhancing the realism and detail of the final extracted mask.

[0070] 2) In the instance segmentation network, the present invention adds a spatiotemporal encoding and decoding module constructed based on a convolutional long short-term memory module. Unlike the instance segmentation model that uses a general convolutional neural network as the backbone network and detection head, the spatiotemporal encoding and decoding module not only uses temporal channel attention to reasonably distribute the visual weights on different frames, but also ensures the continuity of the virtual portrait contour between frames.

[0071] 3) Unlike general recurrent neural networks, the model of the present invention does not directly output results in a single stage, or iteratively optimize the output results by repeating the same module multiple times. Instead, it uses a spatiotemporal encoding and decoding module constructed based on a convolutional long short-term memory module to perform combined input and output and intermediate coding feature calculations for the semantic information, low-resolution detail information, and high-resolution detail information extraction modules. This can fully utilize temporal information while enhancing the extraction effect of the portrait detail part by integrating low-resolution detail and high-resolution detail.

[0072] 4) For detail extraction, a deep-guided filter is used to solve the problem of excessive computational complexity for high-resolution details. The spatiotemporal codec module constructed using the convolutional long short-term memory module is used to distinguish and extract low-resolution detail information and high-resolution detail information in the video semantics to comprehensively repair the details of the virtual portrait.

[0073] 5) A semantic fusion strategy is adopted to fuse semantic information and detail information in multiple steps by using multiple forward jump links of intermediate features. This enables the overall model to directly extract the output video from the video and retain the extracted portrait with complete semantic details in the original video.

[0074] According to the high-resolution video virtual character matting method based on deep spatiotemporal learning in an embodiment of the present invention, the edge details of the virtual portrait can be effectively restored, and the problem of insufficient learning ability of traditional recurrent neural networks for spatial information can be overcome, thereby enhancing the realism and detail of the final extracted mask, completing the overall fusion of detail information and semantic information extracted from the virtual portrait mask, and ensuring the accuracy, efficiency and robustness of the mask extraction.

[0075] In order to implement the above embodiment, Figure 6 As shown, this embodiment also provides a high-resolution video virtual character matting device 10 based on deep spatiotemporal learning. The device 10 includes: a data acquisition module 100, a mask extraction module 200 and a data output module 300.

[0076] The data acquisition module 100 is used to acquire video data in real time; wherein the video data includes a mask image of a virtual character to be extracted;

[0077] The mask extraction module 200 is used to input the video data into the trained matting network model to extract the mask of the virtual character image, wherein the matting network model includes a spatiotemporal encoding and decoding model, a detail extraction model and a semantic information fusion model;

[0078] The data output module 300 is used to obtain first output data based on the video data through the spatiotemporal coding model, obtain second output data based on the first output data through the detail extraction model, and obtain a virtual character mask image of the video data through the semantic information fusion model output based on the second output data; wherein the first output data includes a high-resolution mask image, a low-resolution mask image, high-resolution coding features and low-resolution coding features; and the second output data includes a high-resolution detail mask image and a low-resolution detail mask image.

[0079] Furthermore, before the mask extraction module 200, the following steps are further included:

[0080] A preprocessing module, used for performing data preprocessing on video data;

[0081] The image splitting module is used to group the pre-processed video data into frames according to the similarity of pixel distribution between frames, so as to split the video data into multiple original frame images.

[0082] Furthermore, the above-mentioned spatiotemporal encoding and decoding model includes a spatiotemporal encoder, a spatiotemporal decoder and an ST-ConvLSTM encoder and decoder, and the above-mentioned data output module 300 includes:

[0083] A semantic extraction module is used to input the original frame image into the spatiotemporal encoder after two-way downsampling to extract features and obtain semantic extraction features;

[0084] A tensor prediction module is used to predict the time series tensor features by convolution operation of the semantic extraction features through the ST-ConvLSTM codec;

[0085] The feature output module is used to input the temporal tensor features into the spatiotemporal decoder, and through the first depth-guided filtering operation on the spatiotemporal decoder output data, output a high-resolution mask image and high-resolution encoding features on one path, and output a low-resolution mask image and low-resolution encoding features on the other path.

[0086] Furthermore, the data output module 300 further includes:

[0087] A first splicing module is used to perform channel feature splicing on the two downsampled images of the original frame image, the high-resolution coding features, and the low-resolution coding features through a detail extraction model, and then input the spatiotemporal encoder to obtain a first splicing feature and a second splicing feature;

[0088] A second splicing module is used to perform channel feature splicing on the first splicing feature and the high-resolution mask, and on the second splicing feature and the low-resolution mask image, and then input the spatiotemporal decoder to output a third splicing feature and a fourth splicing feature;

[0089] The detail extraction module is used to perform a second depth-guided filtering operation on the third splicing feature and the fourth splicing feature, and obtain a high-resolution detail mask image through one output of the detail extraction model and a low-resolution detail mask image through another output.

[0090] Furthermore, the data output module 300 further includes:

[0091] A sampling output module is used to upsample the high-resolution mask image through a semantic information fusion model, perform channel feature splicing on the high-resolution encoding features, and then input the result into a first convolutional neural network to output a first overall high-resolution mask image;

[0092] A convolutional stitching module is used to upsample the low-resolution mask image and perform channel feature stitching with the low-resolution encoding features, input the image into the second convolutional neural network, and perform channel feature stitching with the overall high-resolution mask image to obtain a second overall high-resolution mask image;

[0093] The filtering output module is used to input the second overall high-resolution mask image into the third convolutional neural network, perform a third depth-guided filtering operation on the output data of the third convolutional neural network, and output the virtual character mask image of the video data through the semantic information fusion model.

[0094] According to the embodiment of the present invention, the high-resolution video virtual character matting device based on deep spatiotemporal learning can effectively restore the edge details of the virtual portrait, overcome the problem that traditional recurrent neural networks have insufficient learning ability of spatial information, enhance the realism and detail of the final extracted mask, and complete the overall fusion of the detail information and semantic information extracted from the virtual portrait mask, ensuring the accuracy, efficiency and robustness of the mask extraction.

[0095] It should be noted that the aforementioned explanation of the embodiment of the high-resolution video virtual character matting method based on deep spatiotemporal learning is also applicable to the high-resolution video virtual character matting device based on deep spatiotemporal learning in this embodiment, and will not be repeated here.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0097] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0098] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A high-resolution video virtual character matting method based on deep spatiotemporal learning, characterized in that: include: Collecting video data in real time; wherein the video data includes a mask image of a virtual character to be extracted; Inputting the video data into a trained matting network model to extract the virtual character image mask, wherein the matting network model includes a spatiotemporal encoding and decoding model, a detail extraction model, and a semantic information fusion model; Based on the video data, first output data is obtained by the spatiotemporal encoding and decoding model; based on the first output data, second output data is obtained by the detail extraction model; based on the second output data, a virtual character mask image of the video data is obtained by the semantic information fusion model output; wherein, the first output data includes a high-resolution mask image, a low-resolution mask image, high-resolution coding features, and low-resolution coding features; and the second output data includes a high-resolution detail mask image and a low-resolution detail mask image; The step of inputting the video data into a trained matting network model to extract a virtual character image mask comprises: The original video of the virtual character mask to be extracted is used as the model input. After random rotation, translation, and occlusion preprocessing, the frame is grouped into small batches based on the similarity of pixel distribution between frames. The original video is split into groups and input into the model. The input video is downsampled in two ways and then enters the spatiotemporal semantic extraction codec frame by frame. The spatiotemporal semantic extraction codec outputs the extracted high-resolution mask, low-resolution mask, high-resolution coding features, and low-resolution coding features respectively; The high / low resolution encoded features are channel-joined with the downsampled input, and the high / low resolution masks are channel-joined again after temporal encoding. After decoding and depth-guided filtering, two outputs of high-resolution extracted details and low-resolution extracted details are obtained; The output high-resolution mask, low-resolution mask, high-resolution extracted details, and low-resolution extracted details are used as input to the spatiotemporal semantic fusion module and subjected to depth-guided filtering to obtain the final virtual portrait mask extraction video.

2. The method according to claim 1, characterized in that Before inputting the video data into the trained matting network model, the method further includes: performing data preprocessing on the video data; The pre-processed video data is grouped within frames according to inter-frame pixel distribution similarity, so as to split the video data into a plurality of original frame images.

3. The method according to claim 2, characterized in that The spatiotemporal encoding and decoding model includes a spatiotemporal encoder, a spatiotemporal decoder, and an ST-ConvLSTM encoder and decoder, and the first output data is obtained based on the video data through the spatiotemporal encoding and decoding model, including: The original frame image is downsampled in two ways and then input into the spatiotemporal encoder for feature extraction to obtain semantic extraction features; The semantic extraction features are convolved through the ST-ConvLSTM codec to predict the time series tensor features; The temporal tensor features are input into the spatiotemporal decoder, and through a first depth-guided filtering operation on the spatiotemporal decoder output data, a high-resolution mask image and a high-resolution encoding feature are outputted in one path, and a low-resolution mask image and a low-resolution encoding feature are outputted in the other path.

4. The method according to claim 3, characterized in that The obtaining of second output data by the detail extraction model based on the first output data includes: The two downsampled images of the original frame image are spliced ​​with the high-resolution coding features and the low-resolution coding features through a detail extraction model, and then input into the spatiotemporal encoder to obtain a first splicing feature and a second splicing feature; Performing channel feature stitching on the first stitching feature and the high-resolution mask, and on the second stitching feature and the low-resolution mask image, and inputting the resulting stitching into the spatiotemporal decoder to output a third stitching feature and a fourth stitching feature; Through a second depth-guided filtering operation on the third stitching feature and the fourth stitching feature, the detail extraction model outputs one path to obtain the high-resolution detail mask image and another path to obtain the low-resolution detail mask image.

5. The method according to claim 4, characterized in that The step of obtaining a virtual character mask image of the video data by outputting the semantic information fusion model based on the second output data includes: The high-resolution mask image is upsampled by a semantic information fusion model, and then channel features are spliced ​​with the high-resolution coding features, and then input into a first convolutional neural network to output a first overall high-resolution mask image; Upsampling the low-resolution mask image and performing channel feature concatenation with the low-resolution coding features, inputting the result into a second convolutional neural network and performing channel feature concatenation with the overall high-resolution mask image to obtain a second overall high-resolution mask image; The second overall high-resolution mask image is input into a third convolutional neural network, and a third depth-guided filtering operation is performed on the output data of the third convolutional neural network, and the virtual character mask image of the video data is output through the semantic information fusion model.

6. A high-resolution video virtual character matting device based on deep spatiotemporal learning, characterized in that: include: A data acquisition module, configured to acquire video data in real time; wherein the video data includes a mask image of a virtual character to be extracted; A mask extraction module is used to input the video data into a trained matting network model to extract the mask of the virtual character image, wherein the matting network model includes a spatiotemporal encoding and decoding model, a detail extraction model and a semantic information fusion model; A data output module, configured to obtain first output data based on the video data through the spatiotemporal encoding and decoding model, obtain second output data based on the first output data through the detail extraction model, and obtain a virtual character mask image of the video data through the semantic information fusion model output based on the second output data; wherein the first output data includes a high-resolution mask image, a low-resolution mask image, high-resolution coding features, and low-resolution coding features; and the second output data includes a high-resolution detail mask image and a low-resolution detail mask image; The mask extraction module is further used to: The original video of the virtual character mask to be extracted is used as the model input. After random rotation, translation, and occlusion preprocessing, the frame is grouped into small batches based on the similarity of pixel distribution between frames. The original video is split into groups and input into the model. The input video is downsampled in two ways and then enters the spatiotemporal semantic extraction codec frame by frame. The spatiotemporal semantic extraction codec outputs the extracted high-resolution mask, low-resolution mask, high-resolution coding features, and low-resolution coding features respectively; The high / low resolution encoded features are channel-joined with the downsampled input, and the high / low resolution masks are channel-joined again after temporal encoding. After decoding and depth-guided filtering, two outputs of high-resolution extracted details and low-resolution extracted details are obtained; The output high-resolution mask, low-resolution mask, high-resolution extracted details, and low-resolution extracted details are used as input to the spatiotemporal semantic fusion module and subjected to depth-guided filtering to obtain the final virtual portrait mask extraction video.

7. The device according to claim 6, characterized in that Also included before the mask extraction module: A preprocessing module, configured to perform data preprocessing on the video data; The image splitting module is used to perform intra-frame grouping on the pre-processed video data according to the similarity of pixel distribution between frames, so as to split the video data into multiple original frame images.

8. The device according to claim 7, characterized in that The spatiotemporal encoding and decoding model includes a spatiotemporal encoder, a spatiotemporal decoder, and an ST-ConvLSTM encoder and decoder. The data output module includes: A semantic extraction module is used to input the original frame image into the spatiotemporal encoder after two-way downsampling to extract features and obtain semantic extraction features; A tensor prediction module is used to predict the time series tensor features by convolution operation of the ST-ConvLSTM codec on the semantic extraction features; The feature output module is used to input the temporal tensor features into the spatiotemporal decoder, and through a first depth-guided filtering operation on the output data of the spatiotemporal decoder, output a high-resolution mask image and high-resolution encoding features on one path, and output a low-resolution mask image and low-resolution encoding features on the other path.

9. The device according to claim 8, characterized in that The data output module further includes: A first splicing module is configured to perform channel feature splicing on the two downsampled images of the original frame image, the high-resolution coding features, and the low-resolution coding features through a detail extraction model, and then input the spatiotemporal encoder to obtain a first splicing feature and a second splicing feature; A second stitching module is configured to stitch the first stitching feature and the high-resolution mask image, and the second stitching feature and the low-resolution mask image, and input the stitching features into the spatiotemporal decoder to output a third stitching feature and a fourth stitching feature; The detail extraction module is configured to perform a second depth-guided filtering operation on the third and fourth stitching features, and obtain the high-resolution detail mask image through one output of the detail extraction model and the low-resolution detail mask image through another output.

10. The device according to claim 9, characterized in that The data output module further includes: a sampling and output module, configured to upsample the high-resolution mask image through a semantic information fusion model, perform channel feature concatenation on the high-resolution encoding features, and then input the concatenated features into a first convolutional neural network to output a first overall high-resolution mask image; a convolutional stitching module, configured to upsample the low-resolution mask image and perform channel feature stitching with the low-resolution coding features, input the up-sampled image into a second convolutional neural network, and then perform channel feature stitching with the overall high-resolution mask image to obtain a second overall high-resolution mask image; A filtering output module is used to input the second overall high-resolution mask image into a third convolutional neural network, perform a third depth-guided filtering operation on the output data of the third convolutional neural network, and output the virtual character mask image of the video data through the semantic information fusion model.

Images