A real-time high-resolution portrait matting method based on deep neural network

Abstract

The application discloses a kind of real-time high-resolution portrait matting methods based on deep neural network, including obtaining training dataset, and marking generation training groundtruth alpha matte;Training data set is data enhanced;Network model is trained in step phase;Using trained network carries out matting.Through embedding ConvLSTM module in network configuration, using Max Pooling Indices, high-definition detail optimization is carried out using PRM, semantic segmentation task is added, and the core technology of high-precision real-time portrait matting is created, simultaneously, data set and data enhancement method are innovated, and training is carried out in stages, from simple to complex, from rough to fine, the training effect of algorithm is strengthened, the innovation and application of the three aspects interact, mutually unified, the performance and practicality of algorithm are comprehensively improved, and powerful technical support is provided for high-precision real-time portrait matting application.

Description

TECHNICAL FIELD

[0001] The present application relates to a portrait matting method, in particular to a real-time high-resolution portrait matting method based on a deep neural network. BACKGROUND

[0002] The traditional method is green screen matting, that is, the person is photographed in front of the green screen, and the portrait and handheld objects are retained by the matting algorithm, and the background is changed to the target background. However, most of the time, it is not very convenient to set up a green screen, so the matting of complex natural backgrounds becomes a demand. Therefore, algorithms such as Bayesian matting, KNN matting, and Poisson matting have emerged.

[0003] With the rise of deep learning, deep learning-based matting algorithms have become a trend. The first one is Adobe Deep Image Matting[1] from Adobe, which is a two-stage deep learning model. The first stage is an encoder-decoder stage based on a deep convolutional network, and the input is the image to be cut and the trimap of this image, and the output is the predicted alpha map. The second stage is a small convolutional neural network used to refine the alpha map of the first stage to make the edges more sharpened. The input required by Adobe is a trimap that is difficult for people to obtain in reality. Subsequently, there appeared a matting algorithm that takes the background image and the image to be cut as input, represented by: Background Matting:The World is Your GreenScreen[2]. The algorithm of [2] cannot achieve real-time speed due to its overly complex network structure, especially for 4K high-resolution images. Real-time matting algorithms have appeared, such as Real-Time High-Resolution Background Matting[3]. [3] is more streamlined than [2] in terms of network structure, and it consists of two parts: the first part is an encoder-decoder base network, and the second part is a refiner network. Although [3] achieves 4K real-time matting, the input requires not only the image to be cut but also the background image. An algorithm that only requires the image to be cut as input has been produced: Robust High-Resolution Video Matting with Temporal Guidance[4]. The input of [4] only requires the image to be cut, and the ConvGRU module of the network considers the correlation between the previous and subsequent frames, making the entire prediction process more stable.

[0004] The disadvantages of green screen matting are obvious. First, the green screen needs to be obtained and arranged in advance, which is a cumbersome and time-consuming process. Second, it is not suitable to arrange a green screen in some places, such as scenic spots, airports, and coffee shops. Finally, the range of activities of the characters is limited, and the characters can only be shot in front of the green screen and cannot be shot outside the green screen.

[0005] [3][4]The direct transmission of feature maps between the encoder and the decoder of the network occupies a large amount of GPU memory space, which requires a higher memory.

[0006] [1][2][3][4]For a large area of shadow projected by the human body, the background area is similar to the skin color or clothing color of the human body, and the background contains a large amount of texture, which is unsatisfactory.

[0007] The matting work represented by [1] needs a trimap as an additional input, however, which requires manual fine labeling to obtain.

[0008] [2][3]Need background pictures as priori, which requires the background to be static and fixed camera position.

[0009] In some technologies, [7][8]first generate a pseudo-trimap, and then predict the matte according to the pseudo-trimap. However, due to limited training data, these methods cannot generalize well to real-world examples [2]. In addition, they cannot be applied to some real-time tasks, such as real-time background replacement in video conferencing.

[0010] The input of [4] does not need background pictures as priori, nor does it use pseudo-trimap to predict alpha. For 4k resolution pictures, the speed can reach real-time, but the disadvantage is that the edge details of the matting are relatively fuzzy and rough, and the sharpening degree is relatively low, such as hair and shoelaces. One of the reasons for this phenomenon is to pursue faster inference speed. It uses a relatively simple refiner [9]: DGF, which is composed of a simple filter, so the optimization effect on details is not good. SUMMARY

[0011] To solve the above problems, the present application provides a real-time high-resolution portrait matting method based on a deep neural network. For high-precision real-time portrait matting, a new network configuration is proposed, and the algorithm performance is improved from the network, data, and training aspects to better enable application scenarios and effectively solve the shortcomings of the prior art.

[0012] The present application is realized by the following technical solutions: a real-time high-resolution portrait matting method based on a deep neural network, comprising the following steps:

[0013] S1, obtain a training data set and label a groundtruth alpha matte for training;

[0014] S2, data augmentation is performed on the training data set;

[0015] S3, train the network model in steps;

[0016] S4, use the network trained in S3 for matting.

[0017] As a preferred technical solution, the data set is obtained and generated in the following manner:

[0018] S1.1, multiple-angle videos of each instance are taken in multiple different scenes, and each instance video contains multiple body actions and different clothing materials;

[0019] S1.2, multiple-angle and multi-pose videos of each instance are taken in front of a green screen, and each instance video contains multiple body actions and different clothing materials;

[0020] S1.3, the approximate human region is automatically cut out by using image processing software, and then manually corrected to generate a groundtruth alpha matte for training;

[0021] S1.4, using the groundtruth alpha matte of the picture taken in front of the green screen, the foreground picture and the background picture are synthesized to generate a large amount of new training data;

[0022] S1.5, the network model is trained using the high-precision data set to improve the matting ability of the model in processing local details.

[0023] As a preferred technical solution, the data augmentation method is as follows:

[0024] S2.1, a blurred and smooth shadow area is generated near the human body area to handle the case of shadow caused by light projection on the human body;

[0025] S2.2, the main color of the human body area is extracted, and a background block similar to the color is automatically generated to handle the prediction error when the foreground and background colors are similar;

[0026] S2.3, a network crawler is used to collect background pictures with complex textures, and the pictures are synthesized with the human foreground to generate training pictures to handle the case of prediction error when the background contains a large amount of texture;

[0027] S2.4, time dimension data expansion is implemented, including video head and tail flipping, frame rate changing, random pausing and frame jumping, to enhance the stability and continuity of the model when processing video data;

[0028] During the training of the network model, a data enhancement technique is used to prevent network overfitting and maintain good generalization ability on real picture data. Affine deformation, up-down-left-right flipping, rotation, random adjustment of brightness, chroma, saturation and contrast, blurring, sharpening and adding random noise are used. At the same time, the input picture will be randomly cropped to any resolution, so that the network model has good robustness to pictures of any size and resolution.

[0029] As a preferred technical solution, in S3, the network training method specifically includes the following steps:

[0030] S3.1, phase training, from simple to complex, from rough to fine, first train the base-net network portrait segmentation task on a lower resolution dataset, with a larger learning rate, so that the base-net can quickly learn to understand the semantic ability of the portrait, and then guide the matting process, which focuses on the edge area of the segmentation result;

[0031] S3.2, training the base-net portrait matting task on a lower resolution dataset, and inserting the training of the portrait segmentation task at odd iterations to prevent network overfitting on artificially synthesized matting data, while improving the semantic understanding ability of the network, and increasing the length of the input video sequence during the last five epochs;

[0032] S3.3, in the third phase, add a fine matting module, and the complete network training is on a higher resolution dataset;

[0033] S3.4, in the last training phase, use a small amount of high-precision dataset for training to improve the accuracy and details of the matting.

[0034] As a preferred technical solution, the weight optimization method for deep learning portrait matting artificially increases the weight of the portrait edge area to enhance the network's learning ability for portrait edge details. The specific steps are as follows:

[0035] Step one, use distance transformation to calculate the distance from each pixel point to the nearest portrait edge;

[0036] Step two, for each sample's ground truth alpha in the training set, the weight map is calculated as follows using distance transformation:

[0037]

[0038] d represents the distance from pixel point x to the nearest portrait edge. In practical applications, w0 is set to 10 and sigma is set to 5 pixels.

[0039] As a preferred technical solution, a network model High-Resolution Human Matting Network (HHMN) for portrait matting is included, which predicts an alpha matte based on an input picture, and the network model includes:

[0040] An Encoder is responsible for extracting the features of a single picture, which uses ResNet-50 as the backbone and connects an ASPP (Atrous Spatial Pyramid Pooling) module, which extracts features at 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 scales, respectively, and the ASPP module is composed of multiple dilated convolution kernels with different dilated rates (3, 6, and 9) to fuse feature maps of different scales;

[0041] A Decoder with a recurrent module, which up-samples according to the max-pooling indices in the corresponding Encoder layer, preserves necessary local detail information, and reduces memory consumption, each layer of which contains 3*3 convolution, batch normalization, and ReLU modules to fuse features and reduce the number of feature channels, and a convLSTM (Convolutional Long Short-Term Memory) module is embedded to utilize time dimension information, the hidden state of which is updated every iteration, and the convLSTM only uses half of the feature map channels;

[0042] A multi-scale PRM (Progressive Refinement Module) module is used to optimize local details of high-precision matting, for high-resolution pictures, a certain down-sampling is performed before inputting into the Encoder-Decoder network, and then the PRM module is used to reconstruct the high-definition matting with optimized details;

[0043] A part for portrait segmentation task, which uses a 3*3 convolution layer and a 1*1 convolution layer, batch normalization, and ReLU to output a one-dimensional channel segmentation map at the last layer of the Decoder.

[0044] As a preferred technical solution, the network used transmits the details of the information channel, including:

[0045] (1) In the encoder stage, max-pooling operation is used to down-sample the feature maps, and the max-pooling position index is saved;

[0046] (2) The max location indices calculated by the max-pooling layer in the encoder will be saved in memory and passed to the decoder;

[0047] (3) In the decoder, the input feature maps are nonlinearly up-sampled according to the max-pooling indices saved in the encoder. This process not only efficiently utilizes memory / video memory, but also preserves the original detail information;

[0048] (4) This detail information transmission channel is suitable for any encoder-decoder network configuration, which can effectively improve the edge detail sketching, reduce the number of model parameters, and ensure the possibility of end-to-end overall training.

[0049] As a preferred technical solution, the convLSTM module is used in the deep learning network model, specifically as follows:

[0050] (1) In the decoder stage, the convLSTM module is used to fuse and utilize the time series information;

[0051] (2) The use of time series information improves the consistency of inter-frame prediction results and reduces jitter. At the same time, the time series information also improves the robustness of the matting, allowing the model to predict the current frame boundary through the previous frames when the single frame prediction is uncertain. Finally, the use of time series information allows the model to learn more background information;

[0052] (3) Using recurrent structure instead of attention mechanism or only using multiple frames as input, the model adaptively retains long and short time series information and learns how to retain and forget information;

[0053] (4) Using convLSTM significantly reduces the error rate of the first 15 frames and then remains stable, improving the matting quality and consistency. The network will automatically reconstruct the background and retain this information to help future prediction, and will also use other recurrent channels to track the motion history.

[0054] As a preferred technical solution, the PRM method is used in the deep learning network model, specifically including:

[0055] (1) In the decoding process, the uncertain matting region is gradually refined by self-guidance of the PRM module;

[0056] (2) PRM is applied to the alpha output of each layer, selectively fusing the matting output of the previous layer and the current layer, the edge part of the figure needs a lower level of feature to describe the foreground, and the middle area of the figure needs a higher level of feature as guidance;

[0057] (3) For the l layer, the self-guidance mask g_l is generated by the matting output alpha_(l-1) of the previous layer:

[0058]

[0059] The 1 and 0 parts of the alpha of the current layer are the same as those of the previous layer, and the part greater than 0 and less than 1 is the same as the alpha' part of the current layer, so the part determined by the previous layer network as 0 or 1 is retained, and the network only needs to focus on correcting the part greater than 0 and less than 1:

[0060] α l =α′ l g l +α l-1 (1-g l ).;

[0061] (4) The experimental results show that PRM can well coordinate global semantic refinement and local detail refinement. Compared with linearly fused multi-scale alpha, applying multi-level PRM can improve the global and local matting effect.

[0062] The beneficial effects of the present application are:

[0063] (I) The present application greatly reduces the demand of network on GPU memory by transmitting max-pooling indices between the encoder and the decoder, while maintaining the detail information of the image and improving the inference speed;

[0064] (II) A high-precision dataset containing 60 instances is created to improve the model's matting ability for local details;

[0065] (III) A data enhancement strategy is designed for complex environments, enabling the model to achieve satisfactory results in complex environments where the body color is similar to the background color, the background contains a large amount of texture, or the body casts a large shadow;

[0066] (iv) To prevent overfitting and improve generalization ability, a phased training strategy is adopted, from simple to difficult, from rough to fine, and semantic segmentation is added in the training process to improve the semantic understanding ability of the model and prevent the model from overfitting to synthetic matting data, improving the generalization of the model;

[0067] The loss function part increases the weights of the portrait edge region, strengthens the learning ability of the model to the portrait edge details, and realizes accurate matting;

[0068] (v) High-precision detail matting, using PRM (Progressive Refinement Module) module in the Refiner stage for detail reconstruction, realizing high-precision matting at the hair level;

[0069] (vi) Robustness enhancement, the model only needs picture input without other additional input, expanding the application range of the algorithm. At the same time, the convLSTM module is used to fuse historical frame information, enhancing the robustness and time sequence consistency of the model, so that the model can cope with the movement of the camera and the change of the background. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0071] Figure 1 The encoder-decoder grid configuration of the present application is shown in the figure;

[0072] Figure 2 The effect diagram of the present application is shown in the figure;

[0073] Figure 3 The specific operation flowchart of the present application is shown in the figure;

[0074] Figure 4 The PRM module diagram of the present application is shown in the figure;

[0075] Figure 5 The module diagram of the decoder of the network of the present application is shown in the figure;

[0076] Figure 6 The convLSTM model diagram in the network of the present application is shown in the figure;

[0077] Figure 7This is an illustration of the nonlinear upsampling of the decoder in the network of this invention based on max pooling indices. Detailed Implementation

[0078] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0079] Any feature disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0080] The present invention provides a real-time high-resolution portrait matting method based on deep neural networks, comprising the following eight points:

[0081] (1) Design a new encoder-decoder mesh configuration, such as Figure 1 As shown:

[0082] For portrait cutout tasks, given an image I, the cutout process will be formulaic as follows:

[0083] I=αF+(1-α)B, α∈[0,1],;

[0084] Where F and B represent the foreground and background of the image, and 'a' is the alpha matte predicted by the network model, indicating where the foreground is in the image. Using 'a', we can composite the extracted foreground (human image) into a new background image.

[0085] I′=αF+(1-α)B′,

[0086] Where B' is the new background image. Portrait matting requires a network model to predict the alpha matte from the input image, which is a difficult ill-posed problem. Directly deploying deep networks for matting high-resolution images is challenging due to its impractical computational and memory consumption. Figure 1 Generally, the image is very sparse, with most pixels belonging to the foreground (a=1) or background (a=0), and only a small number of areas containing edge details such as hair, accessories, and human contours. Instead of designing a full-size, large-scale deep network, it's better to design two parts: a low-resolution base network and a refiner network to refine the predictions and achieve the original high resolution.

[0087] Compared with the existing network model that needs additional input, such as trimap, the designed network model takes the background image as a priori, and adds a portrait segmentation task to improve the semantic understanding ability of the global portrait. A direct way is to take the segmentation map as the front input of the network, or to design two independent segmentation and matting network modules. However, the matting network module cannot use the rich and useful feature maps learned in the segmentation task, and increases a lot of additional calculation, which cannot meet the real-time requirement of high-precision matting. Therefore, we integrate the portrait segmentation task into the network structure, first train the base-net to do the segmentation task, and then insert the segmentation training in the subsequent matting training.

[0088] For video matting, a recurrent network module is designed to utilize the temporal dimension information in the video to reduce the jitter of the matting result, significantly improve the temporal consistency and quality, and at the same time, cope with the situation of camera displacement, background change, and unclear distinction of single-frame picture. Figure 1

[0089] Based on [3, 4], the network model High-Resolution Human Matting Network (HHMN), as shown in Figure 1 , includes an encoder responsible for extracting the features of single-frame pictures, a decoder with a recurrent module responsible for collecting temporal dimension information, and a PRM module responsible for optimizing the upsampling process of high-precision matting. We choose ResNet-50 as the backbone of the encoder network, and link it with the ASPP (Atrous Spatial Pyramid Pooling) module. The encoder is distributed on the 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 scales to extract features. The ASPP module is composed of multiple layers of dilated convolution kernels with different dilated rates (3, 6, 9) to fuse feature maps of different scales.

[0090] In the decoder, each layer is upsampled according to the max-pooling indices in the corresponding encoder layer, which not only preserves the necessary local detail information, but also greatly reduces the consumption of video memory. Then there are 3*3 convolution, batch normalization and ReLU modules to fuse features and reduce the number of feature channels;

[0091] ​In each layer of the decoder, a convLSTM module is embedded to utilize the time dimension information. The hidden state of the convLSTM is updated each iteration, and the convLSTM only uses half of the feature map channels, which is enough to capture the time dimension information and reduces the number of parameters and computation. For high-resolution pictures, such as 4K pictures, a certain multiple of down-sampling is performed before inputting the encoder-decoder network, and then a PRM (Progressive Refinement Module) module is used to reconstruct a high-definition matting picture with detailed optimization. For the portrait segmentation task, a 3*3 convolution layer and a 1*1 convolution layer, batch normalization and ReLU output a one-channel segmentation map in the last layer of the decoder, as shown in Figure 5

[0092] (2) In view of the fact that the direct transmission of the feature map of the encoder and the decoder of [3] and [4] occupies a large amount of GPU memory, the index of the max-pooling of each layer of the encoder is transmitted to the corresponding layer of the decoder, as in [5], which reduces the network parameters while ensuring that the details are not lost.

[0093] The deep learning model has achieved continuous success due to its huge amount of data and model depth and complexity. For deploying large deep neural networks, practical factors such as memory, computation time, and training time become very important, especially for real-time requirements of high-definition portrait matting applications such as live streaming and AR. Based on this, the designed network must be efficient in terms of computation amount, and must maintain high efficiency in terms of memory consumption and inference time indicators, and can be trained end-to-end as a whole.

[0094] ​The network model is designed for the application scenario of real-time high-definition portrait matting. The model should be able to accurately outline the fine parts of the portrait (such as hair strands), so it is crucial to preserve the edge detail information in the network features. In the encoder part of the network, the down-sampling operation will expand the receptive field of each pixel in the feature map. After multiple down-sampling, the network will establish the invariance to transformation, but at the same time, it will constantly lose the spatial edge detail information. The lack of detailed feature description is extremely disadvantageous to high-precision portrait matting, especially in key local detail parts, so edge detail information must be extracted and preserved in the encoder before up-sampling. If memory limitations are not considered, all feature maps in the encoder can be saved, but this is actually not easy to implement, so an efficient way to store detail information is needed - save the max-pooling indices

[18] . The position of the maximum feature value in the pooling window will be memorized and saved. In practice, 2 bits can be used to correspond to each 2*2 pooling window. This is much more efficient than storing floating-point feature maps.

[0095] The core structure of the network is a decoder network, in which the hierarchy corresponds to that of the encoder. The role of the decoder in the network model is to convert the low-resolution feature maps of the encoder into feature maps of the same resolution as the input image. The learning process of the network is the core of the model. One of the innovative points of the network model is how to up-sample the low-resolution feature maps. Specifically, for each input sample, the max location indices calculated by the max-pooling layer in the encoder are saved in memory and passed to the corresponding decoder. In the decoder, each level uses the max-pooling indices in the corresponding encoder level to perform nonlinear up-sampling on the input feature maps. This idea is inspired by

[18] , and the use of max-pooling indices in the decoder has the following advantages: (1) improves edge details, (2) reduces the number of model parameters to enable end-to-end overall training, (3) this up-sampling method can be combined into any encoder-decoder network configuration by simple modification. The feature maps after up-sampling are sparse and dense maps generated by trainable convolution kernels. After multiple levels of up-sampling, the resolution reaches the input of the encoder. Experimental tests compared with other algorithms show that the network model achieves a good balance between the required memory and model accuracy, and can efficiently use memory while achieving good inference results.

[0096] The inference effect of the network model is more efficient in terms of memory compared to other networks that directly store feature maps. For the method of directly storing feature maps, it may lead to a shortage of memory. For example, storing 64 32-bit floating-point 180*240 feature maps of the first layer requires 11 MB, while storing max pooling indices only requires 0.17 MB (2 bits for each 2*2 pooling window). By comparison, the network using the encoder that directly stores feature maps requires 11 times the memory of the encoder that stores pooling indices, while their model inference results are almost at the same level, as shown in Figure 7 .

[0097] (3) For the case that the prediction is wrong when the human body projects a large shadow area or the background area is similar to the human skin color or clothing color or the background contains a large amount of texture, we designed corresponding data augmentation methods, which are to generate a blurred and smooth shadow area near the human body area, to extract the main color of the human body area and automatically generate a background block of such color, and to collect complex texture background pictures on the network using a crawler and synthesize them with the human foreground to train the pictures.

[0098] For pictures of human body shadows caused by light projection, existing matting methods cannot accurately predict the shadow area. Generally, the shadow is considered to be foreground, so the shadow will be retained when synthesizing the target picture. However, in most cases, people do not want shadows to appear in the synthesized target picture. The reason for this problem is that the training picture is a picture synthesized by foreground and background linearly, and the synthesis process relies on the alpha map of the foreground, which does not contain the shadow area, so the synthesized picture will not retain the shadow area. The network trained with these false synthesized pictures without shadows will not accurately predict the presence of human body shadows in real scenes. To solve this problem, we artificially add shadows to the synthesized pictures before training the network. The method of adding shadows is to shift, scale and affine transform the ground truth alpha with a certain probability, and then generate a shadow mask by random kernel size Gaussian blur. For the background picture, subtract the shadow mask.

[0099] For the case that the prediction is wrong when the foreground and background colors are similar, we use the method of artificially synthesizing backgrounds with similar colors to the foreground when synthesizing training pictures, and then use such backgrounds to synthesize training pictures. The specific method is to randomly select n pixels of the foreground, fill a background picture with the colors of these n pixels, and each pixel has a random filling area, a random position, and a rectangular block shape.

[0100] When training the network model, we use data augmentation techniques to prevent network overfitting and maintain good generalization ability on real picture data. We apply affine deformation, up-down-left-right flipping, rotation, random adjustment of brightness, chroma, saturation, contrast, blur, sharpening and adding random noise. At the same time, the input picture will be randomly cropped to any resolution to make the network model have good robustness to pictures of any size. At the same time, we also use temporal data augmentation, such as video head-tail flipping, frame rate change, random pause and frame skipping.

[0101] (4) Nowadays, there are few high-resolution and high-precision Alpha Mat in the public portrait matting dataset. In order to better adapt to the application scenario of the network model and to make the model have higher matting precision, we create a dataset of 60 instances (people), each instance (person) contains this person in 10 different scenes Multi-angle shooting video, in which the person will have different body movements. In order to deal with professional matting application scenarios such as photography studios, we have added clothing and accessories with more complex modeling materials, such as gauze. We also shot multi-angle and multi-pose videos of the 60 people in front of the green screen background. In order to reduce the workload and time of manual marking, first use Photoshop software to automatically mat the approximate area, and then manually correct it, extract the portrait foreground and network background picture to synthesize a large number of new datasets. Through experiments, we found that our high-precision dataset can effectively improve the network model's ability to mat local details.

[0102] (5) The RNN in the decoder uses convLSTM to fuse information using the time dimension, as shown in Figure 6

[0103]

[0104]

[0105]

[0106]

[0107]

[0108] Most of today's matting methods, even those designed specifically for video, treat each frame as a separate input to the network, outputting its alpha. These methods ignore temporal information, which can effectively improve video matting results. First, temporal information can make the matting results more consistent, and using multiple frame information can reduce inter-frame jitter and improve perceptual quality. Second, temporal information can improve the robustness of network matting. When single-frame prediction is uncertain, such as when the foreground color is similar to the color of moving objects in the background, the model can predict the current frame boundary through previous frames. Finally, as time progresses, temporal information allows the model to learn more background information. When the camera moves, the background behind the person will appear due to changes in the camera's perspective. Even if the camera is fixed, due to the movement of the person, the background behind the person will appear. A better understanding of the background by the network helps matting. Therefore, we use recurrent structure to utilize temporal information. Our method significantly improves the quality and temporal coherence of matting.

[0109] ​We decided to use recurrent structure instead of attention or simply take multiple frames as input, because recurrent mechanism can learn what information should be kept or forgotten in continuous video stream, while attention or multiple frame input can only rely on fixed rules to remove or insert information. The ability of recurrent mechanism to adaptively keep long and short temporal information makes it more suitable for this task.

[0110] The experimental results show that the error rate of the first 15 frames is significantly reduced after using convLSTM, and then remains stable. While MODNet, even if it uses adjacent frames, still has a large fluctuation in video matting. When the recurrent state is assigned to all zeros, the matting quality and consistency become worse, which further shows that temporal information can improve the matting quality and consistency. Because convLSTM is used, the network will automatically reconstruct the background and keep this information to help future prediction, and other recurrent channels will also be used to track the motion history.

[0111] (6) The loss function part increases the weights of the portrait edge area (according to the distance from the portrait edge) to enhance the network's learning ability of the details of the portrait edge.

[0112] For high-precision portrait matting applications (such as professional photography, high-definition online press conferences), accurate matting of portrait details such as hair, complex clothing styles, and accessories is crucial. Existing deep learning-based portrait matting methods are limited by computing resources and cannot directly input 4K or 8K high-definition images. Instead, they first downsample the input image and then input it into the network model, and then use an upsampling refine module to restore it to the original input resolution. Although the loss function can be used on the original resolution during network training, the data distribution of portrait edge details is relatively uneven compared to the main body of the human body, which causes the network to tend to ignore local details during training. To strengthen the network's learning of edge details, a loss function with increased weights is designed for network training. For each training sample, a pixel-level weight map is first calculated, which gives higher importance to edge local pixels to compensate for the frequency of different pixel points in the training set.

[0113] For each sample in the training set, the ground truth alpha, the weight map is calculated as follows using distance transformation:

[0114]

[0115] d represents the distance from pixel x to the nearest portrait edge. In practical applications, w0 is set to 10 and sigma is set to 5 pixels.

[0116] (7) In order to prevent network overfitting in the training process and improve the generalization ability, the training method is designed in stages, from simple to difficult, from rough to fine. At the same time, semantic segmentation is added to prevent the network from overfitting to the artificially synthesized matting data, and to improve the network's semantic understanding ability.

[0117] Automatic portrait matting is cited in many scenarios, and today's deep learning-based portrait matting methods often fail on real photos. The failure is usually due to the failure of portrait semantic segmentation, that is, the inability to accurately understand the portrait area of the picture, especially when dealing with full-body portraits. Today's advanced portrait matting methods [MODNet[3]and RVM[3]all show their weak image semantic understanding ability, which is crucial for stable portrait matting. In order to strengthen the semantic understanding ability of the network model and at the same time preserve the edge details of the matting, a portrait segmentation task module is added to the designed network to guide the matting process. Adding portrait segmentation task has the following benefits: (1), portrait segmentation determines the accuracy of global background and foreground prediction, and local details only appear on the edge of segmentation, which shows that segmentation can actually be replaced by trimap, and can be used as a prerequisite for matting. The segmentation task can provide strong semantic information to effectively help the matting task (2) High-precision matting manual annotation requires professional artists and is very time-consuming, while segmentation annotation is very simple, only the edge line needs to be drawn. Therefore, the public training dataset of matting is much smaller than that of segmentation, and a large amount of training dataset is particularly important for the generalization ability of the network on real photo sets, (3), the training dataset of matting is usually artificially synthesized from foreground and background pictures, which can expand the dataset, but the synthetic pictures do not look real, and there is a visible difference between real photos and synthetic photos. This limits the generalization ability of the trained model on real pictures, while the dataset of portrait segmentation is mostly real pictures. Adding portrait semantic segmentation to the training process prevents the network from overfitting to artificially synthesized matting data. For this reason, we designed a multi-level network structure to realize portrait semantic segmentation and portrait matting, and introduced a lightweight segmentation module that only requires a small amount of additional computation. The portrait segmentation network module is trained first and then guides the matting process, focusing on the edge area of the segmentation result. With the help of portrait semantic understanding ability, the network model can handle challenging matting scenarios. In training, a step-by-step training method is adopted, the first stage is to train the network's portrait segmentation task on the public dataset to achieve stable prediction of portrait segmentation, and the second stage is to train the network's matting task. This can greatly reduce the dependence of the matting network on high-definition annotated samples, and still generate high-quality matting details.

[0118] In order to prevent network overfitting in the training process and improve the generalization ability, the training method is designed in stages, from simple to difficult, from rough to fine.

[0119] Stage1, first train base-net's portrait segmentation task on low resolution dataset with large learning rate, so base-net can quickly learn to understand the semantic of portrait, stage2, train base-net's portrait matting task on low resolution dataset, meanwhile, interpose the training of segmentation task in odd iteration, increase the length of input video sequence in the last 5 epochs, stage3, add the fine matting module, the complete network training on high resolution dataset, stage4, finally train the network using a small amount of high-precision dataset.

[0120] (8) Unlike [3,4], Refiner uses the PRM (Progressive Refinement Module) module of [6] to reconstruct the detailed and optimized high-definition matting, achieving high-precision hair-level portrait matting.

[0121] PRM learns how to guide itself to refine uncertain matting areas step by step during the decoding process. Simply linearly fusing the alpha outputs of different layers cannot well solve the problem of matting refinement. This is because the object edge part needs lower-level features to describe the foreground, while the middle area of the person needs higher-level features as guidance.

[0122] PRM can be applied to the alpha output of each layer, which selectively fuses the alpha output of the previous layer and the current layer.

[0123] As shown in Figure 4 , for layer l, its self-guidance mask g_l is generated from the matting output alpha_(l-1) of the previous layer:

[0124]

[0125] The 1 and 0 parts of the alpha of the current layer are the same as those of the previous layer, and the parts greater than 0 and less than 1 are the same as the parts of alpha' of the current layer. In this way, the part of the network of the previous layer that is determined to be 0 or 1 is preserved, and the network only needs to focus on correcting the part greater than 0 and less than 1:

[0126] α l =α′ l g l +α l-1 (1-g l ).

[0127] The experimental results show that PRM can well coordinate global semantic refinement and local detail refinement. Compared with linearly fusing multi-scale alpha, applying multi-level PRM can improve the global and local matting effect.

[0128] As shown in Figure 3 the specific operation process is as follows:

[0129] Collect data:

[0130] We collected our own training data, details see (4). In addition, we also collected public datasets, including: Distinction-646

[10] , RealWorldPortrait-636

[11] , AIM-500

[12] , VideoMatte240K_JPEG_HD [4], BackgroundVideosTrain [4], BackgroundVideosTest [4], train2017 of COCO

[13] , YouTubeVIS 2021, SPD

[14] , and background images obtained by crawling.

[0131] Design network:

[0132] According to our application scenario, the network is composed of Encoder, Decoder, Matting Head, SegmentationHead, PRM. For the input picture src, first 4 times down-sampling to get src_sm, src_sm as the input of encoder, encoder is composed of layers, these layers output en_1, en_2, en_3, en_4, en_5 respectively, the size is 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64 of src_sm respectively. en_5 as the input of ASPP module, get the output en_5'. Decoder has 5 layers, the output size of each layer is 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64 of src_sm respectively. The input of the second last layer is en_5', and the output is de_5. The input of the third last layer is de_5 and en_4, and the output is de_4. The input of the fourth last layer is de_4 and en_3, and the output is de_3. The input of the third last layer is de_3 and en_2, and the output is de_2. The input of the second last layer is de_2 and en_1, and the output is de_1. After de_1 is up-sampled to the original picture size and added to the original picture, de_0 is obtained. de_1 as the input of segmentation module, after passing through segmentation module, the portrait segmentation result is obtained. de_3 and de_2 are input into PRM1 to obtain p1. p1 and de_1 are input into PRM2 to obtain p2. p2 and de_0 are input into PRM3 to obtain the final alpha output.

[0133] Code implementation:

[0134] Pytorch framework is adopted.

[0135] Training network:

[0136] Stage1: only do segmentation training. For video segmentation, batchsize, T, h, w are consistent with each stage of matting. For picture segmentation, because a single picture is equivalent to T' = 1, T' is 1 will leave a lot of memory space, so you can set batchsize' to be larger, B*T. Because the T' of picture segmentation is 1, this will make the network more robust to the missing of recurrent information. ResNet50 is initialized as the pre-trained weight of ImageNet, the learning rate is 1e^(-4), and the learning rate of the rest of the network is 2e^(-4). Train for 6 epochs. In order to make the network more robust to pictures of different resolutions and aspect ratios, when sampling, the length and width of the picture should be in the range of 256 to 512.

[0137] Stage2: Train HHMN(High-Resolution Human Matting Network) with low resolution, during this stage, the learning rate of PRM is set to 0, because this stage is focused on training encoder and decoder, stage1 to stage4 is a process from simple to complex, PRM is belong to the complex stage. The length of sequence T is set to 15. At the same time, the training of segmentation task is inserted at odd iteration, the parameter setting of segmentation task is the same as stage1. There are 15 eopch, in the last 5 eopch, the length of input video sequence is increased to 50, the learning rate is set to half of stage1.

[0138] Stage3: Open the training of PRM. The picture is high resolution. Train 1 epoch. Because high resolution picture will occupy a lot of video memory, so the length of sequence must be set to low, but low length of sequence will make the network overfitting to short sequence, so long sequence of low resolution and short sequence of high resolution are trained at the same time. The low resolution stage does not open PRM, T is 30, the range of length and width is 256 to 512. High resolution training opens the training of PRM, T is 5, the range of length and width is 1024 to 2048, the learning rate of PRM is 2e^(-4), the learning rate of the rest of the network is 1e^(-5).

[0139] Stage4: The dataset used in stage1 to stage3 is VideoMatte240K_JPEG_HD[4]. The dataset used in this stage is changed to more complex Distiction-646

[10] , RealWorldPortrait-636

[11] , AIM-500

[12] portrait sample. The learning rate of decoder is increased to 5e^(-5). Other settings are the same as stage3.

[0140] Effect evaluation:

[0141]

[0142] The FPS of the model of the application on 3070ti with different input sizes, different down_sample_ratio, different backbones, and different floating point numbers is shown in the above table. The fastest speed can reach 185.5FPS, which is obtained when the input size is 1440p, the downsample_ratio is 0.125, the backbone is mobilenetv3, and the floating point number is 16 bits. For 4K images, the designed network model can still meet the real-time requirement (27.4FPS), which is obtained when the down_sample_ratio is 0.25, the backbone is resnet50, and the floating point number is 32 bits.

[0143] The effects of the model HHM of the application and V3[4], TIMI

[15] , SGHM

[16] and MODNet

[17] are compared, and the results are shown in Figure 2 It can be seen that the designed network model is higher than the mainstream matting algorithm in terms of detail accuracy and accuracy.

[0144] Application landing: the model weight of pytorch is trained. In the landing scene, it is hoped that the inference speed is faster, and the function realized by using the pytorch framework and the python language cannot meet the requirement, so it is necessary to convert the pytorch model into other models with faster inference speed. We choose to convert into a torchscript model, and realize the inference by using the c++ language. The c++ file for calling the torchscript model for inference is encapsulated into a dll. If this function is needed, it is only necessary to import in the header file and then call the encapsulated inference function. In this way, all the implementations are separated from the python language, and are replaced by the c++ language, so that the requirement of the inference speed in the landing scene can be met.

[0145] The detailed usage guide of the specific deep learning network model is divided into the following parts:

[0146] 1) Language version and operation precision selection

[0147] The network model provides python version and compiled and encapsulated dll version, and provides a trained model, pytorch file in pt format and torchscript file easy to convert and deploy, and 16-bit floating point and 32-bit floating point precision. According to the reference of the provided timeliness FPS test chart, the user can select according to the GPU resources. The downloaded model file can be placed in the corresponding directory;

[0148] 2) Python version usage required installation dependencies

[0149] The corresponding dependencies, such as easing_functions, tensorboard, torch, torchvision, tqdm, pims, can be installed using the provided requirements.txt and the 'pip install-r' command. If there is no python environment, you need to install python or anaconda first, and in order to use Nvidia GPU, you need to install Nvidia CUDA and cuDNN first, you can refer to the official instructions of Nvidia.

[0150] 3) Dependencies required by dll version

[0151] The dll version reads the call model and image data through address pointers, and only needs to copy all.dll files and.lib files in the lib directory of the libtouch library to the library directory of the user project. Similarly, Nvidia CUDA and cuDNN need to be installed in order to use GPU, you can refer to the official instructions of Nvidia.

[0152] 4) Network model parameter selection

[0153] The appropriate down-sampling coefficient can be selected according to the following table:

[0154]

[0155] The network model first down-samples the input image according to the set down-sampling, and then uses PRM up-sampling to optimize the high-resolution matting result. For example, the size of the input image is 2160*3840, and the resolution after down-sampling is 540*960. In addition, the setting of the down-sampling coefficient is also based on the content of the image. If the picture content is a portrait of half a person, a lower down-sampling coefficient can be used, and if it is a full-body portrait, a higher down-sampling coefficient should be used. It should be noted that setting a higher down-sampling coefficient does not necessarily lead to better results.

[0156] 5) convLSTM loop states

[0157] This network model is a recurrent network, so each frame of the video should be processed in order and the states of the convLSTM should be updated in a loop. Specifically, these states are the'memory' of the network model, and when processing the next frame of input, the output of the current frame's convLSTM should be used as one of the inputs. It should be noted that for processing a single static image, it can be looped and repeated multiple times, such as 10 times or 100 times, to allow the initial state of the convLSTM to reach a stable state.

[0158] 6) Use of python version

[0159] You can load and test the network model using the provided inference.py file. The line `model.load_state_dict(torch.load('HHM_resnet50.pt'))` loads the trained .pt model. If using a Torchscript model, you can use `model=`

[0160] torch.jit.load('HHM_resnet50.torchscript')', and can use 'model=

[0161] `torch.jit.freeze(model)` freezes the model, enabling graph optimizations such as batch normalization. The frozen model is faster. When using a frozen model, the `device` and `dtype` parameters should be included during model inference.

[0162] 7) DLL version usage

[0163] Usable

[0164] 'BGMv3::GetInstance()->Initialize(TCHAR_TO_ANSI(*(“path_to_model”)),down_ratio,gpuid)' initializes the network, then uses

[0165] 'BGMv3::GetInstance()->SVMCalcAlpha(ImageDataRGB,Width,Height,OutAlphaMask)' runs the network model and outputs the alpha matte.

[0166] References involved in this invention:

[0167] [1]Xu N,Price B,Cohen S,et al.Deep image matting[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2017:2970-2979.

[0168] [2] Sengupta S, Jayaram V, Curless B, et al. Background matting: The world is your green screen [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 2291-2300.

[0169] [3] Lin S, Ryabtsev A, Sengupta S, et al. Real-time high-resolution background matting [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 8762-8771.

[0170] [4] Lin S, Yang L, Saleemi I, et al. Robust high-resolution video matting with temporal guidance [C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2022: 238-247.

[0171] [5] Badrinarayanan, Vijay, Alex Kendall, and Roberto Cipolla.

[0172] Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence 39.12 (2017): 2481-2495.

[0173] [6] Yu Q, Zhang J, Zhang H, et al. Mask guided matting via progressive refinement network [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 1154-1163.

[0174] [7] Chen Q, Ge T, Xu Y, et al. Semantic human matting [C] / / Proceedings of the 26th ACM international conference on Multimedia. 2018: 618-626.

[0175] [8] Shen X, Tao X, Gao H, et al. Deep automatic portrait matting [C] / / Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14. Springer International Publishing, 2016: 92-107.

[0176] [9] Wu H, Zheng S, Zhang J, et al. Fast end-to-end trainable guided filter [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 1838-1847.

[0177]

[10] Qiao Y, Liu Y, Yang X, et al. Attention-guided hierarchical structure aggregation for image matting [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 13676-13685.

[0178]

[11] Yu Q, Zhang J, Zhang H, et al. Mask guided matting via progressive refinement network [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 1154-1163.

[0179]

[12] Li J, Zhang J, Tao D. Deep automatic natural image matting [J]. arXiv preprint arXiv:2107.07235, 2021.

[0180]

[13] Lin T Y, Maire M, Belongie S, et al. Microsoft coco: Common objects in context [C] / / Computer Vision— ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer International Publishing, 2014: 740-755.

[0181]

[14] supervise.ly. Supervisely person dataset. supervise.ly, 2018.

[0182]

[15] Liu Y, Xie J, Shi X, et al. Tripartite information mining and integration for image matting [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 7555-7564.

[0183]

[16] Chen X, Zhu Y, Li Y, et al. Robust Human Matting via Semantic Guidance [C] / / Proceedings of the Asian Conference on Computer Vision. 2022: 2984-2999.

[0184]

[17] Ke Z, Sun J, Li K, et al. Modnet: Real-time trimap-free portrait matting via objective decomposition [C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(1): 1140-1147.

[0185]

[18] Ranzato, Marc'Aurelio, et al. "Unsupervised learning of invariant feature hierarchies with applications to object recognition." 2007 IEEE conference on computer vision and pattern recognition. IEEE, 2007.

[0186] The above merely illustrates the present application, but the protection scope of the present application is not limited thereto, any change or replacement without creative labor should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be limited by the protection scope defined in the claims.

Claims

1. A real-time high-resolution portrait matting method based on a deep neural network, characterized in that, Comprising the following steps: S1, obtaining a training data set, and labeling the generated ground truth alpha matte for training; S2, data augmentation is performed on the training data set; S3, training the network model in stages; S4, using the network trained in S3 for matting; The details of the network used include: In the encoder stage, max-pooling operation is used to downsample the feature maps, and the max-pooling position index is saved; The max location indices calculated by the max-pooling layer in the encoder will be saved in memory and passed to the decoder; In the decoder, the input feature maps are nonlinearly upsampled according to the max-pooling indices saved in the encoder, while the original detail information is preserved by using memory / video memory; The network model High-Resolution Human Matting Network for portrait matting, which predicts alpha matte based on input images, includes: An encoder responsible for extracting features from single images, using ResNet-50 as the backbone and connecting an ASPP module, which extracts features at 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 scales, and the ASPP module consists of multiple dilated convolution kernels with different dilated rates 3, 6, and 9 to fuse feature maps of different scales; A decoder with a recurrent module that upsampled according to the max-pooling indices in the corresponding encoder layer, preserving necessary local detail information and reducing memory consumption, each layer containing 3*3 convolution, batch normalization, and ReLU modules to fuse features and reduce feature channel numbers, and embedding convLSTM modules to utilize temporal dimension information, updating the hidden state of convLSTM at each iteration, and using only half of the feature map channels; Multi-scale PRM module, which is used to optimize local details for high-precision matting, for high-resolution images, downsample by a certain factor before inputting into the encoder-decoder network, and then use the PRM module to reconstruct the high-definition matting with optimized details; Part for portrait segmentation task, using a 3*3 convolution layer and a 1*1 convolution layer, batch normalization and ReLU output one-dimensional channel segmentation map at the last layer of the decoder.

2. The real-time high-resolution portrait matting method based on deep neural network according to claim 1, characterized in that: The data set is obtained and generated by the following methods: S1.1 Capture multi-angle videos of each instance in multiple different scenes, each instance video containing multiple body movements and different clothing materials; S1.2 Capture multi-angle multi-pose videos of each instance in front of a green screen, each instance video containing multiple body movements and different clothing materials; S1.3 Automatically extract the approximate human region using image processing software, then manually correct it to generate the ground truth alpha matte for training; S1.4 Use the ground truth alpha matte of the pictures taken in front of a green screen to synthesize the foreground and background pictures to generate a large amount of new training data; S1.5 Train the network model using high-precision data sets to improve the network's ability to handle local details in matting.

3. The real-time high-resolution portrait matting method based on deep neural network according to claim 1, characterized in that: The data enhancement method is as follows: S2.1 Generate a blurred and smooth shadow area near the human region to handle the case of shadows caused by light projection on the human body; S2.2 Extract the main color of the human region and automatically generate a background block with a similar color to handle prediction errors when the foreground and background colors are similar; S2.3 Use web crawlers to collect background pictures with complex textures and synthesize them with the human foreground to handle cases where the background contains a lot of textures and the prediction is wrong; S2.4 Implement time dimension data expansion, including video head and tail flipping, frame rate changing, random pausing and frame jumping, to enhance the stability and continuity of the model when processing video data; During network model training, data enhancement techniques are used simultaneously to prevent network overfitting and maintain good generalization ability on real picture data. Affine deformation, up-down-left-right flipping, rotation, random adjustment of brightness, color saturation, and contrast, blurring, sharpening, and adding random noise are used. At the same time, the input picture will be randomly cropped to any resolution to make the network model have good robustness to pictures of any size and resolution.

4. The real-time high-resolution portrait matting method based on deep neural network according to claim 1, characterized in that: In S3, the network training method specifically includes the following steps: S3.1 Stage training, from simple to complex, from rough to fine, first train the base-net network's human segmentation task on a lower resolution data set with a larger learning rate, so that the base-net can quickly learn to understand the semantic ability of the human image. After human segmentation training, guide the matting process, which focuses on the edge area of the segmentation result; S3.2 Train the base-net's human matting task on a lower resolution data set. During the odd iteration of training, the human segmentation task is trained in between to prevent the network from overfitting to artificially synthesized matting data, while improving the network's semantic understanding ability. Increase the length of the video sequence input during the last five epochs of training; S3.3 In the third stage, add a fine matting module, and the complete network training is on a higher resolution data set; S3.4 In the last training stage, use a small amount of high-precision data set for training to improve the accuracy and details of matting.

5. The real-time high-resolution portrait matting method based on deep neural network according to claim 1, characterized in that: A weight optimization method for deep learning portrait matting, which artificially increases the weight of the portrait edge region to enhance the network's learning ability of portrait edge details, the specific steps are as follows: Step one, using distance transformation, according to the distance from each pixel point to the nearest portrait edge; Step two, for each sample's ground truth alpha in the training set, the weight map is calculated as follows: D represents the distance from pixel point x to the nearest portrait edge, in practical application, ω0 is set to 10, and σ is set to 5 pixels.

6. The real-time high-resolution portrait matting method based on deep neural network according to claim 5, characterized in that: Use convLSTM module in deep learning network model, as follows: (1) In the decoder stage, use convLSTM module to fuse and use time sequence information; (2) Using time sequence information improves the consistency of inter-frame prediction results, reduces jitter, at the same time, time sequence information also improves the robustness of matting, allows the model to predict the current frame boundary through the previous frame when single frame prediction is uncertain; Finally, the time sequence information used allows the model to learn more background information; (3) Using recurrent structure instead of attention mechanism or only using multiple frames as input, so that the model adaptively retains long and short time sequence information, learns how to retain and forget information; (4) Using convLSTM significantly reduces the error rate of the first 15 frames and then remains stable, improves the quality and consistency of matting, the network will automatically reconstruct the background and retain these information to help future prediction, and also use other recurrent channels to track the motion history.

7. The real-time high-resolution portrait matting method based on deep neural network according to claim 5, characterized in that: The method of using PRM in deep learning network model, specifically includes: (1) In the decoding process, gradually refine the uncertain matting area through the self-guidance of PRM module; (2) PRM is applied to the alpha output of each layer, selectively fusing the matting output of the previous layer and the current layer, the edge part of the figure needs lower level features to describe the foreground, and the middle area of the figure needs higher level features as guidance; (3) For l layer, self-guidance mask g_l is generated by the matting output alpha_(l-1) of the previous layer: The 1 and 0 parts of the current layer's alpha are the same as the 1 and 0 parts of the previous layer, and the parts greater than 0 and less than 1 are the same as the alpha' part of the current layer, in this way, the parts determined as 0 or 1 by the previous layer network are retained, and the network only needs to focus on correcting the parts greater than 0 and less than 1: α l = α' l g l + α l-1 (1 - g l ).

Images