Video portrait matting method and device, equipment and medium

Abstract

The invention relates to the technical field of computer vision, and provides a video portrait matting method and device, equipment and a medium, and the method comprises the steps: obtaining a video frame sequence; inputting the video frame sequence into a video portrait cutout model to obtain a video portrait cutout result output by the video portrait cutout model; wherein the video portrait matting model is obtained by training according to video frame sequence training data and a matting result corresponding to the video frame sequence training data; the video portrait cutout model is used for acquiring a feature map, a mask and a key point detection result of each frame of video image in the video frame sequence so as to generate a video portrait cutout result based on the acquired feature map, the mask and the key point detection result of each frame of video image. According to the method, the video frame sequence is acquired, so that rich inter-frame time sequence information is provided for the video portrait matting model based on the video images with continuous time sequences, consistent matting results are ensured to be generated in time, and the matting accuracy and matting quality are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, equipment and medium for cutting out a video portrait. Background Art

[0002] As an image processing technology, video portrait cutout is mainly used to extract the foreground of the character in the video, separate the foreground of the character from the background, and then perform other operations on the extracted character image, such as creating a new video.

[0003] Currently, the main method for video portrait matting is the End-to-End Video Matting with Transformer (VMFormer) algorithm based on Transformer. After extracting features through the backbone network, it obtains high-dimensional features through the Transformer encoder. It then uses the Transformer decoder and feature-based temporal modeling to output the matting results of multiple frames of data.

[0004] However, these algorithms are typically trained directly on video data, which is complex to annotate and relatively small in size, resulting in limited overall model performance. Furthermore, these algorithms utilize transformers for encoding and decoding, and the transformer's attention-based operations consume a large amount of video memory, resulting in relatively low video resolutions. For today's more common high-definition video, resolution may need to be reduced or a GPU with higher resources required, severely limiting their practical application. Summary of the Invention

[0005] The present invention provides a video portrait cutout method, device, equipment and medium to solve the defect of poor cutout effect of existing algorithms in the prior art. By providing rich inter-frame timing information for video portrait cutout based on time-series continuous video images, it ensures that the cutout results remain consistent in time, thereby improving the accuracy and quality of the cutout.

[0006] The present invention provides a video portrait cutout method, comprising: obtaining a video frame sequence; inputting the video frame sequence into a video portrait cutout model, and obtaining a video portrait cutout result output by the video portrait cutout model; wherein the video portrait cutout model is trained based on video frame sequence training data and the cutout results corresponding to the video frame sequence training data; the video portrait cutout model is used to obtain a feature map, a mask, and a key point detection result of each frame of video image in the video frame sequence, so as to generate a video portrait cutout result based on the obtained feature map, mask, and key point detection of each frame of video image.

[0007] According to a video portrait cutout method provided by the present invention, a video portrait cutout model includes a feature extraction layer, an image segmentation layer, a key point detection layer, a feature fusion layer and a cutout result generation layer; a video frame sequence is input into the video portrait cutout model to obtain a video portrait cutout result output by the video portrait cutout model, including: inputting the video frame sequence into the video portrait cutout model, and when it is determined that there are feature maps, masks and key point detection results of corresponding frame video images in the video frame sequence, obtaining feature maps, masks and key point detection results of corresponding frame video images; wherein the feature maps, masks and key point detection results of corresponding frame video images that exist are obtained and saved by the video portrait cutout model based on the corresponding frame video images in the previously obtained video frame sequence; when it is determined that there are no feature maps, masks and key point detection results of corresponding frame video images in the video frame sequence , input the corresponding frame video image into the feature extraction layer to extract features of the corresponding frame video image, obtain the feature map of the corresponding frame video image and save it; input the feature map of the corresponding frame video image into the image segmentation layer, and combine it with the preset character prompts to perform image segmentation on the corresponding frame video image, obtain the mask of the corresponding frame video image and save it; input the feature map of the corresponding frame video image into the key point detection layer to perform key point detection on the corresponding frame video image, obtain the key point detection result of the corresponding frame video image and save it; input the feature map, mask and key point detection result of each frame video image in the obtained video frame sequence into the feature fusion layer for feature fusion to obtain fusion features; input the fusion features into the cutout result generation layer to predict the transparency value of each pixel, so as to generate the video portrait cutout result according to the predicted transparency value of each pixel.

[0008] According to a video portrait cutout method provided by the present invention, the feature map, mask and key point detection results of each frame video image in the obtained video frame sequence are input into a feature fusion layer for feature fusion to obtain a fusion feature, including: inputting the feature map, mask and key point detection results of each frame video image in the obtained video frame sequence into the feature fusion layer to perform feature fusion on the feature map of each frame video image in the video frame sequence to obtain a feature map fusion feature; performing feature fusion on the mask of each frame video image in the video frame sequence to obtain a mask fusion feature; performing feature fusion on the key point detection results of each frame video image in the video frame sequence to obtain a key point fusion feature; and performing feature fusion on the feature map fusion feature, the mask fusion feature and the key point fusion feature to obtain a fusion feature.

[0009] According to the video portrait matting method provided by the application, the feature fusion is performed on the feature map fusion feature, the mask fusion feature and the key point fusion feature to obtain the fused feature, including: identifying the feature resolution of the feature map fusion feature, the mask fusion feature and the key point fusion feature, and selecting any one of the feature map fusion feature, the mask fusion feature and the key point fusion feature; when it is determined that there is another feature with a feature resolution lower than the selected feature, performing up-sampling on the other feature according to the feature resolution of the selected feature, and splicing the feature obtained by the up-sampling with the selected feature; wherein the other feature is used to represent the feature that is different from the selected feature among the feature map fusion feature, the mask fusion feature and the key point fusion feature; when it is determined that there is another feature with a feature resolution higher than the selected feature, performing down-sampling on the other feature according to the feature resolution of the selected feature, and splicing the feature obtained by the down-sampling with the selected feature; and when it is determined that there is another feature with a feature resolution equal to the selected feature, splicing the other feature with the selected feature.

[0010] According to the video portrait matting method provided by the application, before the video frame sequence is input into the video portrait matting model, the method includes: obtaining video frame sequence training data and a matting result corresponding to the video frame sequence training data; performing data enhancement on the video frame sequence training data based on a preset data enhancement strategy; wherein the preset data enhancement strategy includes at least one of rotation, scaling, translation and cropping; taking the data-enhanced video frame sequence training data as input data for training, taking the matting result corresponding to the video frame sequence training data as a label for training, training the to-be-trained model, and obtaining the video portrait matting model.

[0011] According to a video portrait cutout method provided by the present invention, before performing data enhancement on video frame sequence training data based on a preset data enhancement strategy, the method comprises: performing portrait detection on each frame training image in the video frame sequence training data to obtain a human body detection result of the corresponding frame training image; when determining that the preset data enhancement strategy includes translation, determining a translation parameter based on the human body detection result of each frame training image in combination with a preset translation constraint; wherein the preset translation constraint is used to limit the translation distance to be less than or equal to the distance of the portrait boundary relative to the frame of the corresponding frame training image along the corresponding translation direction; when determining that the preset data enhancement strategy includes cropping, determining a translation parameter based on the human body detection result of each frame training image in combination with a preset translation constraint; wherein the preset translation constraint is used to limit the translation distance to be less than or equal to the distance of the portrait boundary relative to the frame of the corresponding frame training image along the corresponding translation direction; The human detection results of each frame training image in the video frame sequence training data are used to determine the task boundary, and combined with the preset cropping constraints, the cropping size is determined to obtain the cropping parameters; wherein, the preset cropping constraints are used to limit the cropping size to be less than or equal to the distance of the task boundary relative to the border of the corresponding frame training image along the cropping direction; when determining that the preset data augmentation strategy includes rotation, according to the human detection results of each frame training image, combined with the preset rotation constraints, the rotation box is determined to obtain the rotation parameters; wherein, the preset rotation angle constraint is used to limit the maximum distance of the rotation center relative to the rotation box boundary to be less than or equal to the minimum distance of the rotation center relative to the border of the corresponding frame training image.

[0012] According to a video portrait cutout method provided by the present invention, a video frame sequence is obtained, including: obtaining a target frame video image to be cutout; based on the target frame video image, obtaining a preset number of video frames collected before the target frame video image and a preset number of video frames collected after the target frame video image, and combining the target frame video image to obtain a video frame sequence.

[0013] The present invention also provides a video portrait cutout device, comprising: a data acquisition module for acquiring a video frame sequence; a cutout module for inputting the video frame sequence into a video portrait cutout model to obtain a video portrait cutout result output by the video portrait cutout model; wherein the video portrait cutout model is trained based on video frame sequence training data and the cutout results corresponding to the video frame sequence training data; the video portrait cutout model is used to acquire feature maps, masks and key point detection results of each frame of video images in the video frame sequence, so as to generate a video portrait cutout result based on the acquired feature maps, masks and key point detection of each frame of video images.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-described methods for cutting out a video portrait is implemented.

[0015] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the video portrait matting method.

[0016] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the video portrait matting method.

[0017] The video portrait matting method, device, equipment and medium provided by the application can reduce the memory occupation by controlling the batch size of the input video frame sequence of the model, and the obtained video frame sequence is a time-continuous video image, which can provide rich inter-frame time sequence information for the video portrait matting model, help the model fully understand the motion trajectory and change, make the model generate a matting result that is consistent in time, ensure the time sequence consistency, avoid the flickering or jitter phenomenon, and provide multi-level information through the feature map, mask and key point detection result of each video image, so that the model can more comprehensively understand the image content, improve the matting accuracy and matting quality, and through the fusion of the multi-scale feature map, mask and key point detection result, the robustness of the model is enhanced, so that the model can better process various complex scenes and interference. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0019] Figure 1 is a flowchart of the video portrait matting method provided by the application; Figure 2 is an architecture diagram of the video portrait matting model provided by the application; Figure 3 is a structural diagram of the video portrait matting device provided by the application; Figure 4 is a structural diagram of the electronic equipment provided by the application. DETAILED DESCRIPTION

[0020] In order to make the objects, technical solutions and advantages of the application clearer, the technical solutions in the application will be described clearly and completely below with reference to the drawings in the application. Obviously, the described embodiments are some embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0021] Figure 1 : is a flow chart of the video portrait cutout method provided by the present invention, such as Figure 1 As shown, the method includes: S11, obtaining a video frame sequence; S12, inputting the video frame sequence into the video portrait cutout model to obtain the video portrait cutout result output by the video portrait cutout model; wherein, the video portrait cutout model is trained based on the video frame sequence training data and the cutout results corresponding to the video frame sequence training data; the video portrait cutout model is used to obtain the feature map, mask and key point detection results of each frame of the video image in the video frame sequence, so as to generate the video portrait cutout result based on the feature map, mask and key point detection of each frame of the video image obtained.

[0022] It should be noted that the step numbers "S1N" in this manual do not represent the order of the video portrait cutout method. Figure 2 The video portrait cutout method of the present invention is described.

[0023] Step S11, obtaining a video frame sequence.

[0024] In this embodiment, obtaining a video frame sequence includes: obtaining a target frame video image for portrait cutout; according to the target frame video image, obtaining a preset number of video frames collected before the target frame video image and a preset number of video frames collected after the target frame video image, and combining the target frame video image to obtain a video frame sequence.

[0025] It should be noted that the video frame sequence is updated in real time based on the target frame video image. The preset number can be set according to the length of the training data during actual training. For example, if the preset number is 2 frames, the target frame video image is represented by frame t, the preset number of video frames collected before the target frame video image are frame images t-1 and t-2, and the preset number of video frames collected after the target frame video image are frame images t+1 and t+2. The corresponding video frame sequence includes frame video images t-2, t-1, t, t+1, and t+2.

[0026] Correspondingly, the next target frame video image to be cut out for the portrait is the t+1 frame image, and the preset number of video frames collected before the t+1 target frame video image are the t and t-1 frame images, and the preset number of video frames collected after the t+1 target frame video image are the t+2 and t+3 frame images. The corresponding next video frame sequence includes the corresponding frame video images of t-1, t, t+1, t+2 and t+2.

[0027] In addition, images collected from the video can be collected based on a preset acquisition frequency, and when obtaining a video frame sequence, the video frames can be downsampled according to the video memory limitation of the model, that is, the resolution can be reduced to adapt to the video memory limitation.

[0028] Step S12, inputting the video frame sequence into the video portrait cutout model to obtain the video portrait cutout result output by the video portrait cutout model; wherein, the video portrait cutout model is trained based on the video frame sequence training data and the cutout results corresponding to the video frame sequence training data; the video portrait cutout model is used to obtain the feature map, mask and key point detection results of each frame of video image in the video frame sequence, so as to generate the video portrait cutout result based on the feature map, mask and key point detection of each frame of video image obtained.

[0029] In this embodiment, reference Figure 2 , the video portrait cutout model includes a feature extraction layer, an image segmentation layer, a key point detection layer, a feature fusion layer and a cutout result generation layer. Accordingly, the video frame sequence is input into the video portrait cutout model to obtain the video portrait cutout result output by the video portrait cutout model, including: inputting the video frame sequence into the video portrait cutout model, and when it is determined that there are feature maps, masks and key point detection results of the corresponding frame video images in the video frame sequence, the feature maps, masks and key point detection results of the corresponding frame video images are obtained and saved by the video portrait cutout model based on the corresponding frame video images in the video frame sequence obtained previously; when it is determined that there are no feature maps, masks and key point detection results of the corresponding frame video images in the video frame sequence, the corresponding frame video images are input into the feature extraction layer to cut out the corresponding frame video images. Feature extraction is performed on the video image to obtain a feature map of the corresponding frame video image; the feature map of the corresponding frame video image is input into the image segmentation layer, and combined with the preset character prompt, the corresponding frame video image is segmented to obtain the mask of the corresponding frame video image; the feature map of the corresponding frame video image is input into the key point detection layer to perform key point detection on the corresponding frame video image to obtain the key point detection result of the corresponding frame video image; the feature map, mask and key point detection result of each frame video image in the obtained video frame sequence are input into the feature fusion layer for feature fusion to obtain fusion features; the fusion features are input into the cutout result generation layer to predict the transparency value of each pixel to generate the video portrait cutout result according to the predicted transparency value of each pixel.

[0030] It should be noted that by storing features, the calculation of features is reduced, which greatly improves the reasoning speed of the overall model. In addition, after obtaining the feature map, mask and key points of the corresponding frame video image, it includes: saving the feature map, mask and key points of the corresponding frame video image to a preset data storage module; wherein the preset data storage module is previously created based on the video to be cut out of the portrait. Accordingly, when it is determined that there are feature maps, masks and key point detection results of the corresponding frame video image in the video frame sequence, the feature maps, masks and key points of the corresponding saved video image are directly obtained from the preset data storage module.

[0031] It should be noted that when the trained model is applied for the first time, since the model does not store the feature map, mask and key point detection results of the corresponding frame video image, after the video frame sequence is input into the video portrait cutout model, it is determined that there is no feature map, mask and key point detection result of any frame video image in the video frame sequence. Accordingly, the video frame sequence is input into the feature extraction layer to extract features of each frame video image in the video frame sequence, obtain the feature map of each frame video image and save it; input the feature map of each frame video image into the image segmentation layer, and combine it with the preset character prompts to respectively extract the features of the frame video image. Perform image segmentation on each frame of video image, obtain the mask of the corresponding frame of video image and save it; input the feature map of each frame of video image into the key point detection layer to perform key point detection on each frame of video image respectively, obtain the key point detection result of the corresponding frame of video image and save it; fuse the feature map of each frame of video image, the mask of each frame of video image and the key point detection result of each frame of video image in the feature fusion layer to obtain fused features; input the fused features into the cutout result generation layer to predict the transparency value of each pixel, and generate the video portrait cutout result according to the predicted transparency value of each pixel.

[0032] Specifically, the feature map of the corresponding frame video image is input into the image segmentation layer, and the corresponding frame video image is segmented in combination with the preset character prompt to obtain a mask of the corresponding frame video image, including: inputting the feature map of the corresponding frame video image into the image segmentation layer to determine the region of interest of the corresponding frame video image according to the feature map of the corresponding frame video image and the preset character prompt; identifying the feature map of the region of interest to generate a mask of the corresponding frame video image.

[0033] Furthermore, after generating the mask corresponding to the frame video image, the method further includes: filtering the mask of each frame video image by using non-maximum suppression (NMS) to filter out the mask with lower quality.

[0034] It should be noted that the image segmentation layer can adopt the lightweight image and video segmentation (SAM-Lightening) algorithm. By introducing the Dilated Flash Attention mechanism to improve parallel efficiency and using dynamic layer-by-layer distillation technology, it can efficiently transfer knowledge from the basic SAM model without the need for additional training, achieving a significant increase in inference speed and significantly reducing the requirements for memory and computing resources, making it more suitable for deployment on resource-constrained devices.

[0035] Additionally, the Keypoints Detection layer extracts keypoint information from portraits to enhance the model's focus on the human body, preventing missing body parts or fragments from being separated from other accompanying content. The Keypoints Detection layer uses a unified pose estimation (Unipose) model to detect keypoints for any object. In addition to individual video frames, it can also accept text or image prompts. This facilitates encoding different inputs during training, and then learning the relationships between them through a multimodal interactive encoder, making it adaptable to various detection scenarios.

[0036] In addition, the cutout result generation layer can use Mask-to-Matte, which is a technology that converts a rough mask (Mask) into a fine foreground mask (Alpha Matte).

[0037] In an optional embodiment, in order to improve the efficiency of fusion, the feature maps of each frame of video images, the masks of each frame of video images and the key point detection results of each frame of video images are feature fused to obtain fusion features, including: feature fusion of the feature maps of each frame of video images to obtain feature map fusion features; feature fusion of the masks of each frame of video images to obtain mask fusion features; feature fusion of the key point detection results of each frame of video images to obtain key point fusion features; feature fusion of the feature map fusion features, mask fusion features and key point fusion features to obtain fusion features.

[0038] Furthermore, the feature map fusion features, the mask fusion features and the key point fusion features are fused to obtain the fusion features, including: identifying the feature resolutions of the feature map fusion features, the mask fusion features and the key point fusion features, and selecting any one of the feature map fusion features, the mask fusion features and the key point fusion features; when it is determined that there are other features with lower feature resolutions than the selected features, the other features are upsampled according to the feature resolutions of the selected features, and the upsampled features are spliced ​​with the selected features; wherein the other features are used to characterize the features that are different from the selected features in the feature map fusion features, the mask fusion features and the key point fusion features; when it is determined that there are other features with higher feature resolutions than the selected features, the other features are downsampled according to the feature resolutions of the selected features, and the downsampled features are spliced ​​with the selected features; when it is determined that there are other features with the same feature resolutions as the selected features, the other features are spliced ​​with the selected features.

[0039] Furthermore, when it is determined that the feature resolutions of the feature map fusion features, mask fusion features, and keypoint fusion features are consistent, the feature map fusion features, mask fusion features, and keypoint fusion features are concatenated according to a preset order. It should be noted that the preset order can be set based on the actual ordering requirements of the feature map fusion features, mask fusion features, and keypoint fusion features, and is not further limited here.

[0040] For example, when the feature resolutions of feature map fusion features, mask fusion features, and key point fusion features are 512, 256, and 128, respectively, if the selected feature resolution is 512, the features of 256 and 128 are upsampled to a size of 512; if the selected feature resolution is 256, the features of 512 are downsampled to a size of 256, and the features of 128 are upsampled to a size of 256; if the selected feature resolution is 128, the features of 256 and 512 are downsampled to a size of 128, so that the feature map fusion features, mask fusion features, and key point fusion features are unified to the same size and then fused and spliced.

[0041] It should be added that upsampling can use deconvolution operation, and downsampling can use convolution operation. The specific settings can be made according to actual design requirements and are not further limited here.

[0042] In an optional embodiment, before inputting the video frame sequence into the video portrait cutout model, the method includes: obtaining video frame sequence training data and the cutout results corresponding to the video frame sequence training data; performing data enhancement on the video frame sequence training data based on a preset data enhancement strategy; wherein the preset data enhancement strategy includes at least one of rotation, scaling, translation and cropping; using the data-enhanced video frame sequence training data as input data for training, using the cutout results corresponding to the video frame sequence training data as labels for training, training the model to be trained, and obtaining a video portrait cutout model.

[0043] During model training, data augmentation strategies are used to increase dataset diversity, primarily through rotation, scaling, translation, and / or cropping. However, in video portrait cutout tasks, maintaining the integrity of the subject is crucial. If the original image contains the entire person, but only half of the body remains after cropping, or even if part of the face is cut out, the person's presence in the image is too small, which can easily lead to biased model learning. Therefore, before performing data augmentation, it is necessary to properly configure the data augmentation parameters to ensure the integrity of the portrait during data augmentation.

[0044] Specifically, before performing data enhancement on the video frame sequence training data based on the preset data enhancement strategy, the process includes: performing portrait detection on each frame training image in the video frame sequence training data to obtain the human body detection result of the corresponding frame training image; when determining that the preset data enhancement strategy includes translation, determining the translation parameters based on the human body detection results of each frame training image in combination with the preset translation constraint; wherein the preset translation constraint is used to limit the translation distance to be less than or equal to the distance of the portrait boundary relative to the corresponding frame training image border along the corresponding translation direction; when determining that the preset data enhancement strategy includes cropping, determining the human body detection results of each frame training image based on the human body detection results of each frame training image , determine the task boundary of each frame training image in the video frame sequence training data, and determine the cropping size in combination with the preset cropping constraint to obtain the cropping parameters; wherein, the preset cropping constraint is used to limit the cropping size to be less than or equal to the distance of the task boundary relative to the border of the corresponding frame training image along the cropping direction; when determining that the preset data augmentation strategy includes rotation, determine the rotation box according to the human body detection results of each frame training image in combination with the preset rotation constraint to obtain the rotation parameters; wherein, the preset rotation angle constraint is used to limit the maximum distance of the rotation center relative to the rotation box boundary to be less than or equal to the minimum distance of the rotation center relative to the border of the corresponding frame training image.

[0045] It is worth noting that the trained model uses multiple frames of data to predict a single frame. Due to the sequential nature of the video frame sequence training data, feature calculations (masks and keypoints) may be repeated. For example, assuming the input video frame sequence training data consists of five frames, to calculate the cutout result for frame t, the feature maps, masks, and keypoints for frames t-2, t-1, t, t+1, and t+2 must be calculated. Subsequently, to calculate the cutout result for frame t+1, the features for frames t-1, t, t+1, t+2, and t+3 must be calculated. If the cutout results for frame t and frame t+1 are calculated separately, the features for frames t-1, t, t+1, and t+2 will be repeated. Therefore, to avoid repeated calculations, during the training of the trained model, the feature maps, masks, and keypoints for each frame of video image obtained for the video frame sequence training data are saved to avoid repeated calculations for the next video frame sequence training data that involves the same frame of video image from the previous video frame sequence training data.

[0046] For example, when calculating the t+1 frame, the previously saved features are for t-2, t-1, t, t+1, and t+2. At this time, it is only necessary to calculate the feature map, mask, and key points of the newly added t+3 frame video image, and update the features to the features of the images at t-1, t, t+1, t+2, and t+3, thereby reducing the feature calculation to the same number of video frames.

[0047] In summary, the embodiments of the present invention reduce the video memory usage by controlling the batch size of the video frame sequence of the input model, and the obtained video frame sequence is a time-continuous video image, which can provide rich inter-frame timing information for the video portrait cutout model, help the model fully understand the motion trajectory and changes, enable the model to generate temporally consistent cutout results, ensure timing consistency, avoid flickering or jittering, and provide multi-level information through the feature map, mask and key point detection results of each frame of video image, so that the model can understand the image content more comprehensively, improve the cutout accuracy and quality, and enhance the robustness of the model through the fusion of multi-scale feature maps, masks and key point detection results, so that it can better handle various complex scenes and interferences.

[0048] The video portrait cutout device provided by the present invention is described below. The video portrait cutout device described below and the video portrait cutout method described above can be referenced to each other.

[0049] Figure 3 The following is a schematic diagram of the structure of a video portrait cutout device, which includes: The data acquisition module 31 acquires a video frame sequence; The matting module 32 inputs the video frame sequence into a video portrait matting model to obtain a video portrait matting result output by the video portrait matting model; wherein the video portrait matting model is trained according to video frame sequence training data and a matting result corresponding to the video frame sequence training data; the video portrait matting model is used to obtain a feature map, a mask and a key point detection result of each frame of video image in the video frame sequence, so as to generate a video portrait matting result based on the obtained feature map, mask and key point detection result of each frame of video image.

[0050] In the embodiment, the data acquisition module 31 comprises: a target frame acquisition unit that acquires a target frame video image to be subjected to portrait matting; a video frame acquisition unit that acquires a preset number of video frames collected before the target frame video image and a preset number of video frames collected after the target frame video image according to the target frame video image, and combines the target frame video image to obtain a video frame sequence.

[0051] In addition, the matting module 32 comprises: a stored data acquisition unit that inputs the video frame sequence into a video portrait matting model, and acquires a feature map, a mask and a key point detection result of a corresponding frame of video image when the corresponding frame of video image exists in the video frame sequence; wherein the existing feature map, mask and key point detection result of the corresponding frame of video image are obtained and saved by the video portrait matting model based on the corresponding frame of video image in the video frame sequence acquired in advance; a feature extraction unit that inputs the corresponding frame of video image into a feature extraction layer to extract features of the corresponding frame of video image and save the feature map of the corresponding frame of video image when the feature map, mask and key point detection result of the corresponding frame of video image do not exist in the video frame sequence; an image segmentation unit that inputs the feature map of the corresponding frame of video image into an image segmentation layer, and combines a preset character prompt to segment the corresponding frame of video image to obtain the mask of the corresponding frame of video image and save the mask; a key point detection unit that inputs the feature map of the corresponding frame of video image into a key point detection layer to detect key points of the corresponding frame of video image to obtain the key point detection result of the corresponding frame of video image and save the key point detection result; a feature fusion unit that inputs the obtained feature map, mask and key point detection result of each frame of video image in the video frame sequence into a feature fusion layer to perform feature fusion to obtain a fusion feature; and a matting result generation unit that inputs the fusion feature into a matting result generation layer to predict a transparency value of each pixel, so as to generate a video portrait matting result according to the predicted transparency value of each pixel.

[0052] It should be noted that the cutout module 32 further includes a feature storage unit that, after obtaining the feature map, mask, and key points of the corresponding frame of video image, stores the feature map, mask, and key points of the corresponding frame of video image in a preset data storage module. The preset data storage module is previously created based on the video of the person to be cutout. Accordingly, upon determining that a feature map, mask, and key point detection result for the corresponding frame of video image exists in the video frame sequence, the feature map, mask, and key points of the corresponding stored video image are directly retrieved from the preset data storage module.

[0053] Specifically, the image segmentation unit is used to: input the feature map of the corresponding frame video image into the image segmentation layer to determine the region of interest of the corresponding frame video image based on the feature map of the corresponding frame video image and the preset character prompt; identify the feature map of the region of interest and generate a mask of the corresponding frame video image.

[0054] Furthermore, the image segmentation unit is further configured to: after generating the mask corresponding to the frame video image, filter the mask of each frame video image by using non-maximum suppression (NMS) to filter out the mask of lower quality.

[0055] In an optional embodiment, in order to improve the efficiency of fusion, the feature fusion unit is used to: perform feature fusion on the feature maps of each frame of video image to obtain feature map fusion features; perform feature fusion on the masks of each frame of video image to obtain mask fusion features; perform feature fusion on the key point detection results of each frame of video image to obtain key point fusion features; perform feature fusion on the feature map fusion features, mask fusion features and key point fusion features to obtain fusion features.

[0056] Furthermore, the feature fusion unit includes: a resolution identification subunit, which identifies the feature resolution of the feature map fusion feature, the mask fusion feature and the key point fusion feature, and selects any one of the feature map fusion feature, the mask fusion feature and the key point fusion feature; the feature fusion subunit is used to, when it is determined that there are other features with a feature resolution lower than the selected feature, upsample the other features according to the feature resolution of the selected feature, and splice the upsampled features with the selected features; wherein the other features are used to characterize the features that are different from the selected features in the feature map fusion feature, the mask fusion feature and the key point fusion feature; when it is determined that there are other features with a feature resolution higher than the selected feature, downsample the other features according to the feature resolution of the selected feature, and splice the downsampled features with the selected features; when it is determined that there are other features with a feature resolution equal to the selected feature, splice the other features with the selected features.

[0057] In an optional embodiment, the apparatus further comprises: a training sample obtaining module, configured to obtain video frame sequence training data and a matting result corresponding to the video frame sequence training data before inputting the video frame sequence into the video portrait matting model; a data enhancement module, configured to perform data enhancement on the video frame sequence training data based on a preset data enhancement strategy; wherein the preset data enhancement strategy comprises at least one of rotation, scaling, translation and cropping; and a training module, configured to perform training on the to-be-trained model by taking the data-enhanced video frame sequence training data as input data for training and taking the matting result corresponding to the video frame sequence training data as a label for training, to obtain the video portrait matting model.

[0058] Specifically, the apparatus further comprises: a parameter configuration module, configured to perform portrait detection on each frame training image in the video frame sequence training data to obtain a human body detection result of the corresponding frame training image before performing data enhancement on the video frame sequence training data based on the preset data enhancement strategy; when the preset data enhancement strategy comprises translation, the parameter configuration module is further configured to determine a translation parameter according to the human body detection result of each frame training image in combination with a preset translation constraint; wherein the preset translation constraint is configured to limit a translation distance to be less than or equal to a distance of a portrait boundary in a corresponding translation direction relative to a frame boundary of a corresponding frame training image; when the preset data enhancement strategy comprises cropping, the parameter configuration module is further configured to determine a task boundary of each frame training image in the video frame sequence training data according to the human body detection result of each frame training image, and determine a cropping size in combination with a preset cropping constraint to obtain a cropping parameter; wherein the preset cropping constraint is configured to limit the cropping size to be less than or equal to a distance of a task boundary in a cropping direction relative to the frame boundary of the corresponding frame training image; and when the preset data enhancement strategy comprises rotation, the parameter configuration module is further configured to determine a rotation box in combination with a preset rotation constraint according to the human body detection result of each frame training image to obtain a rotation parameter; wherein the preset rotation angle constraint is configured to limit a maximum distance of a rotation center relative to a boundary of the rotation box to be less than or equal to a minimum distance of the rotation center relative to the frame boundary of the corresponding frame training image.

[0059] In addition, the apparatus further comprises: a feature saving module, configured to save the feature map, the mask and the key points of each corresponding frame video image obtained for the video frame sequence training data each time.

[0060] In summary, the embodiments of the present invention reduce the video memory usage by controlling the batch size of the video frame sequence of the input model, and the obtained video frame sequence is a time-continuous video image, which can provide rich inter-frame timing information for the video portrait cutout model, help the model fully understand the motion trajectory and changes, enable the model to generate temporally consistent cutout results, ensure timing consistency, avoid flickering or jittering, and provide multi-level information through the feature map, mask and key point detection results of each frame of video image, so that the model can understand the image content more comprehensively, improve the cutout accuracy and quality, and enhance the robustness of the model through the fusion of multi-scale feature maps, masks and key point detection results, so that it can better handle various complex scenes and interferences.

[0061] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communications bus 440. The processor 410 may call logic instructions in the memory 430 to execute a video portrait cutout method, which includes: obtaining a video frame sequence; inputting the video frame sequence into a video portrait cutout model to obtain a video portrait cutout result output by the video portrait cutout model; wherein the video portrait cutout model is trained based on video frame sequence training data and the cutout results corresponding to the video frame sequence training data; the video portrait cutout model is used to obtain a feature map, a mask, and a key point detection result of each frame of the video image in the video frame sequence, so as to generate a video portrait cutout result based on the obtained feature map, mask, and key point detection of each frame of the video image.

[0062] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0063] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video portrait cutout method provided by the above methods, which includes: obtaining a video frame sequence; inputting the video frame sequence into a video portrait cutout model to obtain a video portrait cutout result output by the video portrait cutout model; wherein the video portrait cutout model is trained based on video frame sequence training data and the cutout results corresponding to the video frame sequence training data; the video portrait cutout model is used to obtain the feature map, mask and key point detection results of each frame of video image in the video frame sequence, so as to generate a video portrait cutout result based on the feature map, mask and key point detection of each frame of video image obtained.

[0064] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the video portrait cutout method provided by the above-mentioned methods, the method comprising: obtaining a video frame sequence; inputting the video frame sequence into a video portrait cutout model to obtain a video portrait cutout result output by the video portrait cutout model; wherein the video portrait cutout model is trained based on video frame sequence training data and the cutout results corresponding to the video frame sequence training data; the video portrait cutout model is used to obtain feature maps, masks and key point detection results of each frame of video images in the video frame sequence, so as to generate a video portrait cutout result based on the feature maps, masks and key point detection of each frame of video images obtained.

[0065] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0066] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for cutting out a portrait from a video, characterized in that: include: Get video frame sequence; Inputting the video frame sequence into a video portrait cutout model to obtain a video portrait cutout result output by the video portrait cutout model; wherein the video portrait cutout model is trained based on video frame sequence training data and the cutout results corresponding to the video frame sequence training data; The video portrait cutout model is used to obtain the feature map, mask and key point detection results of each frame of video image in the video frame sequence, so as to generate a video portrait cutout result based on the obtained feature map, mask and key point detection of each frame of video image.

2. The video portrait cutout method according to claim 1, wherein: The video portrait cutout model includes a feature extraction layer, an image segmentation layer, a key point detection layer, a feature fusion layer and a cutout result generation layer; Inputting the video frame sequence into a video portrait cutout model to obtain a video portrait cutout result output by the video portrait cutout model includes: Inputting the video frame sequence into the video portrait cutout model, and when determining that a feature map, mask, and key point detection result of a corresponding frame video image in the video frame sequence exists, obtaining the feature map, mask, and key point detection result of the corresponding frame video image; wherein the existing feature map, mask, and key point detection result of the corresponding frame video image are obtained and stored by the video portrait cutout model based on the corresponding frame video image in the previously obtained video frame sequence; When it is determined that there is no feature map, mask, and key point detection result of the corresponding frame video image in the video frame sequence, the corresponding frame video image is input into the feature extraction layer to perform feature extraction on the corresponding frame video image to obtain a feature map of the corresponding frame video image; Inputting the feature map of the corresponding frame video image into the image segmentation layer, and performing image segmentation on the corresponding frame video image in combination with a preset character prompt to obtain a mask of the corresponding frame video image; Inputting the feature map of the corresponding frame video image into the key point detection layer to perform key point detection on the corresponding frame video image to obtain a key point detection result of the corresponding frame video image; Inputting the obtained feature map, mask and key point detection results of each frame of the video image in the video frame sequence into the feature fusion layer for feature fusion to obtain fusion features; The fusion feature is input into the cutout result generation layer to predict the transparency value of each pixel, so as to generate a video portrait cutout result according to the predicted transparency value of each pixel.

3. The video portrait cutout method according to claim 2, characterized in that: Inputting the obtained feature map, mask and key point detection results of each frame of the video image in the video frame sequence into the feature fusion layer for feature fusion to obtain fusion features, including: Inputting the obtained feature map, mask and key point detection results of each frame of the video image in the video frame sequence into the feature fusion layer to perform feature fusion on the feature map of each frame of the video image in the video frame sequence to obtain feature map fusion features; Performing feature fusion on the masks of the video images of each frame in the video frame sequence to obtain mask fusion features; Performing feature fusion on key point detection results of each frame of video image in the video frame sequence to obtain key point fusion features; The feature map fusion feature, the mask fusion feature and the key point fusion feature are fused to obtain a fusion feature.

4. The video portrait cutout method according to claim 3, characterized in that: The feature map fusion feature, the mask fusion feature and the key point fusion feature are fused to obtain a fusion feature, including: Identifying feature resolutions of the feature map fusion feature, the mask fusion feature, and the key point fusion feature, and selecting any one of the feature map fusion feature, the mask fusion feature, and the key point fusion feature; When it is determined that there are other features having a feature resolution lower than that of the selected feature, upsampling the other features according to the feature resolution of the selected feature, and concatenating the upsampled features with the selected feature; wherein the other features are used to represent features that are different from the selected feature in the feature map fusion feature, the mask fusion feature, and the key point fusion feature; When it is determined that there are other features having a feature resolution higher than the selected feature, down-sampling the other features according to the feature resolution of the selected feature, and concatenating the down-sampled features with the selected feature; When it is determined that there are other features having a feature resolution equal to that of the selected feature, the other features are spliced ​​with the selected feature.

5. The video portrait cutout method according to claim 1, characterized in that: Before inputting the video frame sequence into the video portrait cutout model, the process includes: Obtaining video frame sequence training data and a cutout result corresponding to the video frame sequence training data; Based on a preset data enhancement strategy, data enhancement is performed on the video frame sequence training data; wherein the preset data enhancement strategy includes at least one of rotation, scaling, translation and cropping; The data-enhanced video frame sequence training data is used as input data for training, and the cutout results corresponding to the video frame sequence training data are used as labels for training. The training model is trained to obtain a video portrait cutout model.

6. The video portrait cutout method according to claim 5, characterized in that: Before performing data enhancement on the video frame sequence training data based on a preset data enhancement strategy, the method includes: Performing human portrait detection on each frame training image in the video frame sequence training data to obtain a human body detection result for the corresponding frame training image; When it is determined that the preset data augmentation strategy includes translation, a translation parameter is determined based on the human body detection results of each frame of the training image in combination with a preset translation constraint; wherein the preset translation constraint is used to limit the translation distance to be less than or equal to the distance between the portrait boundary and the border of the corresponding frame training image along the corresponding translation direction; When determining that the preset data augmentation strategy includes cropping, determining the task boundary of each training image frame in the video frame sequence training data based on the human body detection results of each training image frame, and determining the cropping size in combination with the preset cropping constraint to obtain the cropping parameter; wherein the preset cropping constraint is used to limit the cropping size to be less than or equal to the distance of the task boundary relative to the border of the corresponding training image frame along the cropping direction; When it is determined that the preset data enhancement strategy includes rotation, the rotation box is determined based on the human body detection results of the training images of each frame and combined with the preset rotation constraints to obtain rotation parameters; wherein the preset rotation angle constraint is used to limit the maximum distance of the rotation center relative to the boundary of the rotation box to be less than or equal to the minimum distance of the rotation center relative to the border of the corresponding frame training image.

7. The video portrait cutout method according to claim 1, characterized in that: Get a sequence of video frames, including: Obtain the target frame video image for portrait cutout; According to the target frame video image, a preset number of video frames collected before the target frame video image and a preset number of video frames collected after the target frame video image are obtained, and combined with the target frame video image to obtain a video frame sequence.

8. A video portrait cutout device, characterized in that: include: Data acquisition module, acquiring video frame sequence; a cutout module, which inputs the video frame sequence into a video portrait cutout model to obtain a video portrait cutout result output by the video portrait cutout model; wherein the video portrait cutout model is trained based on video frame sequence training data and the cutout results corresponding to the video frame sequence training data; The video portrait cutout model is used to obtain the feature map, mask and key point detection results of each frame of video image in the video frame sequence, so as to generate a video portrait cutout result based on the obtained feature map, mask and key point detection of each frame of video image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the video portrait cutout method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video portrait cutout method according to any one of claims 1 to 7 is implemented.

Images