Adaptive video matting method and system based on deep learning

Abstract

The invention discloses a self-adaptive video matting method and system based on deep learning. The method comprises the following steps: S1, inputting a video; s2, decoding a frame of picture; s3, carrying out rough matting on the decoded picture, and reserving a part of possibly interested objects in the picture; s4, object segmentation is carried out on the picture after rough matting, and the picture is divided into several independent areas; s5, if the frame is the first frame, continuing processing through S6; otherwise, skipping to S10; s6, if the user sets the interested object area, continuing S7; otherwise, jumping to S8; s7, finding a region which has the maximum intersection with the set region and the minimum non-intersection with the set region, and marking the region as an interested object region; and skipping to S9; s8, finding an independent region with the minimum middle position of the picture, and marking the minimum independent region as an interested object region; s9, initializing an object tracker by using the interested object area of the first frame; skipping to S12; and S10, obtaining the specific position of the interested object in the frame.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular relates to an adaptive video cutout method and system based on deep learning. Background Art

[0002] Current video clip Figure 1 Generally, the image cutout method is used, that is, decoding the video into frames of pictures, and cutting out the object of interest in each frame. This method does not take advantage of the correlation of the video, resulting in multiple complete non-interesting objects being retained in some pictures. In some cases, it cannot be used directly and wastes manpower to process, such as using videos or continuous pictures to build mesh scenes. In addition, some deep learning-based video cutouts are for human body cutouts, which are not suitable for object cutouts. Summary of the invention

[0003] In view of the above problems, the present invention provides an adaptive video cutout method and system based on deep learning, which is used to make full use of the correlation between video frames and the characteristics of each deep learning model, so that only objects of interest are left in each frame, which can greatly improve the accuracy of video cutout and significantly reduce manual operations.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] On the one hand, the present invention provides an adaptive video cutout method based on deep learning, comprising the following steps:

[0006] S1, video input;

[0007] S2, decodes a frame of picture;

[0008] S3, uses an open source deep learning cutout network to roughly cut out the decoded image, retaining some objects of interest in the image;

[0009] S4, performing object segmentation on the roughly cut-out image, dividing the image into several independent regions;

[0010] S5, if it is the first frame, continue processing through S6; otherwise jump to S10;

[0011] S6, if the user sets the object area of ​​interest, continue to S7; otherwise jump to S8;

[0012] S7, find the area with the largest intersection and the smallest non-intersection with the set area, mark this area as the object of interest area; and jump to S9;

[0013] S8, find the smallest independent area in the middle of the image, and mark the smallest independent area as the object of interest area;

[0014] S9, using the object region of interest in the first frame, initialize the object tracker; jump to S12;

[0015] S10, tracking the object of interest through an object tracker to obtain a specific position of the object of interest in this frame;

[0016] S11, finding a region with the largest intersection and the smallest non-intersection with the specific position of the object of interest, and setting this region as the object of interest region;

[0017] S12, removal of independent regions of non-interest;

[0018] S13, using another deep learning image cutout network to remove the non-interesting area overlapping with the object of interest; and outputting the cutout image;

[0019] S14, determine whether all frames of the video have been cut out, if so, continue to S15; otherwise jump to S2;

[0020] S15, video cutout is completed.

[0021] In a possible implementation, the image is divided into several independent regions in S4 using a grayscale image threshold binary method.

[0022] On the other hand, the present invention provides an adaptive video cutout system based on deep learning, including a video decoder, a coarse cutout module, an object tracker, an object segmenter and a coarse cutout module, wherein the video decoder decodes the input video into frames of pictures; the coarse cutout module uses a deep learning method to roughly cut out each frame, retaining some objects of possible interest in the frame; the object tracker is used to locate the specific position of the object of interest in each frame; the object segmenter is used to divide the result of the coarse cutout into several independent areas; the fine cutout module is used to remove the independent areas that do not contain the object of interest according to the result of the object tracker, and use another deep learning method to cut out the part overlapping with the object of interest, retaining only the object of interest.

[0023] In a possible implementation, the object segmenter is used to divide the rough cutout result into several independent regions by using a grayscale image threshold binary division method.

[0024] In a possible implementation, after the rough cutout and before the fine cutout, it is necessary to determine whether it is the first frame. If so, it is further determined whether the user has set an object of interest region. Otherwise, the object of interest is tracked by an object tracker to obtain the specific position of the object of interest in this frame. The region with the largest intersection and the smallest non-intersection with the specific position of the object of interest is found, and this region is set as the object of interest region.

[0025] If the user sets an object of interest region, find the region with the largest intersection and the smallest non-intersection with the set region, mark this region as the object of interest region, and use the object of interest region of the first frame to initialize the object tracker; otherwise find the smallest independent region in the middle of the picture, mark this smallest independent region as the object of interest region, and use the object of interest region of the first frame to initialize the object tracker;

[0026] Finally, independent regions of non-interest are removed.

[0027] The present invention has the following beneficial effects: by making full use of the correlation between video frames and the characteristics of each deep learning model, only the object of interest is left in each frame, which can greatly improve the accuracy of video cutout and greatly reduce manual operation. The accuracy of video cutout is greatly improved. For the scene of building Mesh, no later manual operation is required, and the quality of the built Mesh is perfectly improved. In addition, this algorithm can also be applied to other scenes that require video cutout, such as video synthesis, etc., and can also be used for continuous picture sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flowchart of the steps of the deep learning-based adaptive video cutout method according to an embodiment of the present invention;

[0029] Figure 2 4 is a principle block diagram of an adaptive video cutout system based on deep learning according to an embodiment of the present invention;

[0030] Figure 3 This is a cutout effect diagram using the prior art method in a specific application example;

[0031] Figure 4 The figure is a cutout effect diagram of a specific application example using the method of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Reference Figure 1 , which is a flowchart of a method for adaptive video cutout based on deep learning according to an embodiment of the present invention, comprising the following steps:

[0034] S1, video input;

[0035] S2, decodes a frame of picture;

[0036] S3, uses an open source deep learning cutout network to roughly cut out the decoded image, retaining some objects of interest in the image;

[0037] S4, performing object segmentation on the roughly cut-out image, dividing the image into several independent regions;

[0038] S5, determine whether it is the first frame, if so, continue to process through S6; otherwise jump to S10;

[0039] S6, determine whether the user has set the object of interest area, if yes, proceed to S7; otherwise jump to S8;

[0040] S7, find the area with the largest intersection and the smallest non-intersection with the set area, mark this area as the object of interest area; and jump to S9;

[0041] S8, find the smallest independent area in the middle of the image, and mark the smallest independent area as the object of interest area;

[0042] S9, using the object region of interest in the first frame, initialize the object tracker; jump to S12;

[0043] S10, tracking the object of interest through an object tracker to obtain a specific position of the object of interest in this frame;

[0044] S11, finding a region with the largest intersection and the smallest non-intersection with the specific position of the object of interest, and setting this region as the object of interest region;

[0045] S12, removal of independent regions of non-interest;

[0046] S13, using another deep learning image cutout network to remove the non-interesting area overlapping with the object of interest; and outputting the cutout image;

[0047] S14, determine whether all frames of the video have been cut out, if so, continue to S15; otherwise jump to S2;

[0048] S15, video cutout is completed.

[0049] Specifically, the deep learning cutout network uses open source networks, such as transparent-background or removebg. Other open source image cutout networks can also be used. They are cutout networks based on deep learning. After rough cutout, the image is left with one or several relatively independent objects and black areas, so the image is divided into several independent areas using grayscale threshold binary method. The physical tracker can use the CSRT algorithm in opencv.

[0050] For method embodiments, see Figure 2 , shown is a principle block diagram of an adaptive video cutout system based on deep learning according to another embodiment of the present invention, comprising a video decoder 10, a coarse cutout module 20, an object tracker 30, an object segmenter 40 and a coarse cutout module 50, wherein the video decoder 10 decodes the input video into frames of images; the coarse cutout module 20 uses a deep learning method to roughly cut out each frame of the image, retaining some objects of possible interest in the frame; the object tracker 40 is used to locate the specific position of the object of interest in each frame; the object segmenter is used to divide the result of the coarse cutout 30 into several independent areas; the fine cutout module 40 is used to remove the independent areas that do not contain the object of interest according to the result of the object tracker, and use another deep learning method to cut out the part overlapping with the object of interest, and only retain the object of interest.

[0051] Specifically, after the rough cutout and before the fine cutout, it is necessary to determine whether it is the first frame. If so, it is further determined whether the user has set the object of interest region; otherwise, the object of interest is tracked by the object tracker to obtain the specific position of the object of interest in this frame; the region with the largest intersection and the smallest non-intersection with the specific position of the object of interest is found, and this region is set as the object of interest region;

[0052] If the user sets an object of interest region, find the region with the largest intersection and the smallest non-intersection with the set region, mark this region as the object of interest region, and use the object of interest region of the first frame to initialize the object tracker; otherwise find the smallest independent region in the middle of the picture, mark this smallest independent region as the object of interest region, and use the object of interest region of the first frame to initialize the object tracker;

[0053] Finally, independent regions of non-interest are removed.

[0054] Specifically, both the coarse cutout module and the fine cutout module can use open source deep learning cutout networks, such as transparent-background or removebg, or other open source image cutout networks. After coarse cutout, the image is left with one or several relatively independent objects and black areas, so the image is divided into several independent areas using grayscale threshold binary division. The physical tracker can use the CSRT algorithm in opencv.

[0055] Figure 3 and Figure 4 The result diagrams respectively display the result of applying the existing cutout method and the method of the embodiment of the present invention, which greatly improves the accuracy of video cutout and only retains the area of ​​interest to the user.

[0056] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. An adaptive video cutout method based on deep learning, characterized in that: The following steps are involved: S1, video input; S2, decodes a frame of picture; S3, uses an open source deep learning cutout network to roughly cut out the decoded image, retaining some objects of interest in the image; S4, performing object segmentation on the roughly cut-out image, dividing the image into several independent regions; S5, if it is the first frame, continue processing through S6; otherwise jump to S10; S6, if the user sets the object area of ​​interest, continue to S7; otherwise jump to S8; S7, find the area with the largest intersection and the smallest non-intersection with the set area, mark this area as the object of interest area; and jump to S9; S8, find the smallest independent area in the middle of the image, and mark the smallest independent area as the object of interest area; S9, using the object region of interest in the first frame, initialize the object tracker; jump to S12; S10, tracking the object of interest through an object tracker to obtain a specific position of the object of interest in this frame; S11, finding a region with the largest intersection and the smallest non-intersection with the specific position of the object of interest, and setting this region as the object of interest region; S12, removal of independent regions of non-interest; S13, using another deep learning image cutout network to remove the non-interesting area overlapping with the object of interest; and outputting the cutout image; S14, determine whether all frames of the video have been cut out, if so, continue to S15; otherwise jump to S2; S15, video cutout is completed.

2. The deep learning-based adaptive video cutout method according to claim 1, characterized in that: In S4, the image is divided into several independent regions using the grayscale image threshold binary method.

3. An adaptive video cutout system based on deep learning, characterized in that: The system comprises a video decoder, a rough cutout module, an object tracker, an object segmenter and a rough cutout module. The video decoder decodes the input video into frames of pictures; the rough cutout module uses a deep learning method to roughly cut out each frame, retaining some objects of possible interest in the frame; the object tracker is used to locate the specific position of the object of interest in each frame; the object segmenter is used to divide the result of the rough cutout into several independent areas; The fine-cutting module is used to remove independent areas that do not contain the object of interest based on the results of the object tracker, and use another deep learning method to remove the parts overlapping with the object of interest, so as to only retain the object of interest.

4. The deep learning-based adaptive video cutout system according to claim 3, characterized in that: The object segmenter is used to divide the rough cutout result into several independent areas by using the grayscale image threshold binary method.

5. The deep learning-based adaptive video cutout system according to claim 3 or 4, characterized in that: After the rough cutout and before the fine cutout, it is necessary to determine whether it is the first frame. If so, it is necessary to determine whether the user has set the region of interest. Otherwise, the object of interest is tracked by the object tracker to obtain the specific position of the object of interest in this frame. The region with the largest intersection and the smallest non-intersection with the specific position of the object of interest is found, and this region is set as the region of interest. If the user sets an object of interest region, find the region with the largest intersection and the smallest non-intersection with the set region, mark this region as the object of interest region, and use the object of interest region of the first frame to initialize the object tracker; Otherwise, find the smallest independent area in the middle of the image, mark this smallest independent area as the object of interest area, and use the object of interest area of ​​the first frame to initialize the object tracker; Finally, independent regions of non-interest are removed.

Images