A method for training a matting network and a matting method

Abstract

The present application relates to the field of computer vision technology, and in particular, to a matte network training method and a matte method. The method includes: obtaining a training sample set and an initial network; the training sample set includes a plurality of training samples, each training sample includes an input image sample, the input image sample includes a to-be-matted image sample with a foreground, a background image sample, and a soft segmentation sample of the foreground, the soft segmentation sample is generated by subtracting the depth image corresponding to the background image sample from the depth image corresponding to the to-be-matted image sample; the initial network includes at least one stage network; the stage network includes a cascaded environment combination module, a backbone block, and a prediction module; using the training sample set, training the initial network to obtain a matte network. The embodiments of the present application adopt a lightweight model and achieve high-precision foreground segmentation.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular, to a matte network training method and a matte method. Background Art

[0002] In the field of computer vision technology, matting is a commonly used processing method.

[0003] In traditional matting methods, the static image matting algorithm (trimap) is widely used to guide color feature extraction. The color features of the foreground and background are used to constrain the transition region, thereby determining the segmentation of the foreground. According to the way of using color features, traditional matting methods can be divided into two categories: sampling-based methods and similarity-based methods. Sampling-based methods use a pair of foreground or background pixels to represent the pixels in the transition region to obtain the foreground segmentation. Similarity-based methods determine the foreground boundary through the similarity between certain labels and adjacent pixels in the transition region. Neither of these two matting methods involves semantic information, and the computational complexity is relatively large. Moreover, when the foreground and background have similar color features, the prediction effects of these two matting methods will deteriorate.

[0004] With the development of deep learning, it has greatly promoted the development of matting algorithms. In deep learning-based matting algorithms, trimap-based methods have been widely studied. Although trimap-based methods have high accuracy, they require manual annotation of the given image to add additional constraints to the matting problem. This manual annotation method is very unfriendly to users, so its practicability is poor; in addition, the computational complexity is relatively large.

[0005] In recent years, trimap-free matting methods have received more attention. The mainstream solutions in the past two years are to directly predict the foreground segmentation using a single RGB image. However, such solutions have a relatively large computational complexity, and their accuracy does not exceed that of trimap-based methods. In addition, they are also sensitive to the scene, and the generalization ability still needs to be improved. Especially when the input contains unknown objects or multiple foregrounds, the performance of the network will deteriorate.

[0006] In order to balance the advantages and disadvantages of trimap-based methods and trimap-free methods, currently, a foreground soft segmentation of a background image and a portrait is used to replace trimap as the prior of the background segmentation algorithm. This method has a large computational complexity, is slow, and the effect will deteriorate when dealing with people holding objects or non-human scenes. Summary of the Invention

[0007] In view of this, the embodiments of this application provide a matte network training method and a matte method, which can solve at least one technical problem in the related art.

[0008] In a first aspect, an embodiment of the present application provides a matte extraction network training method, including:

[0009] Obtain a training sample set and an initial network; the training sample set includes a plurality of training samples, each training sample includes an input image sample, the input image sample includes a matte image sample with a foreground, a background image sample, and a soft segmentation sample of the foreground, and the soft segmentation sample is generated by subtracting the depth image corresponding to the background image sample from the depth image corresponding to the matte image sample; the initial network includes at least one stage network; the stage network includes an environment combination module, a backbone block, and a prediction module connected in series, the input image sample is input into the environment combination module, the environment combination module is used to output low-level features and high-level features after feature exchange, the backbone block is used to fuse the low-level features and the high-level features based on an attention mechanism to obtain a fused feature, and the prediction module is used to output a predicted foreground segmentation according to the fused feature;

[0010] Use the training sample set to train the initial network to obtain a matte extraction network.

[0011] This embodiment uses the soft segmentation including the foreground as a prior for model training. Since the soft segmentation prior is adopted, background matte extraction becomes a task that depends less on semantics but more on structural information. Therefore, the network does not need to be too deep, which is beneficial to the lightweight of the network and can be deployed on a chip with low computing power; the backbone block can achieve better high-level feature fusion, and the environment combination module is more lightweight than its corresponding residual network, effectively exchanging the input environmental features, which is beneficial to the fusion process of context information. Both of these modules improve the accuracy to a certain extent and achieve more reliable foreground segmentation prediction.

[0012] As an implementation of the first aspect, the training sample set includes a plurality of labeled training samples, and each labeled training sample includes the input image sample and its annotation;

[0013] Using the training sample set to train the initial network to obtain a matte extraction network includes:

[0014] Perform supervised training on the initial network on the training sample set to obtain a matte extraction network.

[0015] In this implementation, performing supervised training is beneficial to obtaining a matte extraction network with higher accuracy.

[0016] As an implementation of the first aspect, the training sample set includes a plurality of labeled training samples and a plurality of unlabeled training samples, each labeled training sample includes the input image sample and its annotation; each unlabeled training sample includes the input image sample;

[0017] Using the training sample set, training the initial network to obtain a matting network, including:

[0018] After performing supervised training on the initial network using multiple labeled training samples, then performing unsupervised knowledge distillation using multiple unlabeled training samples to obtain a matting network.

[0019] In this implementation, combining supervised training and distillation learning can make up for the differences between the synthetic dataset and real data, which is beneficial to further improving the segmentation accuracy of the matting network and providing a network with good generalization.

[0020] As an implementation of the first aspect, the initial network includes multiple cascaded stage networks; the input image sample is used as the input of the first stage network, and the image sample to be matted, the background image sample, and the predicted foreground segmentation output by the previous stage network are used as the input of the next stage network.

[0021] In this implementation, the initial network includes multiple cascaded stage networks, which can predict more fine-grained structural details, thereby further improving the accuracy of foreground segmentation prediction.

[0022] As an implementation of the first aspect, the stage network includes 3 downsamplings.

[0023] In this implementation, only 3 downsamplings are required for the input, which can retain rich structural clues, and thanks to the incorporation of background information, the speed and accuracy can be better balanced.

[0024] As an implementation of the first aspect, the backbone block includes a feature fusion module based on an attention mechanism.

[0025] As an implementation of the first aspect, the training uses a hybrid loss function, and the hybrid loss function includes mean squared error loss, structural similarity loss, and intersection over union loss.

[0026] In this implementation, due to the use of the hybrid loss function, the foreground and boundaries can be detected more accurately, thereby further improving the accuracy of foreground segmentation prediction.

[0027] In a second aspect, an embodiment of the present application provides a matting method, including:

[0028] Obtaining an image to be matted including a foreground, a background image, and a soft segmentation of the foreground;

[0029] Input the image to be matte, the background image, and the matte network obtained by using the matte network training method described in the first aspect or any implementation manner of the first aspect, and output the foreground segmentation of the image to be matte; the matte network includes at least one stage network; the stage network includes a cascaded context combination module, backbone block, and prediction module. The image to be matte, the background image, and the soft segmentation of the foreground are input into the context combination module. The context combination module is configured to output low-level features and high-level features after feature exchange. The backbone block is configured to fuse the low-level features and the high-level features based on an attention mechanism to obtain fused features. The prediction module is configured to output the foreground segmentation according to the fused features.

[0030] In a third aspect, an embodiment of the present application provides a matte network training apparatus, including:

[0031] An acquisition module, configured to acquire a training sample set and an initial network; the training sample set includes a plurality of training samples, each training sample includes an input image sample, and the input image sample includes an image to be matte sample with a foreground, a background image sample, and a soft segmentation sample of the foreground. The soft segmentation sample is generated by subtracting the depth image corresponding to the background image sample from the depth image corresponding to the image to be matte sample. The initial network includes at least one stage network. The stage network includes a cascaded context combination module, backbone block, and prediction module. The input image sample is input into the context combination module. The context combination module is configured to output low-level features and high-level features after feature exchange. The backbone block is configured to fuse the low-level features and the high-level features based on an attention mechanism to obtain fused features. The prediction module is configured to output a predicted foreground segmentation according to the fused features.

[0032] A training module, configured to train the initial network using the training sample set to obtain a matte network.

[0033] In a fourth aspect, an embodiment of the present application provides a matte apparatus, including:

[0034] An acquisition module, configured to acquire an image to be matte including a foreground, a background image, and a soft segmentation of the foreground.

[0035] The matte extraction module includes a matte extraction network. The matte extraction module is used to input the image to be matte-extracted, the background image, and the soft segmentation into the matte extraction network, and output the foreground segmentation of the image to be matte-extracted. The matte extraction network includes at least one stage network. The stage network includes an environmental combination module, a backbone block, and a prediction module connected in series. The image to be matte-extracted, the background image, and the soft segmentation of the foreground are input into the environmental combination module. The environmental combination module is used to output low-level features and high-level features after feature exchange. The backbone block is used to fuse the low-level features and the high-level features based on the attention mechanism to obtain fused features. The prediction module is used to output the foreground segmentation according to the fused features.

[0036] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the matte extraction network training method as described in the first aspect or any implementation manner of the first aspect; or implements the matte extraction method as described in the second aspect.

[0037] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the matte extraction network training method as described in the first aspect or any implementation manner of the first aspect; or implements the matte extraction method as described in the second aspect.

[0038] In a seventh aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, it causes the electronic device to execute the matte extraction network training method as described in the first aspect or any implementation manner of the first aspect, or execute the matte extraction method as described in the second aspect.

[0039] It should be understood that the beneficial effects of the second aspect to the seventh aspect can be referred to the relevant descriptions of the first aspect and the implementation manners of the first aspect, and will not be elaborated here. Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0041] Figure 1 It is a schematic flowchart of the implementation of a matte extraction network training method provided by an embodiment of the present application;

[0042] Figure 2It is a schematic structural diagram of a matting model provided by an embodiment of the present application;

[0043] Figure 3 is a single-channel heat map from the F layer provided by an embodiment of the present application; H layer;

[0044] Figure 4 It is a schematic diagram of a real-world image set provided by an embodiment of the present application;

[0045] Figure 5 is the comparison result of the speed and accuracy levels of different models on the Composition-1k dataset;

[0046] Figure 6 is a schematic diagram of the qualitative comparison result between different methods on the Composite-1k test set.

[0047] Figure 7 is a schematic diagram of the comparison result between the method provided by an embodiment of the present application and the BM method on real-world images;

[0048] Figure 8 It is a schematic structural diagram of a matting network training device provided by an embodiment of the present application;

[0049] Figure 9 It is a schematic structural diagram of a matting device provided by an embodiment of the present application;

[0050] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0051] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0052] The term "and / or" used in the specification and appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0053] "An embodiment" or "some embodiments" described in the specification of this application means that in one or more embodiments of this application, specific features, structures, or characteristics described in connection with that embodiment are included. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0054] In addition, in the description of this application, "a plurality of" means two or more. Terms such as "first" and "second" are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0055] In order to illustrate the technical solutions described in this application, the following will be described through specific embodiments.

[0056] Figure 1 is a schematic implementation flowchart of a matte extraction network training method provided by an embodiment of this application. The matte extraction network training method in this embodiment can be executed by an electronic device. The electronic device includes but is not limited to a computer, a tablet computer, a server, a mobile phone, a camera, or a wearable device, etc. Among them, the server includes but is not limited to an independent server or a cloud server, etc. As Figure 1 shown, the matte extraction network training method may include steps S110 to S120.

[0057] S110, obtain a training sample set and an initial network.

[0058] S120, use the training sample set to train the initial network to obtain a matte extraction network.

[0059] The initial network is pre-stored in the electronic device as a network model to be trained. The initial network contains a set of network parameters to be learned. After the initial network is trained, a matte extraction network is obtained. The matte extraction network may have the same network structure as the initial network, or may have a simpler network structure than the initial network, and the network parameters of the two are different.

[0060] The initial network (or the matte extraction network) may include a neural network model based on deep learning. For example, backbone networks such as ResNet or VGG.

[0061] It should be noted that current background matting networks all have a large amount of redundancy. Because backbone networks such as ResNet or VGG are usually adopted, and these networks were originally designed for image classification tasks that highly rely on semantics. Therefore, these networks generally downsample 5 times to extract strong semantic features. However, in the embodiments of the present application, since soft segmentation serves as the prior feature for matting, the background matting problem becomes a task with less semantic dependence and higher structural dependence. Therefore, these networks have relatively large redundancy to a certain extent.

[0062] In one embodiment, the initial network (or matting network) adopts a lightweight progressive refinement network (LRN). The initial network (or matting network) includes one stage network or multiple cascaded stage networks. This initial network uses the RGB image I including the foreground, the RGB background image B, and the soft segmentation S of the foreground as priors. The soft segmentation can be generated from the depth image, and lightweight is achieved through specific network designs. The model output undergoes multiple levels of progressive refinement to achieve more reliable foreground segmentation predictions.

[0063] As an implementation, the initial network can be defined as including three input images: the RGB image I, the RGB background image B, and the soft segmentation S of the foreground, representing the network parameters to be determined during training. The network structure of the initial network is as Figure 2 shown, including one stage network or multiple cascaded stage networks. One stage includes a context combining module (CCM), a stem block, and a predictor. The three input images are input into the CCM module, and the CCM module is used to output low-level features and high-level features after feature exchange; the stem block preferably includes a feature fusion module (FFM) for fusing the low-level features and high-level features based on the attention mechanism to obtain fused features, and the predictor is used to output the predicted foreground segmentation according to the fused features.

[0064] Specifically, the CCM module is used to perform feature exchange on the above three input images to generate two output features, a low-level feature F L and a high-level feature F H . And the low-level feature corresponding to each input image is separately encoded as the feature F 1I =E 1I (I), F 1B =E 1B (B) and F 1S =E1S (S), and then splice them to generate an overall low-level feature F L = Cat(F 1I , F 1B , F 1S ), where Cat represents the concatenation operation. In another branch, each low-level feature is further downsampled to a single high-level feature F 2I = E 2I (F 1I ), F 2B = E 2B (F 1B ) and F 2S = E 2S (F 1S ). By fusing the image feature with the remaining two features, F IS = C Is (Cat(F 2I , F 2S )) and F IB = C IB (Cat(F 2I , F 2B )) are obtained. And F H = C ISB (Cat(F IB , F IS )) is used to obtain the overall high-level feature. In the stem block of the network, first, the high-level feature F H is downsampled by the encoder E3, then upsampled by the decoder D3, and then fused by the feature fusion module to obtain the feature F2 = FFM(F H , D3(E3(F H ))). Further, the low-level feature F L is merged into the stem block to obtain the fused feature F1 = Cat(F L , D2(F2)). Finally, α = D1(F1) is used to obtain the predicted foreground segmentation for subsequent image synthesis using the foreground segmentation and the preset background image.

[0065] The above process is called a stage. To predict more refined structural details, in some embodiments of the present application, another stage can be used to further refine the output of the previous stage. The predicted foreground segmentation of the previous stage is used as the prior soft segmentation of the next stage, while the RGB image I and the RGB background image B input to the previous stage continue to be the priors of the next stage. This process can be repeated multiple times to form a cascaded network structure. To clearly represent the network architecture, C, B, and S are used to represent the number of channels in the convolutional layer, the number of convolutional blocks in a residual block, and the number of stages, respectively. The network is further represented as LRN-C-B-S. For example, LRN-16-4-3 represents an LRN constructed from a 16-channel convolutional layer, 4 blocks, and 3 stages. The advantage of LRN is that it can easily balance accuracy and speed by adjusting C, B, and S.

[0066] It should be noted that Figure 2 A lighter backbone, i.e., a stage, is provided, which only requires 3 downsamplings. A simple foreground soft segmentation is input, which is a binary image obtained by subtracting the background depth image (e.g., the depth image corresponding to the RGB background image B) from the foreground depth image (e.g., the depth image corresponding to the RGB image I). The soft segmentation, as a simple off-the-shelf feature, also provides an attention mechanism for the object to be cropped, so that feature extraction can be achieved with fewer weights. In addition, the stage network adopts a CCM module to exchange the context information of image + soft segmentation, image + background, and image + soft segmentation + background, and fully extracts boundary features. Since late fusion of the image and trimap is more effective for feature extraction of the matting network than early fusion, in this example, it is extended to 3 inputs. After pairwise late fusion of the single input image by the CCM module, further late fusion is performed on the fused features, enabling more effective matching and learning of the feature information of different inputs. Additionally, the CCM module uses fewer convolutional layers to extract the features of a single input and then concatenates them. Through such a design, the CCM module is lighter than the corresponding ResNet block module because fewer convolutional channels are introduced before cascading.

[0067] In addition, the FFM module responsible for feature fusion is used to replace the traditional splicing operation. The FFM module utilizes the attention mechanism to achieve better fusion of encoder features and decoder features. In the encoder, the structural features are extracted layer by layer to form high-order semantic features, which are beneficial for the model to utilize a wider range of receptive fields to judge the position of the segmentation edge. For example, when the foreground and background colors are similar (such as a black human head and a black background), it is difficult to judge the boundary directly based on local structural information (the boundary between the head and the black background), but the high-order semantic information can assist in determining the segmentation position by using the empirical information of image recognition (such as a human head is usually round). The FFM module converts the high-order semantic features of the encoder into a spatial attention mask for guiding the restoration of structural information in the decoder. Since only high-order semantic features can provide accurate spatial attention, the low-order features from the encoder are not suitable for using the FFM module. Therefore, in network design, the FFM module is only applied to the inner skip connection, rather than the outer skip connection.

[0068] It can be seen that the network has been designed for lightweight in three aspects. First, the network depth is relatively shallow. Different from traditional backbone networks such as ResNet and GoogleNet, which perform 5 times of downsampling on the input to extract rich semantic clues, due to the use of soft segmentation prior, background matting becomes a task that depends less on semantics but more on structural information. Therefore, the network does not need to be too deep. Only 3 times of downsampling are performed on the input, and the obtained semantic information is already sufficient, while rich structural clues are retained. Second, the number of channels is less. Because background matting is not a classification task, one channel can serve multiple objects. For example, Figure 3 gives a single-channel heatmap from the F H layer. It can be noticed that this channel captures foregrounds of different categories indiscriminately. In addition, the input soft segmentation provides ready-made features, so fewer channels are required for information extraction. Third, the CCM module is lighter than its corresponding residual network. Through comparison experiments, it is proved that if each feature is concatenated and the residual module is used to extract the high-order feature F H , compared with using the CCM module, this method generates an additional 1.8G FLOPs of computation and 0.3M additional parameters, and the model performance becomes worse than using the CCM module.

[0069] In some embodiments, the training sample set includes a plurality of training samples, and each training sample includes an input image sample and its annotation (ground truth). The input image sample includes three, namely, the image sample I to be matte extracted with a foreground, the background image sample B, and the soft segmentation sample S of the foreground. Annotation α * can be the ground truth foreground segmentation manually annotated. For example, the annotation includes the standard transparency mask corresponding to the image sample to be matte extracted. In these embodiments, since the training samples with annotations are used, step S120 includes: performing supervised training on the initial network on the training sample set to obtain the matte extraction network.

[0070] As an implementation manner of these embodiments, the initial network is trained using the Adobe dataset containing 493 foreground objects, and a synthetic dataset is created. The image sample to be matte extracted can select non-transparent objects from the Adobe dataset (such as excluding glass products, etc.), or, further, one or a combination of methods such as cropping, rotating, flipping, and adding Gaussian noise can be used to randomly augment it. The background image sample can be randomly selected from the MS COCO dataset and augmented by one or a combination of methods such as gamma correction and adding Gaussian noise to avoid strong dependence on fixed values of the background. The soft segmentation sample of the foreground can be generated by subtracting the depth image corresponding to the background image sample from the depth image corresponding to the image sample to be matte extracted. For example, a binary image obtained by subtracting the depth image corresponding to the background image sample from the depth image corresponding to the image sample to be matte extracted. Alternatively, the input soft segmentation sample of the foreground can be simulated with a defective real-world segmentation by one or a combination of operations such as erosion, dilation, and blurring on the ground truth foreground segmentation.

[0071] The supervised training task on the synthetic dataset can be defined as updating the network parameters to minimize the loss function L:

[0072]

[0073] In one embodiment, a network is trained using a mixed loss function that includes a variety of different loss functions, such as mean squared error (MSE) loss, structural similarity (SSIM) loss, and intersection over union (IoU) loss. The MSE loss function is a conventional pixel regression loss for segmentation supervision. The SSIM loss function imposes constraints on the mean and standard deviation to better predict structural consistency. The IoU loss function commonly used in image segmentation tasks pays more attention to the optimization of the global structure. The SSIM loss function is used to predict finer boundaries, while the IoU loss function is used to predict more complete foregrounds. Due to the use of the mixed loss function, the foreground and boundaries can be detected more accurately. In one embodiment, the weighted sum of three different loss functions is used as the mixed loss function, also known as the joint loss function, which is defined as:

[0074] L = λ1L MSE + λ2L SSIM + λ3L IoU .

[0075] Where λ1, λ2, and λ3 are the weight coefficients of the three different loss functions respectively. In one embodiment, the weight coefficients of the three loss functions can be assigned as λ1 = 2, λ2 = 2, and λ3 = 5. L MSE is the MSE loss, and L MSE is defined as:

[0076]

[0077] Where H and W represent the image height and width respectively; α i,j and represent the predicted and prior foreground segmentations. L SSIM is the SSIM loss, and L SSIM is defined as:

[0078]

[0079] Where μ, σ and μ * , σ * are the mean and deviation of α i,j and . The constants c1 = 0.01 2 and c2 = 0.03 2 are used to avoid division by zero. L IoU is the IoU loss, and L IoU is defined as:

[0080]

[0081] Where the parameter γ can be set to 5, and θ i,j is the difficulty index of pixel (i, j), which can be determined by the following formula:

[0082]

[0083] Among them, A i,j Represents the neighboring pixels of pixel (i, j).

[0084] In order to make up for the difference between synthetic data and real data, in some other embodiments of the present application, in addition to using labeled synthetic images for supervised training, unlabeled real images can also be used for unsupervised knowledge distillation.

[0085] At this time, the training sample set includes multiple labeled training samples and multiple unlabeled training samples, each labeled training sample includes an input image sample and its label; each unlabeled training sample includes an input image sample. It should be noted that in these embodiments, the input image samples also include 3, namely, a foreground image sample I to be cut out, a background image sample B, and a foreground soft segmentation sample S. In these embodiments, labeled and unlabeled training samples, i.e., a mixed data set, are used, so step S120 includes: using multiple labeled training samples to perform supervised training on the initial network, and then using multiple unlabeled training samples to perform unsupervised knowledge distillation to obtain a cutout network.

[0086] As an implementation of these embodiments, a real-world human-held object dataset is created, including 1259 labeled images as a test set and 11255 unlabeled images as a knowledge distillation training set. All images are recorded using a depth camera. Figure 4 Shown are RGB and depth images of the background and foreground, representing a real-world image dataset. From top left to bottom right, they are depth background, depth image, soft segmentation, color background, color image, and ground truth foreground segmentation. The soft segmentation is a binary image obtained by subtracting the background depth from the image depth. The 1,259 annotated images are from 11 scenes, with an average of 2.5 people per scene, each displaying over 30 items in one to three poses. This dataset enables qualitative evaluation of algorithms on real-world datasets.

[0087] A mixed dataset consisting of 10,000 labeled synthetic images and 11,255 unlabeled real-world images is used. Supervised training and unsupervised knowledge distillation are performed simultaneously on the mixed dataset. The network trained on the synthetic dataset is used as the teacher model, which can be a complex network model such as ResNet or VGG. For labeled data,

[0088]

[0089] To train; for unlabeled data, that is,

[0090]

[0091] to perform distillation learning. Among them, represents the teacher network trained on the synthetic dataset, represents the student network, which can be the lightweight matting network of the present application, and L is the hybrid loss function or the joint loss function.

[0092] Another embodiment of the present application provides a matting method. The matting method can be applied to an electronic device, and the electronic device is pre-deployed with a matting network. In some embodiments, the matting network can adopt an untrained initial network. In some other embodiments, in order to improve the matting accuracy, the matting network can be an initial network, and the matting network can also be trained by the method of the foregoing embodiments. The matting network includes at least one stage network; the stage network includes a cascaded CCM module, a backbone block stem, and a prediction module predictor. When using the matting network to perform background matting on the image to be matted, first obtain three input images, which include: the image to be matted including the foreground, the background image, and the soft segmentation of the foreground; input the three images into the matting network to output the foreground segmentation of the image to be matted. Specifically, the three images are input into the CCM module, and the CCM module is used to output low-order features and high-order features after feature exchange. The backbone block is used to fuse the low-order features and high-order features based on the attention mechanism to obtain the fused features, and the prediction module is used to output the foreground segmentation according to the fused features.

[0093] It should be understood that the process of using the matting network to perform background matting can refer to the relevant description of the foregoing matting network training process, which will not be elaborated here.

[0094] The embodiment of the present application proposes a lightweight real-time background matting network. The network has a relatively shallow structure design, and at the same time, two network modules are proposed. The FFM module can achieve better high-level feature fusion, and the CCM module is more lightweight than the corresponding traditional residual module, which is beneficial to the fusion process of context information. These two modules improve the accuracy to a certain extent. In order to achieve better boundary prediction and foreground prediction, a hybrid loss function is introduced, which combines the advantages of MSE, SSIM, and IoU losses. A real-world dataset containing 1259 labeled images and 11255 unlabeled images is created for quantitative evaluation and knowledge distillation. Experiments on the synthetic dataset and the real dataset show that the method achieves real-time performance on both PC (111FPS) and Amlogic A311D chip (45FPS).

[0095] Based on the method provided by the embodiment of the present application for experiments, using a learning rate of 10 -3The Adam optimizer was used to train the LRN-32-4-4 model with 26,900 synthetic images. The LRN-32-4-4 was chosen because it can well balance accuracy and speed. The model was trained for 100 epochs on 4 RTX2080ti GPUs with a batch size of 16 and an input resolution of 512×512. A test dataset consisting of 1,000 synthetic images (Adobe's test set, also known as Composite-1k) was used to evaluate the model performance. After the supervised training of the LRN-32-4-4 model on the synthetic dataset, the trained LRN-32-4-4 model was further distilled to obtain a more lightweight LRN-16-4-3 model, and the parameter settings for distillation learning were the same as those for supervised learning.

[0096] Four metrics, MSE t , SAD t , MSE e and SAD e were used in the experiment to evaluate the model accuracy. MSE and SAD represent the mean squared error and the sum of absolute differences respectively. The subscripts "t" and "e" denote the evaluation errors in the trimap region and the whole image. Previous studies only used MSE t and SAD t metrics, which were sufficient for trimap-based methods because the foreground regions were known. However, for trimap-free methods that need to predict both the foreground and unknown regions simultaneously, the introduction of MSE e and SAD e metrics was to obtain a more comprehensive evaluation. On the Composition-1k dataset, the method of the embodiment of the present application was compared with 4 other learning-based models, including trimap-based CAM and DIM, and trimap-free LFM and BM. On the real dataset, the model provided by the embodiment of the present application was also compared with the CAM, DIM and BM models. It should be noted that the comparison with traditional methods was excluded because they have been proven to be far less accurate than learning-based methods.

[0097] Specifically, during the experiment, the foreground segmentation error, FLOPs (based on a resolution of 288×288) and the number of parameters Param. were evaluated on the Composite-1k test set. Figure 5The comparison results of the speed and accuracy levels of different models on the Composition-1k dataset are shown. The comparison results of the errors and speeds of different models on the Composition-1k dataset are shown in Table 1 below. Here, the model ours in the embodiments of the present application adopts the LRN-32-4-4 model. The comparison results of the errors and speeds of different models on the real dataset are shown in Table 2 below. Here, the model ours in the embodiments of the present application adopts the LRN-16-4-3 model. Since CAM and DIM are trimap-based methods, only SAD t and MSE t metrics. As can be seen from Table 1 and Figure 5 it can be seen that the model (LRN-32-4-4) provided in the embodiments of the present application is superior to other methods in all 4 metrics, and it is significantly more lightweight. For example, at an input resolution of 288×288, the method in the embodiments of the present application has 13.0G of FLOPs and 2.2M of parameters. Compared with the BM method, the FLOPs are reduced by 89.9%, and the number of parameters Param. is reduced by 87.7%. Model inference at 39FPS is achieved on a GTX1060ti GPU, meeting the real-time inference requirement. Real-time means that the inference speed is greater than 30FPS.

[0098] Table 1

[0099]

[0100] Table 2

[0101]

[0102]

[0103] Figure 6 The schematic diagram of the qualitative comparison results between different methods on the Composite-1k test set is shown. The method provided in the embodiments of the present application has strong robustness to background interference. For example, it shows better foreground and background discrimination ability and can detect small background areas surrounded by the foreground. Figure 7 The schematic diagram of the comparison results between the method provided in an embodiment of the present application and the BM method on real-world images is shown. From Figure 7 it can be seen that the BM method is difficult to detect a foreground with the same color as the background. For example, a white box in front of a white wall.

[0104] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0105] An embodiment of the present application further provides a matte extraction network training device. For details not described in this matte extraction network training device, please refer to the description in the foregoing embodiment of the matte extraction network training method.

[0106] See Figure 8 , Figure 8 FIG. is a schematic block diagram of a matte extraction network training device provided by an embodiment of the present application. The matte extraction network training device includes: an acquisition module 81 and a training module 82.

[0107] Among them, the acquisition module 81 is used to acquire a training sample set and an initial network; the training sample set includes a plurality of training samples, each training sample includes an input image sample, the input image sample includes a to-be-matted image sample with a foreground, a background image sample, and a soft segmentation sample of the foreground, and the soft segmentation sample is generated by subtracting the depth image corresponding to the background image sample from the depth image corresponding to the to-be-matted image sample; the initial network includes at least one stage network; the stage network includes a cascaded environment combination module, a backbone block, and a prediction module, the input image sample is input into the environment combination module, the environment combination module is used to output low-order features and high-order features after feature exchange, the backbone block is used to fuse the low-order features and the high-order features based on an attention mechanism to obtain a fused feature, and the prediction module is used to output a predicted foreground segmentation according to the fused feature.

[0108] The training module 82 is used to train the initial network with the training sample set to obtain a matte extraction network.

[0109] In some embodiments, the training sample set includes a plurality of labeled training samples, and each labeled training sample includes the input image sample and its label.

[0110] The training module 82 is specifically used for:

[0111] Supervisedly train the initial network on the training sample set to obtain a matte extraction network.

[0112] In some embodiments, the training sample set includes a plurality of labeled training samples and a plurality of unlabeled training samples, each labeled training sample includes the input image sample and its label; each unlabeled training sample includes the input image sample.

[0113] The training module 82 is specifically used for:

[0114] After supervisedly training the initial network with a plurality of labeled training samples, then perform unsupervised knowledge distillation with a plurality of unlabeled training samples to obtain a matte extraction network.

[0115] In some embodiments, the initial network includes a plurality of cascaded stage networks; the input image sample serves as the input of the first stage network, and the image sample to be matte-extracted, the background image sample, and the predicted foreground segmentation output by the previous stage network serve as the input of the next stage network.

[0116] In some embodiments, the stage network includes 3 downsamplings.

[0117] In some embodiments, the backbone block includes a feature fusion module based on an attention mechanism.

[0118] In some embodiments, the training module 82 employs a hybrid loss function, and the hybrid loss function includes mean squared error loss, structural similarity loss, and intersection over union loss.

[0119] An embodiment of the present application further provides a matte extraction device. For details not described in this matte extraction device, please refer to the description in the foregoing matte extraction method embodiments.

[0120] See Figure 9 , Figure 9 is a schematic block diagram of a matte extraction device provided by an embodiment of the present application. The matte extraction device includes: an acquisition module 91 and a matte extraction module 92.

[0121] Among them, the acquisition module 91 is used to acquire an image to be matte-extracted including a foreground, a background image, and a soft segmentation of the foreground;

[0122] The matte extraction module 92 includes a matte extraction network. The matte extraction module 92 is used to input the image to be matte-extracted, the background image, and the soft segmentation into the matte extraction network and output the foreground segmentation of the image to be matte-extracted; the matte extraction network includes at least one stage network; the stage network includes a cascaded context combination module, a backbone block, and a prediction module. The image to be matte-extracted, the background image, and the soft segmentation of the foreground are input into the context combination module. The context combination module is used to output low-level features and high-level features after feature exchange. The backbone block is used to fuse the low-level features and the high-level features based on an attention mechanism to obtain fused features. The prediction module is used to output the foreground segmentation according to the fused features.

[0123] An embodiment of the present application further provides an electronic device. As Figure 10 shown, the electronic device may include one or more processors 100 ( Figure 10Only one is shown in the figure), a memory 101, and a computer program 102 stored in the memory 101 and executable on one or more processors 100. For example, a program for matte network training and / or a program for image matting. When the one or more processors 100 execute the computer program 102, each step in the matte network training method and / or the matte method embodiments can be implemented. Alternatively, when the one or more processors 100 execute the computer program 102, the functions of each module / unit in the matte network training device and / or the matte device embodiments can be implemented, which is not limited here.

[0124] Those skilled in the art can understand that Figure 10 These are merely examples of electronic devices and do not constitute a limitation on electronic devices. An electronic device may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, an electronic device may also include input / output devices, network access devices, a bus, etc.

[0125] In one embodiment, the so-called processor 100 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0126] In one embodiment, the memory 101 may be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 101 may also include both the internal storage unit and the external storage device of the electronic device. The memory 101 is used to store the computer program and other programs and data required by the electronic device. The memory 101 may also be used to temporarily store data that has been output or will be output.

[0127] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be repeated here.

[0128] An embodiment of this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the matte painting network training method and / or the matte painting method embodiment can be implemented.

[0129] An embodiment of this application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the matte painting network training method and / or the matte painting method embodiment.

[0130] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0131] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0132] In the embodiments provided in this application, it should be understood that the disclosed device / electronic device and method can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0133] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, in each embodiment of this application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0135] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of this application, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0136] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for training a matte extraction network, characterized in that Comprising: Obtaining a training sample set and an initial network; Wherein, the training sample set includes a plurality of training samples, each of the training samples includes an input image sample, the input image sample includes a to-be-matted image sample with a foreground, a background image sample, and a soft segmentation sample of the foreground as a prior, and the soft segmentation sample is generated by subtracting the depth image corresponding to the background image sample from the depth image corresponding to the to-be-matted image sample; The initial network adopts a lightweight progressive refinement network, and the initial network includes a plurality of cascaded stage networks; the input image sample is used as the input of the first stage network, and the to-be-matted image sample, the background image sample, and the predicted foreground segmentation output by the previous stage network are used as the input of the next stage network; the stage network includes a cascaded context combination module, a backbone block, and a prediction module, the input image sample is input into the context combination module, the context combination module is used to output low-level features and high-level features after feature exchange, the backbone block is used to fuse the low-level features and the high-level features based on an attention mechanism to obtain a fused feature, and the prediction module is used to output a predicted foreground segmentation according to the fused feature; Using the training sample set to train the initial network to obtain a matting network.

2. The matte extraction network training method according to claim 1, wherein The training sample set includes a plurality of labeled training samples, and each of the labeled training samples includes the input image sample and its annotation; Using the training sample set to train the initial network to obtain a matting network, including: Performing supervised training on the initial network on the training sample set to obtain a matting network.

3. The matte extraction network training method according to claim 1, wherein The training sample set includes a plurality of labeled training samples and a plurality of unlabeled training samples, each of the labeled training samples includes the input image sample and its annotation; each of the unlabeled training samples includes the input image sample; Using the training sample set to train the initial network to obtain a matting network, including: Performing supervised training on the initial network using a plurality of labeled training samples, and then performing unsupervised knowledge distillation using a plurality of unlabeled training samples to obtain a matting network.

4. The matte extraction network training method according to any one of claims 1 to 3, characterized in that The stage network includes 3 times of downsampling.

5. The method for training a matte extraction network according to any one of claims 1 to 3, characterized in that, The training adopts a hybrid loss function, and the hybrid loss function includes mean squared error loss, structural similarity loss, and intersection over union loss.

6. A matte extraction method, characterized in that, Comprising: Obtaining a to-be-matted image including a foreground, a background image, and a soft segmentation of the foreground; Using the to-be-matted image, the background image, and the soft segmentation as priors to input into the matting network, and outputting a foreground segmentation of the to-be-matted image; the matting network is obtained by training the initial network; Among them, the initial network adopts a lightweight progressive refinement network, and the initial network includes multiple cascaded stage networks; the input image sample is used as the input of the first stage network, and the image sample to be matteed, the background image sample, and the predicted foreground segmentation output by the previous stage network are used as the input of the next stage network; the stage network includes a cascaded context combination module, a backbone block, and a prediction module. The image sample to be matteed, the background image, and the soft segmentation of the foreground are input into the context combination module, and the context combination module is used to output low-level features and high-level features after feature exchange. The backbone block is used to fuse the low-level features and the high-level features based on the attention mechanism to obtain fused features, and the prediction module is used to output the foreground segmentation according to the fused features.

7. An apparatus for training a matting network, characterized in that Comprising: An acquisition module, configured to acquire a training sample set and an initial network; The training sample set includes multiple training samples, and each training sample includes an input image sample. The input image sample includes an image sample to be matteed with a foreground, a background image sample, and a soft segmentation sample of the foreground as a prior. The soft segmentation sample is generated by subtracting the depth image corresponding to the background image sample from the depth image corresponding to the image sample to be matteed. The initial network adopts a lightweight progressive refinement network, and the initial network includes multiple cascaded stage networks. The input image sample is used as the input of the first stage network, and the image sample to be matteed, the background image sample, and the predicted foreground segmentation output by the previous stage network are used as the input of the next stage network. The stage network includes a cascaded context combination module, a backbone block, and a prediction module. The input image sample is input into the context combination module, and the context combination module is used to output low-level features and high-level features after feature exchange. The backbone block is used to fuse the low-level features and the high-level features based on the attention mechanism to obtain fused features, and the prediction module is used to output the predicted foreground segmentation according to the fused features; A training module, configured to use the training sample set to train the initial network to obtain a matteing network.

8. A matte extraction device, characterized in that, Comprising: An acquisition module, configured to acquire an image sample to be matteed with a foreground, a background image, and a soft segmentation of the foreground; A matteing module, including a matteing network. The matteing module is configured to use the image sample to be matteed, the background image, and the soft segmentation as priors to input into the matteing network and output the foreground segmentation of the image sample to be matteed. The matteing network is obtained by training the initial network; Among them, the initial network adopts a lightweight progressive refinement network, and the initial network includes multiple cascaded stage networks; the input image sample is used as the input of the first stage network, and the image sample to be matte, the background image sample, and the predicted foreground segmentation output by the previous stage network are used as the inputs of the next stage network; the stage network includes a cascaded context combination module, a backbone block, and a prediction module. The image sample to be matte, the background image, and the soft segmentation of the foreground are input into the context combination module, and the context combination module is used to output low-level features and high-level features after feature exchange. The backbone block is used to fuse the low-level features and the high-level features based on the attention mechanism to obtain fused features, and the prediction module is used to output the foreground segmentation according to the fused features.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the matte network training method according to any one of claims 1 to 5, or implements the matte method according to claim 6.

10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the matte network training method according to any one of claims 1 to 5, or implements the matte method according to claim 6.

Images