Image cutout method, device, electronic device and storage medium

By designing a three-branch encoder model, extracting and integrating the features of color images, three-part images and overlay images, the problems of large amount of image cutouts and low accuracy in the existing technology are solved, and better cutout details and speed.

CN115035135BActive Publication Date: 2024-12-06HANGZHOU WANGDAO HLDG CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210788336.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-06
Publication Date
2024-12-06
Estimated Expiration
2042-07-06

AI Technical Summary

Technical Problem

The existing image cutout technology has a large amount of calculation and low accuracy when dealing with complex backgrounds. The conventional methods ignore the global information in the three-point image, resulting in poor detail and slow cutout.

Method used

A three-branch encoder model is designed, and the features of color maps, three-branch maps and overlay maps are extracted through three encoder branches, and the features are fused between high-frequency and low-frequency information to output an extinction map.

Benefits of technology

It effectively supplements the coordination between global and local information, improves the detail and speed of cutouts, and enhances the accuracy of background removal.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115035135B_ABST
    Figure CN115035135B_ABST
Patent Text Reader

Abstract

The present invention provides an image matting method, device, electronic device and storage medium, which can pre-train a three-branch encoder to obtain a matting model, wherein the first encoder branch in the matting model is used to extract the features of a color image; the second encoder branch in the matting model is used to extract the features of a tri-plot; the third encoder branch in the matting model is used to extract the features of a superimposed image of the color image and the tri-plot, fuse the features of the color image and the tri-plot in the features of the superimposed image, and output an extinction image based on the features of the fused superimposed image. For a target color image to be processed and its corresponding target tri-plot, by inputting the target color image, the target tri-plot, and the target superimposed image of the target color image and the target tri-plot into the first encoder branch, the second encoder branch and the third encoder branch in the matting model respectively, the target extinction image output by the third encoder branch in the matting model can be obtained, thereby completing background removal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of software technology, and more specifically, to an image cutout method, device, electronic device and storage medium. Background Art

[0002] Image matting (also known as image background removal) is mainly used in the fields of film making, video editing, and digital image processing.

[0003] The current common practice is to use a green screen to create a pure color background venue after selecting the scene. After completing the shooting task, the background of the shooting content is mainly removed and replaced manually. However, there are almost no pure color scenes in the real world, which leads to a narrow landing scene for green screen cutouts. In addition, the manual background removal mode requires a lot of time to interact with tools to complete the work. Summary of the invention

[0004] In view of this, in order to solve the above problems, the present invention provides an image cutout method, device, electronic device and storage medium, and the technical solution is as follows:

[0005] An image cutout method, the method comprising: obtaining a target color image to be processed, and a target trisection image corresponding to the target color image;

[0006] Retrieve the cutout model;

[0007] The cutout model is obtained by pre-training a three-branch encoder; the first encoder branch in the cutout model is used to extract features of a color image; the second encoder branch in the cutout model is used to extract features of a trisection image; the third encoder branch in the cutout model is used to extract features of a superimposed image of the color image and the trisection image, fuse features of the color image and features of the trisection image into features of the superimposed image, and output an extinction image based on the features of the fused superimposed image;

[0008] The target color map, the target tri-map, and the target superposition map of the target color map and the target tri-map are respectively input into the first encoder branch, the second encoder branch, and the third encoder branch in the cutout model, and the target extinction map output by the third encoder branch in the cutout model is obtained.

[0009] Preferably, the process of training the three-branch encoder to obtain the cutout model includes:

[0010] Obtain a sample color image for this training, and a sample trisection image corresponding to the sample color image, wherein the sample color image and the sample trisection image are annotated with a standard extinction image;

[0011] The sample color map, the sample trisection map, and the sample superposition map of the sample color map and the sample trisection map are respectively input into the first encoder branch, the second encoder branch, and the third encoder branch in the three-branch encoder to achieve:

[0012] The first encoder branch in the three-branch encoder extracts the features of the sample color image; the second encoder branch in the three-branch encoder extracts the features of the sample trisection image; the third encoder branch in the three-branch encoder extracts the features of the sample superposition image, and in the downsampling stage / upsampling stage of extracting the features of the sample superposition image, the features of the sample color image and the features of the sample trisection image are fused in the features mined in the downsampling stage / reconstructed in the upsampling stage between the high-frequency information and the low-frequency information;

[0013] Obtaining a predicted extinction map output by a third encoder branch in the three-branch encoder, and determining a loss function value of the three-branch encoder according to the labeled extinction map and the predicted extinction map;

[0014] If the loss function value does not satisfy the corresponding condition, return to the step of obtaining the sample color image used for this training and the sample trisection image corresponding to the sample color image;

[0015] If the loss function value satisfies the corresponding condition, the training of the three-branch encoder is terminated, and the trained three-branch encoder is used as the cutout model.

[0016] Preferably, the features mined in the downsampling stage / the features reconstructed in the upsampling stage are integrated with the features of the sample color image and the features of the sample trigraph, including:

[0017] Using the features of the sample color image as a query vector and a key-value vector, and determining the similarity between the features of the sample color image and the features mined in the downsampling stage / the features reconstructed in the upsampling stage through a preset similarity algorithm; and using the features of the sample trigraph as a query vector and a key-value vector, and determining the similarity between the features of the sample trigraph and the features mined in the downsampling stage / the features reconstructed in the upsampling stage through a preset similarity algorithm;

[0018] Determining the weights of the features of the sample color image and the features of the sample tri-image, respectively, according to the similarities corresponding to the features of the sample color image and the similarities corresponding to the features of the sample tri-image;

[0019] According to the respective weights of the features of the sample color image and the features of the sample trisection image, the features of the sample color image and the features of the sample trisection image are fused into the features mined in the downsampling stage / the features reconstructed in the upsampling stage.

[0020] Preferably, the first encoder branch in the three-branch encoder includes a residual network Resnet34, and the convolution layer of the Resnet34 is set to three consecutive first convolution layers, second convolution layers and third convolution layers; wherein,

[0021] The kernel sizes of the first convolutional layer, the second convolutional layer, and the third convolutional layer are all 3, the number of channels are 16, 24, and 128 respectively, and the stride is 2.

[0022] Preferably, the second encoder branch in the three-branch encoder comprises Resnet18.

[0023] Preferably, the third encoder branch in the three-branch encoder includes Resnet50, a three-party information feature mining component and a three-party information feature aggregation component; wherein,

[0024] The Resnet50 is used to extract the features of the sample overlay image; the tripartite information feature mining component is used to fuse the features of the sample color image and the features of the sample tri-map with the features mined in the down-sampling stage between high-frequency information and low-frequency information during the up-sampling stage of extracting the features of the sample overlay image; the tripartite information feature aggregation component is used to fuse the features of the sample color image and the features of the sample tri-map with the features reconstructed in the up-sampling stage between high-frequency information and low-frequency information during the down-sampling stage of extracting the features of the sample overlay image.

[0025] An image cutout device, comprising:

[0026] A cutout model training module, used for training a three-branch encoder to obtain a cutout model; wherein the first encoder branch in the cutout model is used to extract features of a color image; the second encoder branch in the cutout model is used to extract features of a trisection image; the third encoder branch in the cutout model is used to extract features of a superimposed image of the color image and the trisection image, fuse features of the color image and features of the trisection image into features of the superimposed image, and output an extinction image based on the features of the fused superimposed image;

[0027] An image acquisition module, used to acquire a target color image to be processed and a target three-part image corresponding to the target color image;

[0028] The image cutout module is used to retrieve the cutout model; input the target color image, the target tri-image, and the target superposition image of the target color image and the target tri-image into the first encoder branch, the second encoder branch, and the third encoder branch in the cutout model respectively, and obtain the target extinction image output by the third encoder branch in the cutout model.

[0029] Preferably, the cutout model training module is specifically used for:

[0030] A sample color image and a sample trigraph corresponding to the sample color image are obtained for this training, wherein the sample color image and the sample trigraph are marked with a standard extinction image; the sample color image, the sample trigraph, and a sample superposition image of the sample color image and the sample trigraph are respectively input into a first encoder branch, a second encoder branch, and a third encoder branch in the three-branch encoder, so as to achieve: the first encoder branch in the three-branch encoder extracts features of the sample color image; the second encoder branch in the three-branch encoder extracts features of the sample trigraph; the third encoder branch in the three-branch encoder extracts features of the sample superposition image, and extracts features of the sample superposition image at a downsampling stage. In the segment / upsampling stage, the features of the sample color image and the features of the sample tri-map are integrated into the features mined in the down-sampling stage / reconstructed in the up-sampling stage between the high-frequency information and the low-frequency information; the predicted extinction map output by the third encoder branch in the three-branch encoder is obtained, and the loss function value of the three-branch encoder is determined according to the labeled extinction map and the predicted extinction map; if the loss function value does not meet the corresponding conditions, the step of obtaining the sample color image used for this training and the sample tri-map corresponding to the sample color image is returned to be executed; if the loss function value meets the corresponding conditions, the training of the three-branch encoder is terminated, and the trained three-branch encoder is used as the cutout model.

[0031] An electronic device comprises: at least one memory and at least one processor; the memory stores an application program, the processor calls the application program stored in the memory, and the application program is used to implement the image cutout method.

[0032] A storage medium stores computer program code, which implements the image cutout method when executed.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention provides an image cutout method, device, electronic device and storage medium, which can pre-train a three-branch encoder to obtain a cutout model, wherein the first encoder branch in the cutout model is used to extract features of a color image; the second encoder branch in the cutout model is used to extract features of a tri-plot; the third encoder branch in the cutout model is used to extract features of a superimposed image of the color image and the tri-plot, fuse the features of the color image and the tri-plot in the features of the superimposed image, and output an extinction map based on the features of the fused superimposed image. For a target color image to be processed and its corresponding target tri-plot, by inputting the target color image, the target tri-plot, and the target superimposed image of the target color image and the target tri-plot into the first encoder branch, the second encoder branch and the third encoder branch in the cutout model respectively, the target extinction map output by the third encoder branch in the cutout model can be obtained, thereby completing background removal. The present invention designs a three-branch encoder based on a neural network to comprehensively mine the input information, and can mine the color image, the tri-plot and the superimposed image. Figure 3 The feature fusion is carried out based on the local information, which effectively supplements the coordination between the ignored global information and the local information, making the cutout details better and faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0036] Figure 1 A method flow chart of an image matting method provided by an embodiment of the present invention;

[0037] Figure 2 A partial method flow chart of the image matting method provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the structure of a three-branch encoder provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the structure of an image cutout device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] With the development of deep convolutional neural networks, image matting has entered a new stage. The solution mainly uses (RGB) color images and (Trimap) tri-maps as input. The tri-map consists of foreground, background and area to be solved. The foreground, background and area to be solved are represented as white, black and dark in the image respectively. This is because the direct matting of color images requires a large amount of calculation and is subject to too many external interference factors, resulting in low accuracy. Therefore, the use of the tri-map can calculate the area to be solved, reducing the amount of calculation and reducing the difficulty of solving.

[0043] However, most current studies only focus on feature extraction of the region to be solved, while ignoring the global information composed of foreground and background in the trisection map, thereby ignoring the inherent coordination between global information and local information of the region to be solved. The present invention proposes a tripartite information mining and integration network to formally coordinate the interaction between global and local information. To this end, the present invention designs a novel three-branch encoder to comprehensively mine the input information. This structure can effectively supplement the neglected coordination between global and local information, making the cutout details better and faster.

[0044] See also Figure 1 , Figure 1 A method flow chart of an image cutout method provided by an embodiment of the present invention, wherein the image cutout method comprises the following steps:

[0045] S101, obtaining a target color image to be processed and a target three-part image corresponding to the target color image.

[0046] In the embodiment of the present invention, the target color image is the color image to be cut out, and the target three-division image is the three-division image corresponding to the target color image.

[0047] S102, retrieve the cutout model; wherein,

[0048] The cutout model is obtained by pre-training a three-branch encoder; the first encoder branch in the cutout model is used to extract the features of the color image; the second encoder branch in the cutout model is used to extract the features of the triplicated image; the third encoder branch in the cutout model is used to extract the features of the superimposed image of the color image and the triplicated image, fuse the features of the color image and the triplicated image in the features of the superimposed image, and output an extinction image based on the features of the fused superimposed image.

[0049] The traditional learning-based model mainly focuses on superimposing the color image and the triplicated image and inputting them into the convolutional neural network for learning. However, this process cannot mine the different information provided by the two. Therefore, the present invention first proposes a three-party information extraction framework to mine the information of the color image, the triplicated image and the superimposed image (the color image superimposed on the triplicated image). The present invention designs a novel three-branch encoder based on a neural network to comprehensively mine the input information. Specifically, in order to extract the respective features of the color image and the triplicated image, one branch of the three-branch encoder (i.e., the first encoder branch) and the other branch (i.e., the second encoder branch) are used to model and sample the two respectively. At the same time, in order to ensure that information is not lost, another branch of the three-branch encoder (i.e., the third encoder branch) is used to model and sample the superimposed image to obtain the features of the superimposed image. In this process, the color image, the triplicated image and the superimposed image are mined through three-party information feature mining. Figure 3 The high-frequency information is extracted by mining the three-way information features, and the color map, three-way map and overlay are aggregated by the three-way information features. Figure 3 Experiments show that this three-branch encoder structure can effectively supplement the coordination between the ignored global information and local information, making the cutout details better and faster.

[0050] In the specific implementation process, the process of training the three-branch encoder to obtain the cutout model can be carried out in the following steps. The method flow chart is as follows: Figure 2 As shown:

[0051] S201, obtaining a sample color image used for this training and a sample trisection image corresponding to the sample color image, wherein the sample color image and the sample trisection image are annotated with a standard extinction image.

[0052] In the embodiment of the present invention, the sample color image is the color image used as the training sample, and the sample three-dimensional image is the three-dimensional image corresponding to the sample color image. A set of corresponding sample color images and sample three-dimensional images are pre-labeled with a standard extinction image, i.e., a standard extinction image, which is an image with the background removed.

[0053] S202, inputting the sample color image, the sample trisection image, and the sample superposition image of the sample color image and the sample trisection image into the first encoder branch, the second encoder branch, and the third encoder branch of the three-branch encoder respectively, to achieve:

[0054] The first encoder branch in the three-branch encoder extracts features of the sample color image; the second encoder branch in the three-branch encoder extracts features of the sample tri-map; the third encoder branch in the three-branch encoder extracts features of the sample superposition image, and in the downsampling stage / upsampling stage of extracting features of the sample superposition image, the features of the sample color image and the features of the sample tri-map are fused in the features mined in the downsampling stage / the features reconstructed in the upsampling stage between the high-frequency information and the low-frequency information.

[0055] See also Figure 3 , Figure 3 A schematic diagram of the structure of a three-branch encoder provided by an embodiment of the present invention. A color image (GRB) and a trimap are input, and the color image and the trimap are input to the first encoder branch (RGB-Unit) and the second encoder branch (Trimap-Unit), respectively. A superposition operation is performed on the color image and the trimap (as shown in "C" in the figure) to obtain a superposition image, and the superposition image is input to the third encoder branch. Figure 3 , the short cut of the first encoder branch is represented by a thin solid line, the short cut of the second encoder branch is represented by a dotted line, and the short cut of the third encoder branch is represented by a thick solid line.

[0056] In practical applications, the first encoder branch in the three-branch encoder of the present invention can use Resnet (Residual Network) 34, and considering that the color image mainly provides texture and color features, it does not require an overly complex architecture. In the present invention, each convolutional layer of Resnet34 is set to three consecutive convolutional layers, namely the first convolutional layer, the second convolutional layer and the third convolutional layer. Among them, the kernel size of the first convolutional layer, the second convolutional layer and the third convolutional layer is 3, and the number of channels is 16, 24, and 128 respectively, which not only reduces the weight of the model but also ensures the extraction of features, and the stride of the first convolutional layer, the second convolutional layer and the third convolutional layer is set to 2, thereby achieving 8 times downsampling to be compatible with resolution and calculation. When the color image features are changed from 512*512*3 to a feature map of 64*64*128 through three convolutional layers, the feature extraction of the color image is completed.

[0057] Since the triplicate map is different from the color map, it focuses more on the expression of clear foreground information and the area to be solved (i.e., the unknown area). Therefore, the triplicate map provides structural features with global information, which is often ignored. Therefore, the second encoder branch in the three-branch encoder in the present invention uses Resnet18 for feature extraction. It should be noted that the feature sizes of the triplicate map, the color map, and the superimposed map should be consistent to facilitate the subsequent mining of features from three different information.

[0058] Continue to see Figure 3 , the third encoder branch consists of three parts, from input to output, it is a network structure for downsampling (the part shown in area AA in the figure), an empty pyramid multi-scale hole convolution (ASPP), and a network structure for upsampling (the part shown in area BB in the figure). In addition, the network structure for downsampling in the third encoder branch also includes a three-party information feature mining component (the part shown in area CC in the figure), and the network structure for upsampling in the third encoder branch also includes a three-party information feature aggregation component (the part shown in area DD in the figure).

[0059] In practical applications, in order to retain the traditional superposition information extraction, Resnet50 (i.e. Figure 3 However, as the high-frequency information of the model downsamples features more, the traditional method of only processing the overlay information will produce confusing features, which hinders the model from better extracting the information extracted by the color map and the tri-map, thereby affecting the prediction. In addition, since the extraction of features by the neural network is a transition process from low frequency to high frequency, the present invention introduces the features provided by the original high and low frequency information of the color map and the features provided by the tri-map at the stage between the high and low frequency information.

[0060] To this end, a tripartite information feature mining component is introduced into the network structure of Resnet50 for downsampling. Resnet50 extracts features of the overlay image in the downsampling stage, and the tripartite information feature mining component fuses the features of the color image and the tripartite image with the features mined in the downsampling stage between the high-frequency information and the low-frequency information.

[0061] Continue to see Figure 3After the three-party information feature mining component, two downsamplings are continued to be completed to obtain enhanced high-level features. Then Resnet50 performs conventional upsampling based on these features, but direct upsampling will lead to feature loss and reconstruction failure. The usual way is to connect the features of the downsampling stage in the way of Unet to ensure feature perfection. Here, the present invention follows this mechanism, but such an architecture is still based on a single branch result with only superposition map feature extraction, and does not take into account the features of the color map and the three-part map extracted previously. In order to maximize feature reuse, the present invention introduces a three-party information feature aggregation component into the network structure of Resnet50 for downsampling, and Resnet50 performs feature extraction in the upsampling stage. The three-party information feature aggregation component fuses the features of the sample color map and the features of the sample three-part map reconstructed in the upsampling stage between the high-frequency information and the low-frequency information. Specifically, in order to maximize the use of features in the upsampling stage, in addition to the features learned by the model itself, the previous features are also directed to the upsampling stage, ensuring that interpolation between high-frequency and low-frequency information is performed to finally complete the aggregation of three-party information. After the information is enhanced, three times of double upsampling are performed to complete the restoration of the feature map, followed by conventional supervised learning.

[0062] In addition, the features extracted from the overlay image also include some features of the color image and some features of the trigraph. The traditional method cannot well distinguish the importance of the features of the color image and the trigraph in the features of the overlay image. Therefore, in the process of fusing the features of the color image and the trigraph in the features mined in the downsampling stage / the features reconstructed in the upsampling stage, the present invention can regard the features of the color image / the trigraph as queries and keys, obtain the importance of the features of the color image / the trigraph to the features of the overlay image through similarity calculation, and then fuse them. The advantage of this is that in addition to the original features of the overlay image, the present invention uses the texture features of the color image and the structural features of the trigraph to guide the overlay image to perform feature enhancement, which greatly improves the feature extraction of the cutout method and makes the cutout effect better. Specifically:

[0063] Continue to see Figure 3In the area CC in the image, the features of the color image are multiplied as query vectors and key-value vectors (as shown by "×" in the figure), and then the similarity between the features of the color image and the features mined in the downsampling stage is determined by a preset similarity algorithm (as shown by "S" in the figure). Specifically, the features of the trigraph are first multiplied as query vectors and key-value vectors, and then the similarity between the features of the trigraph and the features mined in the downsampling stage is determined by Softmax. Similarly, the features of the trigraph are multiplied as query vectors and key-value vectors (as shown by "×" in the figure), and then the similarity between the features of the trigraph and the features mined in the downsampling stage is determined by Softmax (as shown by "×" in the figure); according to the similarity corresponding to the features of the color image and the similarity corresponding to the features of the trigraph, the weights of the features of the color image and the features of the trigraph are determined, and the similarity is positively correlated with the weight, that is, the higher the similarity, the greater the weight and the higher the corresponding importance; continue to refer to Figure 3 In the area CC in the image, the features of the color image and their corresponding weights are multiplied (as shown by “×” in the figure), and the features of the tri-map and their corresponding weights are multiplied (as shown by “×” in the figure). Then, the multiplication results of the features of the color image and the tri-map with the weights are added to the features of the overlay image (as shown by “+” in the figure), thereby integrating the features of the sample color image and the features of the sample tri-map into the features mined in the downsampling stage.

[0064] Continue to see Figure 3 In the area DD in the image, the features of the color image are multiplied as the query vector and the key vector (as shown by "×" in the figure), and then the similarity between the features of the color image and the features reconstructed in the upsampling stage is determined by a preset similarity algorithm (as shown by "S" in the figure). Specifically, the features of the trigraph are first multiplied as the query vector and the key vector, and then the similarity between the features of the trigraph and the features reconstructed in the upsampling stage is determined by Softmax. Similarly, the features of the trigraph are multiplied as the query vector and the key vector (as shown by "×" in the figure), and then the similarity between the features of the trigraph and the features reconstructed in the upsampling stage is determined by Softmax (as shown by "×" in the figure); according to the similarity corresponding to the features of the color image and the similarity corresponding to the features of the trigraph, the weights of the features of the color image and the features of the trigraph are determined, and the similarity is positively correlated with the weight, that is, the higher the similarity, the greater the weight and the higher the corresponding importance; continue to refer to Figure 3In the area CC in the image, the features of the color image and their corresponding weights are multiplied (as shown by “×” in the figure), and the features of the tri-map and their corresponding weights are multiplied (as shown by “×” in the figure). Then, the multiplication results of the features of the color image and the tri-map with the weights are added to the features of the superposition image (as shown by “+” in the figure), thereby merging the features of the sample color image and the features of the sample tri-map into the features reconstructed in the upsampling stage.

[0065] Based on the above description, the sample color map, the sample trisection map, and the sample superposition map (i.e., the superposition map of the sample color map and the sample trisection map) are respectively input into the first encoder branch, the second encoder branch, and the third encoder branch in the three-branch encoder, which can be achieved:

[0066] The first encoder branch (Resnet 34) in the three-branch encoder extracts features of the sample color map. The second encoder branch (Resnet 18) in the three-branch encoder extracts features of the sample tri-map. The third encoder branch (Resnet50) in the three-branch encoder extracts features of the sample superposition map; the three-party information feature mining component fuses features of the sample color map and features of the sample tri-map in the features mined in the downsampling stage between high-frequency information and low-frequency information in the downsampling stage of extracting features of the sample superposition map; the three-party information feature aggregation component fuses features of the sample color map and features of the sample tri-map in the features reconstructed in the upsampling stage between high-frequency information and low-frequency information in the upsampling stage of extracting features of the sample superposition map.

[0067] Correspondingly, when integrating the features of the sample color image and the features of the sample triplicate image into the features mined in the downsampling stage, on the one hand, the features of the sample color image are used as query vectors and key-value vectors, and the similarity between the features of the sample color image and the features mined in the downsampling stage is determined through Softmax; on the other hand, the features of the sample triplicate image are used as query vectors and key-value vectors, and the similarity between the features of the triplicate image and the features mined in the downsampling stage is determined through Softmax; then, according to the similarity corresponding to the features of the sample color image and the similarity corresponding to the features of the sample triplicate image, the respective weights of the features of the sample color image and the features of the sample triplicate image are determined; further, according to their respective weights, the features of the sample color image and the features of the sample triplicate image are integrated into the features mined in the downsampling stage.

[0068] Correspondingly, when fusing the features of the sample color map and the features of the sample triplicate map into the features reconstructed in the upsampling stage, on the one hand, the features of the sample color map are used as query vectors and key-value vectors, and the similarity between the features of the sample color map and the features reconstructed in the upsampling stage is determined through Softmax; on the other hand, the features of the sample triplicate map are used as query vectors and key-value vectors, and the similarity between the features of the sample triplicate map and the features reconstructed in the upsampling stage is determined through Softmax; then, according to the similarity corresponding to the features of the sample color map and the similarity corresponding to the features of the sample triplicate map, the respective weights of the features of the sample color map and the features of the sample triplicate map are determined; further, according to their respective weights, the features of the sample color map and the features of the sample triplicate map are fused into the features reconstructed in the upsampling stage.

[0069] S203, obtain the predicted extinction map output by the third encoder branch in the three-branch encoder, and determine the loss function value of the three-branch encoder based on the labeled extinction map and the predicted extinction map; if the loss function value does not meet the corresponding conditions, return to execute step S201; if the loss function value meets the corresponding conditions, execute step S204.

[0070] Continue to see Figure 3 For the sample color image and sample trisection image input by the three-branch encoder, the third encoder branch in the three-branch encoder can predict the corresponding extinction image based on the restored feature image, that is, the predicted extinction image. Then, the loss function value of the three-branch encoder is calculated according to the labeled extinction image and the predicted extinction image to determine the prediction accuracy of the three-branch encoder. If the loss function value does not meet the corresponding conditions, that is, the prediction accuracy does not meet the preset accuracy threshold, then return to step S201 and continue to train the three-branch encoder; if the loss function value meets the corresponding conditions, that is, the prediction accuracy meets the preset accuracy threshold, then the training is terminated and the trained three-branch encoder is used as the cutout model.

[0071] S204, ending the training of the three-branch encoder, and using the trained three-branch encoder as a cutout model.

[0072] S103, input the target color image, the target tri-image, and the target superposition image of the target color image and the target tri-image into the first encoder branch, the second encoder branch, and the third encoder branch in the cutout model respectively, and obtain the target extinction image output by the third encoder branch in the cutout model.

[0073] In the embodiment of the present invention, the target color map, the target trisection map, and the target overlay map (overlay map of the target color map and the target trisection map) are respectively input into the first encoder branch, the second encoder branch, and the third encoder branch in the matting model to achieve:

[0074] The first encoder branch in the cutout model extracts features of the target color image; the second encoder branch in the cutout model extracts features of the target tri-image; the third encoder branch in the cutout model extracts features of the target superposition image. On the one hand, in the downsampling stage of extracting features of the target superposition image, features of the target color image and features of the target tri-image are fused with features mined in the downsampling stage between high-frequency information and low-frequency information. On the other hand, in the upsampling stage of extracting features of the target superposition image, features of the target color image and features of the target tri-image are fused with features reconstructed in the upsampling stage between high-frequency information and low-frequency information, and then the corresponding extinction map, i.e., the target extinction map, is predicted and output based on the feature map restored by the fused features.

[0075] The embodiment of the present invention provides an image matting method, which designs a three-branch encoder based on a neural network to fully mine the input information and can perform image matting on color images, three-branch images and superposition images. Figure 3 The feature fusion is carried out based on the information of each party, so as to effectively supplement the coordination between the ignored global information and the local information, making the cutout details better and faster.

[0076] Based on the image cutout method provided in the above embodiment, an embodiment of the present invention provides a device for executing the above image cutout method. The structural diagram of the device is shown in FIG. Figure 4 As shown, including:

[0077] The cutout model training module 10 is used to train the three-branch encoder to obtain the cutout model; wherein the first encoder branch in the cutout model is used to extract the features of the color image; the second encoder branch in the cutout model is used to extract the features of the tri-plot; the third encoder branch in the cutout model is used to extract the features of the superimposed image of the color image and the tri-plot, fuse the features of the color image and the tri-plot in the features of the superimposed image, and output the extinction image based on the features of the fused superimposed image;

[0078] An image acquisition module 20 is used to acquire a target color image to be processed and a target three-dimensional image corresponding to the target color image;

[0079] The image cutout module 30 is used to retrieve the cutout model; input the target color image, the target tri-image, and the target superposition image of the target color image and the target tri-image into the first encoder branch, the second encoder branch, and the third encoder branch in the cutout model respectively, and obtain the target extinction image output by the third encoder branch in the cutout model.

[0080] Optionally, the cutout model training module 10 is specifically used for:

[0081] A sample color image and a sample trisection image corresponding to the sample color image for this training are obtained, wherein the sample color image and the sample trisection image are annotated with a standard extinction image; the sample color image, the sample trisection image, and the sample superposition image of the sample color image and the sample trisection image are respectively input into the first encoder branch, the second encoder branch, and the third encoder branch of the three-branch encoder, so as to achieve: the first encoder branch of the three-branch encoder extracts the features of the sample color image; the second encoder branch of the three-branch encoder extracts the features of the sample trisection image; the third encoder branch of the three-branch encoder extracts the features of the sample superposition image, and in the downsampling stage / In the upsampling stage, the features of the sample color image and the features of the sample tri-map are fused between the high-frequency information and the low-frequency information in the features mined in the downsampling stage / the features reconstructed in the upsampling stage; the predicted extinction map output by the third encoder branch in the three-branch encoder is obtained, and the loss function value of the three-branch encoder is determined according to the labeled extinction map and the predicted extinction map; if the loss function value does not meet the corresponding conditions, the step of obtaining the sample color image used for this training and the sample tri-map corresponding to the sample color image is returned to execute; if the loss function value meets the corresponding conditions, the training of the three-branch encoder is terminated, and the trained three-branch encoder is used as the cutout model.

[0082] Optionally, the cutout model training module 10 for fusing the features of the sample color image and the features of the sample trisection image in the features mined in the downsampling stage / the features reconstructed in the upsampling stage is specifically used for:

[0083] The features of the sample color image are used as query vectors and key-value vectors, and the similarity between the features of the sample color image and the features mined in the downsampling stage / the features reconstructed in the upsampling stage is determined through a preset similarity algorithm; and the features of the sample trigraph are used as query vectors and key-value vectors, and the similarity between the features of the sample trigraph and the features mined in the downsampling stage / the features reconstructed in the upsampling stage is determined through a preset similarity algorithm; the weights of the features of the sample color image and the features of the sample trigraph are determined according to the similarities corresponding to the features of the sample color image and the similarities corresponding to the features of the sample trigraph; and the features of the sample color image and the features of the sample trigraph are fused into the features mined in the downsampling stage / the features reconstructed in the upsampling stage according to their respective weights.

[0084] Optionally, the first encoder branch in the three-branch encoder includes a residual network Resnet34, and the convolution layer of Resnet34 is set to three consecutive first convolution layers, second convolution layers and third convolution layers; wherein,

[0085] The kernel size of the first, second, and third convolutional layers is 3, the number of channels is 16, 24, and 128 respectively, and the stride is 2.

[0086] Optionally, the second encoder branch in the three-branch encoder includes Resnet18.

[0087] Optionally, the third encoder branch in the three-branch encoder includes Resnet50, a three-party information feature mining component and a three-party information feature aggregation component; wherein,

[0088] Resnet50 is used to extract the features of the sample overlay map. The tripartite information feature mining component is used to fuse the features of the sample color map and the sample tri-map with the features mined in the down-sampling stage between the high-frequency information and the low-frequency information in the down-sampling stage of extracting the features of the sample overlay map. The tripartite information feature aggregation component is used to fuse the features of the sample color map and the sample tri-map with the features reconstructed in the up-sampling stage between the high-frequency information and the low-frequency information in the up-sampling stage of extracting the features of the sample overlay map.

[0089] It should be noted that the detailed functions of each module in the embodiment of the present invention can be found in the corresponding disclosed part of the above-mentioned image cutout method embodiment, and will not be repeated here.

[0090] Based on the image cutout method provided in the above embodiment, an embodiment of the present invention provides an electronic device, which includes: at least one memory and at least one processor; the memory stores an application, the processor calls the application stored in the memory, and the application is used to implement the image cutout method.

[0091] Based on the image cutout method provided in the above embodiment, an embodiment of the present invention provides a storage medium correspondingly, wherein the storage medium stores computer program code, and the image cutout method is implemented when the computer program code is executed.

[0092] The above is a detailed introduction to an image cutout method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0093] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0094] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device that includes a series of elements is inherent to the elements, or also includes elements inherent to these processes, methods, articles or devices. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0095] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image cutout method, characterized in that: The method comprises: Obtaining a target color image to be processed and a target three-part image corresponding to the target color image; Retrieve the cutout model; The cutout model is obtained by pre-training a three-branch encoder; the first encoder branch in the cutout model is used to extract features of a color image; the second encoder branch in the cutout model is used to extract features of a trisection image; the third encoder branch in the cutout model is used to extract features of a superimposed image of the color image and the trisection image, fuse features of the color image and features of the trisection image into features of the superimposed image, and output an extinction image based on the features of the fused superimposed image; Inputting the target color map, the target trisection map, and the target superposition map of the target color map and the target trisection map into the first encoder branch, the second encoder branch, and the third encoder branch in the cutout model, respectively, and obtaining the target extinction map output by the third encoder branch in the cutout model; The process of training the three-branch encoder to obtain the cutout model includes: Obtaining a sample color image for this training and a sample trisection image corresponding to the sample color image, wherein the sample color image and the sample trisection image are annotated with a standard extinction image; The sample color map, the sample trisection map, and the sample superposition map of the sample color map and the sample trisection map are respectively input into the first encoder branch, the second encoder branch, and the third encoder branch in the three-branch encoder to achieve: The first encoder branch in the three-branch encoder extracts the features of the sample color image; the second encoder branch in the three-branch encoder extracts the features of the sample trisection image; the third encoder branch in the three-branch encoder extracts the features of the sample superposition image, and in the downsampling stage / upsampling stage of extracting the features of the sample superposition image, the features of the sample color image and the features of the sample trisection image are fused in the features mined in the downsampling stage / reconstructed in the upsampling stage between the high-frequency information and the low-frequency information; Obtaining a predicted extinction map output by a third encoder branch in the three-branch encoder, and determining a loss function value of the three-branch encoder according to the labeled extinction map and the predicted extinction map; If the loss function value does not satisfy the corresponding condition, return to the step of obtaining the sample color image used for this training and the sample trisection image corresponding to the sample color image; If the loss function value satisfies the corresponding condition, the training of the three-branch encoder is terminated, and the trained three-branch encoder is used as the cutout model; The feature mining in the downsampling stage / the feature reconstructed in the upsampling stage are integrated with the feature of the sample color image and the feature of the sample tri-image, including: Using the features of the sample color image as a query vector and a key-value vector, and determining the similarity between the features of the sample color image and the features mined in the downsampling stage / the features reconstructed in the upsampling stage through a preset similarity algorithm; and using the features of the sample trigraph as a query vector and a key-value vector, and determining the similarity between the features of the sample trigraph and the features mined in the downsampling stage / the features reconstructed in the upsampling stage through a preset similarity algorithm; Determining the weights of the features of the sample color image and the features of the sample trigraph, respectively, according to the similarities corresponding to the features of the sample color image and the similarities corresponding to the features of the sample trigraph; According to the respective weights of the features of the sample color image and the features of the sample trisection image, the features of the sample color image and the features of the sample trisection image are fused into the features mined in the downsampling stage / the features reconstructed in the upsampling stage.

2. The method according to claim 1, characterized in that The first encoder branch in the three-branch encoder includes a residual network Resnet34, and the convolution layer of the Resnet34 is set to three consecutive first convolution layers, second convolution layers and third convolution layers; wherein, The kernel sizes of the first convolutional layer, the second convolutional layer, and the third convolutional layer are all 3, the numbers of channels are 16, 24, and 128 respectively, and the strides are all 2.

3. The method according to claim 1, characterized in that The second encoder branch in the three-branch encoder includes Resnet18.

4. The method according to claim 1, characterized in that: The third encoder branch in the three-branch encoder includes Resnet50, a three-party information feature mining component and a three-party information feature aggregation component; wherein, The Resnet50 is used to extract the features of the sample overlay image; the tripartite information feature mining component is used to fuse the features of the sample color image and the features of the sample tri-map with the features mined in the down-sampling stage between high-frequency information and low-frequency information during the down-sampling stage of extracting the features of the sample overlay image; the tripartite information feature aggregation component is used to fuse the features of the sample color image and the features of the sample tri-map with the features reconstructed in the up-sampling stage between high-frequency information and low-frequency information during the up-sampling stage of extracting the features of the sample overlay image.

5. An image cutout device, characterized in that: The device comprises: A cutout model training module, used for training a three-branch encoder to obtain a cutout model; wherein the first encoder branch in the cutout model is used to extract features of a color image; the second encoder branch in the cutout model is used to extract features of a trisection image; the third encoder branch in the cutout model is used to extract features of a superimposed image of the color image and the trisection image, fuse features of the color image and features of the trisection image into features of the superimposed image, and output an extinction image based on the features of the fused superimposed image; An image acquisition module, used to acquire a target color image to be processed and a target three-part image corresponding to the target color image; an image matting module, for retrieving the matting model; inputting the target color image, the target trisection image, and the target superposition image of the target color image and the target trisection image into the first encoder branch, the second encoder branch, and the third encoder branch in the matting model, respectively, and obtaining the target extinction image output by the third encoder branch in the matting model; The cutout model training module is specifically used for: A sample color image and a sample trigraph corresponding to the sample color image are obtained for this training, wherein the sample color image and the sample trigraph are marked with a standard extinction image; the sample color image, the sample trigraph, and a sample superposition image of the sample color image and the sample trigraph are respectively input into a first encoder branch, a second encoder branch, and a third encoder branch in the three-branch encoder, so as to achieve: the first encoder branch in the three-branch encoder extracts features of the sample color image; the second encoder branch in the three-branch encoder extracts features of the sample trigraph; the third encoder branch in the three-branch encoder extracts features of the sample superposition image, and extracts features of the sample superposition image at a downsampling stage. In the segment / upsampling stage, the features of the sample color image and the features of the sample trisection image are integrated into the features mined in the downsampling stage / reconstructed in the upsampling stage between the high-frequency information and the low-frequency information; the predicted extinction map output by the third encoder branch in the three-branch encoder is obtained, and the loss function value of the three-branch encoder is determined according to the labeled extinction map and the predicted extinction map; if the loss function value does not meet the corresponding conditions, the step of obtaining the sample color image used for this training and the sample trisection image corresponding to the sample color image is returned to be executed; if the loss function value meets the corresponding conditions, the training of the three-branch encoder is terminated, and the trained three-branch encoder is used as the cutout model; The feature mining in the downsampling stage / the feature reconstructed in the upsampling stage are integrated with the feature of the sample color image and the feature of the sample tri-image, including: Using the features of the sample color image as a query vector and a key-value vector, and determining the similarity between the features of the sample color image and the features mined in the downsampling stage / the features reconstructed in the upsampling stage through a preset similarity algorithm; and using the features of the sample trigraph as a query vector and a key-value vector, and determining the similarity between the features of the sample trigraph and the features mined in the downsampling stage / the features reconstructed in the upsampling stage through a preset similarity algorithm; Determining the weights of the features of the sample color image and the features of the sample trigraph, respectively, according to the similarities corresponding to the features of the sample color image and the similarities corresponding to the features of the sample trigraph; According to the respective weights of the features of the sample color image and the features of the sample trisection image, the features of the sample color image and the features of the sample trisection image are fused into the features mined in the downsampling stage / the features reconstructed in the upsampling stage.

6. An electronic device, characterized in that: The electronic device comprises: at least one memory and at least one processor; the memory stores an application, the processor calls the application stored in the memory, and the application is used to implement the image cutout method described in any one of claims 1-4.

7. A storage medium, characterized in that: The storage medium stores computer program code, and when the computer program code is executed, the image cutout method according to any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • Background removal method, device and equipment, and readable storage medium

    CN111627038A

  • Foreground matting method and device and electronic equipment

    CN114419091A