Portrait matting method, device and equipment and storage medium

Abstract

The application provides a portrait matting method, device, equipment and storage medium. The method comprises the following steps: obtaining a training set, the training set comprising a plurality of groups of training images, each group of training images comprising an original portrait image and corresponding labeled semantic feature maps, labeled detail feature maps and labeled fusion feature maps; constructing a matting model structure, the matting model structure comprising a detail prediction network based on an attention mechanism; training the matting model structure according to the training set to obtain a trained portrait matting model. The application introduces an attention mechanism to enhance the learning of the portrait matting model on the detail features of the edges of a portrait. The semantic features and the detail features extracted based on the attention mechanism are fused in the fusion network of the portrait matting model, thereby improving the accuracy of the fusion features. The final matting result map is more accurate and the details are more accurate. The boundaries of the foreground of the portrait in the matting result map are smoothed, so that the classification of pixels is more accurate and the edges of the portrait are smoother.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a method, apparatus, device, and storage medium for portrait matting. Background Technology

[0002] Portrait matting involves predicting an accurate foreground mask (alpha matte) and then using it to extract a person from a given image or video. This technique has been widely used in areas such as photo editing, image re-creation, and ID photo generation. Currently, while portrait matting can achieve complete extraction of a person's image, it generally produces some imperfections in the details of the portrait.

[0003] ID photo generation typically involves directly overlaying a cut-out portrait onto a background image of the same color. Because the background image is uniformly colored, it tends to magnify any imperfections in the overlaid portrait. Therefore, the process of generating ID photos demands a higher level of skill in selecting the cut-out portrait. Summary of the Invention

[0004] This application proposes a method, apparatus, device, and storage medium for portrait matting, and introduces an attention mechanism to enhance the learning of detailed features of the portrait matting model on the edges of the person, so that the matting accuracy of the trained portrait matting model is higher and the details are more precise.

[0005] The first aspect of this application proposes a portrait matting method, including:

[0006] Obtain a training set, which contains multiple sets of training images. Each set of training images includes an original human image and a corresponding labeled semantic feature map, labeled detail feature map, and labeled fusion feature map.

[0007] Construct a matting model structure, which includes a detail prediction network based on an attention mechanism;

[0008] Based on the training set, the matting model structure is trained to obtain a trained portrait matting model.

[0009] In some embodiments of this application, the matting model structure is trained based on the training set to obtain a trained portrait matting model, including:

[0010] The original human image is input into the semantic estimation network in the matting model structure to obtain the corresponding semantic feature map;

[0011] Based on the semantic feature map and the original character image, the corresponding detail feature map is obtained through the attention-based detail prediction network.

[0012] Based on the semantic feature map and the intermediate feature map based on the attention mechanism in the detail prediction network, the corresponding fused feature map is obtained through the fusion network in the matting model structure.

[0013] In some embodiments of this application, obtaining the corresponding detail feature map through the attention-based detail prediction network based on the semantic feature map and the original person image includes:

[0014] Upsample the semantic feature map;

[0015] Based on the original image of the person, feature extraction is performed through the attention mechanism module included in the detail prediction network;

[0016] The feature map obtained by upsampling is fused with the feature map output by the attention mechanism module;

[0017] Based on the original image of the person and the fused feature map, the final detail feature map is obtained through the detail prediction network.

[0018] In some embodiments of this application, obtaining the corresponding fused feature map through the fusion network in the matting model structure based on the semantic feature map and the intermediate feature map based on the attention mechanism in the detail prediction network includes:

[0019] In the fusion network of the image matting model structure, the semantic feature map is upsampled;

[0020] The upsampled feature map is fused with the intermediate feature map based on the attention mechanism in the detail prediction network;

[0021] Upsample the fused feature map to obtain the corresponding fused feature map.

[0022] In some embodiments of this application, the method further includes:

[0023] Based on the labeled semantic feature map, labeled detail feature map, labeled fusion feature map, the semantic feature map, the detail feature map, and the fusion feature map corresponding to the original character image, calculate the overall loss value for the current training cycle;

[0024] If the overall loss value reaches the preset convergence condition, the matting model structure and the current model parameters will be used to form a trained portrait matting model;

[0025] If the overall loss value does not reach the preset convergence condition, the current model parameters are adjusted, and the matting model structure is trained again based on the adjusted model parameters.

[0026] In some embodiments of this application, after obtaining the trained portrait matting model, the process further includes:

[0027] Obtain the image of the person to be cut out;

[0028] The person image is input into the portrait matting model to obtain the semantic feature map, detail feature map and fusion feature map corresponding to the person image;

[0029] The semantic feature map, the detail feature map, and the fusion feature map corresponding to the person image are fused to obtain the cutout result map corresponding to the person image.

[0030] In some embodiments of this application, after obtaining the cutout result image corresponding to the person image, the method further includes:

[0031] Determine the foreground boundary pixels and the radius corresponding to the foreground boundary pixels in the matting result image;

[0032] Draw a circle corresponding to the foreground boundary pixel with the foreground boundary pixel as the center and the radius.

[0033] Determine the centroid of the cutout region where the foreground boundary pixels are located;

[0034] If the centroid is located within the circle corresponding to the foreground boundary pixel, then the foreground boundary pixel is determined to be a foreground pixel; otherwise, the foreground boundary pixel is determined to be a background pixel.

[0035] In some embodiments of this application, after obtaining the cutout result image corresponding to the person image, the method further includes:

[0036] Pixels whose classification confidence scores fall within a preset range in the cutout result image are identified as pixels to be processed.

[0037] Determine the radius corresponding to the pixel to be processed;

[0038] Draw a circle corresponding to the pixel to be processed with the pixel as the center and the radius as the radius;

[0039] Determine the centroid of the cutout region where the pixel to be processed is located;

[0040] If the centroid is located within the circle corresponding to the pixel to be processed, then the pixel to be processed is determined to be a foreground pixel; otherwise, the pixel to be processed is determined to be a background pixel.

[0041] In some embodiments of this application, after obtaining the cutout result image corresponding to the person image, the method further includes:

[0042] Determine the radius corresponding to the first pixel and the centroid of the cutout region where the first pixel is located; the first pixel is any pixel in the cutout result image.

[0043] Draw a circle corresponding to the first pixel with the first pixel as the center and the radius specified.

[0044] If the centroid is located within the circle corresponding to the first pixel, then the first pixel is determined to be a foreground pixel;

[0045] If the centroid is not located within the circle, the classification result of the first pixel remains unchanged.

[0046] An embodiment of the second aspect of this application provides a portrait cutout device, comprising:

[0047] The training acquisition module is used to acquire a training set, which contains multiple sets of training images. Each set of training images includes an original human image and a corresponding labeled semantic feature map, labeled detail feature map and labeled fusion feature map.

[0048] The model building module is used to build the matting model structure, which includes a detail prediction network based on an attention mechanism.

[0049] The model training module is used to train the matting model structure based on the training set to obtain a trained portrait matting model.

[0050] An embodiment of the third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0051] An embodiment of the fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method described in the first aspect above.

[0052] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0053] In this embodiment, an attention mechanism module is added to the detail prediction network of the portrait matting model. This module enhances the model's ability to extract and learn detailed features of the subject's edges. The fusion network of the portrait matting model integrates semantic features and detail features extracted based on the attention mechanism, improving the accuracy of the final fused features. This results in a more accurate matting result image with more precise details. Furthermore, this application performs boundary smoothing on the foreground of the subject in the matting result image, making pixel classification more accurate and the subject edges smoother.

[0054] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0055] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0056] In the attached diagram:

[0057] Figure 1 A schematic flowchart of a portrait cutout method provided in an embodiment of this application is shown;

[0058] Figure 2 A schematic diagram of the matting model structure provided in an embodiment of this application is shown;

[0059] Figure 3 This illustration shows a schematic diagram of boundary smoothing processing through pixels to be processed provided in an embodiment of this application;

[0060] Figure 4 This illustration shows a structural schematic diagram of a portrait cutout device provided in one embodiment of this application;

[0061] Figure 5 This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0062] Figure 6 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0063] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0064] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0065] The following description, in conjunction with the accompanying drawings, outlines a method, apparatus, device, and storage medium for portrait cutout based on embodiments of this application.

[0066] Currently, while portrait cutout technology can achieve complete extraction of a person's image, some imperfections may still remain in the details. ID photo generation typically involves directly overlaying the cut-out image onto a background of the same color. Because the background image in an ID photo is uniformly colored, it tends to magnify any imperfections in the overlaid image. Therefore, the process of generating ID photos places higher demands on the cut-out image.

[0067] Based on this, this application provides a portrait matting method. This method introduces an attention mechanism into the detail prediction network of the matting model to enhance the model's extraction and learning of detailed features of the portrait image, thereby improving the comprehensiveness and accuracy of the final matted foreground in terms of detail. Furthermore, after the foreground is matted out using the matting model in this method, boundary smoothing processing is performed on the foreground, making the classification of pixels at the boundaries more accurate.

[0068] See Figure 1 This method trains an attention-based portrait matting model through the following steps 101-103, specifically including:

[0069] Step 101: Obtain the training set, which contains multiple sets of training images. Each set of training images includes an original human image and corresponding labeled semantic feature map, labeled detail feature map, and labeled fusion feature map.

[0070] First, a large number of original human images are acquired. These can be downloaded from the internet and / or captured by photographing people using a camera or mobile phone camera. After acquiring the original images, semantic features are annotated. These semantic features include foreground and background features. Foreground features refer to the portion of the image occupied by the person; annotations distinguish the foreground and background, resulting in an annotated semantic feature map. Edges of the foreground figures, such as the edges of clothing and hair, are also annotated, resulting in an annotated detail feature map. Finally, a fusion feature map combining semantic and detail features is annotated, such as annotating arms, clothing, neck, and head, resulting in an annotated fusion feature map.

[0071] The original image of the person and its corresponding labeled semantic feature map, labeled detail feature map, and labeled fused feature map are combined into a set of training images. Multiple sets of training images obtained in the above manner constitute the training set.

[0072] Step 102: Construct the matting model structure, which includes a detail prediction network based on an attention mechanism.

[0073] The aforementioned matting model structure includes a semantic estimation network, a detail prediction network, and a fusion network. An attention mechanism is incorporated into the detail prediction network. Connecting the semantic estimation network to the fusion network and the attention-based detail prediction network, and then connecting the fusion network and the attention-based detail prediction network, constructs the matting model structure to be trained.

[0074] Figure 2 The details of the constructed matting model structure are shown, such as Figure 2 As shown, the detail prediction network includes a first downsampling layer, an attention mechanism module, a first fusion layer, a first upsampling layer, a second fusion layer, and a second upsampling layer connected in sequence, as well as a first staggered network layer connected in parallel between the input of the first downsampling layer and the second fusion layer. The first fusion layer is also connected to a third upsampling layer.

[0075] By connecting the semantic estimation network to the third upsampling layer in the detail prediction network, connecting the output of the semantic estimation network to the input of the fusion network, and connecting the output of the first upsampling layer in the detail prediction network to the fusion network, the matting model structure can be obtained.

[0076] Specifically, such as Figure 2As shown, the semantic estimation network includes a second downsampling layer, a third fusion layer, and a fourth upsampling layer connected in sequence, as well as a second staggered network layer connected in parallel between the output of the second downsampling layer and the third fusion layer. The output of the fourth upsampling layer is connected to the input of the third upsampling layer in the detail prediction network.

[0077] like Figure 2 As shown, the fusion network includes a fifth upsampling layer, a fourth fusion layer, and a sixth upsampling layer connected in sequence. The input of the fifth upsampling layer is connected to the output of the fourth upsampling layer in the semantic estimation network. The input of the fourth fusion layer is connected to the output of the first upsampling layer in the detail prediction network.

[0078] exist Figure 2 The network includes a preprocessing structure before the input of the detail prediction network, comprising a third downsampling layer and a fifth fusion layer connected in sequence. The output of the fifth fusion layer is connected to the inputs of the first downsampling layer and the first staggered network layer in the detail prediction network. The input of the fifth fusion layer is also connected to the output of the seventh upsampling layer, and the input of the seventh upsampling layer is connected to the output of the second downsampling layer in the semantic estimation network.

[0079] The attention mechanism module in the aforementioned detail prediction network can be a CBAM (Convolutional Block Attention Module), which includes a channel attention module and a spatial attention module. This not only saves parameters and computational power, but also ensures that it can be integrated into the network architecture of the matting model structure as a plug-and-play module.

[0080] In this embodiment, the semantic estimation network can also employ a MobileNet-based network architecture, such as MobileNetV1 or MobileNetV2, to extract high-level semantics through an encoder (i.e., the low-resolution branch of MODNet). The detail prediction network combines information from the original input image of the person with the semantic features output by the semantic estimation network, and performs separate constraint learning on the person's edges through an encoder-decoder process. To reduce computational cost, the encoder-decoder structure is relatively lightweight, processing only the downsampled scale of the original image of the person. The fusion network combines the semantic features output by the semantic estimation network and the detail prediction results output by the detail prediction network to obtain the final fusion result.

[0081] This step constructs an attention-based image matting model structure. By adding an attention mechanism module to the detail prediction network, the image matting model can significantly enhance the extraction and learning of detailed features of people in images.

[0082] Step 103: Train the matting model structure based on the training set to obtain a trained portrait matting model.

[0083] Multiple sets of training images are obtained from the training set obtained in step 101. Each set of training images includes the original image of the person and its corresponding labeled semantic feature map, labeled detail feature map, and labeled fusion feature map. Each set of training images is then input into the matting model structure constructed in step 102.

[0084] Specifically, the original image of the person is input into the semantic estimation network in the matting model structure to obtain the corresponding semantic feature map. Based on the semantic feature map and the original image of the person, the corresponding detail feature map is obtained through the detail prediction network based on the attention mechanism. Based on the semantic feature map and the intermediate feature map based on the attention mechanism in the detail prediction network, the corresponding fused feature map is obtained through the fusion network in the matting model structure.

[0085] For the acquisition of detail feature maps, the semantic feature map output by the semantic estimation network is upsampled. Based on the original image of the person, features are extracted using the attention mechanism module included in the detail prediction network. The upsampled feature map and the feature map output by the attention mechanism module are fused. Based on the original image of the person and the fused feature map, the detail prediction network is used to obtain the final detail feature map.

[0086] like Figure 2 As shown, the specific process for obtaining the detail feature map is as follows: The original image of the person is input into the third downsampling layer and the second downsampling layer in the semantic estimation network. After the second downsampling layer downsamples the original image of the person, it is then upsampled through the seventh upsampling layer. The five fusion layer fuses the two feature maps output by the third downsampling layer and the seventh upsampling layer, and the fused feature map is input into the detail prediction network. In the detail prediction network, the first downsampling layer downsamples the feature map output by the fifth fusion layer, and then the attention mechanism module extracts features from the downsampled feature map. The semantic feature map output by the semantic estimation network is input into the third upsampling layer for upsampling, and then the first fusion layer fuses the two feature maps output by the attention mechanism module and the third upsampling layer. Then, the first upsampling layer upsamples the feature map output by the first fusion layer. The first staggered network also extracts features from the feature map output by the fifth fusion layer, and then the second fusion layer fuses the two feature maps output by the first staggered network and the first upsampling layer. The second upsampling layer upsamples the feature map output by the second fusion layer to obtain the final detail feature map.

[0087] In the above process, an attention mechanism module was added to enhance the model's ability to extract and learn detailed features such as the edges of clothing and hair of people in the image, which helps to improve the accuracy of the finally trained model in detail prediction.

[0088] For the process of obtaining the fused feature map, in the fusion network of the image matting model structure, the semantic feature map is upsampled, and the upsampled feature map is fused with the intermediate feature map based on the attention mechanism in the detail prediction network. The fused feature map is then upsampled again to obtain the corresponding fused feature map.

[0089] like Figure 2 As shown, the specific process for obtaining the fused feature map is as follows: the semantic feature map output by the semantic estimation network is input into the fifth upsampling layer of the fusion network for upsampling; the feature map output by the fifth upsampling layer and the feature map output by the first upsampling layer in the detail prediction network are fused through the fourth fusion layer; and the feature map output by the fourth fusion layer is upsampled through the sixth upsampling layer to obtain the final fused feature map. The intermediate feature map based on the attention mechanism in the detail prediction network is the feature map output by the first upsampling layer in the detail prediction network.

[0090] Since the fourth fusion layer incorporates the feature map output from the first upsampling layer in the detail prediction network, and the feature map output from the first upsampling layer is a feature map processed by the attention mechanism module, it has more accurate detail features. Therefore, it also increases the accuracy of the fused features obtained by the fusion network by fusing semantic features and detail features.

[0091] The specific process of obtaining semantic feature maps, such as Figure 2 As shown, the original image of the person is input into the second downsampling layer of the semantic estimation network. After the second downsampling layer downsamples the original image of the person, the second staggered network layer extracts features from the feature map output by the second downsampling layer. The third fusion layer fuses the feature map output by the second downsampling layer and the feature map output by the second staggered network layer. Finally, the fourth upsampling layer upsamples the feature map output by the third fusion layer to obtain the final semantic feature map.

[0092] Following the above method, the constructed matting model predicts the semantic features, detail features, and fusion features of the original human images, obtaining the semantic feature map, detail feature map, and fusion feature map corresponding to the original human images. Then, based on the labeled semantic feature map, labeled detail feature map, labeled fusion feature map corresponding to the original human images in the training set, as well as the semantic feature map, detail feature map, and fusion feature map predicted by the matting model structure, the overall loss value for the current training cycle is calculated.

[0093] For the semantic estimation network, the semantic loss value can be calculated separately based on the labeled semantic feature map corresponding to the original person image and the semantic feature map output by the semantic estimation network. Specifically, L2 loss function or L1 loss function can be used to calculate the semantic loss value. For the detail prediction network, the detail loss value can be calculated separately based on the labeled detail feature map corresponding to the original person image and the detail feature map output by the detail prediction network. Specifically, cross-entropy loss function can be used for calculation. For the fusion network, the fusion loss value can be calculated separately based on the labeled fusion feature map corresponding to the original person image and the fusion feature map output by the fusion network. Specifically, L2 loss function or L1 loss function can be used to calculate the fusion loss value.

[0094] After calculating the semantic loss, detail loss, and fusion loss, the three are weighted and summed to obtain the overall loss value. It is then determined whether the current overall loss value meets a preset convergence condition, which may include a preset threshold. If the current overall loss value is less than the preset threshold, the overall loss value is determined to have met the preset convergence condition. Otherwise, the preset convergence condition is determined not to have been met.

[0095] If the overall loss value reaches the preset convergence condition, the matting model structure and the current model parameters are used to construct a trained portrait matting model. If the overall loss value does not reach the preset convergence condition, the current model parameters are adjusted based on the current overall loss value. Based on the adjusted model parameters, the matting model structure is trained again in a loop according to the above process until a trained portrait matting model is obtained.

[0096] After obtaining the trained portrait matting model through the above process, the model is deployed on a server or a terminal requiring portrait matting services. When portrait matting is needed, the image of the person to be matted is acquired. The image is input into the portrait matting model to obtain the semantic feature map, detail feature map, and fused feature map corresponding to the image. Then, the semantic feature map, detail feature map, and fused feature map are fused to obtain the matted result image.

[0097] To improve the continuity of the figure's edges in the obtained matting result image, this application embodiment also performs blurring and smoothing processing on the matting result image. In one implementation, the foreground boundary pixels and their corresponding radii in the matting result image are first determined. Each pixel in the matting result image obtained by the portrait matting model has its own classification confidence score, which is the probability that the portrait matting model predicts that the pixel is a foreground pixel. In this application embodiment, pixels with classification confidence scores greater than a certain value are determined as foreground pixels, and this certain value can be 60%, 70%, 80%, etc. All foreground pixels are determined based on the classification confidence score of each pixel in the matting result image. The image area occupied by all foreground pixels is the foreground region, and the pixels located at the edge of the foreground region are the foreground boundary pixels.

[0098] For each identified foreground boundary pixel, determine which target region of the person in the matting result image that foreground boundary pixel belongs to, such as the person's head, arm, chest, or leg. Determine the smallest bounding rectangle of the target region where the foreground boundary pixel is located. Select the shorter side of this bounding rectangle, calculate the product between a preset coefficient and the length of the shorter side, and use this product as the radius corresponding to the foreground boundary pixel. This preset coefficient can be 0.1, 0.15, or 0.2, etc.

[0099] After determining the foreground boundary pixel and its corresponding radius, draw a circle with the foreground boundary pixel as the center and its corresponding radius. Then, determine the centroid of the cutout area where the foreground boundary pixel is located; this cutout area is the target area. If the centroid is located inside the circle corresponding to the foreground boundary pixel, then the foreground boundary pixel is determined as a foreground pixel; otherwise, the foreground boundary pixel is determined as a background pixel.

[0100] For each foreground boundary pixel in the matting result image, the classification result of the foreground boundary pixel is re-determined as a foreground pixel or a background pixel in the above manner to improve the accuracy of foreground boundary pixel classification and make the foreground edge extracted in the matting result image more accurate and smooth.

[0101] In another implementation, pixels whose classification confidence scores fall within a preset range in the cutout result image are identified as pixels to be processed. The preset range can be 50%-70%, 45%-65%, or 55%-75%, etc. This application embodiment does not limit the specific value of the preset range, and it can be set according to the requirements in actual applications.

[0102] For each identified pixel to be processed, determine its location within the target region of the figure in the cutout result image. Determine the minimum bounding rectangle of that target region. Calculate the product of a preset coefficient and the length of the shorter side of this minimum bounding rectangle, and use this product as the radius corresponding to the pixel to be processed. Draw a circle with the pixel to be processed as its center and its corresponding radius. Determine the centroid of the cutout region containing the pixel to be processed; this cutout region is the aforementioned target region. If the centroid is located within the circle corresponding to the pixel to be processed, then the pixel to be processed is determined as a foreground pixel; otherwise, it is determined as a background pixel.

[0103] In the matting result image obtained by the portrait matting model, pixels with classification confidence scores falling within a preset range may be predicted as foreground pixels or background pixels. For example... Figure 3 As shown, each small rectangle in the image represents a pixel. White pixels are accurately predicted background pixels, black pixels are accurately predicted foreground pixels, and the gray rectangles located between the white and black pixels are the pixels to be processed. These pixels are in the blurred area between the accurately predicted foreground and background pixels, and the prediction errors for these pixels are relatively large. Following the above method, a circle with a defined radius is drawn with the pixel to be processed as the center. The classification result of the pixel to be processed is further determined based on whether the centroid of the target area containing the pixel is within this circle. This improves the accuracy of the pixel classification and makes the foreground edges extracted in the matting result image more accurate and smooth.

[0104] In another implementation, the distinction between foreground boundary pixels and pixels to be processed can be omitted. Instead, the classification result for each pixel in the matting result image can be further determined. In this implementation, any pixel in the matting result image is called the first pixel, and the radius corresponding to the first pixel and the centroid of the matting region containing the first pixel are determined. The method for determining the radius is the same as that for determining the radius corresponding to the foreground boundary pixels and pixels to be processed, as described above, and will not be repeated here. A circle corresponding to the first pixel is drawn with the first pixel as the center and its corresponding radius. If the centroid is located inside the circle corresponding to the first pixel, then the first pixel is determined to be a foreground pixel. If the centroid is not located inside the circle, the classification result of the first pixel remains unchanged.

[0105] By further determining the classification result of each pixel, the accuracy of each pixel classification is improved, making the foreground edges of the person extracted in the cutout image more accurate and smooth.

[0106] After obtaining the final cutout result image through the embodiments of this application, the cutout foreground image can be used to create images of people against different backgrounds. For example, the cutout foreground image can be blended with a solid-color background image of blue, white, or red to obtain an ID photo image. Alternatively, the cutout foreground image can be blended with different life scenes or fictional scenes to obtain images of people against different backgrounds.

[0107] Because the foreground of the person extracted through the embodiments of this application is highly accurate and the edges are smoother, the foreground of the person and the background are well blended in the ID photo image or the image of the person in different background scenes produced in the above manner, and the visual effect is more natural.

[0108] In this embodiment, an attention mechanism module is added to the detail prediction network of the portrait matting model. This module enhances the model's ability to extract and learn detailed features of the subject's edges. The fusion network of the portrait matting model integrates semantic features and detail features extracted based on the attention mechanism, improving the accuracy of the final fused features. This results in a more accurate matting result image with more precise details. Furthermore, this application performs boundary smoothing on the foreground of the subject in the matting result image, making pixel classification more accurate and the subject edges smoother.

[0109] This application also provides a portrait cutout apparatus for performing the portrait cutout method provided in any of the above embodiments. Figure 4 As shown, the device includes:

[0110] The training acquisition module 201 is used to acquire the training set, which contains multiple sets of training images. Each set of training images includes an original human image and a corresponding labeled semantic feature map, labeled detail feature map and labeled fusion feature map.

[0111] Model building module 202 is used to build the matting model structure, which includes a detail prediction network based on an attention mechanism.

[0112] The model training module 203 is used to train the matting model structure based on the training set to obtain a trained portrait matting model.

[0113] The model training module 203 is used to input the original human image into the semantic estimation network in the matting model structure to obtain the corresponding semantic feature map; based on the semantic feature map and the original human image, the corresponding detail feature map is obtained through the detail prediction network based on the attention mechanism; based on the semantic feature map and the intermediate feature map based on the attention mechanism in the detail prediction network, the corresponding fusion feature map is obtained through the fusion network in the matting model structure.

[0114] The model training module 203 is used to upsample the semantic feature map; extract features from the original image of the person through the attention mechanism module included in the detail prediction network; fuse the upsampled feature map and the feature map output by the attention mechanism module; and obtain the final detail feature map through the detail prediction network based on the original image of the person and the fused feature map.

[0115] The model training module 203 is used to upsample the semantic feature map in the fusion network of the image matting model structure; fuse the upsampled feature map with the intermediate feature map based on the attention mechanism in the detail prediction network; and upsample the fused feature map to obtain the corresponding fused feature map.

[0116] The model training module 203 is used to calculate the overall loss value of the current training cycle based on the labeled semantic feature map, labeled detail feature map, labeled fusion feature map, semantic feature map, detail feature map and fusion feature map corresponding to the original human image. If the overall loss value reaches the preset convergence condition, the matting model structure and the current model parameters are used to form the trained human image matting model. If the overall loss value does not reach the preset convergence condition, the current model parameters are adjusted and the matting model structure is trained again based on the adjusted model parameters.

[0117] The device also includes: a model application module for acquiring the image of the person to be cut out; inputting the image of the person into the portrait cutting out model to obtain the semantic feature map, detail feature map and fusion feature map corresponding to the image of the person; and fusing the semantic feature map, detail feature map and fusion feature map corresponding to the image of the person to obtain the cutting out result image corresponding to the image of the person.

[0118] The device also includes: a smoothing module, used to determine the foreground boundary pixel and the radius corresponding to the foreground boundary pixel in the cutout result image; draw a circle corresponding to the foreground boundary pixel with the radius as the center; determine the centroid of the cutout area where the foreground boundary pixel is located; if the centroid is located inside the circle corresponding to the foreground boundary pixel, then the foreground boundary pixel is determined as a foreground pixel, otherwise the foreground boundary pixel is determined as a background pixel.

[0119] The smoothing module is also used to identify pixels whose classification confidence in the cutout result image falls within a preset range as pixels to be processed; determine the radius corresponding to the pixels to be processed; draw a circle corresponding to the pixels to be processed with the pixels to be processed as the center and the radius; determine the centroid of the cutout area where the pixels to be processed are located; if the centroid is located inside the circle corresponding to the pixels to be processed, then the pixels to be processed are identified as foreground pixels, otherwise the pixels to be processed are identified as background pixels.

[0120] The smoothing module is also used to determine the radius corresponding to the first pixel and the centroid of the cutout area where the first pixel is located. The first pixel is any pixel in the cutout result image. A circle corresponding to the first pixel is drawn with the first pixel as the center and the radius. If the centroid is located inside the circle corresponding to the first pixel, the first pixel is determined to be a foreground pixel. If the centroid is not located inside the circle, the classification result of the first pixel remains unchanged.

[0121] The portrait cutout device and the portrait cutout method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods used, run or implemented by the applications stored therein.

[0122] This application also provides an electronic device for performing the above-described portrait cutout method. Please refer to... Figure 5 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 5 As shown, the electronic device 8 includes: a processor 800, a memory 801, a bus 802, and a communication interface 803. The processor 800, the communication interface 803, and the memory 801 are connected via the bus 802. The memory 801 stores a computer program that can run on the processor 800. When the processor 800 runs the computer program, it executes the portrait matting method provided in any of the foregoing embodiments of this application.

[0123] The memory 801 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 803 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0124] Bus 802 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs. After receiving an execution instruction, the processor 800 executes the program. The portrait matting method disclosed in any of the foregoing embodiments of this application can be applied to the processor 800, or implemented by the processor 800.

[0125] The processor 800 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 800 or by instructions in software form. The processor 800 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 801. Processor 800 reads the information in memory 801 and, in conjunction with its hardware, completes the steps of the above method.

[0126] The electronic device provided in this application embodiment and the portrait cutout method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0127] This application also provides a computer-readable storage medium corresponding to the portrait matting method provided in the foregoing embodiments. Please refer to... Figure 6 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the portrait cutout method provided in any of the foregoing embodiments.

[0128] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0129] The computer-readable storage medium provided in the above embodiments of this application and the portrait cutout method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0130] It should be noted that:

[0131] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0132] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0133] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0134] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for image cutout of a human figure, characterized in that, include: Obtain a training set, which contains multiple sets of training images, each set of training images including original human images; Construct a matting model structure, which includes a detail prediction network based on an attention mechanism; Based on the training set, the matting model structure is trained to obtain a trained portrait matting model; The matting model structure includes a semantic estimation network, a detail prediction network, and a fusion network. The detail prediction network includes a first downsampling layer, an attention mechanism module, a first fusion layer, a first upsampling layer, a second fusion layer, and a second upsampling layer connected in sequence, as well as a first staggered network layer connected in parallel between the input of the first downsampling layer and the second fusion layer. The first fusion layer is also connected to a third upsampling layer. The semantic estimation network is connected to the third upsampling layer, the output of the semantic estimation network is connected to the input of the fusion network, and the output of the first upsampling layer is connected to the fusion network to obtain the matting model structure. The process of obtaining the trained portrait matting model further includes: Obtain the image of the person to be cut out; input the image of the person into the portrait cutting out model to obtain the semantic feature map, detail feature map and fusion feature map corresponding to the image of the person; fuse the semantic feature map, detail feature map and fusion feature map corresponding to the image of the person to obtain the cutout result map corresponding to the image of the person.

2. The method according to claim 1, characterized in that, Based on the training set, the matting model structure is trained to obtain a trained portrait matting model, including: The original human image is input into the semantic estimation network in the matting model structure to obtain the corresponding semantic feature map; Based on the semantic feature map and the original character image, the corresponding detail feature map is obtained through the attention-based detail prediction network. Based on the semantic feature map and the intermediate feature map based on the attention mechanism in the detail prediction network, the corresponding fused feature map is obtained through the fusion network in the matting model structure; The intermediate feature map is the feature map output by the first upsampling layer.

3. The method according to claim 2, characterized in that, The step of obtaining the corresponding detail feature map through the attention-based detail prediction network based on the semantic feature map and the original character image includes: Upsample the semantic feature map; Based on the original image of the person, feature extraction is performed through the attention mechanism module included in the detail prediction network; The feature map obtained by upsampling is fused with the feature map output by the attention mechanism module; Based on the original image of the person and the fused feature map, the final detail feature map is obtained through the detail prediction network.

4. The method according to claim 2, characterized in that, The step of obtaining the corresponding fused feature map through the fusion network in the matting model structure based on the semantic feature map and the intermediate feature map based on the attention mechanism in the detail prediction network includes: In the fusion network of the image matting model structure, the semantic feature map is upsampled; The upsampled feature map is fused with the intermediate feature map based on the attention mechanism in the detail prediction network; Upsample the fused feature map to obtain the corresponding fused feature map.

5. The method according to any one of claims 2-4, characterized in that, Each training image set includes an original image of a person and a corresponding labeled semantic feature map, labeled detail feature map, and labeled fusion feature map; The method further includes: Based on the labeled semantic feature map, labeled detail feature map, labeled fusion feature map, the semantic feature map, the detail feature map, and the fusion feature map corresponding to the original character image, calculate the overall loss value for the current training cycle; If the overall loss value reaches the preset convergence condition, the matting model structure and the current model parameters will be used to form a trained portrait matting model; If the overall loss value does not reach the preset convergence condition, the current model parameters are adjusted, and the matting model structure is trained again based on the adjusted model parameters.

6. The method according to claim 1, characterized in that, After obtaining the cutout result image corresponding to the person image, the process further includes: Determine the foreground boundary pixels and the radius corresponding to the foreground boundary pixels in the matting result image; Draw a circle corresponding to the foreground boundary pixel with the foreground boundary pixel as the center and the radius. Determine the centroid of the cutout region where the foreground boundary pixels are located; If the centroid is located within the circle corresponding to the foreground boundary pixel, then the foreground boundary pixel is determined to be a foreground pixel; otherwise, the foreground boundary pixel is determined to be a background pixel.

7. The method according to claim 1, characterized in that, After obtaining the cutout result image corresponding to the person image, the process further includes: Pixels whose classification confidence scores fall within a preset range in the cutout result image are identified as pixels to be processed. Determine the radius corresponding to the pixel to be processed; Draw a circle corresponding to the pixel to be processed with the pixel as the center and the radius as the radius; Determine the centroid of the cutout region where the pixel to be processed is located; If the centroid is located within the circle corresponding to the pixel to be processed, then the pixel to be processed is determined to be a foreground pixel; otherwise, the pixel to be processed is determined to be a background pixel.

8. The method according to claim 1, characterized in that, After obtaining the cutout result image corresponding to the person image, the process further includes: Determine the radius corresponding to the first pixel and the centroid of the cutout region where the first pixel is located; the first pixel is any pixel in the cutout result image. Draw a circle corresponding to the first pixel with the first pixel as the center and the radius specified. If the centroid is located within the circle corresponding to the first pixel, then the first pixel is determined to be a foreground pixel; If the centroid is not located within the circle, the classification result of the first pixel remains unchanged.

9. A portrait cutout device, characterized in that, include: The training acquisition module is used to acquire a training set, which contains multiple sets of training images. Each set of training images includes an original human image and a corresponding labeled semantic feature map, labeled detail feature map and labeled fusion feature map. The model building module is used to build the matting model structure, which includes a detail prediction network based on an attention mechanism. A model training module is used to train the matting model structure based on the training set to obtain a trained portrait matting model. The matting model structure includes a semantic estimation network, a detail prediction network, and a fusion network. The detail prediction network includes a first downsampling layer, an attention mechanism module, a first fusion layer, a first upsampling layer, a second fusion layer, and a second upsampling layer connected in sequence, as well as a first staggered network layer connected in parallel between the input of the first downsampling layer and the second fusion layer. The first fusion layer is also connected to a third upsampling layer. The semantic estimation network is connected to the third upsampling layer, the output of the semantic estimation network is connected to the input of the fusion network, and the output of the first upsampling layer is connected to the fusion network to obtain the matting model structure. The process of obtaining the trained portrait matting model further includes: Obtain the image of the person to be cut out; input the image of the person into the portrait cutting out model to obtain the semantic feature map, detail feature map and fusion feature map corresponding to the image of the person; fuse the semantic feature map, detail feature map and fusion feature map corresponding to the image of the person to obtain the cutout result map corresponding to the image of the person.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-8.

Images