Introducing ESA's 2D digital portrait cutout method and system

Abstract

This invention discloses a 2D digital human portrait matting method and system using ESA, belonging to the field of image processing technology. The method aims to solve the technical problem of achieving performance-delay balanced portrait matting based on a spatial attention mechanism. The method comprises the following steps: acquiring datasets, including the coco8-seg dataset and the open-source P3M-10k portrait matting dataset; constructing a sample dataset, which includes images with and without facial privacy issues; performing data enhancement and image preprocessing on the images and corresponding masks; introducing the ESA module into the RepVit network model to obtain the ESA-RepViT backbone network model; using the ESA-RepViT backbone network model as the backbone network and constructing a portrait matting network model based on the YOLOv8 network; and performing feature extraction on the trained portrait matting network model to output the corresponding mask.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a 2D digital human portrait cutout method and system using ESA. Background Art

[0002] In neural networks, the attention mechanism is an important technology used to enhance the model's attention to input information, thereby improving the model's performance when handling complex tasks. Traditional neural network models treat all input information equally and are unable to effectively distinguish the importance of input information. This leads to performance degradation and increased computational costs when processing large-scale spatial information. Therefore, the introduction of the attention mechanism has become an effective way to solve this problem. The SA spatial self-attention mechanism is currently a widely used attention mechanism and has shown good results when processing spatial information. It learns the relationship between the input spatial information and dynamically assigns attention weights to different positions, allowing the model to focus on the most important parts of the input information.

[0003] Portrait Matting is a high-frequency application example of semantic segmentation. As an important component of pre- or post-processing for many tasks, it plays an irreplaceable role. It is particularly prominent in the full-process technology of 2D digital human production. It determines the final visualization effect in the post-processing of face retalking in various TFG models, such as Wav2Lip, VideoRetalking, DINet, and SyncTalkFace. Current portrait matting algorithms mainly include trimap-based and trimap-free. The latter is less effective than the former. However, considering the difficulty of obtaining prior knowledge in practical applications, trimap-free methods are still chosen in most cases, such as FaceParsing, MODNet, SAM, and SAM-Track, which is improved based on SAM.

[0004] The underlying idea behind the spatial self-attention mechanism is to help the model focus on features that are strongly relevant to the task, enhancing the network's learning capabilities and thus improving the model's expressive power. However, existing self-attention mechanism designs often rely on data path design strategies, namely feedforward propagation mechanisms, while the essence of learning is achieved through the reverse gradient propagation chain differentiation rule. Specifically, different task objectives and loss constraint strategies have a greater impact on the network's learning content, rather than the influence of relevant layers in the feedforward sequence.

[0005] How to achieve performance-delay balance in portrait cutout based on the spatial attention mechanism is a technical problem that needs to be solved. Summary of the Invention

[0006] The technical task of the present invention is to address the above shortcomings and provide a 2D digital human portrait cutout method and system that introduces ESA to solve the technical problem of how to achieve performance-delay balanced portrait cutout based on the spatial attention mechanism.

[0007] The present invention provides a 2D digital human portrait cutout method using ESA, comprising the following steps:

[0008] S100 data collection: Data sets were collected, including the coco8-seg dataset and the open-source P3M-10k portrait cutout dataset. For the P3M-10k portrait cutout dataset, the images in its training set all protect facial privacy, while the images in its two test sets are privacy-free. The coco8-seg dataset contains semantic segmentation data for multiple categories of objects.

[0009] S200 dataset construction: A sample dataset was constructed based on the coco8-seg dataset and the open-source P3M-10k portrait cutout dataset. The sample dataset includes images that protect facial privacy and images that do not have facial privacy issues. The sample dataset was then divided into training, validation, and test sets.

[0010] S300 Image preprocessing: For each image and corresponding mask in the training validation set, perform data augmentation and image preprocessing on the image and the corresponding mask to obtain a preprocessed training validation set;

[0011] S400 model construction: Introducing the ESA module into the RepVit network model to obtain the ESA-RepViT backbone network model, using the ESA-RepViT backbone network model as the backbone network, and building a portrait cutout network model based on the YOLOv8 network;

[0012] S500 model training: The portrait cutout network model is trained and verified based on the preprocessed training and validation sets, and the trained portrait cutout network model is tested based on the test set to obtain the trained portrait cutout network model;

[0013] S600 Portrait Cutout: For a 2D portrait to be processed, data enhancement and image preprocessing are performed on the image to obtain a preprocessed image. The preprocessed image is used as input, feature extraction is performed through the trained portrait cutout network model, and the corresponding mask is output.

[0014] Preferably, constructing the data set includes the following steps:

[0015] For the two test sets without face privacy issues in the P3M-10k portrait cutout dataset, α images are manually screened out, where the image corresponding to the i-th data is A i, where i∈{1,2,…,α};

[0016] For the person data in the coco8-seg dataset, β images are manually screened out, and the jth image is B j , where j∈{1,2,…,β};

[0017] Mixed filtered image A i and image B j , build guidance dataset D m , where, m∈{1,2,…,α+β};

[0018] For the guidance dataset D m , select γ images from them, and select δ images that protect the privacy of faces from the P3M-10k portrait cutout dataset, and construct a training and verification set F of γ+δ images k , where k∈{1,2,…,γ+δ}, and the training data and validation data in the training validation set are divided proportionally;

[0019] Construct a test set based on the remaining (α+β-γ) images.

[0020] Preferably, performing data enhancement and image preprocessing on the image and the corresponding mask comprises the following steps:

[0021] For image F k , use the ColorJitter function in Python to enhance the image, and together with the corresponding mask M k Perform random cropping, random translation, and horizontal flipping operations to obtain the enhanced image F i k and mask M i k ;

[0022] The enhanced image F i k and its mask M i k Scale to 640×640 pixels and use Python to i k Perform tensor transformation, normalization, dimension expansion, and dimension inversion operations in sequence to obtain the preprocessed image F' k and mask M' k .

[0023] Preferably, the portrait cutout network model includes an ESA-RepViT network structure, an SPPF module, a Neck part and a Head part, wherein the neck part includes a PAN network structure. The portrait cutout network model is used to perform the following feature extraction and mask output on the input image:

[0024] L100, image I in Input ESA-RepViT network structure, pass through Stage2, Stage3, Stage4 modules in sequence, and get FM s2 ,FM s3 and FM s4 ;

[0025] L200, FM s4 Bring in the SPPF module and get FM sppf ,in,

[0026]

[0027] L300, FM sppf With FM s2 ,FM s3 The multi-scale features are respectively input into the PAN network for fusion and extraction, and the output of the PAN network is input into the Head part, and the final portrait cutout MASK is obtained through the output layer.

[0028] Preferably, step L100 includes the following operations:

[0029] L110, Image I in Input the Stem module, and pass through the convolution block with a convolution kernel of 3×3 and a step size of 2, the activation function module, and the convolution block with a convolution kernel of 3×3 and a step size of 2 to obtain FM 1-1 ,FM 1-2 and FM 1-3 ,in,

[0030] L120, FM 1-3 The input Stage1 module passes through two RepViTBlock network structures in sequence. The two RepViTBlock network structures are the first RepViTBlock1 and the second RepViTBlock2. The first RepViTBlock1 passes through the Token Mixer, ESA module, SE module, and serial 1x1 expansion convolution layer and 1x1 projection layer to realize the FFN module of interaction between channels, respectively.

[0031] In the second RepViTBlock2, the Token Mixer, which is composed of parallel 3×3 and 1×1 depth convolutions and integrates spatial information, the serial 1x1 expansion convolution layer and the 1x1 projection layer are used to realize the FFN module for interaction between channels, and the following are obtained:

[0032] L130, FM 2-6 Input DownSample module, pass through RepViTBlock2, depth convolution block with convolution kernel of 3×3 and stride of 2, 1×1 convolution block and FFN module, and get

[0033] L140, FM 3-4 Input Stage2 module, pass through two RepViTBlocks in sequence, the first RepViTBlock1 and the second RepViTBlock2, in the first RepViTBlock1 pass through TokenMixer, ESA module, SE module and FFN module in sequence, and get In the second RepViTBlock2, after passing through the TokenMixer and FFN modules, we can get

[0034] L150, FM 4-6 Input DownSample module, pass through RepViTBlock2, depth convolution block with convolution kernel of 3×3 and stride of 2, 1×1 convolution block and FFN module, and get

[0035] L160, FM 5-4 Input Stage3 module, pass through two RepViTBlocks in sequence, and in the first RepViTBlock1 pass through Token Mixer, ESA module, SE module and FFN module in sequence, and get In the second RepViTBlock2, after passing through the TokenMixer and FFN modules, we can get

[0036] L170, FM 6-6Input DownSample module, pass through RepViTBlock2, depth convolution block with convolution kernel of 3×3 and stride of 2, 1×1 convolution block and FFN module, and get

[0037] L180, FM 7-4 Input Stage4 module, pass through two RepViTBlocks in sequence, and in the first RepViTBlock1 pass through Token Mixer, ESA module, SE module and FFN module in sequence, and get In the second RepViTBlock2, after passing through the TokenMixer and FFN modules, we can get

[0038] Preferably, step L120 includes the following steps:

[0039] FM 2-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and

[0040] FM 2-1-1 and FM 2-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain

[0041] The supervision mask M' k Scale to 160 x 160 pixels and use FM 2-1-4 Construct LMSE loss constraints;

[0042] FM 2-1-4 Heavy action on FM 2-1 Perform Hadamard product to get FM 2-2 .

[0043] Preferably, in step L140, the ESA module performs the following:

[0044] FM 4-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and

[0045] FM 4-1-1 and FM 4-1-2By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain

[0046] The supervision mask M' k Scale to 80×80 pixels and use FM 4-1-4 Construct LMSE loss constraints;

[0047] FM 4-1-4 Heavy action on FM 4-1 Perform Hadamard product to get FM 4-2 .

[0048] Preferably, in step L160, the ESA module performs the following:

[0049] FM 6-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and

[0050] FM 6-1-1 and FM 6-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain

[0051] The supervision mask M' k Scale to 40×40 pixels and use FM 6-1-4 Construct LMSE loss constraints;

[0052] FM 6-1-4 Heavy action on FM 6-1 Perform Hadamard product to get FM 6-2 .

[0053] Preferably, in step L180, the ESA module performs the following operations:

[0054] FM 8-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and

[0055] FM 8-1-1 and FM 8-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain

[0056] The supervision mask M' k Scale to 20×20 pixels and use FM 8-1-4 Construct LMSE loss constraints;

[0057] FM 8-1-4 Heavy action on FM 8-1 Perform Hadamard product to get FM 8-2 .

[0058] In a second aspect, the present invention provides a 2D digital human portrait cutout system using an ESA, which is used to cutout a 2D digital human portrait using a 2D digital human portrait cutout method using an ESA as described in any one of the first aspects, wherein the system includes a data acquisition module, a data set construction module, an image preprocessing module, a model construction module, a model training module, and a portrait cutout module;

[0059] The data acquisition module is used to perform the following: collect data sets, including the coco8-seg data set and the open source P3M-10k portrait cutout data set. For the P3M-10k portrait cutout data set, the images in its training set are all images that protect the privacy of human faces, and the images in the two test sets are images without privacy issues. The coco8-seg data set contains semantic segmentation data of multiple categories of objects;

[0060] The dataset construction module is used to perform the following: construct a sample dataset based on the coco8-seg dataset and the open source P3M-10k portrait cutout dataset, the sample dataset including images that protect face privacy and images without face privacy issues, and divide the sample dataset into training validation sets and test sets;

[0061] The image preprocessing module is used to perform the following: for each image and corresponding mask in the training validation set, perform data enhancement and image preprocessing on the image and the corresponding mask to obtain a preprocessed training validation set;

[0062] The model construction module is used to perform the following: introduce the ESA module into the RepVit network model to obtain the ESA-RepViT backbone network model, use the ESA-RepViT backbone network model as the backbone network, and build a portrait cutout network model based on the YOLOv8 network;

[0063] The model training module is used to perform the following: perform model training and model verification on the portrait cutout network model based on the preprocessed training verification set, and test the trained portrait cutout network model based on the test set to obtain the trained portrait cutout network model;

[0064] The portrait cutout module is used to perform the following: for a 2D portrait to be processed, data enhancement and image preprocessing are performed on the image to obtain a preprocessed image, the preprocessed image is used as input, feature extraction is performed through the trained portrait cutout network model, and the corresponding mask is output.

[0065] The 2D digital human portrait cutout method and system using ESA of the present invention have the following advantages:

[0066] 1. The proposed ESA attention mechanism, guided by the gradient path design strategy, introduces high-quality masks for auxiliary supervision. Compared with the data path design strategy, which designs a deeper network structure to obtain rich gradient flows, it significantly reduces the network size and improves the efficiency of inference feedforward propagation. It also has good performance and is suitable for real-time processing on the device.

[0067] 2. The ESA-RepViT network, which is modified by introducing the ESA mechanism, gives global information a second incremental attention, which is more in line with the subjective cognition of the portrait cutout task and improves the representation ability of global fine-grained information and semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] The present invention will be further described below with reference to the accompanying drawings.

[0070] Figure 1 This is a structural block diagram of the ESA module in a 2D digital portrait cutout method using ESA in Example 1;

[0071] Figure 2 This is a structural block diagram of a portrait cutout network model in a 2D digital portrait cutout method that introduces ESA in Example 1. DETAILED DESCRIPTION

[0072] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0073] The embodiments of the present invention provide a 2D digital human portrait cutout method and system that introduces ESA, which is used to solve the technical problem of how to achieve performance-delay balanced portrait cutout based on a spatial attention mechanism.

[0074] Example 1:

[0075] The present invention provides a 2D digital human portrait cutout method introducing ESA, which includes six steps: data acquisition, data set construction, image preprocessing, model construction, model training and portrait cutout.

[0076] S100 data collection: Data sets are collected, including the coco8-seg dataset and the open-source P3M-10k portrait cutout dataset. For the P3M-10k portrait cutout dataset, the images in its training set are all images that protect facial privacy, and the images in the two test sets are images without privacy issues. The coco8-seg dataset contains semantic segmentation data for multiple categories of targets.

[0077] Among them, the P3M-10k portrait cutout dataset covers 10,421 training sets that protect facial privacy and two test sets without portrait privacy issues. The coco8-seg dataset contains semantic segmentation data of 80 categories of targets, and only uses the person category data.

[0078] S200 dataset construction: A sample dataset was constructed based on the coco8-seg dataset and the open source P3M-10k portrait cutout dataset. The sample dataset includes images that protect facial privacy and images without facial privacy issues. The sample dataset is divided into training validation sets and test sets.

[0079] In this embodiment, constructing a data set includes the following steps:

[0080] (1) For the two test sets without face privacy issues in the P3M-10k portrait cutout dataset, α images are manually screened out, where the image corresponding to the i-th data is A i , where i∈{1,2,…,α};

[0081] (2) For the person data in the coco8-seg dataset, β images are manually screened out, and the jth image is B j , where j∈{1,2,…,β};

[0082] (3) Mixed filtered image A i and image B j , build guidance dataset D m , where, m∈{1,2,…,α+β};

[0083] (4) For the guidance dataset D m , select γ images from them, and select δ images that protect the privacy of faces from the P3M-10k portrait cutout dataset, and construct a training and verification set F of γ+δ imagesk , where k∈{1,2,…,γ+δ}, and the training data and validation data in the training validation set are divided proportionally;

[0084] (5) Construct a test set based on the remaining (α+β-γ) images.

[0085] S300 Image preprocessing: For each image and corresponding mask in the training validation set, data enhancement and image preprocessing are performed on the image and the corresponding mask to obtain a preprocessed training validation set.

[0086] In this embodiment, data enhancement and image preprocessing of the image and mask include the following operations:

[0087] (1) For image F k , use the ColorJitter function in Python to enhance the image, and together with the corresponding mask M k Perform random cropping, random translation, and horizontal flipping operations to obtain the enhanced image F i k and mask M i k ;

[0088] (2) The enhanced image F i k and its mask M i k Scale to 640×640 pixels and use Python to i k Perform tensor transformation, normalization, dimension expansion, and dimension inversion operations in sequence to obtain the preprocessed image F' k and mask M' k .

[0089] S400 model construction: The ESA module is introduced into the RepVit network model to obtain the ESA-RepViT backbone network model. The ESA-RepViT backbone network model is used as the backbone network, and a portrait cutout network model is constructed based on the YOLOv8 network.

[0090] In this embodiment, the overall structure of the ESA module is as follows: Figure 1 As shown, the overall structure of the ER-YOLOv8 portrait cutout network is as follows Figure 2As shown in the figure, it includes the ESA-RepViT network structure, SPPF module, Neck and Head. The neck includes the PAN network structure. Its logical relationship is that the Backbone network of ER-YOLOv8 is ESA-RepViT. The network structure of the Neck and Head parts is consistent with the network structure of the Neck and Head parts of the standard YOLOv8. The ESA module is introduced into the ESA-RepViT network structure. k and its mask M' k Input into the ER-YOLOv8 network and output the portrait mask G k .

[0091] As a specific implementation, the portrait cutout network model is used to perform the following feature extraction and mask output on the input image:

[0092] L100, image I in Input ESA-RepViT network structure, pass through Stage2, Stage3, Stage4 modules in sequence, and get FM s2 ,FM s3 and FM s4 ;

[0093] L200, FM s4 Bring in the SPPF module and get FM sppf ,in,

[0094] L300, FM sppf With FM s2 ,FM s3 The multi-scale features are respectively input into the PAN network for fusion and extraction, and the output of the PAN network is input into the Head part, and the final portrait cutout MASK is obtained through the output layer.

[0095] Step L100 includes the following operations:

[0096] L110, Image I in Input the Stem module, and pass through the convolution block with a convolution kernel of 3×3 and a step size of 2, the activation function module, and the convolution block with a convolution kernel of 3×3 and a step size of 2 to obtain FM 1-1 ,FM 1-2 and FM 1-3 ,in,

[0097] L120, FM 1-3The input Stage1 module passes through two RepViTBlock network structures in sequence. The two RepViTBlock network structures are the first RepViTBlock1 and the second RepViTBlock2. The first RepViTBlock1 passes through the Token Mixer, ESA module, SE module, and serial 1x1 expansion convolution layer and 1x1 projection layer to realize the FFN module of interaction between channels, respectively. In the second RepViTBlock2, the Token Mixer, which is composed of parallel 3×3 and 1×1 depth convolutions and integrates spatial information, the serial 1x1 expansion convolution layer and the 1x1 projection layer are used to realize the FFN module for interaction between channels, and the following are obtained:

[0098] L130, FM 2-6 Input DownSample module, pass through RepViTBlock2, depth convolution block with convolution kernel of 3×3 and stride of 2, 1×1 convolution block and FFN module, and get

[0099] L140, FM 3-4 Input Stage2 module, pass through two RepViTBlocks in sequence, the first RepViTBlock1 and the second RepViTBlock2, in the first RepViTBlock1 pass through TokenMixer, ESA module, SE module and FFN module in sequence, and get In the second RepViTBlock2, after passing through the TokenMixer and FFN modules, we can get

[0100] L150, FM 4-6 Input DownSample module, pass through RepViTBlock2, depth convolution block with convolution kernel of 3×3 and stride of 2, 1×1 convolution block and FFN module, and get

[0101] L160, FM 5-4Input Stage3 module, pass through two RepViTBlocks in sequence, and in the first RepViTBlock1 pass through Token Mixer, ESA module, SE module and FFN module in sequence, and get In the second RepViTBlock2, after passing through the TokenMixer and FFN modules, we can get

[0102] L170, FM 6-6 Input DownSample module, pass through RepViTBlock2, depth convolution block with convolution kernel of 3×3 and stride of 2, 1×1 convolution block and FFN module, and get

[0103] L180, FM 7-4 Input Stage4 module, pass through two RepViTBlocks in sequence, and in the first RepViTBlock1 pass through Token Mixer, ESA module, SE module and FFN module in sequence, and get In the second RepViTBlock2, after passing through the TokenMixer and FFN modules, we can get

[0104] Step L120 includes the following steps:

[0105] L121, FM 2-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and

[0106] L122, FM 2-1-1 and FM 2-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain

[0107] L123, will monitor the mask M' k Scale to 160 x 160 pixels and use FM 2-1-4 Construct LMSE loss constraints;

[0108] L124, FM 2-1-4 Heavy action on FM2-1 Perform Hadamard product to get FM 2-2 .

[0109] In step L140, the ESA module performs the following:

[0110] L141, FM 4-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and

[0111] L142, FM 4-1-1 and FM 4-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain

[0112] L143, will monitor the mask M' k Scale to 80×80 pixels and use FM 4-1-4 Construct LMSE loss constraints;

[0113] L144, FM 4-1-4 Heavy action on FM 4-1 Perform Hadamard product to get FM 4-2 .

[0114] In step L160, the ESA module performs the following:

[0115] L161, FM 6-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and

[0116] L162, FM 6-1-1 and FM 6-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain

[0117] L163, will monitor the mask M' k Scale to 40×40 pixels and use FM 6-1-4 Construct LMSE loss constraints;

[0118] L164, FM 6-1-4 Heavy action on FM 6-1 Perform Hadamard product to get FM6-2 .

[0119] In step L180, the ESA module performs the following operations:

[0120] L181, FM 8-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and

[0121] L182, FM 8-1-1 and FM 8-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain

[0122] L183, will monitor the mask M' k Scale to 20×20 pixels and use FM 8-1-4 Construct LMSE loss constraints;

[0123] L184, FM 8-1-4 Heavy action on FM 8-1 Perform Hadamard product to get FM 8-2 .

[0124] S500 model training: The portrait cutout network model is trained and verified based on the preprocessed training validation set, and the trained portrait cutout network model is tested based on the test set to obtain the trained portrait cutout network model.

[0125] S600 Portrait Cutout: For a 2D portrait to be processed, data enhancement and image preprocessing are performed on the image to obtain a preprocessed image. The preprocessed image is used as input, feature extraction is performed through the trained portrait cutout network model, and the corresponding mask is output.

[0126] Example 2:

[0127] The present invention provides a 2D digital human portrait cutout system introducing ESA, comprising a data acquisition module, a data set construction module, an image preprocessing module, a model construction module, a model training module and a portrait cutout module.

[0128] The data acquisition module is used to perform the following: collect datasets, including the coco8-seg dataset and the open-source P3M-10k portrait cutout dataset. For the P3M-10k portrait cutout dataset, the images in its training set are all images that protect facial privacy, and the images in the two test sets are images without privacy issues. The coco8-seg dataset contains semantic segmentation data for multiple categories of targets.

[0129] The dataset construction module is used to perform the following: construct a sample dataset based on the coco8-seg dataset and the open-source P3M-10k portrait cutout dataset. The sample dataset includes images that protect facial privacy and images that do not have facial privacy issues. The sample dataset is then divided into training, validation, and test sets.

[0130] The image preprocessing module is used to perform the following: for each image and corresponding mask in the training verification set, perform data enhancement and image preprocessing on the image and the corresponding mask to obtain a preprocessed training verification set.

[0131] The model construction module is used to perform the following: introduce the ESA module into the RepVit network model to obtain the ESA-RepViT backbone network model, use the ESA-RepViT backbone network model as the backbone network, and build a portrait cutout network model based on the YOLOv8 network.

[0132] The model training module is used to perform the following: perform model training and model verification on the portrait cutout network model based on the preprocessed training verification set, and test the trained portrait cutout network model based on the test set to obtain the trained portrait cutout network model.

[0133] The portrait cutout module is used to perform the following: for the 2D portrait to be processed, data enhancement and image preprocessing are performed on the image to obtain a preprocessed image, the preprocessed image is used as input, feature extraction is performed through the trained portrait cutout network model, and the corresponding mask is output.

[0134] The system of this embodiment can cut out the 2D digital human portrait using the method disclosed in Example 1 to obtain a corresponding mask.

[0135] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A 2D digital human portrait cutout method using ESA, characterized in that: The steps include: S100 data collection: Data sets were collected, including the coco8-seg dataset and the open-source P3M-10k portrait cutout dataset. For the P3M-10k portrait cutout dataset, the images in its training set all protect facial privacy, while the images in its two test sets are privacy-free. The coco8-seg dataset contains semantic segmentation data for multiple categories of objects. S200 dataset construction: A sample dataset was constructed based on the coco8-seg dataset and the open-source P3M-10k portrait cutout dataset. The sample dataset includes images that protect facial privacy and images that do not have facial privacy issues. The sample dataset was then divided into training, validation, and test sets. S300 Image preprocessing: For each image and corresponding mask in the training validation set, perform data augmentation and image preprocessing on the image and the corresponding mask to obtain a preprocessed training validation set; S400 model construction: Introducing the ESA module into the RepVit network model to obtain the ESA-RepViT backbone network model, using the ESA-RepViT backbone network model as the backbone network, and building a portrait cutout network model based on the YOLOv8 network; S500 model training: The portrait cutout network model is trained and verified based on the preprocessed training and validation sets, and the trained portrait cutout network model is tested based on the test set to obtain the trained portrait cutout network model; S600 Portrait Cutout: For a 2D portrait to be processed, data enhancement and image preprocessing are performed on the image to obtain a preprocessed image. The preprocessed image is used as input, feature extraction is performed through the trained portrait cutout network model, and the corresponding mask is output.

2. The 2D digital human portrait cutout method using ESA according to claim 1, characterized in that: Building a dataset includes the following steps: For the two test sets without face privacy issues in the P3M-10k portrait cutout dataset, α images are manually screened out, where the image corresponding to the i-th data is A i , where i∈{1,2,…,α}; For the person data in the coco8-seg dataset, β images are manually screened out, and the jth image is B j , where j∈{1,2,…,β}; Mixed filtered image A i and image B j , build guidance dataset D m , where, m∈{1,2,…,α+β}; For the guidance dataset D m , select γ images from them, and select δ images that protect the privacy of faces from the P3M-10k portrait cutout dataset, and construct a training and verification set F of γ+δ images k , where k∈{1,2,…,γ+δ}, and the training data and validation data in the training validation set are divided proportionally; Construct a test set based on the remaining α+β-γ images.

3. The 2D digital human portrait cutout method using ESA according to claim 1, characterized in that: Performing data enhancement and image preprocessing on the image and the corresponding mask includes the following steps: For image F k , use the ColorJitter function in Python to enhance the image, and together with the corresponding mask M k Perform random cropping, random translation, and horizontal flipping operations to obtain the enhanced image F i k and mask M i k ; The enhanced image F i k and its mask M i k Scale to 640×640 pixels and use Python to i k Perform tensor transformation, normalization, dimension expansion, and dimension inversion operations in sequence to obtain the preprocessed image F , k and mask M , k .

4. The 2D digital human portrait cutout method using ESA according to claim 1, characterized in that: The portrait cutout network model includes an ESA-RepViT network structure, an SPPF module, a Neck part, and a Head part. The neck part includes a PAN network structure. The portrait cutout network model is used to perform the following feature extraction and mask output on the input image: L100, image I in Input ESA-RepViT network structure, pass through Stage2, Stage3, Stage4 modules in sequence, and get FM s2 ,FM s3 and FM s4 ; L200, FM s4 Bring in the SPPF module and get FM sppf ,in, L300, FM sppf With FM s2 ,FM s3 The multi-scale features are respectively input into the PAN network for fusion and extraction, and the output of the PAN network is input into the Head part, and the final portrait cutout MASK is obtained through the output layer.

5. The 2D digital human portrait cutout method using ESA according to claim 3, characterized in that: Step L100 includes the following operations: L110, Image I in Input the Stem module, and pass through the convolution block with a convolution kernel of 3×3 and a step size of 2, the activation function module, and the convolution block with a convolution kernel of 3×3 and a step size of 2 to obtain FM 1-1 ,FM 1-2 and FM 1-3 ,in, L120, FM 1-3 The input Stage1 module passes through two RepViTBlock network structures in sequence. The two RepViTBlock network structures are the first RepViTBlock1 and the second RepViTBlock2. The first RepViTBlock1 passes through the Token Mixer, ESA module, SE module, and serial 1x1 expansion convolution layer and 1x1 projection layer to realize the FFN module of interaction between channels, respectively. In the second RepViTBlock2, the Token Mixer, which is composed of parallel 3×3 and 1×1 depth convolutions and integrates spatial information, the serial 1x1 expansion convolution layer and the 1x1 projection layer are used to realize the FFN module for interaction between channels, and the following are obtained: L130, FM 2-6 Input DownSample module, pass through RepViTBlock2, depth convolution block with convolution kernel of 3×3 and stride of 2, 1×1 convolution block and FFN module, and get L140, FM 3-4 Input Stage2 module, pass through two RepViTBlocks in sequence, the first RepViTBlock1 and the second RepViTBlock2, in the first RepViTBlock1 pass through TokenMixer, ESA module, SE module and FFN module in sequence, and get In the second RepViTBlock2, after passing through the TokenMixer and FFN modules, we can get L150, FM 4-6 Input DownSample module, pass through RepViTBlock2, depth convolution block with convolution kernel of 3×3 and stride of 2, 1×1 convolution block and FFN module, and get L160, FM 5-4 Input Stage3 module, pass through two RepViTBlocks in sequence, and in the first RepViTBlock1 pass through Token Mixer, ESA module, SE module and FFN module in sequence, and get In the second RepViTBlock2, after passing through the TokenMixer and FFN modules, we can get L170, FM 6-6 Input DownSample module, pass through RepViTBlock2, depth convolution block with convolution kernel of 3×3 and stride of 2, 1×1 convolution block and FFN module, and get L180, FM 7-4 Input Stage4 module, pass through two RepViTBlocks in sequence, and in the first RepViTBlock1 pass through Token Mixer, ESA module, SE module and FFN module in sequence, and get In the second RepViTBlock2, after passing through the TokenMixer and FFN modules, we can get 6. The 2D digital human portrait cutout method using ESA according to claim 4, characterized in that: Step L120 includes the following steps: FM 2-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and FM 2-1-1 and FM 2-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain The supervision mask M , k Scale to 160 x 160 pixels and use FM 2-1-4 Construct LMSE loss constraints; FM 2-1-4 Heavy action on FM 2-1 Perform Hadamard product to get FM 2-2 .

7. The 2D digital human portrait cutout method using ESA according to claim 4, characterized in that: In step L140, the ESA module performs the following: FM 4-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and FM 4-1-1 and FM 4-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain The supervision mask M , k Scale to 80×80 pixels and use FM 4-1-4 Construct LMSE loss constraints; FM 4-1-4 Heavy action on FM 4-1 Perform Hadamard product to get FM 4-2 .

8. The 2D digital human portrait cutout method using ESA according to claim 4, characterized in that: In step L160, the ESA module performs the following: FM 6-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and FM 6-1-1 and FM 6-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain The supervision mask M , k Scale to 40×40 pixels and use FM 6-1-4 Construct LMSE loss constraints; FM 6-1-4 Heavy action on FM 6-1 Perform Hadamard product to get FM 6-2 .

9. The 2D digital human portrait cutout method using ESA according to claim 4, characterized in that: In step L180, the ESA module performs the following operations: FM 8-1 The parallel input is fed into the C-GAP and C-GMP modules for global average pooling and global maximum pooling along the channel respectively, and the obtained and FM 8-1-1 and FM 8-1-2 By juxtaposition by channel, we get Then 1×1 convolution and Sigmoid activation function are performed to obtain The supervision mask M , k Scale to 20×20 pixels and use FM 8-1-4 Construct LMSE loss constraints; FM 8-1-4 Heavy action on FM 8-1 Perform Hadamard product to get FM 8-2 .

10. A 2D digital human portrait cutout system using ESA, characterized in that: Used to perform cutout on a 2D digital human portrait by using a 2D digital human portrait cutout method introducing ESA as described in any one of claims 1 to 8, the system comprising a data acquisition module, a data set construction module, an image preprocessing module, a model construction module, a model training module, and a portrait cutout module; The data acquisition module is used to perform the following: collect data sets, including the coco8-seg data set and the open source P3M-10k portrait cutout data set. For the P3M-10k portrait cutout data set, the images in its training set are all images that protect the privacy of human faces, and the images in the two test sets are images without privacy issues. The coco8-seg data set contains semantic segmentation data of multiple categories of objects; The dataset construction module is used to perform the following: construct a sample dataset based on the coco8-seg dataset and the open source P3M-10k portrait cutout dataset, the sample dataset including images that protect face privacy and images without face privacy issues, and divide the sample dataset into training validation sets and test sets; The image preprocessing module is used to perform the following: for each image and corresponding mask in the training validation set, perform data enhancement and image preprocessing on the image and the corresponding mask to obtain a preprocessed training validation set; The model construction module is used to perform the following: introduce the ESA module into the RepVit network model to obtain the ESA-RepViT backbone network model, use the ESA-RepViT backbone network model as the backbone network, and build a portrait cutout network model based on the YOLOv8 network; The model training module is used to perform the following: perform model training and model verification on the portrait cutout network model based on the preprocessed training verification set, and test the trained portrait cutout network model based on the test set to obtain the trained portrait cutout network model; The portrait cutout module is used to perform the following: for a 2D portrait to be processed, data enhancement and image preprocessing are performed on the image to obtain a preprocessed image, the preprocessed image is used as input, feature extraction is performed through the trained portrait cutout network model, and the corresponding mask is output.

Images