A portrait segmentation method, apparatus, device, medium and product

Abstract

This invention discloses a method, apparatus, device, medium, and product for portrait segmentation. The method involves performing two deformable depthwise separable convolution calculations on the portrait image to be segmented, obtaining a preliminary feature map and a refined feature map, respectively. The preliminary and refined feature maps are then fused, and the fused result, the feature maps of the preliminary and refined feature maps are summed pixel by pixel. The pixel sum is then transformed to obtain the final portrait segmentation image. This solution improves the segmentation accuracy of single portrait segmentation and effectively reduces model complexity.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method, apparatus, device, medium, and product for human portrait segmentation. Background Technology

[0002] Human image segmentation is a technique that separates the human image from the background in an image. It is widely used in image processing, special effects production, and human image enhancement.

[0003] Currently, portrait segmentation technology primarily employs supervised learning to train deep models, utilizing large labeled datasets for training and parameter tuning. This allows the model to fully acquire high-level semantic information while preserving features. In portrait segmentation, adjusting the network structure aims to improve model performance and efficiency. Two common methods for adjusting the network structure are adjusting the structure within the convolutional layers and adjusting the overall network structure. A common method for adjusting the structure within the convolutional layers is to use deformable convolutions. Deformable convolutions allow the convolutional kernels to dynamically adjust their shape based on input features to adapt to different geometric changes, making them particularly suitable for handling scale variations, deformations, and occlusions of objects in images. A common method for adjusting the overall network structure is to use multi-branch networks. Multi-branch network structures have unique advantages in deep learning, especially when handling tasks that require simultaneously considering high-level semantic information and low-level feature information. However, both deformable convolutions and multi-branch network structures have a high number of parameters and high model complexity, consuming significant computational resources. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a method, apparatus, device, medium, and product for human portrait segmentation, which can improve the segmentation accuracy of single human portrait segmentation and effectively reduce model complexity.

[0005] This application provides a portrait segmentation method, the method comprising: The human image to be segmented is subjected to two deformable depthwise separable convolution calculations to obtain a preliminary feature map and a refined feature map, respectively. The preliminary feature map and the enhanced feature map are fused together, and the fused result, the feature map of the preliminary feature map, and the feature map of the enhanced feature map are added pixel by pixel; The pixel summation result is then subjected to information transformation processing to obtain the human image segmentation result image.

[0006] Preferably, the step of performing two deformable depthwise separable convolution calculations on the portrait image to be segmented to obtain a preliminary feature map and a refined feature map includes: The image of the person to be segmented is input into a pre-constructed deformable depthwise separable convolutional block for calculation, capturing the preliminary features of the image and generating the preliminary feature map; The preliminary feature map is then input again into the deformable depthwise separable convolutional block for calculation, semantic information is extracted, and the deepened feature map is obtained.

[0007] Preferably, the calculation process of the deformable depth-separable convolutional block specifically includes: The input feature map is subjected to pointwise convolution to adjust the feature depth of the input feature map; The results of pointwise convolution are computed in parallel twice. The results of the two parallel computations are concatenated to restore the original feature depth. The depth-restored image is residually concatenated with the input feature map, and the calculated image is then passed through an activation function.

[0008] Preferably, the parallel computing process specifically includes: The input feature map is pooled, and the offset is extracted from the pooled feature map to obtain an offset feature map with the same resolution but a preset depth. The offset feature map is processed using a preset depth self-attention convolution operation model; The processed feature map is subjected to point self-attention convolution to change the number of channels, and then bilinear interpolation is performed to restore the resolution of the feature values, resulting in the processed feature map.

[0009] Preferably, the deep self-attention convolution operation model includes: ; in, Indicates the current pixel after processing. The pixel values, where D represents the set of receptive fields. This represents the nth offset position within the sensing field. For each offset position The learned offset, This indicates the valid pixel offset. Indicates the convolution kernel at the offset position The weight parameters at that location, Indicates the sampling position after offset. The pixel value.

[0010] Preferably, the step of performing two deformable depthwise separable convolution calculations on the portrait image to be segmented to obtain a preliminary feature map and a refined feature map further includes: The feature maps of the preliminary feature map and the refined feature map are calculated by retaining the deformable depth separable convolutional blocks respectively.

[0011] Preferably, the step of fusing the preliminary feature map and the refined feature map, and then adding the fused result, the feature map of the preliminary feature map, and the feature map of the refined feature map pixel by pixel, includes: Unify the preliminary feature map and the refined feature map to the same resolution; Information fusion is performed through an information supplementation module based on category confidence, and the fused result is then pixel-wise added to the feature map obtained by two deformable depthwise separable convolutions.

[0012] Preferably, the information fusion via the category-based confidence-based information supplementation module includes: The input feature map with two channels is processed by an activation function, and the confidence probability value of each pixel with respect to the two categories is calculated. The pixel locations with the highest confidence probability value less than the first preset value are stored as the first information set, and the pixel locations with the difference between two confidence probability values ​​less than the second preset value are stored as the second information set. A third information set is obtained by performing an OR operation on the first information set and the second information set. The third information set is multiplied at the pixel level with the deepened feature map, and the result is then concatenated with the preliminary feature map to obtain the fusion result.

[0013] Preferably, the step of performing information transformation processing on the pixel addition result to obtain the portrait segmentation result image includes: Upsample the pixel summation result to restore the image to the input image size; The upsampled image is processed point by point, and the number of channels in the feature mapping is controlled to match the number of categories. A two-channel matrix is ​​output to obtain the human image segmentation result.

[0014] This invention also provides a human image segmentation device, the device comprising: The convolution module is used to perform two deformable depthwise separable convolution calculations on the portrait image to be segmented, to obtain a preliminary feature map and a refined feature map, respectively. The fusion module is used to fuse the preliminary feature map and the enhanced feature map, and to add the fused result, the feature map of the preliminary feature map and the feature map of the enhanced feature map pixel by pixel; The transformation module is used to perform information transformation processing on the pixel summation result to obtain the human image segmentation result image.

[0015] This invention also provides a communication device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the portrait segmentation method as described in any of the above embodiments.

[0016] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to perform the portrait segmentation method as described in any of the above embodiments.

[0017] This invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0018] This invention provides a method, apparatus, device, medium, and product for portrait segmentation. The method involves performing two deformable depthwise separable convolution calculations on the portrait image to be segmented, obtaining a preliminary feature map and a refined feature map, respectively. The preliminary and refined feature maps are then fused, and the fused result, the feature maps of the preliminary and refined feature maps are summed pixel by pixel. The pixel sum is then subjected to information transformation processing to obtain the portrait segmentation result image. This solution improves the segmentation accuracy of single portrait segmentation and effectively reduces model complexity. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the portrait segmentation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the implementation process of the portrait segmentation method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the workflow of the deformable depth-separable convolution block provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the parallel computing process provided in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the information fusion process performed by the information supplementation module based on category confidence provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of the structure of a human image segmentation device provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a communication device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] To address the aforementioned technical issues, this case provides a human image segmentation method, see [link to relevant documentation]. Figure 1 This is a flowchart illustrating the portrait segmentation method provided in an embodiment of the present invention. The method includes: Step S1: Perform two deformable depthwise separable convolution calculations on the portrait image to be segmented to obtain a preliminary feature map and a refined feature map, respectively. Step S2: The preliminary feature map and the enhanced feature map are fused, and the fused result, the feature map of the preliminary feature map, and the feature map of the enhanced feature map are added pixel by pixel; Step S3: Perform information transformation processing on the pixel summation result to obtain the human image segmentation result image.

[0022] In a specific implementation of this embodiment, the method includes the following implementation process: Information extraction involves inputting the image to be segmented into the network. The image undergoes two consecutive calculations using deformable depthwise separable convolutional modules, each preserving its feature maps to obtain a preliminary feature map and a refined feature map. The image is then fed into the network and processed through two consecutive deformable depthwise separable convolutional modules. After the first calculation, the basic contours and structural information of the image are captured, and the feature maps are preserved, resulting in a preliminary feature map. After the second calculation, the feature maps are preserved, and further detailed textures and semantic information of the image are extracted, resulting in a refined feature map. Deformable convolution breaks away from the fixed grid sampling of traditional convolution, resulting in more accurate feature extraction. It can adaptively fit the irregular contours of the image, effectively capturing edge details. Depthwise separable convolution significantly reduces computational cost and parameter count while maintaining efficient feature extraction, making the model more lightweight.

[0023] Information fusion unifies the two feature maps from the previous stage to the same resolution. This is achieved through an information supplementation module based on class confidence, followed by pixel-by-pixel addition of the fused result to the feature map calculated by two deformable depthwise separable convolutional modules. The initial and refined feature maps are unified to the same resolution through upsampling or downsampling operations. This fusion is then performed using the class confidence-based information supplementation module. This module dynamically allocates the contribution of the two feature maps based on the confidence level of each pixel belonging to either "portrait" or "background," achieving precise information complementarity. The class confidence-based fusion module avoids simple feature concatenation or addition, dynamically supplementing information based on pixel category affiliation, thus improving the segmentation accuracy at the boundary between the portrait and background. Pixel-by-pixel addition with the feature map after two convolutions further enhances the features of key regions and suppresses noise interference.

[0024] Pixel-level feature addition: The fused feature map is added pixel-by-pixel to the feature map calculated by two deformable depthwise separable convolution modules, enhancing the feature representation of key regions. The pixel-by-pixel feature map is then upsampled to restore the same resolution as the input image. Pointwise convolution to control channels: A pointwise convolution is performed to control the number of channels in the feature map to match the number of target categories, typically two, namely "portrait" and "background". This results in a dual-channel probability matrix, with each channel corresponding to the category probability of "portrait" and "background" respectively, ultimately generating the portrait segmentation result map. Upsampling restores the input size, ensuring spatial alignment between the segmentation result and the original image. Pointwise convolution controls the number of channels and directly outputs the category probabilities, resulting in a final segmentation map with sharp edges and complete details, effectively improving the visual effect and quantitative indicators of portrait segmentation. The pixel summation result from the previous stage is upsampled once to restore the input image size before the information extraction stage. Then, a pointwise convolution calculation is performed, and the number of channels in the feature mapping is controlled to match the number of categories. A two-channel matrix is ​​output to obtain the final portrait segmentation result image.

[0025] See Figure 2 This is a schematic diagram illustrating the implementation process of the portrait segmentation method provided in this embodiment of the invention. Based on a portrait segmentation network structure supplemented by deformable depthwise separable convolution and category confidence information, the portrait segmentation method specifically includes the following steps: Initial feature extraction is performed on the input image of the person to be segmented. First, a standard convolutional layer is used to perform preliminary feature extraction on the input image to obtain a basic feature map.

[0026] The first step involves a deformable depthwise separable convolution, where the base feature map obtained in the previous step is input into this module. This module combines the advantages of deformable convolution and depthwise separable convolution, adaptively extracting features that better fit the human face contour while maintaining efficient computation.

[0027] Branch processing: Left branch: After one standard convolution, a preliminary feature map is obtained and input into the "information supplementation module based on class confidence". Right branch: Continue with subsequent feature deepening.

[0028] The second deformable depthwise separable convolution, standard convolution: performs a standard convolution on the feature map of the right branch to further refine the features.

[0029] A deformable depthwise separable convolutional module is used again to obtain a deeper feature map, which is used to capture more refined semantic information.

[0030] The enhanced feature map is then subjected to another standard convolution to prepare for subsequent fusion.

[0031] The category confidence-based information supplementation module: The preliminary feature map of the left branch is input into this module, which dynamically supplements and corrects the feature information based on the confidence that each pixel belongs to the portrait or the background.

[0032] Pixel-level addition ⊕ adds the output of the information supplementation module to the feature map of the right branch after two deformable convolutions and a standard convolution, pixel by pixel, to achieve information fusion and enhance the feature expression of key regions.

[0033] A standard convolution is performed on the fused feature map to integrate global information. This generates the final human image segmentation result map, which is usually a two-channel probability map, corresponding to the predicted probabilities of "human image" and "background" respectively.

[0034] This application proposes a deformable depthwise separable convolution module, which not only effectively reduces the number of parameters and computational cost generated by deformable convolution, but also focuses more on detailed features that traditional convolution cannot cover. Secondly, through an information supplementation module based on class confidence, confidence thresholds are set separately for the two classes to capture details of lost low-level features, avoiding the introduction of redundant information and enhancing the effectiveness of feature fusion. Through these optimization measures, this scheme not only improves the segmentation accuracy of single-portrait segmentation, but also effectively reduces model complexity and runtime. Furthermore, this scheme utilizes cloud computing resources, using a cloud phone to access high-performance computing resources in the cloud to execute data-intensive portrait segmentation algorithms without consuming computing power on local devices.

[0035] In another embodiment of the present invention, the information extraction stage is the core of the portrait segmentation technology. In this embodiment, when extracting the feature map, two deformable depth-separable convolution calculations are used.

[0036] Specifically, the image of the person to be segmented is input into a pre-constructed deformable depthwise separable convolutional block for calculation, capturing the preliminary features of the image and generating the preliminary feature map; The preliminary feature map is then input again into the deformable depthwise separable convolutional block for calculation, semantic information is extracted, and the deepened feature map is obtained.

[0037] A deformable depthwise separable convolution module is employed to perform multi-level feature analysis on the input portrait image. First, through a single deformable depthwise separable convolution, the model captures preliminary features of the image, including texture, color, and basic shape, thereby generating a preliminary feature map. Deformable convolution can flexibly adjust the spatial range of input data that neurons at different levels can utilize, covering more detailed information and ensuring accurate extraction of key features.

[0038] Subsequently, the image is again processed through deformable depthwise separable convolution. This operation focuses on refining deeper semantic information, such as contours, structure, and details, to obtain a deepened feature map. The deepened feature map not only retains the details of the initial feature map but also adds an abstract understanding of the portrait features, providing rich and accurate feature representations for subsequent information fusion stages, ensuring high accuracy and quality of the segmentation results. Through two convolutional processes, the model can efficiently extract and deepen key features from the original image, laying a solid foundation for accurate portrait segmentation.

[0039] In another embodiment of the present invention, the calculation process of the deformable depth-separable convolution block specifically includes: The input feature map is subjected to pointwise convolution to adjust the feature depth of the input feature map; The results of pointwise convolution are computed in parallel twice. The results of the two parallel computations are concatenated to restore the original feature depth. The depth-restored image is residually concatenated with the input feature map, and the calculated image is then passed through an activation function.

[0040] For specific implementation, please refer to Figure 3 This is a schematic diagram of the workflow of a deformable depth-separable convolutional block provided in an embodiment of the present invention. The calculation process of the deformable depth-separable convolutional block includes the following steps: Pointwise convolution: The input feature map first undergoes pointwise convolution (1×1 convolution) to perform linear transformation and information fusion on the channel dimensions, preparing for subsequent depthwise convolution.

[0041] The core computational unit comprises two parallel branches: deformable depthwise convolution, which performs depthwise convolution on the feature map after pointwise convolution, and introduces offset learning for deformable convolution, allowing the convolution kernel to adaptively adjust the sampling position according to the input content, thereby better capturing geometric deformations; and standard depthwise convolution, which, as a parallel branch, performs standard depthwise convolution, providing a stable and regular feature extraction path.

[0042] The concatenation method combines the two sets of feature maps output by deformable depthwise convolution and standard depthwise convolution along the channel dimension, fusing features extracted in two different ways to enrich feature representation.

[0043] Residual connections are performed by concatenating the spliced ​​feature maps with the original pointwise convolution outputs (element-wise addition), which alleviates the gradient vanishing problem in deep network training and ensures the smooth flow of information.

[0044] PReLU activation applies the PReLU activation function to the result after residual connection, introducing a nonlinear transformation and enhancing the expressive power of the model.

[0045] The feature map processed by the above steps is output as the final output of this module, and can be used as input for the next layer to continue the calculation.

[0046] Convolutional operations excel at capturing fine information within local regions, extracting features by assigning weights to each pixel and summing them. Traditionally, the local field of view of convolution typically employs a fixed geometric configuration, such as a 3×3 or 5×5 matrix. The size of the convolutional kernel and its operational method directly affect the computational requirements of the network. To optimize computational efficiency and accelerate the processing, lightweight models tend to use modified convolutional techniques as the core computational component.

[0047] Several advanced convolutional techniques can significantly reduce computational burden, such as dilated convolution and spatially separable convolution. However, they may be limited by their own field of view during downsampling, failing to fully capture the subtle features of the data. In contrast, depthwise separable convolution is more favored in lightweight network designs. Depthwise separable convolution decomposes the standard convolution's operation of adjusting the resolution and depth of the feature map into two independent operations: channel-wise convolution and pointwise convolution.

[0048] Channel-wise convolution is essentially a variant of grouped convolution. It groups the input feature map by channel, with each group containing only one channel. This means there's no need to limit the number of channels in the output feature map, and the computation process maintains the feature map's depth. Pointwise convolution, on the other hand, uses a 1×1 kernel, changing only the number of channels in the input feature map without affecting its spatial dimensions. Through this decomposition, depthwise separable convolution significantly reduces the number of parameters and computational complexity, resulting in a marked improvement in efficiency compared to traditional standard convolution.

[0049] Deformable convolution, a type of convolution technique, is characterized by abandoning a fixed-shape receptive field and instead employing a dynamically adjusted mechanism. This mechanism allows the receptive field position to be adaptively learned through backpropagation, without additional guidance or supervision. This characteristic enables deformable convolution to focus on details that traditional methods might overlook, effectively overcoming the limitations of traditional convolution's limited field of view and inability to capture all key details at once. By giving convolutional operations greater flexibility, deformable convolution enhances the accuracy and comprehensiveness of feature extraction, making the feature extraction results more effective.

[0050] The deformable depthwise separable convolutional block contains two core computational units that combine deformable convolution and depthwise separable convolution. This module first performs a pointwise convolution to adjust the depth of the input feature map to half of the original input. Then, it substitutes this into the two core computational units for parallel computation. The calculated results are then concatenated to restore the depth of the original input feature map. After residual connection with the original input feature map, the result is activated by a PReLU function before being output.

[0051] In another embodiment of the present invention, during the above-mentioned parallel computation, the deformable depthwise separable convolutional block includes two core computational units that combine deformable convolution and depthwise separable convolution, used for parallel computation operations on the feature map, specifically: The input feature map is pooled, and the offset is extracted from the pooled feature map to obtain an offset feature map with the same resolution but a preset depth. The offset feature map is processed using a preset depth self-attention convolution operation model; The processed feature map is subjected to point self-attention convolution to change the number of channels, and then bilinear interpolation is performed to restore the resolution of the feature values, resulting in the processed feature map.

[0052] Specifically, see Figure 4 This is a schematic diagram of the parallel computing process provided in an embodiment of the present invention. The parallel computing process of deformable channel-wise convolution can be divided into two parallel branches, which are finally fused and output through point-wise convolution: Branch 1: Deformable Channel-by-Channel Convolution. The input feature map is directly fed into deformable channel-by-channel convolution. This is the core operation, which performs convolution independently along the channel dimension. At the same time, the sampling position of the convolution kernel is adaptively adjusted according to the learned offset to better capture deformation information.

[0053] Branch 2: Offset Generation. Pooling is used to downsample the input feature map, reducing spatial resolution and subsequent computation. Channel-wise convolution is performed on the pooled features to extract spatial features. Pointwise convolution fuses channel information through 1×1 convolutions. Offset Set Extraction: From the output of pointwise convolution, the offset parameters required for deformable convolution are parsed and generated to guide the sampling positions of deformable channel-wise convolutions in the main path.

[0054] The fusion and output process involves pointwise convolution, which performs pointwise convolution between the output of deformable channel-wise convolution and the generated offset information to achieve feature fusion.

[0055] Upsampling involves upsampling the fused feature map to restore it to the same spatial resolution as the original input.

[0056] The upsampled result is the final output of this module and can be used in subsequent network layers.

[0057] The core computing unit first performs pooling on the input feature map to further reduce computational load and memory consumption. Then, it uses depthwise separable convolution to extract the offset from the pooled feature map. The resolution of the output feature map remains unchanged, but the depth becomes 2K. 2 K is the size of the depthwise separable convolution kernel. Here, every 2K... 2 Each pixel at the same position corresponds to the receptive field offset of the corresponding pixel in the resulting feature map. The input feature map after pooling is processed by depthwise self-attention convolution, and then point self-attention convolution is performed to change the number of channels. Finally, bilinear interpolation, corresponding to the pooling operation, is performed to restore the resolution of the resulting feature map to the size of the input.

[0058] Through parallel computation of two paths, the main path focuses on extracting deformable features, enhancing the ability to model geometric deformations. The auxiliary path efficiently generates offset parameters, providing adaptive guidance for the main path. The two paths achieve information fusion through pointwise convolution and upsampling, ultimately outputting a high-quality feature map.

[0059] In another embodiment of the present invention, for a traditional convolutional kernel with a receptive field of 3×3 and an expansion of 1, its receptive field... The formula is expressed as follows: ; For each pixel of the output feature map The formula is expressed as follows: ; in, Traversal For deformable convolution, it first obtains 18 offsets for each pixel through convolution (the convolution kernel size is 3, and each pixel has two-dimensional offsets), and then uses bilinear interpolation to ensure that each offset position falls on the pixel, resulting in a depthwise self-attention convolution operation model. Each pixel after deformable convolution... The formula is expressed as follows: ; in, Indicates the current pixel after processing. The pixel values, where D represents the set of receptive fields. This represents the nth offset position within the sensing field. For each offset position The learned offset, This indicates the valid pixel offset. Indicates the convolution kernel at the offset position The weight parameters at that location, Indicates the sampling position after offset. The pixel value.

[0060] While deformable convolution can make feature extraction more effective, it requires calculating the offset of each pixel, which increases the number of parameters exponentially. Therefore, it is generally not used in research where equipment is limited.

[0061] This application combines deformable convolution with depthwise separable convolution, proposing a deformable depthwise separable convolution block. The increased parameters and computational cost of deformable convolution are effectively reduced by depthwise separable convolution. The deformable depthwise separable convolution block can focus on detailed features that traditional convolution cannot cover without introducing additional computational burden, and even reduces the number of parameters and computational cost compared to traditional convolution.

[0062] In another embodiment of the present invention, when performing convolution calculations using variable depth separable convolutional blocks, the image of the person to be segmented is input into the network, and two consecutive deformable depth separable convolutions are performed to output preliminary feature maps and enhanced feature maps, respectively, and the feature maps corresponding to the two convolutions are retained respectively.

[0063] The preliminary feature map and the refined feature map are fused, and then the fusion result is added to the feature map retained by the two convolutions at the pixel level.

[0064] The features after pixel addition are processed by information transformation to finally output the human portrait segmentation result image.

[0065] Deformable depthwise separable convolutions maintain lightweight design and low computational cost while adaptively capturing human poses and contour deformations, enhancing feature extraction capabilities for irregular human structures. By performing two consecutive convolutions and retaining feature maps separately, dual feature retention—both shallow details and deep semantics—is achieved, preventing information loss.

[0066] The process first fuses pixels and then adds them together, enhancing key features and suppressing noise, resulting in more accurate image edges and more complete segmentation. The overall structure is lightweight and efficient, balancing segmentation accuracy and inference speed, making it suitable for real-time image segmentation scenarios.

[0067] In another embodiment provided by the present invention, step S2 specifically includes: Unify the preliminary feature map and the refined feature map to the same resolution; Information fusion is performed through an information supplementation module based on category confidence, and the fused result is then pixel-wise added to the feature map obtained by two deformable depthwise separable convolutions.

[0068] Specifically, after adjusting the feature maps of the initial feature map and the refined feature map to the same resolution, they are both fed into an information supplementation module based on class confidence. This module intelligently fuses information from different feature maps by evaluating the confidence that each pixel belongs to a specific class, thereby supplementing and enhancing feature representation. The fused feature map is then added to the refined feature map pixel by pixel, further strengthening key features and providing richer and more accurate input for subsequent information transformation steps, thus improving the overall model performance and segmentation accuracy.

[0069] In another embodiment of the present invention, the process of information fusion through the information supplementation module based on category confidence is specifically executed by the information supplementation module based on category confidence, see [link to relevant documentation]. Figure 5 This is a flowchart illustrating the information fusion process performed by the information supplementation module based on category confidence provided in this embodiment of the invention. Specifically: First, parallel feature extraction is performed. Input A is fed into two parallel branches P1 and P2, which generate information set A, such as fine-grained spatial features, and information set B, such as category confidence features.

[0070] Preliminary information fusion involves integrating information set A and information set B through fusion operations such as splicing and element-by-element addition to obtain information set C, which is the comprehensive feature of the fused set.

[0071] Category confidence weighting supplementation involves multiplying the information set C element-wise with the input category confidence map B to achieve feature weighting based on category confidence, thereby strengthening the feature representation of high-confidence regions.

[0072] The weighted result is added element by element to the original information set C to supplement the information, preserving the original features while strengthening the key areas.

[0073] Output the final fusion result: Obtain the fused enhanced feature map, which can be used for subsequent tasks, such as human image segmentation and semantic segmentation.

[0074] Semantic segmentation models widely employ an encoder-decoder architecture. The encoder downsamples the input image, extracting details and reducing the feature map resolution; the decoder upsamples the smaller feature maps to restore details and resolution. However, the details lost during downsampling are difficult to fully recover during upsampling. Most models concatenate high-resolution and low-resolution feature maps to supplement high-level semantic details, thereby improving segmentation accuracy. However, in skip connections, simply concatenating, summing elements, or using low-resolution attention vectors to enhance high-resolution feature maps, while introducing details ignored by downsampling, also introduces noise filtered out by downsampling, resulting in suboptimal segmentation quality.

[0075] To address this issue, this patent proposes an information supplementation module based on category confidence. This module calculates the confidence level of the low-resolution feature map and sets two confidence thresholds to control the position of pixels in the high-resolution feature map, effectively blocking noise interference with the segmentation results. Specifically, the module processes the input feature map with two channels (i.e., two categories) using the Softmax function, calculating the confidence probability value of each pixel with respect to the two categories. Then, the positions of pixels with a maximum probability value less than P1 are stored as information set A, and the positions of pixels with a difference between two probability values ​​less than P2 are stored as information set B. By performing an OR operation on A and B, a set C for filtering detailed information is obtained. Set C is then multiplied pixel-level by the high-resolution feature map, and the result is concatenated with the low-resolution feature map. This effectively avoids the intrusion of noise-generating pixel information while introducing low-level feature details into high-level semantics.

[0076] In another embodiment provided by the present invention, step S3 specifically includes the following steps: The pixel summation result from the previous stage is upsampled to restore the image to the size of the input image. The upsampled image is processed point by point, and the number of channels in the feature mapping is controlled to match the number of categories. A two-channel matrix is ​​output to obtain the human image segmentation result.

[0077] The information fusion result is upsampled to restore the feature map to the original input image size, ensuring the segmentation result accurately corresponds to the original image. Following upsampling, pointwise convolution is performed, adjusting the number of channels in the feature map to the required number of categories (human and background) using a 1x1 convolution kernel, forming a dual-channel matrix. Each pixel in this matrix contains two values, corresponding to the probabilities of human and background, respectively. This process ultimately generates a high-resolution segmentation result map, achieving fine-grained human edge segmentation and high-quality image segmentation.

[0078] See Figure 6 A schematic diagram of a human face segmentation device provided in an embodiment of the present invention, the device comprising: The requirement parsing module is used to input the acquired input sequence into a preset multimodal model and parse the user's processing requirements. The data determination module is used to determine the data to be processed in the mobile cloud drive based on the processing requirements. The processing module is used to input the data to be processed into a preset multimodal machine learning model for processing and output the business processing results.

[0079] The portrait segmentation device provided in this embodiment can perform all the steps and functions of the portrait segmentation method provided in any of the above embodiments. The specific functions of the device will not be described in detail here.

[0080] See Figure 7 This is a schematic diagram of a communication device provided in an embodiment of the present invention. The communication device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a portrait segmentation program. When the processor executes the computer program, it implements the steps in each of the above-described embodiments of a portrait segmentation method, for example... Figure 1 The steps shown. Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-described device embodiments.

[0081] For example, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the portrait segmentation device. For example, the computer program can be divided into several modules, the specific functions of which have been described in detail in the portrait segmentation method provided in any of the above embodiments; therefore, the specific functions of the device will not be repeated here.

[0082] The communication device described can be a desktop computer, laptop, handheld computer, or cloud server, etc. The communication device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the schematic diagram is merely an example of a communication device and does not constitute a limitation on a human image segmentation device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the communication device may also include input / output devices, network access devices, buses, etc.

[0083] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the facial recognition device, connecting all parts of the device via various interfaces and lines.

[0084] The memory can be used to store the computer program and / or modules. The processor implements various functions of the human image segmentation device by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0085] If the integrated module of the aforementioned human image segmentation device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0086] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the functional network element implementing the method described in the above embodiments.

[0087] The computer program product provided in this embodiment can execute all the steps and functions of the portrait segmentation method provided in any of the above embodiments. The specific functions of the product will not be described in detail here.

[0088] It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered to be within the scope of protection of this invention.

Claims

1. A method for human face segmentation, characterized in that, The method includes: The human image to be segmented is subjected to two deformable depthwise separable convolution calculations to obtain a preliminary feature map and a refined feature map, respectively. The preliminary feature map and the enhanced feature map are fused together, and the fused result, the feature map of the preliminary feature map, and the feature map of the enhanced feature map are added pixel by pixel; The pixel summation result is then subjected to information transformation processing to obtain the human image segmentation result image.

2. The portrait segmentation method according to claim 1, characterized in that, The process of performing two deformable depthwise separable convolutions on the human image to be segmented to obtain a preliminary feature map and a refined feature map includes: The image of the person to be segmented is input into a pre-constructed deformable depthwise separable convolutional block for calculation, capturing the preliminary features of the image and generating the preliminary feature map; The preliminary feature map is then input again into the deformable depthwise separable convolutional block for calculation, semantic information is extracted, and the deepened feature map is obtained.

3. The portrait segmentation method according to claim 2, characterized in that, The calculation process for the deformable depth-separable convolutional block specifically includes: The input feature map is subjected to pointwise convolution to adjust the feature depth of the input feature map; The results of pointwise convolution are computed in parallel twice. The results of the two parallel computations are concatenated to restore the original feature depth. The depth-restored image is residually concatenated with the input feature map, and the calculated image is then passed through an activation function.

4. The portrait segmentation method according to claim 3, characterized in that, The parallel computing process specifically includes: The input feature map is pooled, and the offset is extracted from the pooled feature map to obtain an offset feature map with the same resolution but a preset depth. The offset feature map is processed using a preset depth self-attention convolution operation model; The processed feature map is subjected to point self-attention convolution to change the number of channels, and then bilinear interpolation is performed to restore the resolution of the feature values, resulting in the processed feature map.

5. The portrait segmentation method according to claim 4, characterized in that, The deep self-attention convolution operation model includes: ; in, Indicates the current pixel after processing. The pixel values, where D represents the set of receptive fields. This represents the nth offset position within the sensing field. For each offset position The learned offset, This indicates the valid pixel offset. Indicates the convolution kernel at the offset position The weight parameters at that location, Indicates the sampling position after offset. The pixel value.

6. The portrait segmentation method according to claim 2, characterized in that, The step of performing two deformable depthwise separable convolution calculations on the portrait image to be segmented to obtain a preliminary feature map and a refined feature map also includes: The feature maps of the preliminary feature map and the refined feature map are calculated by retaining the deformable depth separable convolutional blocks respectively.

7. The portrait segmentation method according to claim 1, characterized in that, The step of fusing information from the preliminary feature map and the refined feature map, and then adding the fused result, the feature map of the preliminary feature map, and the feature map of the refined feature map pixel by pixel, includes: Unify the preliminary feature map and the refined feature map to the same resolution; Information fusion is performed through an information supplementation module based on category confidence, and the fused result is then pixel-wise added to the feature map obtained by two deformable depthwise separable convolutions.

8. The portrait segmentation method according to claim 7, characterized in that, The information fusion through the information supplementation module based on category confidence includes: The input feature map with two channels is processed by an activation function, and the confidence probability value of each pixel with respect to the two categories is calculated. The pixel locations with the highest confidence probability value less than the first preset value are stored as the first information set, and the pixel locations with the difference between two confidence probability values ​​less than the second preset value are stored as the second information set. A third information set is obtained by performing an OR operation on the first information set and the second information set. The third information set is multiplied at the pixel level with the deepened feature map, and the result is then concatenated with the preliminary feature map to obtain the fusion result.

9. The portrait segmentation method according to claim 1, characterized in that, The step of performing information transformation processing on the pixel summation result to obtain the human image segmentation result image includes: Upsample the pixel summation result to restore the image to the input image size; The upsampled image is processed point by point, and the number of channels in the feature mapping is controlled to match the number of categories. A two-channel matrix is ​​output to obtain the human image segmentation result.

10. A human face segmentation device, characterized in that, The device includes: The convolution module is used to perform two deformable depthwise separable convolution calculations on the portrait image to be segmented, to obtain a preliminary feature map and a refined feature map, respectively. The fusion module is used to fuse the preliminary feature map and the enhanced feature map, and to add the fused result, the feature map of the preliminary feature map and the feature map of the enhanced feature map pixel by pixel; The transformation module is used to perform information transformation processing on the pixel summation result to obtain the human image segmentation result image.

11. A communication device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the human image segmentation method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the portrait segmentation method as described in any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 9.

Images