A feature weighting real-time portrait segmentation method based on adaptive up-sampling strategy

Abstract

This invention discloses a feature-weighted real-time portrait segmentation method based on an adaptive upsampling strategy. The method involves acquiring portrait images in real time via an image acquisition module and sending these images to a core computing unit. The core computing unit uses an improved lightweight neural network algorithm module to process the acquired portrait images, obtaining feature maps and corresponding weights to generate and mix convolutional weights. Conditional weights are used for cross-channel information exchange. The feature maps obtained by the improved lightweight neural network algorithm need to be upsampled for high-resolution prediction. An adaptive upsampling module is designed to acquire non-uniform points with high uncertainty after receiving CNN feature maps from multiple channels. For each difficult point, point-by-point feature representation is used, and then a multilayer perceptron is used to predict the corresponding category point by point.

Description

Technical Field

[0001] This invention relates to the field of deep learning image processing technology, and in particular to a feature-weighted real-time portrait segmentation method based on an adaptive upsampling strategy. Background Technology

[0002] Human face segmentation is one of the fundamental technologies in computer vision. This technology uses sensors (such as cameras) to separate the human face from the background in a single frame of image / video data, forming foreground and background regions, which are then labeled and presented with different tags. This provides the foundation for subsequent downstream tasks (background blurring, background replacement, portrait beautification, etc.).

[0003] With the rapid development of deep learning technology in recent years, deep convolutional neural network models have been widely used in image segmentation of portrait regions. However, current portrait segmentation techniques generally suffer from drawbacks such as unsmooth edge fitting, loss of foreground and background parts of the portrait within the same color family, and poor segmentation performance in backlit and backlit scenes. Furthermore, current portrait segmentation techniques rely on high-scale network models to improve accuracy, leading to increased algorithm complexity, excessively large output feature maps, slow training and inference speeds, and low segmentation efficiency and high latency on mobile devices. This makes it difficult to achieve a good balance between performance and accuracy, and thus fails to meet the demands of current real-time applications. Therefore, existing technologies require further improvement and enhancement.

[0004] Based on the above technical background, a feature-weighted real-time portrait segmentation method based on an adaptive upsampling strategy is proposed, which can smoothly segment portrait edges in real time on a low-cost hardware platform.

[0005] Existing semantic segmentation models typically output a predicted image at a 1:n ratio to the original image. For example, within a CNN, the image is downsampled by a factor of 16 relative to the input image, and then padded to the n-fold resolution using linear interpolation. This approach is highly detrimental to the prediction of detailed edges. Even when replacing upsampling layers with dilated convolutions to achieve higher accuracy, the required memory and computational resources are often substantial. Furthermore, existing portrait segmentation techniques oversample low-frequency pixel regions belonging to the same label category (such as large, continuous background areas), while sampling high-frequency regions near object boundaries (such as hair edges and complex gestures) is too sparse, resulting in jagged, blocky, and unsmooth segmented boundaries that do not reflect reality and have poor accuracy. On the hardware side, recognition speed is slow on low-cost hardware platforms (such as mobile phones and tablets), and upstream and downstream algorithmic processes can cause application lag and frame drops, significantly impacting the user experience. Summary of the Invention

[0006] Based on the above shortcomings, this invention is mainly aimed at portrait segmentation for mobile / embedded devices. It adopts a lightweight deep learning algorithm and uses pixel rendering to avoid jagged, blocky, and unsmooth problems in complex portrait edges. The hardware only uses a CPU and a monocular camera to complete low-cost real-time foreground and background segmentation of portraits and backgrounds, without the need for GPUs or high-definition cameras such as Kinect.

[0007] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention discloses a feature-weighted real-time portrait segmentation method based on an adaptive upsampling strategy, the method comprising:

[0008] The image acquisition module is used as the input to acquire portrait images in real time, and the acquired portrait images are sent to the core computing unit. The core computing unit uses an improved lightweight neural network algorithm module to operate on the acquired portrait images to obtain feature maps and corresponding weights, thereby generating and mixing convolution weights. Conditional weights are used to exchange information across channels, while following the high-resolution model of HRNet. In the process, high-resolution representation is maintained while replacing the computationally expensive pointwise convolution.

[0009] The feature maps obtained by the improved lightweight neural network algorithm need to be upsampled to output high-resolution predictions. An adaptive strategy upsampling module is designed. After receiving CNN feature maps with multiple channels, the adaptive strategy upsampling module uses an adaptive strategy to obtain non-uniform points with strong uncertainty. For each difficult point, a point-by-point feature representation is used, and then a multilayer perceptron is used to predict the corresponding category point by point.

[0010] Furthermore, the image acquisition module is any monocular camera, and the core computing unit is a mobile CPU.

[0011] Furthermore, the improved lightweight neural network algorithm module further includes applying the improved shuffle block in ShuffleNet to the HRNet high-resolution network. The improved shuffle block is divided into two partitions according to channels. One partition is passed sequentially through a lightweight channel-weighted unit whose complexity is linearly related to the number of channels, a 3*3 depthwise convolution, and then through another lightweight channel-weighted unit. After the output of this branch is processed, it is stacked and concatenated with the other half of the partition that is not processed on a channel-by-channel basis, and finally the shuffle operation is performed.

[0012] Furthermore, the shuffle operation further includes: since the shuffle operation and depthwise convolution do not involve information exchange between channels, the 1*1 convolution mainly undertakes the task of information exchange between channels. Lightweight channel weighting units are set up to perform information exchange by calculating weights across channels and resolutions. In the k parallel resolution branches in the k-stage HRNet, the bit-weighted QS of the ki branch is shown in Equation 1.

[0013] Q s =W s *X s (Formula 1)

[0014] Among them, X S and Q S The input and output mappings are represented respectively. Ws represents a weight mapping matrix with shape WS (width) x HS (height) x CS (number of channels), where the symbol * indicates bitwise multiplication.

[0015] Furthermore, the channel weight mapping under different resolutions is calculated using the P-function, as shown in Equation 2.

[0016] P s (X1,X2,…,X s ) = (W1, W2, ..., W S )(Formula 2)

[0017] Where X is the input mapping of the resolution corresponding to the s branch.

[0018] Furthermore, the P function performs adaptive pooling operations on each input to output a feature map with specified width and height. This feature map is then processed by the ReLU and sigmold functions to generate a weight map containing s partitions. The definitions of ReLU and sigmold are shown in Equations 3 and 4.

[0019] ReLU(x) = max(0,x) (Formula 3)

[0020]

[0021] Where x represents the independent variables of the ReLU function and the sigmold function, respectively.

[0022] Furthermore, the weights depend on the total number of pixels in the input channel at the resolution, and are calculated using the function H, as shown in Equation 5:

[0023] H(X s ) = w s (Formula 5)

[0024] The function H performs global average pooling on the input mapping to collect spatial features from all locations, and then obtains spatial weights through functions such as ReLU and sigmold.

[0025] Furthermore, the adaptive strategy upsampling module adopts a uniform coverage country generation sampling strategy during training, randomly sampling candidate points from the feature map according to a uniform distribution, ensuring that uncertain areas are covered while maintaining a certain degree of uniform coverage of the global area, and the process of calculating the loss function is only performed on the generated candidate points.

[0026] Furthermore, during output, the output mask is iteratively applied to perform coarse predictions on the points on the grid. If the predicted value is greater than a threshold, the point set is removed; otherwise, bilinear interpolation is used to upsample the segments predicted in the previous iteration. Then, N uncertain points with prediction probabilities close to 0.5 are selected. The upsampling module then calculates features for the N points and predicts labels. This process iterates until the segments are upsampled to the required resolution. The bilinear interpolation formula is shown in Formula 6.

[0027]

[0028] Among them, Q 11 Q 12 Q 21 Q 22 These are four point values.

[0029] Furthermore, the specific steps of the adaptive strategy upsampling module strategy execution are as follows:

[0030] Step 1: Use Boolean judgment to determine whether the feature resolution output by the backbone feature extraction network is less than the original resolution of the input image. If true, proceed to step 2; otherwise, terminate upsampling and output the feature map.

[0031] Step 2: Perform bilinear interpolation upsampling on the output feature map to obtain a rough prediction value of the pixel points within the closure region of the uncertain region;

[0032] Step 3: Select N uncertain points with a prediction probability close to 0.5 within the closure region, that is, points on the edge of the object whose labels are different from those of their neighbors.

[0033] Step 4: For each difficult point, interpolation is performed on the bottom feature map using the coordinates of the point to obtain low-level features. High-level features are obtained by upsampling in step 1. The high-level features and low-level features together form the representation vector of the pixel.

[0034] Step 5: Use a multivariate perceptron to make new predictions on the composed representation vectors, update the coarse prediction values, and then jump back to step 1 after the prediction ends.

[0035] Compared to existing technologies, the advantages of this invention are significant. Existing semantic segmentation techniques typically output a predicted image at a 1:n ratio to the original image. For example, CNNs may downsample the input image by a factor of 16 and then use linear interpolation to fill in the n-fold resolution. This approach is particularly detrimental to the prediction of detailed edges. Even when using dilated convolutions to replace upsampling layers for higher accuracy, the required memory and computational resources are often substantial. Furthermore, existing portrait segmentation techniques oversample low-frequency pixel regions (such as large, continuous background areas) belonging to the same label category, while sampling high-frequency regions near object boundaries (such as hair edges and complex gestures) is too sparse, resulting in jagged, blocky, and unsmooth segmented boundaries that are unrealistic and inaccurate. On the hardware side, recognition speed is slow on low-cost hardware platforms (such as mobile phones and tablets), and upstream and downstream algorithm interactions can cause application lag and frame drops, significantly impacting the user experience.

[0036] Therefore, the beneficial effects of the present invention are: without reducing the prediction of detail edges, it reduces the memory and computing power required by the hardware. At the same time, the sampling in the high-frequency region near the object boundary in the image data center is significantly better than the sparse sampling in the prior art, which leads to jagged and blocky uneven boundary segments. It can be applied to low-cost hardware platforms such as mobile phones, tablets and other devices to solve the problems of slow recognition speed, application lag and frame dropping caused by low-end hardware, and improve the user experience.

[0037] The beneficial effects of this invention can be more specifically summarized as follows: by using an adaptive strategy upsampling module to focus on edge region information, combining low-level and high-level features, finely dividing the edge points according to their features, gradually refining them at different scales, and utilizing the inaccurate values ​​of the network reclassification boundary, combining fine-grained and coarse-grained features, the jagged and uneven phenomenon caused by the network's inaccurate edge prediction is improved. Attached Figure Description

[0038] The invention will be further understood from the following description taken in conjunction with the accompanying drawings. The components in the drawings are not necessarily drawn to scale, but rather the emphasis is on illustrating the principles of the embodiments. In the drawings, the same reference numerals designate corresponding parts in different views.

[0039] Figure 1 This is a diagram of the core modules of the real-time portrait segmentation method based on deep learning rendering of the present invention.

[0040] Figure 2 This is a structural diagram of a lightweight channel weighting unit according to an embodiment of the present invention;

[0041] Figure 3 This is a structural diagram of a multi-sensor according to an embodiment of the present invention. Detailed Implementation

[0042] Example 1

[0043] In this embodiment, the core module of the real-time portrait segmentation method based on deep learning rendering is as follows: Figure 1 As shown, the system includes an image acquisition module as the input, a core computing unit, a lightweight neural network algorithm module, and a pixel rendering output module. The image acquisition module uses any monocular camera, and the core computing unit uses a mobile CPU. The core design of this invention consists of the lightweight neural network algorithm module and the pixel rendering output module, which are used to ensure real-time performance on low-cost hardware.

[0044] I. Lightweight Neural Network Algorithm Module

[0045] This lightweight neural network algorithm module improves upon the shuffle block in ShuffleNet and applies it to the HRNet high-resolution network, achieving better performance than popular lightweight networks (such as MobileNet and ShuffleNet) without losing image information. The improved shuffle block is divided into two partitions based on channels. Each partition sequentially passes through a lightweight channel-weighted unit (whose complexity is linearly related to the number of channels), a 3x3 depthwise convolution, and then another lightweight channel-weighted unit. The output of this branch is then stacked and concatenated with the other half (which is unprocessed) on a channel-by-channel basis before finally performing the shuffle operation.

[0046] Traditional 1x1 convolutions in a shuffle block perform matrix-vector multiplication at each position, resulting in a time complexity on the order of the number of channels. In contrast, 3x3 depthwise convolutions have a linear time complexity. The two 1x1 convolutions in the shuffle block consume significant computational resources. However, since shuffle operations and depthwise convolutions do not involve inter-channel information exchange, the 1x1 convolution primarily handles this task. Lightweight channel-weighted units, on the other hand, exchange information by calculating weights across channels and resolutions. In the k parallel resolution branches of a k-stage HRNet, the bit-weighted Q-squared weights of the ki branch... S As shown in Formula 1, where X S and Q S W represents the mapping between input and output, respectively. s Represents a shape of W S (width) xH S (height) x C SThe weight mapping matrix of (number of channels), where the symbol * indicates bitwise multiplication.

[0047] Q s =W s *X s (Formula 1)

[0048] Its overall complexity is linear with respect to the number of channels C, lower than the power-order computation time complexity of pointwise 1*1 convolution. The weights calculated include the weights of individual resolution feature maps and the weights of the overall resolution feature maps. Since HrNet itself has s parallel resolutions in the s stage, that is, s corresponding resolution weight mappings, the channel weight mapping under different resolutions is calculated using the calculation function P. The specific implementation is shown in Equation 2, where X is the input mapping of the resolution corresponding to the s branch.

[0049] P s (X1,X2,…,X s ) = (W1, W2, ..., W S )(Formula 2)

[0050] The P function performs adaptive pooling on each input and outputs a feature map with specified width and height. After processing by the ReLU and sigmold functions, a weight map containing s partitions is generated, where the definitions of ReLU and sigmold are shown in Equations 3 and 4.

[0051] ReLU(x) = max(0,x) (Formula 3)

[0052]

[0053] For each resolution, the lightweight channel weighting unit also performs spatial weighting to ensure spatial consistency. The weights depend on the total number of pixels in the input channel at the resolution and are calculated by the function H, as shown in Equation 5. The calculation function H performs global average pooling on the input mapping to collect spatial features from all locations, and then obtains the spatial weights through functions such as ReLU and sigmold.

[0054] H(X s ) = w s (Formula 5)

[0055] Based on the aforementioned cross-resolution and spatial weights, the lightweight channel-weighted unit realizes the generation and mixing of convolutional weights. The structure diagram of the lightweight channel-weighted unit is shown below. Figure 2 As shown, conditional weights are used for cross-channel information exchange, while always adhering to the high-resolution model of HRNet. During the process, high-resolution representation is maintained while replacing computationally expensive pointwise convolution.

[0056] II. Adaptive Strategy Upsampling Module

[0057] The feature maps obtained by the lightweight neural network algorithm module need to be upsampled before outputting high-resolution predictions. To avoid jagged edges in object segmentation caused by conventional direct bilinear interpolation upsampling, this technical solution proposes a rendering upsampling module. After receiving CNN feature maps with multiple channels, the upsampling module uses an adaptive strategy to obtain non-uniform points with strong uncertainty. For each difficult point, a point-by-point feature representation is used, and then a simple multilayer perceptron is used to predict the corresponding category point by point.

[0058] Pixels of uncertain categories are mainly distributed in high-frequency areas such as image edges. When lightweight networks sacrifice input and output resolution for timeliness, the jagged edges and lack of smoothness in uncertain areas are amplified. The adaptive upsampling module adopts a uniform coverage sampling strategy during training, randomly sampling candidate points from the feature map according to a uniform distribution. This ensures coverage of uncertain areas while maintaining a certain degree of uniform coverage of the global region. The loss function calculation is only performed on the generated candidate points. During output, the output mask is iteratively applied to perform coarse predictions on the points in the grid. If the predicted value is greater than a threshold, the point set is removed; otherwise, bilinear interpolation is used to upsample the segments predicted in the previous iteration. Then, N uncertain points with prediction probabilities close to 0.5 are selected. The upsampling module then calculates features for these N points and predicts labels, iterating until the segments are upsampled to the required resolution. The bilinear interpolation formula is shown in Equation 6. 11 Q 12 Q 21 Q 22 These are four point values.

[0059]

[0060] The implementation process of the adaptive strategy upsampling module is shown below.

[0061] (1) Use a Boolean check to determine if the feature resolution output by the backbone feature extraction network is less than the original resolution of the input image. If true, proceed to step (2); otherwise, terminate upsampling and output the feature map.

[0062] (2) Perform bilinear interpolation upsampling on the output feature map to obtain a rough prediction value of the pixel points within the closure region of the uncertain region.

[0063] (3) Select N uncertain points with a prediction probability close to 0.5 within the closure region, that is, points whose results at the edge of the object are different from the labels of their neighboring points.

[0064] (4) For each difficult point, the low-level features are obtained by interpolation in the low-level feature map through the coordinates of the point, and the high-level features are obtained by upsampling in step (1). The high-level features and the low-level features together form the representation vector of the pixel.

[0065] (5) Use a multivariate perceptron to make new predictions on the composed representation vectors and update the coarse prediction values. The multivariate perceptron structure diagram is shown below. Figure 3 As shown. After the prediction ends, jump back to step (1).

[0066] The adaptive strategy upsampling module focuses on edge region information, combines low-level and high-level features, and performs fine segmentation based on the features of edge points. It gradually refines the segmentation at different scales and utilizes the inaccurate values ​​of the network reclassification boundary to combine fine-grained and coarse-grained features, thereby improving the jagged and uneven phenomenon caused by the network's inaccurate edge prediction.

[0067] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0068] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] While the invention has been described above with reference to various embodiments, it should be understood that many changes and modifications can be made without departing from the scope of the invention. Therefore, the detailed description above is intended to be illustrative rather than restrictive, and it should be understood that the following claims (including all equivalents) are intended to define the spirit and scope of the invention. These embodiments should be understood as illustrative only and not as limiting the scope of protection of the invention. After reading the description of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent changes and modifications also fall within the scope defined by the claims of this invention.

Claims

1. A feature weighting real-time portrait segmentation method based on adaptive up-sampling strategy, characterized in that, The method includes: The image acquisition module is used as the input to acquire portrait images in real time, and the acquired portrait images are sent to the core computing unit. The core computing unit uses an improved lightweight neural network algorithm module to operate on the acquired portrait images to obtain feature maps and corresponding weights, thereby generating and mixing convolution weights. Conditional weights are used to exchange information across channels. At the same time, following the HRNet high-resolution network, high-resolution representation is maintained during feature extraction, replacing the computationally expensive pointwise convolution. The feature map obtained by the improved lightweight neural network algorithm module needs to be upsampled to output a high-resolution prediction. An adaptive strategy upsampling module is designed. After receiving CNN feature maps with multiple channels, the adaptive strategy upsampling module uses an adaptive strategy to obtain non-uniform points with strong uncertainty. For each non-uniform point, a point-by-point feature representation is used, and then a multilayer perceptron is used to predict the corresponding category point by point.

2. The feature weighted real-time portrait segmentation method based on adaptive up-sampling strategy according to claim 1, characterized in that, The image acquisition module is any monocular camera, and the core computing unit is a mobile CPU.

3. The feature weighted real-time portrait segmentation method based on adaptive up-sampling strategy according to claim 2, characterized in that, The improved lightweight neural network algorithm module further includes applying the improved shuffle block in ShuffleNet to the HRNet high-resolution network. The improved shuffle block is divided into two partitions according to channels. One partition is passed through a lightweight channel weighting unit whose complexity is linearly related to the number of channels, a 3*3 depthwise convolution, and then through another lightweight channel weighting unit. The output of this partition is stacked and concatenated with another unprocessed partition on a channel-by-channel basis, and finally the shuffle operation is performed.

4. The feature weighted real-time portrait segmentation method based on adaptive up-sampling strategy of claim 3, wherein, The shuffle operation further includes: since shuffle operation and depthwise convolution do not involve inter-channel information exchange, 1*1 convolution undertakes the task of inter-channel information exchange. Lightweight channel weighting units are set up, and information exchange is performed by calculating weights across channels and resolutions. In the k parallel resolution branches of the k-stage HRNet, the bit-weighted Q-factor of the ki branch... s As shown in Formula 1, Q s = W s X s (Formula 1) Among them, X S and Q S The input and output mappings are respectively represented. Ws represents a weight mapping matrix with shape WS × height HS × number of channels CS, where the symbol * indicates bitwise multiplication.

5. The feature weighted real-time portrait segmentation method based on adaptive up-sampling strategy according to claim 4, characterized in that, The channel weight mapping under different resolutions is calculated using the calculation function P, as shown in Equation 2. P s (X1,X2,…,X s ) = (W1, W2, ..., W S )(Formula 2) Where X is the input mapping of the resolution corresponding to the s branch.

6. The feature weighted real-time portrait segmentation method based on adaptive up-sampling strategy according to claim 5, characterized in that, The P function internally performs adaptive pooling operations on each input and outputs a feature map with a specified width and height. After processing by the ReLU and sigmold functions, a weight map containing s branches is generated, where ReLU and sigmold are defined as shown in Equations 3 and 4: ReLU(x) = max(0,x) (Formula 3) Where x represents the independent variables of the ReLU function and the sigmold function, respectively.

7. The feature weighted real-time portrait segmentation method based on adaptive up-sampling strategy of claim 6, wherein, Spatial weights depend on the total number of pixels in the input channel at the resolution, and are calculated using a function H, as shown in Equation 5: H(Xs) = W S (5) The function H performs global average pooling on the input mapping to collect spatial features from all locations, and then obtains spatial weights through ReLU and sigmold functions.

8. The feature-weighted real-time portrait segmentation method based on an adaptive upsampling strategy as described in claim 7, characterized in that, The adaptive strategy upsampling module adopts a uniform coverage transitional generation sampling strategy during training. It randomly samples candidate points from the feature map according to a uniform distribution, ensuring that it can cover uncertain areas while maintaining a certain degree of uniform coverage of the global area. The process of calculating the loss function is only performed on the generated candidate points.

9. The feature weighted real-time portrait segmentation method based on adaptive up-sampling strategy of claim 8, wherein, During output, the output mask is iteratively applied to make coarse predictions of the points on the grid. If the predicted value is greater than a threshold, the point set is removed; otherwise, bilinear interpolation is used to upsample the segments predicted in the previous iteration. Then, N non-uniform points with a prediction probability close to 0.5 are selected. The adaptive strategy upsampling module then calculates features for the N points and predicts labels. This process iterates until the segments are upsampled to the required resolution. The bilinear interpolation formula is shown in Formula 6. where Q 11 Q 12 Q 21 Q 22 are four point values, respectively.

10. The feature weighted real-time portrait segmentation method based on adaptive up-sampling strategy of claim 9, wherein, The specific steps of the adaptive strategy upsampling module's strategy execution are as follows: Step 1: Use Boolean judgment to determine whether the feature resolution output by the backbone feature extraction network is less than the original resolution of the input image. If true, proceed to step 2; otherwise, terminate upsampling and output the feature map. Step 2: Perform bilinear interpolation upsampling on the output feature map to obtain a rough prediction value of the pixel points within the closure region of the uncertain region; Step 3: Select N uncertain non-uniform points with a prediction probability close to 0.5 within the closure region, that is, points located at the edge of the object whose labels are different from those of their neighbors. Step 4: For each non-uniform point, interpolation is performed on the bottom-level feature map using the point's coordinates to obtain low-level features. High-level features are obtained through upsampling in Step 1. The high-level features and low-level features together form the representation vector of this pixel; Step 5: Use a multivariate perceptron to make new predictions on the composed representation vectors, update the coarse prediction values, and then jump back to step 1 after the prediction ends.

Images