Fast Portrait Segmentation Method, Device, and Medium Based on FastPortrait Model

Abstract

This invention discloses a method, device, and medium for fast video portrait segmentation based on the FastPortrait model. The method involves: decomposing the video to be segmented into multiple consecutive frames; and performing portrait segmentation using a pre-trained portrait segmentation model. The portrait segmentation model employs a FastPortrait model with an encoder-decoder structure, where a memory matching module is located between the encoder and decoder at the lowest level. This memory matching module extracts and enhances portrait features from the feature images output by the encoder based on a historical image feature memory library. This invention enables fast and accurate predictive segmentation of portraits in videos.

Description

Technical Field

[0001] This invention belongs to the field of image intelligent processing technology, specifically relating to a method, device, and medium for fast video portrait segmentation based on the FastPortrait model. Background Technology

[0002] Currently, lightweight model architectures available for image segmentation include U2-Net, Lite-HRNet, STDCBiSeNet, ISNet, HyperSeg, and PP-HumanSeg. While these models have made significant progress, various technical challenges remain in video portrait segmentation tasks.

[0003] (1) In existing model methods, downsampling techniques still use convolutional or pooling layers with a stride greater than 1, which results in insufficient clarity of feature maps after multiple downsamplings and significant semantic loss between pixels, making them less than ideal for accurate portrait segmentation tasks.

[0004] (2) Most current video portrait segmentation models are based on a single image, and the semantic relationship between video frames is often ignored, resulting in poor segmentation stability.

[0005] (3) In existing research, the backbone networks of lightweight real-time semantic segmentation models are basically MobileNet or EfficientNet series. The main building block of the model is the mobile inverse bottleneck convolution (MBConv). Using the original SE attention structure, it cannot capture the positional relationship between pixels and has insufficient semantic understanding ability.

[0006] (4) Some methods propose using boundary loss to improve the model's ability to learn the shape of human figures. These methods mainly use boundary-based cross-entropy loss. However, calculating cross-entropy loss on the boundary does not reflect the actual segmentation effect of the model well, and the loss design is flawed. Summary of the Invention

[0007] This invention provides a method, device, and medium for fast video portrait segmentation based on the FastPortrait model, which can quickly and accurately predict and segment portraits in videos.

[0008] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0009] A fast video portrait segmentation method based on the FastPortrait model decomposes the video to be segmented into consecutive multi-frame images and uses a pre-trained portrait segmentation model for portrait segmentation. The portrait segmentation model adopts the FastPortrait model with an encoder-decoder structure, and a memory matching module is set between the encoder and decoder at the lowest layer. The memory matching module extracts and enhances portrait features in the feature images output by the encoder based on a historical image feature memory library.

[0010] Furthermore, the encoder's convolutional layers use EfficientNetV2's building blocks, which consist of moving backward bottleneck convolutions and fused moving backward bottleneck convolutions, to extract human image semantics from the input video frame images.

[0011] Furthermore, the input layer of the moving inverse bottleneck convolution uses depthwise separable convolution, and after the depthwise separable convolution operation, a coordinate attention mechanism is used to refine the feature map.

[0012] Furthermore, the encoder downsampling employs an inverted subpixel downsampling module:

[0013] (1) The input feature map of the downsampled sample Perform a 1×1 convolution to generate a channel number of The feature map x' is then grouped into pixels according to equation (1) to obtain the feature map x'.

[0014] z m,n =x i,j ' where i mod 2=m and jmod 2=n (1)

[0015] In the formula, i and j represent the subscript indices of the image, and m and n are used to group the pixels.

[0016] (2) The feature map z m,n The feature map is obtained by concatenating the data along the channel dimension.

[0017] (3) Perform a 1×1 convolution on the feature map z to reduce the number of channels. Let it be denoted as , and the feature map is obtained.

[0018] (4) Perform a downsampling operation on the downsampled input feature map x with C output channels;

[0019] (5) Concatenate the outputs of steps (3) and (4), and then perform a 3×3 convolution to obtain the inverted subpixel downsampled output feature map. As shown in equation (2):

[0020]

[0021] In the formula, This indicates a channel-dimensional concatenation operation, where F1 and F3 are 1×1 and 3×3 convolutions, respectively, and DS is downsampling.

[0022] Furthermore, a lightweight aggregation encoder and a memory update module are employed to update the historical image feature memory in real time when the decoder outputs the previous frame prediction map, based on the historical frame prediction map, the current frame feature map output by the encoder, and the current frame original map.

[0023] Furthermore, the method for real-time updating the historical image feature memory is as follows:

[0024] (1) The lightweight clustering encoder uses convolution and depthwise separable convolution to perform feature transformation and downsampling on the historical frame prediction map output by the portrait segmentation model according to the feature map size generated by the backbone encoder, and obtains a set of feature maps; the backbone encoder refers to the encoder in the encoder-decoder-based portrait segmentation model.

[0025] (2) Cut the original image into patches and convolve it with the predicted image output by the decoder to generate a feature map. Then, after concatenating the generated feature map with the feature map generated after each downsampling of the backbone encoder in the channel dimension, perform 1×1 convolution fusion to generate a key and value matrix with the semantic information of the human image in the current frame, which is called the current frame image feature memory. The key matrix is ​​used to match the semantic association of the human image, and the value matrix stores the complete image features. For the first frame, the key and value of the current frame are directly used as the image feature memory. For frames from the second frame onwards, the memory update method in steps (2)-(6) is used to generate the historical image feature memory.

[0026] (3) Starting from the second frame, query the key value of the current predicted frame generated by the backbone network encoder of the current frame. and memory key values ​​in historical image feature memory bank First, K S With K mem Calculate cosine similarity to obtain the similarity matrix. As shown in equation (11):

[0027]

[0028] Where n is the sequence length of the historical image feature memory, ||·|| represents the second norm of the vector, and i and j represent the matrix K respectively. Sand K mem Indexes along the sequence dimension;

[0029] (4) For K S For each row i in the array, take its sum with K. mem The maximum value in the cosine similarity is obtained. As shown in equation (12):

[0030]

[0031] (5) For K mem For each column j, take it and K S The maximum cosine similarity was obtained. As shown in equation (13):

[0032]

[0033] (6) For m S-mem Sort the vectors and select the top β-proportions with the lowest cosine similarity. S-mem ', as the update vector, is concatenated into the historical image feature memory; simultaneously, for m mem-S Sort the vectors and retain the top 1-β proportions of the vectors with the highest similarity. mem-S '; Ultimately, M S-mem 'with M mem-S 'Concatenate the matrices to obtain a key matrix whose size remains unchanged after the update.' and value matrix

[0034] Furthermore, a memory update trigger is set before the lightweight aggregation encoder to determine whether the current frame image triggers an update of the historical image feature memory. The specific determination method is as follows:

[0035] (1) For the input image x of two adjacent frames t-1 x t With output prediction graph y t-1 y t The differences between adjacent input frames are calculated using equations (14) and (15), respectively. and output adjacent frame differences

[0036]

[0037]

[0038] In the formula, c represents any R, G, B channel of the image; i represents any pixel in the image;

[0039] (2) For each pixel i in the input image, use equation (16) to calculate the degree of change P of that pixel. i :

[0040]

[0041] Among them, th x With th y The preset threshold for change;

[0042] (3) The degree of change P of all pixels in the image i The summation yields the image change degree P, which is then compared with a preset image change threshold P. th The comparison only occurs when the input frame image x t When the degree of change P exceeds the threshold, the historical image feature memory is updated.

[0043] Furthermore, the human face segmentation model is trained using a composite loss function consisting of weighted cross-entropy loss, IOU loss, and boundary-aware InverseForm loss; the composite loss function is specifically as follows:

[0044] L = L CE +γ1L IOU +γ2L IF

[0045]

[0046] Among them, L CE For weighted cross-entropy loss, L IOU For IOU loss, L IF For InverseForm loss, γ1 and γ2 are the weights of the loss, b pred For the predicted graph boundary, b true This represents the true label boundary.

[0047] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor enables the processor to implement any of the above-described methods for fast video portrait segmentation based on the FastPortrait model.

[0048] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for fast video portrait segmentation based on the FastPortrait model.

[0049] Beneficial effects

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] 1. The Inverted Sub-Pixel Down-Sampling module is used to reduce the semantic loss between pixels caused by the downsampling process, thereby improving the prediction accuracy of video portrait segmentation.

[0052] 2. A memory matching module is used for video portrait segmentation. Temporal semantic associations are used to assist the model in segmentation and improve its stability. Simultaneously, to improve computational efficiency, a bidirectional memory update module can efficiently update portrait semantic information in real time.

[0053] 3. Improve the attention mechanism of the MBConv module in the encoder by using the same lightweight coordinate attention to enhance the model's ability to capture spatial location context information. This can more accurately locate the human image region and thus improve the accuracy of human image segmentation.

[0054] 4. The InverseForm loss function based on boundary awareness is used to train the model, which makes the portrait segmentation model more capable of capturing the shape of portraits. Attached Figure Description

[0055] Figure 1 This is a structural diagram of the FastPortrait model described in the embodiments of this application;

[0056] Figure 2 This is a structural diagram of the inverted subpixel downsampling module of the FastPortrait model;

[0057] Figure 3 This is a structural diagram of the lightweight aggregation encoder of the FastPortrait model;

[0058] Figure 4 This is a structural diagram of the memory matching module of the FastPortrait model;

[0059] Figure 5 This is a structural diagram of the bidirectional memory update module of the FastPortrait model;

[0060] Figure 6 This is a schematic diagram illustrating the boundary loss caused by a small spatial offset in the prediction map;

[0061] Figure 7The segmentation prediction results of each model are compared. The models corresponding to the numbers are: (a) FastPortrait, (b) U2-Net Lite, (c) Lite HRNet, (d) STDC, (e) ISNet, (f) HyperSeg, and (g) PP-HumanSeg.

[0062] Figure 8 This is a graph showing the relationship between the number of parameters and mIOU in the FastPortrait model. Detailed Implementation

[0063] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.

[0064] This invention provides a fast video portrait segmentation method based on the FastPortrait model. The video to be segmented is decomposed into a series of consecutive frames, and a pre-trained portrait segmentation model is used for portrait segmentation. The portrait segmentation model adopts the FastPortrait model with an encoder-decoder structure, and a memory matching module is set between the encoder and the decoder at the lowest level. The memory matching module extracts and enhances the portrait features in the feature image output by the encoder according to the historical image feature memory library.

[0065] This invention addresses the practical application characteristics of video portrait segmentation, ensuring accuracy while maintaining computational efficiency. It innovatively proposes the FastPortrait model to construct a portrait segmentation model, applying it to video portrait segmentation for fast and accurate segmentation.

[0066] The FastPortrait model consists of two main parts: the SegmentModule, which is responsible for inference, and the MemoryModule, which is responsible for image semantic memory.

[0067] The SegmentModule is the backbone of the FastPortrait model, consisting of an encoder, a memory match module, and a multi-scale fusion decoder. The encoder uses EfficientNetV2's building blocks—moving backward bottleneck convolution (MBConv) and fused MBConv—to extract human image semantics from the input video frames. It also employs an innovative inverted subpixel downsampling module (ISP) to reduce semantic loss during downsampling. Inspired by the Transformer model, the memory match module fully utilizes historical and current image semantics for feature vector matrix multiplication, enhancing human image features and filtering out non-human background. The multi-scale fusion decoder, composed of atrous spatial pyramid pooling and a multi-scale residual refinement module, extracts, analyzes, and decodes the features after memory matching.

[0068] In video-based portrait segmentation tasks, the semantic information of portraits extracted from historical video frames often plays a crucial role in improving segmentation accuracy. Therefore, this invention uses a MemoryModule to learn portrait semantic extraction patterns end-to-end. The MemoryModule consists of a Lightweight Aggregation Encoder (LAE) and a Memory Update Module (MUM). The LAE fuses the portrait features of the previous frame with the predicted segmentation map to generate fused portrait features, which are cached as key-value pairs for use in subsequent video segmentation frames. For each frame's generated memory portrait features, simple feature concatenation leads to increasingly high resource consumption and computational load during long video runs, failing to meet the requirements of speed and lightweight design. To address this, this embodiment uses a Memory Update Module to ensure that the key-value pair dimensions in the memory features remain unchanged during inference, while also ensuring that the most effective features are retained.

[0069] The FastPortrait model, which is applied to fast video portrait segmentation in this application, will be described in detail below.

[0070] 1. Inverted Sub-Pixel Down-Sampling (ISP) module

[0071] Current semantic segmentation models primarily employ Encoder-Decoder or VisonTransformer (ViT) structures. ViT, in particular, uses image patch-based matrix multiplication to capture semantic relationships between pixels, offering significant advantages in semantic understanding. However, modeling based on pixel sequences is computationally expensive. Therefore, the Encoder-Decoder structure remains the optimal choice for building an efficient real-time semantic segmentation model. The Encoder-Decoder approach first performs progressive downsampling using the Encoder, followed by hierarchical upsampling using the Decoder. To compensate for semantic loss during downsampling and to better extract low-level semantic information, some studies use skip-connections to fuse intermediate feature maps generated by the Encoder into the Decoder, such as U-Net and U2-Net. While these methods have achieved some success, the information loss during image downsampling is irreversible, and the feature maps generated by Decoder upsampling are often coarse. Therefore, the coarse fusion using skip-connections still yields unsatisfactory results. In response, this embodiment innovatively proposes an inverted subpixel downsampling structure, which minimizes information loss during the downsampling process.

[0072] The inverted sub-pixel downsampling module in this embodiment refers to... Figure 2 As shown, the specific method is as follows:

[0073] (1) The input feature map of the downsampled sample Perform a 1×1 convolution to generate a channel number of The feature map x' is then grouped into pixels according to equation (1) to obtain the feature map x'.

[0074] z m,n =x i,j ' where i mod 2=m and jmod 2=n (1)

[0075] In the formula, i and j represent the subscript indices of the image, and m and n are used to group the pixels.

[0076] (2) The feature map z m,n The feature map is obtained by concatenating the data along the channel dimension. At this point, the pixel information in the original image is preserved in different channels, while the image size is reduced to 1 / 4 of the original size, which is the same as the output of normal downsampling, but minimizes the loss of pixel information during downsampling.

[0077] (3) Perform a 1×1 convolution on the feature map z to reduce the number of channels to 1 / 2, and obtain the feature map.

[0078] (4) Perform a downsampling operation on the downsampled input feature map x with C output channels;

[0079] (5) Concatenate the outputs of steps (3) and (4), and then perform a 3×3 convolution to obtain the inverted subpixel downsampled output feature map. As shown in equation (2):

[0080]

[0081] In the formula, This indicates a channel-dimensional concatenation operation, where F1 and F3 are 1×1 and 3×3 convolutions, respectively, and DS is downsampling.

[0082] The inverted subpixel downsampling module combines two downsampling methods to compensate for pixel loss and channel loss, preserving effective information to the greatest extent and improving portrait segmentation accuracy.

[0083] 2. Moving Inverse Bottleneck Convolution (MBConv)

[0084] For a traditional convolutional layer with kernel size m×n, input channels c1, and output channels c2, the number of parameters is m×n×c1×c2 (ignoring bias). When the number of input and output channels is large, the total number of parameters for a convolutional layer often reaches hundreds of thousands, resulting in a large model size. To address this, the MBConv module uses depthwise separable convolution at the input layer to effectively reduce the number of parameters and computational cost. Depthwise separable convolution consists of Depthwise Convolution and Pointwise Convolution. Depthwise Convolution uses an independent kernel for each channel, with a parameter count of m×n×c1; Pointwise Convolution uses a 1×1 traditional kernel, with a parameter count of c1×c2. With the same kernel size and the same number of input and output channels, depthwise separable convolution significantly reduces the number of parameters compared to traditional convolution, allowing for the construction of deeper network structures with less computational overhead.

[0085] Following the depthwise separable convolution operation, MBConv uses an attention mechanism to refine the input feature map. The original MBConv module employs the classic Squeeze and Excitation (SE) attention structure. However, this attention structure ignores the relative positional features present in the input feature map, applying the same weight to pixels within each channel. This embodiment improves upon this by using a Coordinate Attention (CA) mechanism, introducing horizontal and vertical attention factors. This allows the attention layer to learn contextual information between pixels, improving model performance with a smaller increase in the number of parameters. The structure of coordinate attention will be detailed in Section 3.

[0086] Overall, MBConv employs a strategy of first increasing and then decreasing the number of channels to extract and analyze fine-grained features of the input data in the middle part of the module before aggregation, achieving a good balance between running efficiency and model performance. Regarding the activation function, MBConv replaces the original ReLU function with the swish activation function, resulting in smoother gradients and improved convergence, as shown in equation (3).

[0087] δ(x)=x·σ(x) (3)

[0088] Where σ is the Sigmoid function.

[0089] The FastPortrait model uses the MBConv module to build the backbone of the network. Compared with traditional convolutional modules, it uses fewer parameters and less computation to build more network layers, making it suitable for computing terminals with lower computing power.

[0090] 3. Coordinate Attention (CA)

[0091] In recent years, with the development of deep learning, novel attention structures have been proposed, significantly improving the prediction accuracy of models. Considering computationally limited scenarios, the most widely used attention mechanism remains Squeeze and Excitation (SE) attention. SE generates a feature vector through 2D global pooling, multiplying channels containing effective information in the input feature map with high weights to improve the model's feature extraction capability. However, SE attention only considers information between channels, ignoring the relative positional information between pixels within the same channel. The subsequently proposed CBAM attention uses convolutional layers to extract positional features of the input image, but convolutional operations struggle to capture long-range dependencies in images, which is crucial for portrait segmentation tasks. Therefore, this embodiment uses coordinate attention to replace the attention structure in the traditional MBConv module.

[0092] In SE attention, the 2D global average pooling of the input image is shown in Equation (4).

[0093]

[0094] in, The input is an image, where c represents the image channel, and C, H, and W represent the number of input channels, image height, and image width, respectively.

[0095] In coordinate attention, in order to enable the attention structure to retain and learn long-range dependency information in the input image, this embodiment decomposes the pooling process into pooling in two directions, horizontal and vertical, as shown in equations (5) and (6).

[0096]

[0097]

[0098] By using 1D pooling, we can obtain two feature tensors that aggregate information from the horizontal and vertical directions, respectively. and

[0099] The generated feature tensors are first concatenated in terms of spatial dimensions and then convolved using a shared convolution kernel, as shown in Equation (7).

[0100]

[0101] in, δ represents the concatenation of tensors in spatial dimensions, F1 represents the shared 1×1 convolution operation, and δ is the swish activation function.

[0102] Next, the obtained output z is re-segmented into horizontal and vertical tensors, and convolution and sigmoid activation operations are performed on them respectively to obtain... and At this time, z h With z w Horizontal and vertical pixel features were aggregated separately, while preserving the relative positional correlation between the vertical and horizontal elements. Finally, the original input image was compared with z... h and z w Multiply the results to obtain the feature-weighted attention output.

[0103] Coordinate attention, while considering information between channels, can also learn the encoded information of spatial location associations. Compared with SE attention, it can more accurately locate the human face region, thereby improving the model's segmentation performance.

[0104] 4. Lightweight Aggregation Encoder (LAE)

[0105] For video portraits, the semantic relationships between frames often help the model to more accurately identify the portrait region. Inspired by the real-time video object segmentation model SwiftNet, this embodiment introduces a lightweight aggregation encoder module to extract the semantics of the portrait in the current frame and assist in the segmentation of subsequent video frames.

[0106] This embodiment performs aggregate encoding on the intermediate outputs of the encoder part of the FastPortrait model backbone and the final prediction results of the model, referring to... Figure 3 As shown, the LAE uses ordinary convolution and depthwise separable convolution to perform feature transformation and downsampling on the model's prediction map, obtaining a set of feature maps with the same size as the intermediate results generated by the backbone encoder. Then, the input image is segmented into patches, which are convolved with the prediction map output by the decoder to generate feature maps. These generated feature maps and the intermediate results generated after each downsampling by the backbone encoder are then concatenated along the channel dimension and fused using a 1×1 convolution to generate a key-value matrix containing the semantic information of the human image in the current frame, denoted as the current frame image feature memory. The key matrix is ​​used to match semantic associations of the human image, and the value matrix stores the complete image features to assist in identifying the human image region in subsequent frames. For the first frame, the key and value of the current frame are directly used as image features for memory. For frames starting from the second frame, the details are described in the bidirectional memory update module section.

[0107] 5. Memory Match Module

[0108] The feature map generated by the encoder in the backbone section contains human image information and other background information. In the memory matching module, the human image features are extracted and enhanced using the feature map and the previously generated key and value.

[0109] Let the historically generated key and value be and Where d key and d value Let be the dimension of the feature vectors for the key and value, and n be the length of the feature vector sequence. For the feature map S output by the backbone encoder, a set of depthwise separable convolutions is first used to generate... Q S K S V represents the query vector and key vector, used for semantic information matching. S V is a value vector used to store image features. SSimilar to the idea behind shuffle nets, it's used to form residual connections. Then K... mem With K S V mem With V S Concatenate along the sequence dimension to obtain and Inspired by the Transformer model, this embodiment uses the method shown in Equation (8) to implement memory matching query and enhancement.

[0110]

[0111] After memory matching, this embodiment will obtain V S "with V" S 'Concatenate and perform convolution operations along the channel dimension to generate the final output V, as shown in Equation (9).

[0112]

[0113] 6. Atrous Spatial Pyramid Pooling (ASPP) Module

[0114] ASPP was first proposed in the DeepLabV2 paper and has been improved in subsequent versions of the DeepLab model, such as by introducing normalization and adding depthwise separable convolutions. Traditional convolutional neural networks can extract abstract high-level semantic information well, but due to the limited receptive field of the convolutional kernel, some detailed spatial information is ignored. In addition, due to the diversity of target scales in human image semantic segmentation, traditional convolutional structures encounter difficulties in segmentation, resulting in insufficient accuracy. ASPP effectively solves the above problems through multi-scale spatial pyramid pooling layers, enabling the model to capture multi-scale targets in the input feature map well and improve the model's human image segmentation performance.

[0115] ASPP uses convolutional kernels with varying degrees of dilation to achieve a larger receptive field, compensating for the smaller receptive field of ordinary convolutional layers and enabling it to better capture spatial location information between pixels. As a gain module, ASPP uses depth-separable dilated convolutions in FastPortrait to reduce the number of parameters, and uses ordinary convolutions in the final layer of ASPP fusion to reduce semantic loss.

[0116] 7. Multi-Scale Residual Refine Module (MRRM)

[0117] Inspired by the U-Net model, the low-level semantic information output from shallow networks often greatly aids in model prediction. Therefore, this embodiment uses a multi-scale residual extraction module in the backbone network decoder to fuse low-level and high-level feature maps.

[0118] For each intermediate result generated by the trunk encoder during downsampling, this embodiment uses a bilinear interpolation algorithm to upsample it and add it to the feature map before downsampling by the trunk encoder. Then, the features of the added result are transformed and fused through MBConv and depthwise separable convolutional layers.

[0119] The multi-scale residual extraction module utilizes skip-connection to enable the model to have receptive fields of different sizes, resulting in a stronger ability to recognize multi-scale human targets.

[0120] Where F1 represents a 1×1 ordinary convolution, This represents the channel dimension concatenation operation of a tensor.

[0121] During training, the model will learn the human features in the key and value, and perform background filtering and human enhancement on the semantic information output in the current frame.

[0122] For the first frame of the video, there are no memory features, so the above matching operation degenerates into the standard Transformer self-attention structure, as shown in Equation (10).

[0123]

[0124] For K T Softmax processing of the matrix multiplication with Q ensures that the expected value and variance of the generated matrices are the same when matching the first frame and subsequent frames, which is beneficial for model convergence. During the training phase, this embodiment will use a random two-frame training method to ensure that the model achieves good results in both the first and subsequent frames where there is no memory information. Specific training methods will be described in detail in the experimental section.

[0125] 8. Bidirectional Memory Update (BMU)

[0126] In traditional video segmentation, the target semantics generated from historical video frames are typically processed in two ways. The first approach caches all semantic information from each video frame for prediction of the next frame. This method results in significant spatiotemporal redundancy in the cached historical semantic information, leading to high computational costs when processing long videos. The second, mainstream approach fuses the semantics of the first and previous frames for segmentation of the next frame. This method indiscriminately discards the semantics of intermediate frames, ignoring any useful information they contain. To address this, the recent paper SwiftNet proposes a memory-update-based model framework that effectively filters out spatiotemporal redundancy, extracts useful information from all historical frames, and improves model efficiency. Building upon this, this embodiment improves the memory update module for the specific application scenario of human face segmentation, proposing a bidirectional memory update structure to ensure constant model efficiency. The original memory update module of SwiftNet will be introduced first.

[0127] The query key value for the current predicted frame generated by the backbone network encoder Memory key value of historical frames generated by the memory module (where n is the sequence length of the historical image feature memory), SwiftNet's memory update first updates K... S With K mem Calculate cosine similarity to obtain the similarity matrix. As shown in equation (11).

[0128]

[0129] Where ||·|| represents the second norm of the vector, and i and j represent the matrix K respectively. S and K mem Indexes for the sequence dimension.

[0130] For K S For each row i in the dataset, this embodiment takes the maximum value among a set of cosine similarities between it and historical data, and obtains... As shown in equation (12).

[0131]

[0132] Then, for m S-mem Sort the vectors and select the top β-proportions with the lowest cosine similarity. S-mem The updated key and value are then appended to the historical memory data as an update vector. Thus, in this embodiment, the updated key and value are obtained: Unlike previous methods, the memory update method proposed by SwiftNet selects a set of vectors with large semantic changes by using cosine similarity and adds them to the historical memory matrix, which effectively reduces spatiotemporal redundancy and improves computational efficiency.

[0133] However, the memory update method proposed by the SwiftNet model directly concatenates vectors with a β ratio. In scenarios involving long videos, portraits, and frequently changing backgrounds, the size of the key-value matrix continues to increase, leading to higher memory consumption and reduced computational efficiency. SwiftNet was proposed for a generalized target segmentation task based on few-shot learning (specifically, one-shot learning), which requires preserving as much semantic information as possible to improve segmentation accuracy. However, when focusing on specific portrait segmentation tasks, it was found that the model itself can learn some fixed portrait patterns. Therefore, historical semantic information can be largely simplified and used as auxiliary information to help the model segment. Thus, this embodiment proposes an improved Bidirectional Memory Update (BMU) module to ensure that the size of the key and value remains constant, avoiding the problem of model efficiency decreasing over time.

[0134] As mentioned earlier, SwiftNet uses the vectors from the first beta proportion as update vectors triggered by scene changes to enrich historical memory information. However, as the video scene progresses, some cached feature vectors in historical memory become redundant and may even interfere with subsequent predictions. To address this, the bidirectional memory update module utilizes another dimension of the similarity matrix to determine which memory information should be discarded.

[0135] For K mem For each column j, in this embodiment, we take its value and K. S The maximum cosine similarity was obtained. As shown in equation (13).

[0136]

[0137] Similarly, for m mem-S Sort the data and retain the historical memory features with the highest similarity (1-β) to obtain M. mem-S Finally, M S-mem 'with M mem-S 'By concatenating them, we obtain a memory key and value of the same size:'

[0138] The bidirectional memory update module discards redundant information from the old scene while accepting the semantics of the new scene, ensuring that the size of the memory key and value remains unchanged, thus keeping the resource consumption of the model at a low level during operation.

[0139] 9. Memory Update Trigger (MUT)

[0140] During the prediction process of a video, consecutive frames often contain extremely similar scenes and portraits. Therefore, bidirectional memory updates are not necessary after each frame prediction. To address this, this embodiment uses a memory update trigger algorithm to determine whether a new frame requires a memory update, reducing unnecessary computation.

[0141] For two adjacent frames of input image x t-1 x t With output prediction graph y t-1 y t In this embodiment, the differences between them are calculated using equations (14) and (15).

[0142]

[0143]

[0144] For each pixel i in the image, this embodiment uses equation (16) to calculate the degree of change of the pixel.

[0145]

[0146] Among them, th x With th y This is a preset threshold for change.

[0147] Then all P i The summation yields the image change degree P, which is then compared with a preset image change threshold P. th In comparison, bidirectional memory updates are only performed on frames that exceed the threshold, reducing unnecessary computation.

[0148] 10. Loss Function

[0149] This embodiment uses a composite loss function consisting of weighted cross-entropy loss, IOU loss, and boundary-aware InverseForm loss to train the human image segmentation model.

[0150] (1) Weighted cross-entropy loss

[0151] To balance the sample labels, this embodiment uses a weighted cross-entropy loss function. The formula for the weighted cross-entropy loss is shown in Equation (17).

[0152] L CE =-αy gt log(y pred )-(1-α)(1-y gt log(1-y) pred (17)

[0153] Among them, y gt For real labels, y pred α represents the predicted label, and α is the proportion of negative samples in the batch data.

[0154] (2) IOU loss

[0155] In addition to the weighted cross-entropy loss, this embodiment also uses IOU loss for optimization, and its calculation formula is shown in Equation (18).

[0156]

[0157] (3)InverseFormLoss

[0158] Both cross-entropy loss and IOU loss rigidly teach the model which points represent a person and which do not, thus ignoring information such as the overall shape and spatial position of the person. Therefore, this embodiment uses the boundary-aware InverseForm loss function to improve upon this approach.

[0159] In human face segmentation tasks, the boundary information of the target human face often plays a crucial role in improving segmentation accuracy and learning human appearance features. Previous boundary loss functions directly calculate the difference between the model's prediction and the ground truth label using cross-entropy loss. In most cases, the better the model's prediction, the lower the loss calculated by the loss function should be. However, Figure 6 In the case shown, the traditional boundary loss function does not conform to this characteristic.

[0160] Even a small spatial shift in the prediction map will result in a very large loss value calculated using traditional boundary loss. Figure 6 The loss between x2 and t(x2) is calculated, while the loss at another boundary that is completely unrelated to this region is much smaller. Figure 6 The loss between x1 and x2 (in the original text) does not accurately reflect the boundary segmentation effect of the image. Therefore, this embodiment uses InverseForm to better perceive the boundary shape loss.

[0161] For the model's prediction graph boundary b pred In this embodiment, it is assumed that the boundary b with the real label can be reduced by the affine transformation shown in equation (19). trueDifferences between them:

[0162]

[0163] in, It is the boundary b of the prediction graph pred The coordinates of each point on the graph. It is the true label boundary b true The coordinates of each point on the map. Let be the affine transformation matrix.

[0164] In this way, the difference between the predicted boundary and the true boundary can be determined by whether the affine matrix is ​​close to the identity transformation matrix (i.e., by applying an identity transformation to the image). To determine this. InverseForm learns how to use a pre-trained deep neural network to determine b. pred and b true To determine the affine matrix between them (similar to the idea of ​​GAN), and to calculate the second norm of this affine matrix and the unit affine matrix, a boundary-aware loss function is generated, as shown in Equation (20).

[0165]

[0166] Finally, the loss function for model training is shown in equation (21).

[0167] L = L CE +γ1L IOU +γ2L IF (twenty one)

[0168] Where γ1 and γ2 are the weights of the loss, a suitable combination of values ​​here is: This is the experimental configuration used in this embodiment.

[0169] 11. Experiment and Results Analysis

[0170] (1) Experimental dataset, the sources of which are shown in Table 1.

[0171] Table 1

[0172]

[0173] This experiment involved manual screening and organization of the collected experimental dataset, selecting videos containing human images and creating metadata (meta.json). Low-quality videos, such as those depicting multiple people but only labeling one or a few individuals, were excluded. Ultimately, approximately 20,000 high-quality images containing human images were obtained, and 25% of these were randomly selected as the validation set.

[0174] To reduce overfitting, random data augmentation is performed on the collected dataset. Augmentation methods include rotation, flipping, random cropping, random scaling, and adding Gaussian noise and salt-and-pepper noise at different proportions.

[0175] (2) Training methods and parameter configuration

[0176] Model training was performed using PyTorch, and the experimental environment and hardware configuration are shown in Table 2.

[0177] Table 2

[0178] Configuration items Configuration Description GPU 2*Tesla V100 32GB CPU Intel(R)Xeon(R)CPU E5-2698 v4@2.20GHz Memory capacity 48GB operating system Ubuntu Server 18.04 LTS Python Python 3.8 PyTorch 1.10.0 CUDA 11.3

[0179] Since the dataset contains single-image data, the model's segmentation module is first pre-trained on a single image to obtain a trained segmentation backbone. Then, the memory module is trained using a random two-frame training method, using the key and value generated in the previous frame to assist the segmentation in the next frame. Finally, backpropagation is used to optimize the LAE and the key and value generation module, as well as to fine-tune the segmentation backbone network.

[0180] The hyperparameter configurations for training are shown in Table 3.

[0181] Table 3

[0182]

[0183] (3) Results Comparison and Analysis

[0184] The model was tested using 20 test videos provided by the company. From the test results, simple single-person scenarios, complex single-person scenarios, and complex multi-person scenarios were selected as test examples. The output of each model was then displayed and analyzed. Figure 7 As shown.

[0185] As can be seen, in both simple single-person scenes (the first video) and complex single-person scenes (the second and third videos), the FastPortrait model has more detail in the edge shape of the portrait and the segmentation of limbs (such as fingers), and performs better. Meanwhile, in complex multi-person scenes (the fourth and fifth videos), FastPortrait can accurately segment the shape features of multiple human bodies and has better stability.

[0186] The quantitative evaluation metrics of each model on the segmented validation set are shown in Table 4.

[0187] Table 4

[0188] Model Number of parameters (#Params) mIOU (%) <![CDATA[U 2 -Net Lite]]> 1.4M 93.1 Lite HRNet 1.8M 91.3 STDC 1.7M 92.8 ISNet 2.1M 91.4 HyperSeg 3.5M 92.6 PP-HumanSeg 1.3M 90.5 FastPortrait (Ours) 1.4M 93.7

[0189] Meanwhile, the relationship between the number of model parameters and mIOU is as follows: Figure 8 As shown.

[0190] As can be seen from the quantitative indicators and comparison results, the FastPortrait model achieves a good balance in terms of model parameter count, running efficiency, and accuracy (mIOU). It has stronger robustness in various portrait segmentation scenarios and is an efficient and accurate video portrait segmentation model.

[0191] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.

Claims

1. A fast video portrait segmentation method based on the FastPortrait model, characterized in that, The video to be segmented is decomposed into a series of consecutive frames, and a pre-trained portrait segmentation model is used for portrait segmentation. The portrait segmentation model adopts the FastPortrait model with an encoder-decoder structure, and a memory matching module is set between the encoder and the decoder at the lowest level. The memory matching module extracts and enhances the portrait features in the feature image output by the encoder based on the historical image feature memory library. The encoder downsampling uses an inverted subpixel downsampling module: (1) The input feature map of the downsampled sample Perform a 1×1 convolution to generate a channel number of The feature map x' is then grouped into pixels according to equation (1) to obtain the feature map x'. z m,n =x i,j ' whereimod2=mandjmod2=n (1) In the formula, i and j represent the subscript indices of the image, and m and n are used to group the pixels. (2) The feature map z m,n The feature map is obtained by concatenating the data along the channel dimension. (3) Perform a 1×1 convolution on the feature map z to reduce the number of channels. Let be the feature map obtained. (4) Perform a downsampling operation on the downsampled input feature map x with C output channels; (5) Concatenate the outputs of steps (3) and (4), and then perform a 3×3 convolution to obtain the inverted subpixel downsampled output feature map. As shown in equation (2): In the formula, This indicates a channel-dimensional concatenation operation, where F1 and F3 are 1×1 and 3×3 convolutions, respectively, and DS is downsampling.

2. The video portrait fast segmentation method according to claim 1, characterized in that, The encoder's convolutional layers use EfficientNetV2's building blocks, which consist of moving backward bottleneck convolutions and fused moving backward bottleneck convolutions, to extract human image semantics from the input video frame images.

3. The method for fast video portrait segmentation according to claim 2, characterized in that, The input layer of the moving inverse bottleneck convolution uses depthwise separable convolution, and after the depthwise separable convolution operation, a coordinate attention mechanism is used to refine the feature map.

4. The method for fast video portrait segmentation according to claim 1, characterized in that, A lightweight aggregation encoder and a memory update module are used to update the historical image feature memory in real time when the decoder outputs the previous frame prediction map, based on the historical frame prediction map, the current frame feature map output by the encoder, and the current frame original map.

5. The method for fast video portrait segmentation according to claim 4, characterized in that, The method for real-time updating of the historical image feature memory is as follows: (1) The lightweight clustering encoder uses convolution and depthwise separable convolution to perform feature transformation and downsampling on the historical frame prediction map output by the portrait segmentation model according to the feature map size generated by the backbone encoder, and obtains a set of feature maps; the backbone encoder refers to the encoder in the encoder-decoder-based portrait segmentation model. (2) Cut the original image into patches and convolve it with the predicted image output by the decoder to generate a feature map. Then, after concatenating the generated feature map with the feature map generated after each downsampling of the backbone encoder in the channel dimension, perform 1×1 convolution fusion to generate a key and value matrix with the semantic information of the human image in the current frame, which is called the current frame image feature memory. The key matrix is ​​used to match the semantic association of the human image, and the value matrix stores the complete image features. For the first frame, the key and value of the current frame are directly used as the image feature memory. For frames from the second frame onwards, the memory update method in steps (2)-(6) is used to generate the historical image feature memory. (3) Starting from the second frame, query the key value of the current predicted frame generated by the backbone network encoder of the current frame. and memory key values ​​in historical image feature memory bank First, K S With K mem Calculate cosine similarity to obtain the similarity matrix. As shown in equation (11): Where n is the sequence length of the historical image feature memory, ||·|| represents the second norm of the vector, and i and j represent the matrix K respectively. S and K mem Indexes along the sequence dimension; (4) For K S For each row i in the array, take its sum with K. mem The maximum value in the cosine similarity is obtained. As shown in equation (12): (5) For K mem For each column j, take it and K S The maximum cosine similarity was obtained. As shown in equation (13): (6) For m S-mem Sort the vectors and select the top β-proportions with the lowest cosine similarity. S-mem ', as the update vector, is concatenated into the historical image feature memory; simultaneously, for m mem-S Sort the vectors and retain the top 1-β proportions of the vectors with the highest similarity. mem-S '; Ultimately, M S-mem 'with M mem-S 'Concatenate the matrices to obtain a key matrix whose size remains unchanged after the update.' and value matrix 6. The method for fast video portrait segmentation according to claim 5, characterized in that, A memory update trigger is also set before the lightweight aggregation encoder to determine whether the current frame image triggers an update of the historical image feature memory. The specific determination method is as follows: (1) For the input image x of two adjacent frames t-1 x t With output prediction graph y t-1 y t The differences between adjacent input frames are calculated using equations (14) and (15), respectively. and output the difference between adjacent frames In the formula, c represents any R, G, B channel of the image; i represents any pixel in the image; (2) For each pixel i in the input image, use equation (16) to calculate the degree of change P of that pixel. i : Among them, th x With th y The preset threshold for change; (3) The degree of change P of all pixels in the image i The summation yields the image change degree P, which is then compared with a preset image change threshold P. th The comparison only occurs when the input frame image x t When the degree of change P exceeds the threshold, the historical image feature memory is updated.

7. The method for fast video portrait segmentation according to claim 1, characterized in that, The human image segmentation model is trained using a composite loss function consisting of weighted cross-entropy loss, IOU loss, and boundary-aware InverseForm loss; the composite loss function is as follows: L=L CE +γ1L IOU +γ2L IF Among them, L CE For weighted cross-entropy loss, L IOU For IOU loss, L IF For InverseForm loss, γ1 and γ2 are the weights of the loss, b pred For the predicted graph boundary, b true This represents the true label boundary.

8. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images