Live background replacement real-time optimization method and system based on portrait tracking

Abstract

The present application provides a live background replacement real-time optimization method and system based on portrait tracking, relates to the technical field of image processing, and comprises the following steps: acquiring a real-time picture frame in a live video stream, constructing a feature pyramid for portrait detection to obtain an initial portrait area, inputting a double-path tracking network, and obtaining a target tracking frame; inputting an RGB image and a corresponding depth map of a region corresponding to the target tracking frame into a double-mode segmentation network to generate a foreground mask, correcting the foreground mask, and obtaining a portrait segmentation result; and fusing the portrait segmentation result with a preset background, performing time sequence smoothing processing, and obtaining a background replacement result.

Description

Technical Field

[0001] This invention relates to image processing technology, and more particularly to a method and system for real-time optimization of live background replacement based on human face tracking. Background Technology

[0002] Existing live streaming background replacement technologies mainly rely on chroma keying or single-modal semantic segmentation networks. Chroma keying requires a strict green screen environment, which is difficult to adapt to home live streaming scenarios. On the other hand, semantic segmentation methods based on single RGB images have limited segmentation accuracy under complex backgrounds, similar color regions, or lighting changes, which can easily lead to "ghosting" and affect live streaming quality and user experience.

[0003] Existing background replacement techniques typically employ fixed-density feature extraction and full-frame processing strategies, which are computationally burdensome and make it difficult to achieve high frame rate real-time processing on resource-constrained mobile devices. Meanwhile, common target tracking algorithms lack stability when dealing with fast-moving and occluded scenes, leading to the easy loss of tracked targets and resulting in discontinuous background replacement effects.

[0004] Existing technologies often overlook the issue of lighting consistency between the foreground and the new background during background replacement, resulting in a strong visual disharmony due to direct superposition. In addition, the independent processing between frames and the lack of an effective mechanism for utilizing temporal information cause the replacement effect to exhibit jitter and instability in the video sequence, especially in the edge areas, making it difficult to meet the requirements of high-quality live streaming for background replacement effects. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a real-time optimization method and system for live streaming background replacement based on human face tracking, which can solve the problems in existing technologies.

[0006] This invention provides a real-time optimization method for live streaming background replacement based on human face tracking, including: Real-time frames from a live video stream are acquired; sparse optical flow estimation is performed on the real-time frames to obtain a region motion intensity map; the sampling density distribution is determined based on the region motion intensity map; a feature pyramid is constructed to perform human face detection to obtain an initial human face region; the initial human face region is input into a dual-path tracking network, where the main path extracts target features based on an attention mechanism, and the auxiliary path predicts motion information based on historical frame features, and the two paths are fused to obtain a target tracking box. A monocular depth estimation network is used to generate a corresponding depth map from an RGB image. The RGB image and the corresponding depth map of the region corresponding to the target tracking box are input into a bimodal segmentation network to generate a foreground mask. The consistency value of the edge features of the RGB image and the edge features of the depth map is calculated, and the foreground mask is corrected to obtain the portrait segmentation result. Based on the portrait segmentation result, the illumination parameter vector of the corresponding region is extracted from the preset background image, and illumination mapping is performed on the foreground portrait region; the illuminated foreground portrait is fused with the preset background, and temporal smoothing processing is performed to obtain the background replacement result.

[0007] Optionally, the initial human image region is obtained by determining the sampling density distribution based on the regional motion intensity map and constructing a feature pyramid for human image detection, including: Based on the motion intensity map of the region, an adaptive sampling probability density function is constructed. The probability density function includes a motion intensity term, a uniform distribution term, and a Gaussian prior term. The weight coefficients of each term are adaptively adjusted according to the degree of motion in the scene. A feature pyramid is constructed based on the number of sampling points with exponential decay, and the position of feature points is obtained by importance sampling based on the probability density function at each level. Multimodal feature vectors are extracted and normalized at the feature point locations. The detection confidence of each level is calculated. Based on the scale consistency and confidence distribution of the detection boxes at each level, inter-layer fusion weights are constructed. The detection boxes with different weights are weighted and fused to obtain the initial human image region.

[0008] Optionally, inputting the initial human image region into a dual-path tracking network to obtain a target tracking bounding box includes: A convolution operation is performed on the initial human image region to obtain a feature map. The feature map is divided into a fixed grid. An average pooling operation is performed on the feature map in each grid to obtain a pooled feature. The pooled feature is then multiplied by the feature map to obtain a grid attention feature. Weight coefficients are assigned to the grid attention features of different grids according to the importance of the human body region. The weighted summation is then used to obtain the main path target feature. Fast optical flow estimation is performed on the historical frame sequence of the initial portrait region to obtain the displacement field. Based on the amplitude of the displacement field, the feature update coefficient is calculated. The current frame feature and the previous frame feature are subjected to an exponential moving average through the feature update coefficient to obtain the auxiliary path motion feature. The magnitude of the displacement field is calculated and normalized to obtain the motion confidence. The main path target feature is convolved with the motion confidence to obtain the feature fusion weight. The main path target feature and the auxiliary path motion feature are linearly fused according to the feature fusion weight to obtain the final target feature. The position parameters and scale parameters of the target tracking box are obtained by regression based on the final target feature.

[0009] Optionally, generating a corresponding depth map from the RGB image using a monocular depth estimation network, and inputting the RGB image and corresponding depth map of the region corresponding to the target tracking box into a bimodal segmentation network to generate a foreground mask includes: The monocular depth estimation network calculates depth attention coefficients using a query matrix and a key matrix, and generates the depth map by weighted combination of the depth feature map and the depth attention coefficients. The RGB image and depth map of the region corresponding to the target tracking box are input into a bimodal segmentation network to obtain RGB features and depth features. The RGB features and depth features are then aligned across modally using a complementary attention mechanism. The complementary attention mechanism uses the RGB features as a query vector and the depth features as key-value pairs to calculate cross-modal attention weights. The RGB features and depth features are then weighted and combined based on the cross-modal attention weights to obtain the aligned features. Extract edge features from the RGB image and depth map, calculate the edge difference between the RGB edge features and the depth edge features, and obtain the edge fusion weight based on the edge difference; The alignment features are input into a multi-level decoder to generate an RGB mask and a depth mask. The RGB mask and the depth mask are then weighted and fused according to the edge fusion weights to obtain a foreground mask.

[0010] Optionally, the consistency value of the edge features of the RGB image and the edge features of the depth map is calculated to correct the foreground mask, resulting in a portrait segmentation result including: Within a sliding window, the local consistency value of the edge features of the RGB image and the edge features of the depth map is calculated, and the structural similarity between the edge features of the RGB image and the edge features of the depth map is calculated to obtain the global structural consistency value. The local consistency value is multiplied by the first weight and then added to the product of the global structural consistency value and the second weight to obtain the edge consistency value. The optimal threshold is calculated based on the histogram of the edge consistency values, and the pixels are divided into high consistency regions and low consistency regions. In the high consistency region, the edge features of the depth map are used to correct the foreground mask to obtain a first corrected mask, and in the low consistency region, the edge features of the RGB image are used to correct the foreground mask to obtain a second corrected mask. The first and second modified masks are subjected to total variational regularization to obtain the optimized mask. When the iteration difference between the optimized masks obtained in two adjacent iterations is greater than or equal to a preset difference threshold, the next round of iteration optimization is performed. When the iteration difference is less than the preset difference threshold, the final portrait segmentation result is output.

[0011] Optionally, calculating the optimal threshold based on the histogram of the edge consistency values ​​includes: Construct a histogram of the edge consistency values, and generate multiple candidate thresholds based on the histogram; Calculate the structural similarity between the region partitioning results corresponding to each candidate threshold, and determine the candidate threshold with the maximum structural similarity as the optimal threshold for the current frame; Calculate the region overlap between the region segmentation result of the previous frame and the region segmentation result of the current frame, calculate the temporal weight based on the region overlap, multiply the temporal weight by the optimal threshold of the current frame to obtain a first product, multiply the complement of the temporal weight by the optimal threshold of the previous frame to obtain a second product, and add the first product and the second product to obtain the target threshold. Within a preset range of the region boundary corresponding to the target threshold, a smoothing weight is calculated based on the image gradient magnitude. The smoothing weight is multiplied by the target threshold to obtain a third product. The complement of the smoothing weight is multiplied by the local threshold of the region boundary to obtain a fourth product. The third product and the fourth product are added together to obtain the optimal threshold.

[0012] Optionally, extracting the illumination parameter vector of the corresponding region from the preset background image based on the portrait segmentation result, and performing illumination mapping on the foreground portrait region includes: The orientation features of a preset background image are calculated using a structure tensor. Anisotropy coefficients are calculated based on the orientation features. The filtering radius and regularization parameters of the guided filter are adaptively adjusted based on the anisotropy coefficients. An adaptive guided filter is then applied to the preset background image to obtain the illumination component. Construct a Gaussian pyramid of the preset background image, calculate the semantic segmentation score, gradient magnitude and brightness contrast of each layer of the Gaussian pyramid, and combine them in a weighted manner to obtain the region saliency weight; Based on the regional saliency weight and the illumination component, multiple illumination sampling points are selected from the preset background image. The illumination intensity, principal direction and color temperature of each illumination sampling point are calculated to obtain an illumination parameter vector. A local illumination model is constructed based on the illumination parameter vector. The diffuse reflection coefficient is obtained by calculating the ratio of the darkest area brightness value to the ambient light intensity of the foreground portrait area. The specular reflection coefficient is obtained by calculating the difference between the maximum brightness value and the median brightness value of the foreground portrait area. The glossiness parameter is calculated based on the gradient histogram characteristics of the foreground portrait area. The local lighting model is corrected using the diffuse reflection coefficient, specular reflection coefficient, and gloss parameters, and then the corrected local lighting model is applied to the foreground portrait area for lighting mapping.

[0013] Secondly, it provides a real-time optimization system for live streaming background replacement based on human face tracking, including: The first unit is used to acquire real-time frames from the live video stream; perform sparse optical flow estimation on the real-time frames to obtain a region motion intensity map; determine the sampling density distribution based on the region motion intensity map; construct a feature pyramid to perform human face detection to obtain an initial human face region; input the initial human face region into a dual-path tracking network, wherein the main path extracts target features based on an attention mechanism, and the auxiliary path predicts motion information based on historical frame features; and fuse them to obtain a target tracking box. The second unit is used to generate a corresponding depth map from an RGB image using a monocular depth estimation network. The RGB image and the corresponding depth map of the region corresponding to the target tracking box are input into a bimodal segmentation network to generate a foreground mask. The consistency value of the edge features of the RGB image and the edge features of the depth map is calculated, and the foreground mask is corrected to obtain the portrait segmentation result. The third unit is used to extract the illumination parameter vector of the corresponding region from the preset background image based on the portrait segmentation result, perform illumination mapping on the foreground portrait region, merge the illumination-mapped foreground portrait with the preset background, and perform temporal smoothing processing to obtain the background replacement result.

[0014] Thirdly, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0015] This invention achieves efficient detection of human faces in live videos through sparse optical flow estimation and feature pyramid construction, effectively reducing computational resource consumption and improving real-time processing capabilities, making it particularly suitable for mobile live streaming applications.

[0016] This invention innovatively adopts a dual-path tracking network structure, combining the attention mechanism with historical frame feature prediction, which significantly improves the stability of portrait tracking in fast motion and pose change scenarios, reduces the background replacement flickering problem caused by tracking failure, and enhances the continuity and immersion of the user experience.

[0017] This invention introduces depth information as an auxiliary modality, combines an RGB-D dual-modal segmentation network and edge feature consistency calculation, and achieves more accurate portrait edge segmentation under complex lighting and similar background conditions. At the same time, it extracts lighting parameter vectors based on the segmentation results to perform foreground lighting mapping and temporal smoothing, making the background replacement effect more natural and realistic, and solving the problems of lighting mismatch and edge artifacts between the portrait and the new background in traditional methods. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the real-time optimization method for live background replacement based on human face tracking, according to an embodiment of the present invention. Figure 2This is a comparison chart of lighting mapping effects in different scenarios. Detailed Implementation

[0019] The technical solutions of the present invention will be described below with reference to the accompanying drawings. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0020] Figure 1 This is a flowchart illustrating the real-time optimization method for live background replacement based on human face tracking according to the present invention. Figure 1 As shown, the method includes: Real-time frames from a live video stream are acquired; sparse optical flow estimation is performed on the real-time frames to obtain a region motion intensity map; the sampling density distribution is determined based on the region motion intensity map; a feature pyramid is constructed to perform human face detection to obtain an initial human face region; the initial human face region is input into a dual-path tracking network, where the main path extracts target features based on an attention mechanism, and the auxiliary path predicts motion information based on historical frame features, and the two paths are fused to obtain a target tracking box. A monocular depth estimation network is used to generate a corresponding depth map from an RGB image. The RGB image and the corresponding depth map of the region corresponding to the target tracking box are input into a bimodal segmentation network to generate a foreground mask. The consistency value of the edge features of the RGB image and the edge features of the depth map is calculated, and the foreground mask is corrected to obtain the portrait segmentation result. Based on the portrait segmentation result, the illumination parameter vector of the corresponding region is extracted from the preset background image, and illumination mapping is performed on the foreground portrait region; the illuminated foreground portrait is fused with the preset background, and temporal smoothing processing is performed to obtain the background replacement result.

[0021] Optionally, Based on the regional motion intensity map, the sampling density distribution is determined, and a feature pyramid is constructed for human face detection to obtain the initial human face region, which includes: Based on the motion intensity map of the region, an adaptive sampling probability density function is constructed. The probability density function includes a motion intensity term, a uniform distribution term, and a Gaussian prior term. The weight coefficients of each term are adaptively adjusted according to the degree of motion in the scene. A feature pyramid is constructed based on the number of sampling points with exponential decay, and the position of feature points is obtained by importance sampling based on the probability density function at each level. Multimodal feature vectors are extracted and normalized at the feature point locations. The detection confidence of each level is calculated. Based on the scale consistency and confidence distribution of the detection boxes at each level, inter-layer fusion weights are constructed. The detection boxes with different weights are weighted and fused to obtain the initial human image region.

[0022] For example, sparse optical flow estimation is performed on the acquired real-time frame to obtain a regional motion intensity map. The displacement vectors of feature points are calculated between the current frame and the previous frame based on the Lucas-Kanade algorithm. A 16×16 grid is uniformly distributed in the image, and 5-10 corner points are randomly selected as feature points within each grid. The displacement magnitudes of these feature points between adjacent frames are calculated. The displacement magnitudes of all feature points are mapped to the range of 0-1 to generate the regional motion intensity map. For example, if the average displacement magnitude detected in the right region of the image is 5 pixels, and in the middle region it is 2 pixels, then the motion intensity value of the right region is approximately 0.8, and that of the middle region is approximately 0.4.

[0023] An adaptive sampling probability density function is constructed based on the regional motion intensity map. This function consists of three terms: a motion intensity term, a uniform distribution term, and a Gaussian prior term. The motion intensity term represents the probability that a sampling point appears in a high-motion region, and its value is the motion intensity value at that location. The uniform distribution term ensures that the entire image has a certain probability of being sampled, and its value is 1 / the number of pixels in the image. The Gaussian prior term represents the probability that a sampling point appears in the center region of the image, with a value of 0.8 at the center and decreasing towards the edges. The weights of the three terms are adaptively adjusted according to the level of scene motion. Specifically, when the average motion intensity of the entire image exceeds 0.6, the weight of the motion intensity term is set to 0.7, the uniform distribution term to 0.2, and the Gaussian prior term to 0.1; when the average motion intensity of the entire image is below 0.3, the weights of the three terms are set to 0.3, 0.3, and 0.4, respectively.

[0024] A four-layer feature pyramid is constructed, with each layer having a resolution corresponding to the original resolution, half the original resolution, one-quarter of the original resolution, and one-eighth of the original resolution, respectively. The number of sampling points is also set exponentially: if the first layer has 1000 sampling points, the second layer has 500, the third layer has 250, and the fourth layer has 125. At each layer, importance sampling is performed based on the aforementioned probability density function to obtain feature point locations. An accept-reject sampling method is used to generate random point coordinates in the image space, and then the probability density value at that location determines whether to accept the sampling point. For example, if the probability density value of a point is 0.7, then that point is accepted as a feature point with a 70% probability.

[0025] Multimodal feature vectors are extracted at the obtained feature point locations. For each feature point, the following features are extracted: color features: extract the RGB color histogram of the 7×7 pixel region surrounding the feature point, with a total of 24 dimensions; texture features: calculate the HOG features of the region, with a total of 32 dimensions; local shape features: extract the SURF descriptor of the region, with a total of 64 dimensions. These features are concatenated into a 120-dimensional feature vector, and then normalized using the L2 norm to make the magnitude of the feature vector equal to 1.

[0026] The detection confidence score is calculated for each level. A pre-trained portrait detection model is used to classify the feature vectors, obtaining the probability value of each feature point belonging to the portrait region. This model employs a lightweight MobileNet architecture, taking a 120-dimensional feature vector as input and outputting a binary classification result. Based on the classification result, multiple candidate detection boxes are generated at each level. Each detection box contains location coordinates, width, height, and confidence information. For example, a certain level might generate 5 candidate boxes with confidence scores of 0.92, 0.85, 0.77, 0.65, and 0.58, respectively.

[0027] Scale consistency refers to whether the size ratio of detection boxes across different layers conforms to expectations. It calculates the difference between the theoretical size and the actual detected size of each detection box after mapping from the previous layer to the current layer. The smaller the difference, the higher the consistency and the greater the weight. For example, if a detection box in the second layer has a width and height of 100×200 pixels, its theoretical size mapped to the third layer should be 50×100 pixels, but the actual detected size is 48×105 pixels. The difference is approximately 0.05, and the corresponding consistency weight is 0.95. The confidence distribution weight directly uses the confidence value of the detection box.

[0028] All detection boxes at all levels are mapped back to the original resolution, and then a comprehensive weight is calculated based on scale consistency weight and confidence weight. The comprehensive weight calculation formula is: Comprehensive Weight = 0.6 × Confidence Weight + 0.4 × Consistency Weight. Based on the comprehensive weight, the coordinates and size parameters of the detection boxes are fused using a weighted average method. For example, if there are three detection boxes with coordinates (100, 200), (105, 195), and (110, 205) respectively, and comprehensive weights of 0.9, 0.8, and 0.7 respectively, then the fused coordinates will be (104, 199). In this way, the final initial human image region is obtained, which is defined by the coordinates of the top-left corner, width, and height.

[0029] This invention employs an adaptive sampling strategy guided by regional motion intensity maps, which significantly improves the targeting and efficiency of feature extraction; the multi-level feature pyramid structure enhances the adaptability to portraits of different scales; the weighted fusion mechanism based on scale consistency and confidence improves the stability and accuracy of detection results; the method can maintain stable real-time performance on resource-constrained mobile devices, while maintaining high portrait detection accuracy under complex backgrounds and lighting conditions, which greatly improves the quality of subsequent background replacement and user experience.

[0030] Optionally, The initial human image region is input into a dual-path tracking network to obtain a target tracking bounding box, which includes: A convolution operation is performed on the initial human image region to obtain a feature map. The feature map is divided into a fixed grid. An average pooling operation is performed on the feature map in each grid to obtain a pooled feature. The pooled feature is then multiplied by the feature map to obtain a grid attention feature. Weight coefficients are assigned to the grid attention features of different grids according to the importance of the human body region. The weighted summation is then used to obtain the main path target feature. Fast optical flow estimation is performed on the historical frame sequence of the initial portrait region to obtain the displacement field. Based on the amplitude of the displacement field, the feature update coefficient is calculated. The current frame feature and the previous frame feature are subjected to an exponential moving average through the feature update coefficient to obtain the auxiliary path motion feature. The magnitude of the displacement field is calculated and normalized to obtain the motion confidence. The main path target feature is convolved with the motion confidence to obtain the feature fusion weight. The main path target feature and the auxiliary path motion feature are linearly fused according to the feature fusion weight to obtain the final target feature. The position parameters and scale parameters of the target tracking box are obtained by regression based on the final target feature.

[0031] For example, after obtaining the initial human image region, it is input into a dual-path tracking network for accurate tracking. This network consists of a main path and an auxiliary path. The main path extracts target features based on an attention mechanism, while the auxiliary path predicts motion information based on historical frame features. The accurate target tracking bounding box is obtained by fusing the outputs of the two paths. The implementation process of the dual-path tracking network is described in detail below: The initial portrait region is input into the main path. The initial portrait region is cropped from the original image and resized to a standard size of 224×224 pixels. Then, MobileNetV2 is used as the backbone network to perform convolution operations on the resized region, outputting a feature map with dimensions of 14×14×512 (14 pixels wide, 14 pixels high, and 512 channels). This feature map is then used as input for subsequent processing.

[0032] The feature map is divided into a grid and average pooling is performed. A 14×14 feature map is evenly divided into a 7×7 grid, with each grid containing a 2×2 pixel region. Average pooling is performed within each grid, averaging the values ​​of the 2×2 regions to obtain a pooled feature of size 7×7×512. For example, if a channel value at four locations in a certain grid of the feature map is [0.8, 0.6, 0.7, 0.5], then the corresponding pooled feature value for that grid is 0.65.

[0033] The 7×7×512 pooling features are adjusted to a 7×7×1 attention weight map by adjusting the channel dimension, and then a dot product operation is performed with the original feature map to obtain the grid attention features. Specifically, first, a 1×1 convolution is applied to the pooling features to reduce the number of channels from 512 to 1, resulting in a 7×7 attention weight map. Then, the attention weight map is upsampled to 14×14 and multiplied element-wise with the original feature map at each position to obtain a grid attention feature of size 14×14×512. For example, if the value of the upsampled attention weight map at position (3,4) is 0.8, and the value of a certain channel in the original feature map at that position is 0.6, then the value of the grid attention feature at that position and channel is 0.48.

[0034] Different weight coefficients are assigned to different grid areas based on their importance. A weight template is constructed based on prior knowledge of the human body, with higher weights for the head and upper body, followed by the limbs, and the background area has the lowest weight. Specifically, based on the initial position of the human image area, the approximate positions of each part of the body within the grid are estimated, and then weights are assigned. The head area grid weight is set to 0.4, the upper body grid weight to 0.3, the limbs grid weight to 0.2, and the background grid weight to 0.1.

[0035] Weighted Summarization of Grid Attention Features. Based on the weight coefficients assigned in the previous step, the grid attention features are weighted and summed to obtain the main path target features. Specifically, the attention features of the 14×14 grids are weighted and summed according to their respective weight coefficients, and then the features are reduced to a 512-dimensional vector through a global average pooling operation, which is used as the target feature output of the main path. For example, if the attention feature values ​​of four adjacent grids on a certain channel are [0.7, 0.6, 0.5, 0.4], and the corresponding weight coefficients are [0.4, 0.3, 0.2, 0.1], then the weighted summarization value of this channel in this region is 0.7×0.4 + 0.6×0.3 + 0.5×0.2 + 0.4×0.1 = 0.61.

[0036] The historical frame sequence of the auxiliary path is processed in parallel, and the portrait regions of the most recent 5 frames are saved as the historical frame sequence. The resolution of each frame is also adjusted to 224×224 pixels. For the current frame and the previous frame, the Farneback optical flow algorithm is used for fast optical flow estimation to calculate the pixel-level displacement field. The displacement field contains displacement values ​​in both horizontal and vertical directions, with a size of 224×224×2. For example, if a pixel moves from position (100, 150) in the previous frame to position (105, 152) in the current frame, then the displacement field value of that point is (5, 2). The magnitude of the displacement field is calculated, which is the square root of the sum of the squares of the horizontal and vertical displacements of each pixel, resulting in a displacement magnitude map with a size of 224×224. Then, the displacement amplitude is normalized to the range of 0-1, and thresholds are set for adjustment: for regions with displacement amplitude less than 2 pixels, the feature update coefficient is set to 0.9; for regions with displacement amplitude between 2 and 5 pixels, the feature update coefficient is 0.7; for regions with displacement amplitude between 5 and 10 pixels, the feature update coefficient is 0.5; and for regions with displacement amplitude greater than 10 pixels, the feature update coefficient is 0.3. An exponential moving average is performed to calculate the secondary path motion features. The same MobileNetV2 backbone network is used to extract features from the current frame and the previous frame, resulting in two 14×14×512 feature maps. Then, based on the feature update coefficient, the features are updated using the exponential moving average method: Current secondary path motion feature = Feature update coefficient × Previous frame feature + (1 - Feature update coefficient) × Current frame feature. For example, if the feature update coefficient for a certain region is 0.7, the feature value for the previous frame is 0.8, and the feature value for the current frame is 0.6, then the secondary path motion feature value for that region is 0.7 × 0.8 + (1 - 0.7) × 0.6 = 0.74.

[0037] First, the magnitude of the displacement field at each pixel is calculated (the square root of the sum of the squares of the horizontal and vertical displacements). Then, the sigmoid function is used to map the magnitude to the range of 0-1, resulting in a motion confidence map of size 14×14. The larger the motion amplitude, the higher the confidence. For example, a region with a displacement magnitude of 3 pixels has a motion confidence of approximately 0.6; a region with a displacement magnitude of 8 pixels has a motion confidence of approximately 0.9.

[0038] The main path target features are convolved with the motion confidence map to obtain feature fusion weights. Specifically, a 1×1 convolution is used to convert the motion confidence map into fusion weight factors, with values ​​ranging from 0 to 1. Then, linear fusion is performed based on the fusion weights: Final target feature = Fusion weight × Main path target feature + (1 - Fusion weight) × Auxiliary path motion feature. For example, if the fusion weight of a certain channel is 0.65, the main path feature value is 0.8, and the auxiliary path feature value is 0.6, then the final feature value is 0.65 × 0.8 + (1 - 0.65) × 0.6 = 0.73.

[0039] Based on the final target features, a fully connected layer is used to regress the target tracking box to obtain its position and scale parameters. The position parameters include the relative offsets of the center point's x and y coordinates, and the scale parameters include scaling factors for width and height. Using the tracking box from the previous frame as a reference, the tracking box for the current frame is calculated based on the regressed parameters. For example, if the tracking box position in the previous frame is (300, 400), and its width and height are (100, 200), and the regressed offset is (0.05, -0.02), with a scaling factor of (1.03, 0.98), then the tracking box position in the current frame is (300 + 0.05 × 100, 400 - 0.02 × 200) = (305, 396), and its width and height are (100 × 1.03, 200 × 0.98) = (103, 196).

[0040] The main path of this invention effectively captures the feature representation of various parts of the human body through a grid attention mechanism, enhancing the ability to identify local features; the auxiliary path predicts the human body's motion trend through historical frame sequence analysis, improving adaptability to fast-moving and occluded scenes; the feature fusion strategy of the two paths adaptively adjusts the weights according to the motion amplitude, effectively combining static appearance features and dynamic motion information, and significantly improving the stability and accuracy of tracking.

[0041] Optionally, Generating a corresponding depth map from an RGB image using a monocular depth estimation network, and inputting the RGB image and corresponding depth map of the region corresponding to the target tracking box into a bimodal segmentation network to generate a foreground mask includes: The monocular depth estimation network calculates depth attention coefficients using a query matrix and a key matrix, and generates the depth map by weighted combination of the depth feature map and the depth attention coefficients. The RGB image and depth map of the region corresponding to the target tracking box are input into a bimodal segmentation network to obtain RGB features and depth features. The RGB features and depth features are then aligned across modally using a complementary attention mechanism. The complementary attention mechanism uses the RGB features as a query vector and the depth features as key-value pairs to calculate cross-modal attention weights. The RGB features and depth features are then weighted and combined based on the cross-modal attention weights to obtain the aligned features. Extract edge features from the RGB image and depth map, calculate the edge difference between the RGB edge features and the depth edge features, and obtain the edge fusion weight based on the edge difference; The alignment features are input into a multi-level decoder to generate an RGB mask and a depth mask. The RGB mask and the depth mask are then weighted and fused according to the edge fusion weights to obtain a foreground mask.

[0042] For example, a lightweight ResNet18 is used as the backbone network, with an input of a 256×256 pixel RGB image. Multi-level features are extracted through five encoding layers. The feature map size extracted in layer 1 is 128×128×64, layer 2 is 64×64×128, layer 3 is 32×32×256, layer 4 is 16×16×512, and layer 5 is 8×8×1024. A deep feature pyramid is constructed based on these feature maps for subsequent depth estimation.

[0043] The extracted multi-layer features are processed by a convolutional neural network to generate a depth feature map. Specifically, a 1×1 convolution is applied to the 5th layer feature map to reduce the number of channels to 256, resulting in an 8×8×256 feature map; then, a transposed convolution is used for upsampling, expanding its size to 16×16×128; after fusing with the 4th layer feature map, it is further upsampled to 32×32×64; and so on, upsampling is performed layer by layer and fused with the corresponding layer features, finally generating a 256×256×32 depth feature map. Each pixel in the depth feature map contains a 32-dimensional feature vector, which encodes the depth information of that pixel. For example, the depth feature at position (100, 100) might be [0.3, 0.5, 0.1, ..., 0.4], and these values ​​collectively represent the depth characteristics at that position.

[0044] A deep attention mechanism is implemented to calculate deep attention coefficients. For the feature map of layer 5 (8×8×1024), three 1×1 convolutions are first used to generate a query matrix, a key matrix, and a value matrix, each with a size of 8×8×256. The query matrix and key matrix are used to calculate the correlation between feature points, and the value matrix is ​​used to generate the weighted result. The query matrix is ​​reshaped to a 64×256 shape, and the key matrix is ​​transposed to a 256×64 shape. Then, matrix multiplication is performed to obtain a 64×64 attention score matrix. This matrix is ​​normalized using the Softmax function to obtain the deep attention coefficients. For example, if the attention score between points A and B is 0.8, and the score between points A and C is 0.2, then after normalization, the attention coefficient of point A to point B is 0.8, and the attention coefficient to point C is 0.2.

[0045] The previously generated depth feature map (256×256×32) is resized into a 65536×32 matrix (256×256=65536) and multiplied by a block matrix with the depth attention coefficient matrix (64×64). During this process, each 32×32 pixel block of the depth feature map is treated as a processing unit, corresponding to an element of the attention coefficient matrix, and weighted. The weighted feature map is then passed through a 1×1 convolutional layer to reduce the number of channels from 32 to 1, and then the Sigmoid activation function is applied to output a 256×256×1 depth map with pixel values ​​ranging from 0 to 1, where 0 represents the farthest distance and 1 represents the nearest distance. For example, if the average depth feature value within a 32×32 block is [0.4, 0.6, 0.3, ..., 0.5], and the corresponding attention coefficient is 0.8, then the weighted feature value is approximately [0.32, 0.48, 0.24, ..., 0.4]. After convolution and Sigmoid, a depth value of 0.75 may be obtained.

[0046] Based on the target tracking bounding box position, the corresponding region is cropped from the original RGB image and the generated depth map. To retain sufficient contextual information, the tracking bounding box is expanded outwards by 20%, and then the cropped area is adjusted to a standard size of 256×256 pixels. For example, if the tracking bounding box coordinates are (300, 400) and the width and height are (150, 200), the expanded cropped area will be from (270, 360) to (480, 600), and then adjusted to 256×256 pixels.

[0047] RGB images and depth maps are input into a bimodal segmentation network to extract features. The network employs a dual-branch structure, with the RGB and depth branches using the same lightweight MobileNetV3 backbone. The RGB image is input into the RGB branch, and the depth map into the depth branch, where features are extracted separately. Each branch contains five encoding stages, generating feature maps from shallow to deep layers, with the deepest feature map having a size of 8×8×256. For example, the third layer of the RGB branch has a feature map size of 32×32×96, as does the third layer of the depth branch.

[0048] A complementary attention mechanism is implemented to align RGB and depth features across modalities. For each feature map layer, RGB features are used as query vectors, and depth features are used as key-value pairs. Taking layer 3 (32×32×96) as an example, a 1×1 convolution is first used to reduce the number of channels in both RGB and depth features to 48, resulting in a 32×32×48 feature map. Then, the RGB features are reshaped into a 1024×48 (32×32=1024) query matrix, and the depth features are reshaped into 1024×48 key and value matrices. The query matrix is ​​multiplied by the transpose of the key matrix to obtain a 1024×1024 attention score matrix, which is then normalized using the Softmax function to obtain the cross-modal attention weights. For example, if the correlation score between position A in the RGB feature and position B in the depth feature is 0.7, and the score with position C is 0.3, then after normalization, the attention weight of A to B is 0.7, and the weight to C is 0.3.

[0049] Cross-modal attention weights are used to weight and combine RGB and depth features. The attention weight matrix (1024×1024) is multiplied by the value matrix (1024×48) to obtain a 1024×48 weighted result, which is then reshaped into a 32×32×48 feature map. This feature map is then combined with the original RGB features via channel concatenation to obtain a 32×32×96 alignment feature. This process is performed independently on each layer of features, generating alignment features at multiple scales. For example, the second layer generates a 64×64×64 alignment feature, and the fourth layer generates a 16×16×128 alignment feature.

[0050] Edge detection is performed on both the RGB image and the depth map using the Sobel operator, resulting in two 256×256 edge intensity maps. RGB edge features are obtained by applying the Sobel operator to each of the three channels and then taking the maximum value. For example, if the Sobel gradient values ​​for the three RGB channels at position (100, 100) are 0.5, 0.7, and 0.3 respectively, then the RGB edge intensity at that position is 0.7. Depth edge features are obtained by directly applying the Sobel operator to the depth map. The edge intensity values ​​are normalized to the range of 0-1; a larger value indicates a more pronounced edge.

[0051] Calculate the edge difference between RGB edge features and depth edge features. For each pixel location, calculate the absolute difference between the RGB edge intensity and the depth edge intensity, resulting in a 256×256 edge difference map. For example, if the RGB edge intensity at a certain location is 0.8 and the depth edge intensity is 0.3, then the edge difference at that location is 0.5. The larger the edge difference, the greater the difference in edge information between the two modalities at that location. Calculate the edge fusion weight based on the edge difference. Use the Sigmoid function to map the edge difference to the edge fusion weight: Fusion weight = 1 / (1+exp(-5×(edge ​​difference-0.3))). When the edge difference is less than 0.3, the fusion weight is close to 0; when the edge difference is greater than 0.3, the fusion weight increases rapidly and approaches 1. For example, when the edge difference is 0.1, the fusion weight is approximately 0.1, and when the difference is 0.5, the weight is approximately 0.9. Thus, in regions where the edges of the two modalities are consistent, the fusion weight is lower; in regions where the edges are inconsistent, the fusion weight is higher, favoring the use of depth information.

[0052] Alignment features are input into a multi-level decoder to generate segmentation masks. The decoder uses a UNet architecture, progressively restoring the feature map from 8×8 to 256×256 resolution through four upsampling passes. After each upsampling, the feature map is fused with the corresponding scale alignment features via skip connections. Finally, two independent 1×1 convolutional layers generate RGB and depth masks, each 256×256×1 pixels, with pixel values ​​ranging from 0 to 1, where 1 represents foreground and 0 represents background. For example, if the RGB mask value for a human portrait region is 0.95 and the depth mask value is 0.90, then that region is likely foreground.

[0053] For each pixel location, linear interpolation is performed using edge blending weights: Foreground mask = (1 - edge blending weight) × RGB mask + edge blending weight × depth mask. For example, if the edge blending weight at a certain location is 0.7, the RGB mask value is 0.8, and the depth mask value is 0.9, then the foreground mask value at that location is 0.3 × 0.8 + 0.7 × 0.9 = 0.87. Thus, in edge regions, segmentation relies more heavily on depth information; in other regions, both RGB and depth information are utilized.

[0054] This invention utilizes a monocular depth estimation network to generate a depth map, providing additional spatial information for segmentation; a bimodal segmentation network effectively fuses RGB and depth features through a complementary attention mechanism, enhancing the ability to recognize human image edges in complex backgrounds; and a weighted fusion strategy driven by edge feature differences solves the uncertainty problem in edge regions in traditional monomodal segmentation, significantly improving segmentation accuracy and edge detail preservation.

[0055] Optionally, Calculate the consistency value between the edge features of the RGB image and the edge features of the depth map, correct the foreground mask, and obtain the portrait segmentation result, including: Within a sliding window, the local consistency value of the edge features of the RGB image and the edge features of the depth map is calculated, and the structural similarity between the edge features of the RGB image and the edge features of the depth map is calculated to obtain the global structural consistency value. The local consistency value is multiplied by the first weight and then added to the product of the global structural consistency value and the second weight to obtain the edge consistency value. The optimal threshold is calculated based on the histogram of the edge consistency values, and the pixels are divided into high consistency regions and low consistency regions. In the high consistency region, the edge features of the depth map are used to correct the foreground mask to obtain a first corrected mask, and in the low consistency region, the edge features of the RGB image are used to correct the foreground mask to obtain a second corrected mask. The first and second modified masks are subjected to total variational regularization to obtain the optimized mask. When the iteration difference between the optimized masks obtained in two adjacent iterations is greater than or equal to a preset difference threshold, the next round of iteration optimization is performed. When the iteration difference is less than the preset difference threshold, the final portrait segmentation result is output.

[0056] For example, the local consistency value of the edge features of the RGB image and the edge features of the depth map is calculated within a sliding window. A 7×7 pixel sliding window is used, and the normalized cross-correlation coefficient of the RGB edge features and the depth edge features is calculated within the window. Specifically, the RGB edge features and the depth edge features are treated as 7×7 local vectors, and their dot product is calculated and divided by the product of their respective norms. For example, if the mean of the RGB edge features within a certain sliding window is 0.6 and the standard deviation is 0.2; the mean of the depth edge features is 0.5 and the standard deviation is 0.3; and their covariance is 0.05, then the local consistency value of this window is 0.05 / (0.2×0.3)≈0.83. The local consistency value ranges from -1 to 1, with values ​​closer to 1 indicating greater consistency between the two edge features in the local region. Performing the sliding window operation on the entire 256×256 image yields a 256×256 local consistency map.

[0057] We also need to calculate the structural similarity between the edge features of the RGB image and the edge features of the depth image to obtain a global structural consistency value. First, we calculate the global mean and standard deviation of the two edge features. For example, the global mean of the RGB edge features is 0.4, and the standard deviation is 0.25; the global mean of the depth edge features is 0.35, and the standard deviation is 0.3. Then, we calculate the global covariance of the two, assuming it is 0.06. Based on these statistics, we calculate the structural similarity index: first, we calculate the brightness similarity term (2×0.4×0.35) / (0.4²+0.35²)≈0.93; then, we calculate the contrast similarity term (2×0.25×0.3) / (0.25²+0.3²)≈0.91; finally, we calculate the structural similarity term 0.06 / (0.25×0.3)≈0.8. Multiplying these three terms together gives the global structural consistency value 0.93×0.91×0.8≈0.68. The global structural consistency value ranges from 0 to 1. The closer the value is to 1, the more similar the two edge features are in the global structure.

[0058] The first weight is set to 0.7, and the second weight to 0.3. For example, if the local consistency value of a pixel is 0.8 and the global structural consistency value is 0.6, then the edge consistency value at that location is 0.7 × 0.8 + 0.3 × 0.6 = 0.74. This calculation is performed on all pixels to obtain a 256 × 256 edge consistency map, where pixel values ​​range from -1 to 1, with larger values ​​indicating more consistent edge features.

[0059] The optimal threshold is calculated based on the histogram and standard deviation of the edge consistency value. Pixels with an edge consistency value greater than or equal to a set value (such as 0.35) are classified as high consistency regions, and pixels with a consistency value less than 0.35 are classified as low consistency regions.

[0060] In high-consistency regions, edge features from the depth map are used to correct the foreground mask, resulting in a first corrected mask. Specifically, morphological operations are used to process the foreground mask. First, a 3×3 erosion operation is applied to shrink the foreground region; then, conditional dilation is applied at locations where the depth edge feature is greater than 0.5 to restore the edges. For example, in a high-consistency region, the original foreground mask is a binary image (0 or 1), and the depth edge feature is a value between 0 and 1. After erosion, the foreground region shrinks by 3 pixels; during conditional dilation, the foreground is restored only at locations where the depth edge feature is greater than 0.5, thus aligning the foreground boundary with the depth edge.

[0061] In low-consistency regions, edge features from the RGB image are used to correct the foreground mask, resulting in a second corrected mask. Similar to the high-consistency regions, morphological operations are also used, but this time conditioned on RGB edge features. First, a 3×3 erosion operation is applied; then, conditional dilation is applied at locations where the RGB edge features are greater than 0.4. For example, in a low-consistency region, the original foreground mask shrinks by 3 pixels after erosion; during conditional dilation, the foreground is only restored at locations where the RGB edge features are greater than 0.4, aligning the foreground boundary with the RGB edges.

[0062] The goal of total variational regularization (TVLR) is to smooth non-edge regions of a mask while maintaining sharp edges. TVLR is implemented iteratively, with each iteration including the following steps: First, calculate the gradient map of the mask, one horizontally and one vertically; then calculate the magnitude of the gradients. For example, at position (100, 100), the horizontal gradient is 0.3 and the vertical gradient is 0.4, so the magnitude is 0.5; next, calculate the gradient divergence map; finally, update and optimize the mask. The new value equals the original value minus the gradient divergence multiplied by the step size (set to 0.1). For example, if the original mask value at position (100, 100) is 0.8 and the gradient divergence is 0.2, then the updated value is 0.8 - 0.1 × 0.2 = 0.78.

[0063] Determine if the iteration difference between the optimized masks obtained from two consecutive iterations is greater than or equal to a preset difference threshold. Calculate the average absolute difference between the two iterations as the iteration difference. For example, the average absolute difference between the masks in the 5th and 6th iterations is 0.008. Set the preset difference threshold to 0.005. When the iteration difference is greater than or equal to 0.005, continue to the next round of iteration optimization; when the iteration difference is less than 0.005, the optimization is considered to have converged, and the iteration stops.

[0064] Finally, the final portrait segmentation result is output. The optimized mask after total variational regularization is a grayscale image with pixel values ​​ranging from 0 to 1. A threshold of 0.5 is applied for binarization to obtain the final binary portrait segmentation result. Pixel values ​​greater than or equal to 0.5 are marked as foreground (value 1), and values ​​less than 0.5 are marked as background (value 0). For example, if the mask value of a pixel after optimization is 0.73, then this pixel is marked as foreground in the final segmentation result. Small area removal operations can be selectively applied to delete isolated foreground or background regions with an area less than 50 pixels to further improve the quality of the segmentation result.

[0065] This invention achieves high-precision portrait segmentation through edge consistency calculation and masking correction processes; the edge analysis method combining local and global consistency accurately identifies the reliability of edge features in both RGB and depth modalities; the consistency-based adaptive region segmentation strategy effectively utilizes the complementary advantages of different modalities; and the total variational regularization mechanism smooths noise in non-edge regions while maintaining edge sharpness. This multi-level edge perception optimization method significantly improves the accuracy and naturalness of portrait edges, especially excelling in areas such as semi-transparent objects that are difficult to handle with traditional methods.

[0066] Optionally, Calculating the optimal threshold based on the histogram of the edge consistency values ​​includes: Construct a histogram of the edge consistency values, and generate multiple candidate thresholds based on the histogram; Calculate the structural similarity between the region partitioning results corresponding to each candidate threshold, and determine the candidate threshold with the maximum structural similarity as the optimal threshold for the current frame; Calculate the region overlap between the region segmentation result of the previous frame and the region segmentation result of the current frame, calculate the temporal weight based on the region overlap, multiply the temporal weight by the optimal threshold of the current frame to obtain a first product, multiply the complement of the temporal weight by the optimal threshold of the previous frame to obtain a second product, and add the first product and the second product to obtain the target threshold. Within a preset range of the region boundary corresponding to the target threshold, a smoothing weight is calculated based on the image gradient magnitude. The smoothing weight is multiplied by the target threshold to obtain a third product. The complement of the smoothing weight is multiplied by the local threshold of the region boundary to obtain a fourth product. The third product and the fourth product are added together to obtain the optimal threshold.

[0067] For example, we can analyze the distribution characteristics of a histogram of edge consistency values. The range of edge consistency values ​​[-1, 1] is evenly divided into 100 intervals, and the number of pixels in each interval is counted to form a histogram. For example, there are 1520 pixels in the interval [0.4, 0.41) and 1680 pixels in the interval [0.41, 0.42). We can analyze the peaks, valleys, and distribution patterns of the histogram to find possible clustering boundaries. In typical scenarios, the histogram of edge consistency values ​​usually exhibits a bimodal or multimodal distribution, with low peaks corresponding to low consistency regions and high peaks corresponding to high consistency regions. For example, the histogram has a peak near 0.2 (containing approximately 12,000 pixels), another peak near 0.6 (containing approximately 8,000 pixels), and a valley region between 0.35 and 0.45.

[0068] An initial threshold is generated based on the global mean and global standard deviation of the edge consistency values. The mean and standard deviation of all pixel values ​​in the 256×256 edge consistency map are calculated; for example, the global mean is 0.35 and the global standard deviation is 0.25. Using histogram analysis, the initial threshold is found in the valley regions of the histogram. If the valley region is near the mean, the initial threshold is set to the global mean; if the valley region deviates from the mean, the initial threshold is set to the global mean plus 0.5 times the standard deviation and the average of the valley location. Assuming the histogram valley location is at 0.4, the initial threshold is (0.35 + 0.5 × 0.25 + 0.4) / 2 = 0.4375, rounded to 0.44. This initial threshold serves as the starting point for subsequent searches.

[0069] Multiple candidate thresholds are generated within a preset range of the initial threshold. Centered on the initial threshold, a range of 0.2 is expanded upwards and downwards, resulting in nine candidate thresholds evenly generated within the interval [0.44-0.2, 0.44+0.2], or the range [0.24, 0.64]. Specifically, the generated candidate thresholds are: 0.24, 0.29, 0.34, 0.39, 0.44, 0.49, 0.54, 0.59, and 0.64. These candidate thresholds will be used for subsequent optimal threshold selection.

[0070] The edge consistency values ​​are divided into regions based on multiple candidate thresholds, resulting in multiple sets of region divisions. For each candidate threshold, pixels with edge consistency values ​​greater than or equal to that threshold are marked as 1 (high consistency region), and pixels with edge consistency values ​​less than that threshold are marked as 0 (low consistency region), forming a binary region division map. For example, using a threshold of 0.44, 60.2% of the pixels in the edge consistency map are classified as low consistency regions, and 39.8% of the pixels are classified as high consistency regions. This process is performed for all nine candidate thresholds, resulting in nine different sets of region division results.

[0071] The structural similarity between each group of partitioned regions is calculated, and the candidate threshold with the highest structural similarity is determined as the optimal threshold for the current frame. Connectivity analysis and shape descriptors are used to evaluate structural similarity. For each partitioning result, all connected regions with an area greater than 100 pixels are extracted, and shape features such as perimeter-to-area ratio, circularity, and rectangularity are calculated for each connected region. Then, the shape feature differences between the partitioning results corresponding to adjacent thresholds are calculated. For example, the partitioning results corresponding to thresholds 0.39 and 0.44 have a difference of 2 in the number of connected regions, a difference of 0.03 in the average perimeter-to-area ratio, a difference of 0.05 in the average circularity, and a difference of 0.04 in the average rectangularity. These difference indicators are weighted and summed (with weights of 0.3, 0.3, 0.2, and 0.2, respectively) to obtain the structural similarity score. The similarity scores of all adjacent threshold pairs are calculated, and the threshold with the highest average similarity score is selected as the optimal threshold for the current frame. Assuming that threshold 0.44 has the highest average similarity score of 0.92, it is determined as the optimal threshold for the current frame.

[0072] The overlap between the region segmentation results of the previous frame and the current frame is calculated, and a temporal weight is calculated based on the overlap. Assume the optimal threshold for the previous frame is 0.42 and the optimal threshold for the current frame is 0.44. The proportion of overlapping pixels in the region segmentation results obtained using these two thresholds is calculated. Specifically, an AND operation is performed on the segmentation results of the two frames, and the number of pixels with a result of 1 is counted. This count is then divided by the union of the total number of pixels marked as 1 in both frames to obtain the region overlap. For example, if 41,000 pixels in the previous frame are classified as high-consistency regions and 39,800 pixels in the current frame are classified as high-consistency regions, the number of overlapping high-consistency region pixels is 36,500. Therefore, the region overlap is 36,500 / 44,300 ≈ 0.824. The temporal weight is set to the square of the overlap, i.e., 0.824² ≈ 0.679.

[0073] With a temporal weight of 0.679, the optimal threshold for the current frame is 0.44, and the optimal threshold for the previous frame is 0.42. The complement of the temporal weight is 1 - 0.679 = 0.321. The first product is 0.679 × 0.44 = 0.299, the second product is 0.321 × 0.42 = 0.135, and the target threshold is 0.299 + 0.135 = 0.434. This smoothing strategy based on temporal information makes the threshold change more smoothly between frames, reducing jitter.

[0074] The edge consistency map is binarized using a target threshold of 0.434, and then an edge detection operator (such as the Sobel operator) is applied to extract the region boundaries. Within a 5-pixel neighborhood of the boundary pixels, the gradient magnitudes of the RGB image and the depth map are calculated. For example, the average gradient magnitude of the RGB image for a certain boundary region is 0.6, and the average gradient magnitude of the depth map is 0.4. The weighted average of the RGB and depth gradients (with weights of 0.6 and 0.4) is used as the comprehensive gradient magnitude for this region, i.e., 0.6 × 0.6 + 0.4 × 0.4 = 0.52. The smoothing weight is set to the Sigmoid function transformation of the gradient magnitude: 1 / (1 + exp(-5 × (0.52 - 0.3))) ≈ 0.73. The larger the gradient magnitude, the clearer the boundary, and the greater the smoothing weight.

[0075] The local threshold for the region boundary is obtained by calculating the local mean of the edge consistency value within a preset range of the region boundary. The average edge consistency value is calculated within a 5-pixel neighborhood of the boundary pixel. For example, if a boundary region contains 25 pixels, and the average edge consistency value of these pixels is 0.45, then the local threshold for that region is 0.45. Compared to the global threshold, the local threshold better adapts to the characteristics of local regions.

[0076] The smoothing weight is 0.73, the target threshold is 0.434, the local threshold at the region boundary is 0.45, and the complement of the smoothing weight is 1 - 0.73 = 0.27. The third product is 0.73 × 0.434 = 0.317, the fourth product is 0.27 × 0.45 = 0.122, and the final optimal threshold is 0.317 + 0.122 = 0.439. This optimal threshold comprehensively considers global statistical characteristics, histogram distribution, temporal consistency, and local boundary characteristics, and can more accurately divide high-consistency regions into low-consistency regions.

[0077] The edge consistency map is binarized using a calculated optimal threshold of 0.439. Pixels with an edge consistency value greater than or equal to 0.439 are classified as high-consistency regions, while those with a value less than 0.439 are classified as low-consistency regions. For example, in a 256×256 edge consistency map, approximately 40.5% of the pixels are classified as high-consistency regions, and 59.5% are classified as low-consistency regions. This region classification result will be used for subsequent masking correction processing.

[0078] This invention achieves high-precision region segmentation through an adaptive threshold selection strategy; histogram analysis of edge consistency values ​​provides a global perspective on distribution characteristics; multi-candidate threshold generation and structural similarity evaluation ensure the structural rationality of the segmentation results; the introduction of temporal information improves inter-frame consistency and reduces flickering and jitter; local boundary characteristic analysis enhances the segmentation accuracy of boundary regions; this multi-factor integrated threshold optimization method enables adaptive handling of portrait segmentation tasks under different scenes and lighting conditions, especially showing significant advantages in complex backgrounds and blurred boundary regions.

[0079] Optionally, Based on the portrait segmentation result, the illumination parameter vector of the corresponding region is extracted from the preset background image, and illumination mapping is performed on the foreground portrait region, including: The orientation features of a preset background image are calculated using a structure tensor. Anisotropy coefficients are calculated based on the orientation features. The filtering radius and regularization parameters of the guided filter are adaptively adjusted based on the anisotropy coefficients. An adaptive guided filter is then applied to the preset background image to obtain the illumination component. Construct a Gaussian pyramid of the preset background image, calculate the semantic segmentation score, gradient magnitude and brightness contrast of each layer of the Gaussian pyramid, and combine them in a weighted manner to obtain the region saliency weight; Based on the regional saliency weight and the illumination component, multiple illumination sampling points are selected from the preset background image. The illumination intensity, principal direction and color temperature of each illumination sampling point are calculated to obtain an illumination parameter vector. A local illumination model is constructed based on the illumination parameter vector. The diffuse reflection coefficient is obtained by calculating the ratio of the darkest area brightness value to the ambient light intensity of the foreground portrait area. The specular reflection coefficient is obtained by calculating the difference between the maximum brightness value and the median brightness value of the foreground portrait area. The glossiness parameter is calculated based on the gradient histogram characteristics of the foreground portrait area. The local lighting model is corrected using the diffuse reflection coefficient, specular reflection coefficient, and gloss parameters, and then the corrected local lighting model is applied to the foreground portrait area for lighting mapping.

[0080] For example, the directional features of a preset background image are calculated using a structure tensor. A Gaussian filter is applied to the preset background image (assuming a resolution of 1920×1080 pixels) for pre-smoothing, and then the gradients of the image in the x and y directions are calculated. For each pixel, the second-order matrix of the gradient, i.e., the structure tensor, is calculated. The structure tensor contains three components: the sum of squares of the gradients in the x-direction, the sum of squares of the gradients in the y-direction, and the product of the gradients in the x and y directions. A 9×9 Gaussian window is applied to perform a locally weighted average of the structure tensor. For example, the structure tensor at position (500, 600) might be [[0.65, 0.15], [0.15, 0.25]], indicating that this position mainly has strong structural features along the x-direction.

[0081] The eigenvalues ​​and eigenvectors of the structure tensor are calculated. The eigenvalues ​​represent the strength of the gradient in the principal and secondary directions. For example, the eigenvalues ​​of the structure tensor at position (500, 600) might be 0.7 and 0.2, and the eigenvectors might be [0.95, 0.31] and [-0.31, 0.95]. Anisotropy coefficients are calculated based on the eigenvalues: (λ1 - λ2) / (λ1 + λ2 + 0.01), where λ1 and λ2 are the larger and smaller eigenvalues, respectively. For example, the anisotropy coefficient is (0.7 - 0.2) / (0.7 + 0.2 + 0.01) ≈ 0.55. The anisotropy coefficient ranges from 0 to 1; a larger value indicates a stronger directionality of the image at that location.

[0082] The basic radius of the guided filter is set to 15 pixels, and the baseline value of the regularization parameter is set to 0.01. In regions with high anisotropy coefficients (such as regions with rich textures or distinct edges), the filter radius is reduced and the regularization parameter is increased to preserve structural details; in regions with low anisotropy coefficients (such as smooth regions), the filter radius is increased and the regularization parameter is decreased to enhance the smoothing effect. Specifically, the adjusted filter radius is 15 × (1 - 0.5 × anisotropy coefficient), and the adjusted regularization parameter is 0.01 × (1 + 2 × anisotropy coefficient). For example, when the anisotropy coefficient is 0.55, the filter radius is approximately 15 × (1 - 0.5 × 0.55) ≈ 11 pixels, and the regularization parameter is approximately 0.01 × (1 + 2 × 0.55) ≈ 0.021.

[0083] Guided filtering uses the image itself as a guide image and applies an adaptive filter radius and regularization parameters. The filtering process involves calculating the local mean, covariance, and regression coefficients, followed by a linear transformation. Filtering is performed on the R, G, and B channels separately to obtain an illumination component image of the same size as the original image. The illumination component retains the large-scale brightness variations of the original image while filtering out detailed textures. For example, if the average brightness of a region in the original image is [180, 160, 150], the filtered illumination component might be [175, 155, 145].

[0084] The background image is resized to a standard size (e.g., 1024×576 pixels), and then a 5-layer Gaussian pyramid is constructed. Layer 0 is the original size, layer 1 is 512×288, layer 2 is 256×144, layer 3 is 128×72, and layer 4 is 64×36. Each layer is obtained by downsampling the previous layer using a 5×5 Gaussian kernel. For example, the RGB value of a certain location in layer 1 might be [160, 140, 120], obtained by the weighted average of 25 pixels in the corresponding region of layer 0. Semantic segmentation scores, gradient magnitudes, and brightness contrast are calculated for each layer of the Gaussian pyramid, and these are weighted together to obtain the region saliency weight. Scene semantics are extracted from each layer, and common background areas such as furniture, walls, and ceilings are assigned higher semantic scores. Simultaneously, the gradient magnitude (the square root of the sum of the squares of the horizontal and vertical gradients) and local brightness contrast (the difference between the pixel brightness and the average brightness of its 9×9 neighborhood) are calculated for each pixel. These three metrics are normalized to the range of 0-1, and then weighted with weights of 0.5, 0.3, and 0.2 to obtain the region saliency weight. For example, if a pixel has a semantic score of 0.8, a gradient magnitude of 0.3, and a brightness contrast of 0.5, then its saliency weight is 0.5 × 0.8 + 0.3 × 0.3 + 0.2 × 0.5 = 0.61. Through pyramid upsampling, the saliency weights of each layer are merged into the original resolution.

[0085] A threshold operation (threshold set to 0.6) is applied to the regional saliency weights to filter out highly saliency regions. Then, within these regions, stratified sampling is performed based on the brightness values ​​of the illumination components: the brightness range is evenly divided into 5 intervals, and within each interval, 5-10 sampling points are randomly selected according to the probability distribution of the saliency weights. A total of 30-50 illumination sampling points are selected. For example, within the brightness interval [180, 220], sampling points with coordinates (300, 500), (450, 600), (700, 300), etc., might be selected. The illumination intensity, principal direction, and color temperature of each illumination sampling point are calculated to obtain the illumination parameter vector. For each sampling point, within its 11×11 neighborhood: calculate the average brightness of the illumination components as the illumination intensity, for example, the RGB average value is [200, 180, 160], and the brightness is approximately 180; calculate the principal eigenvector of the structure tensor as the principal direction of illumination, for example, [0.7, 0.7], representing a 45-degree angle; calculate the RGB ratio to estimate the color temperature, for example, an R / B ratio of 1.25 corresponds to a color temperature of approximately 5000K. Combine these three parameters into an illumination parameter vector [180, 45, 5000].

[0086] A local illumination model is constructed based on illumination parameter vectors. A radial basis function (RBF) network is used to spatially interpolate multiple sampled illumination parameter vectors to construct a continuous illumination field covering the entire image. The position coordinates (x_i, y_i) of each sampling point are used as the center point of the RBF network, and the illumination parameter vector [intensity_i, direction_i, color temperature_i] is used as the output value. A Gaussian kernel function is set, and the kernel width parameter σ is 1.5 times the average distance between sampling points. For any position (x, y) in the image, the Euclidean distance between this point and the center of all sampling points is calculated. The distance is converted into a weight coefficient exp(-d² / (2σ²)) through the Gaussian kernel function, where d is the Euclidean distance and σ is the width parameter of the Gaussian kernel function. For example, if the distance between point (500, 400) and sampling point (450, 380) is 58 pixels and σ is 150 pixels, then the weight is approximately 0.93. The normalized weights of all sampling points are calculated, and the final illumination parameters are the weighted sum of the parameters of each sampling point. In addition, the directional attenuation of illumination is also considered. For each sampling point, a directional weighting factor is constructed based on its principal illumination direction θ. The adjusted Euclidean distance is d'=d×(1-0.3×cos²(α-θ)), where α is the direction angle from the sampling point to the target point.

[0087] Find the 1% of pixels with the lowest brightness within the portrait area and calculate their average brightness, for example, 40. Then estimate the ambient light intensity by taking the average value of the dark areas of the background image (brightness below 50), for example, 30. The diffuse reflection coefficient is calculated as the ratio of the darkest area brightness to the ambient light: 40 / 30 ≈ 1.33. The diffuse reflection coefficient controls the degree to which light is uniformly scattered by the object's surface. Calculate the average value of the 1% of pixels with the highest brightness within the portrait area, for example, 230; calculate the median brightness of the portrait area, for example, 120. The specular reflection coefficient is calculated as (maximum brightness - median brightness) / 128: (230 - 120) / 128 ≈ 0.86. The specular reflection coefficient controls the intensity of the highlight areas. Calculate the gradient magnitude histogram of the portrait area and count the proportion of pixels with high gradients (greater than a threshold of 0.5), for example, 0.15. Set the glossiness parameter to 4 times this proportion, approximately 0.6. The glossiness parameter controls the concentration and sharpness of the highlight areas.

[0088] The intensity component of the local lighting model is multiplied by the diffuse reflection coefficient, the specular component is multiplied by the specular reflection coefficient, and the distribution range of the specular highlights is adjusted using glossiness parameters. For example, if the parameters of the original lighting model at a certain location are [180,45,5000], after correction it may become [180×1.33,45,5000]=[239,45,5000], which increases the lighting intensity by approximately 33%.

[0089] For each pixel in the portrait area, lighting parameters are queried from the local lighting model based on its position, and then the physically based lighting model is applied for rendering. First, the RGB values ​​are converted to HSV space, the V channel is adjusted to match the target light intensity, the H channel is adjusted to match the target color temperature, and then converted back to RGB space. For example, if the original pixel value is [150, 130, 120], the corresponding HSV is approximately [20, 0.2, 150]. If the target light intensity is 180 and the color temperature corresponds to a color shift of -5 degrees, the mapped HSV might be [15, 0.2, 180], and the converted RGB value would be approximately [180, 156, 144].

[0090] The foreground portrait after illumination mapping is blended with a preset background, and temporal smoothing is performed to obtain the background replacement result. The foreground mask is used as the blending weight, and feathering (Gaussian blur masking the edges) is applied to the edge areas to achieve a smooth transition. To prevent inter-frame jitter, temporal exponential smoothing is applied to the blending results of consecutive frames. The output of the current frame is a weighted average of the current processing result and the output of the previous frame, with weights of 0.8 and 0.2, respectively. For example, if a pixel in the current frame processing result is [180,160,140], and the output of the previous frame is [170,150,130], then the final output is 0.8×[180,160,140]+0.2×[170,150,130]=[178,158,138].

[0091] Figure 2 This chart compares the lighting mapping effects under different scenarios, showing the performance of the lighting mapping technology of this invention compared with three other existing methods in seven different lighting scenarios. The invention includes: a physically based lighting mapping method using structure tensor and local lighting models (solid line); a method based on the Phong lighting model (long dashed line); histogram matching (medium dashed line); and a color transfer method (short dashed line). This invention demonstrates significant advantages in all test scenarios, proving that the lighting mapping method based on structure tensor analysis and physically based lighting models proposed in this invention has significant advantages over existing technologies, especially in complex lighting scenarios.

[0092] This invention achieves high-quality background replacement through illumination consistency processing and fusion steps; structural tensor analysis and adaptive guided filtering accurately extract the lighting environment characteristics of the background image; multi-scale saliency analysis ensures accurate sampling of illumination parameters in key areas; parameterized description based on the physical lighting model achieves accurate transfer of lighting conditions; foreground reflection characteristic estimation and illumination model correction solve the differences in lighting response of different material surfaces; temporal smoothing processing eliminates inter-frame jitter and ensures visual continuity. This invention makes the replaced image have extremely high realism and naturalness, effectively solving the problem of disharmony caused by illumination mismatch in traditional background replacement.

[0093] Optionally, a lightweight model architecture is employed, combined with model quantization and parallel computing techniques, to ensure that the dual-path tracing network and dual-modal segmentation network can run in real time on consumer-grade hardware. Under resource constraints, the processing resolution and feature complexity are dynamically adjusted to prioritize the processing quality of the portrait region. For example, MobileNetV3 is used as the backbone network, and depthwise separable convolutions are applied instead of standard convolutions; 8-bit integer quantization reduces the model parameter precision from 32-bit floating-point to 8-bit integer, reducing memory usage; SIMD instruction set optimization is implemented for the dual-path tracing network and dual-modal segmentation network, utilizing multi-core processors to process independent channels and batch data in parallel. Under resource constraints, the processing strategy is dynamically adjusted: the input resolution is adaptively scaled to between 224×224 and 448×448 based on device performance; lower-complexity feature extraction is applied to the periphery of the portrait region; different numbers of feature channels are used for edge and center regions during multi-level feature fusion, maintaining full channels (96 channels) in the center region and reducing them to 32 channels in the edge region; and a region importance map guides the allocation of computational resources to ensure that key parts of the portrait, such as the face, receive the highest precision processing.

[0094] After background replacement, this invention also supports multi-person online connections and real-time translation. It receives multiple live streams in parallel via the RTMP protocol, with each stream assigned an independent thread for background replacement processing. For multi-host online scenarios, an adaptive bandwidth allocation algorithm is used to dynamically adjust the encoding parameters of each video stream based on the complexity of the video feed and network conditions. In terms of speech processing, a WebRTC audio engine is used to extract speech signals. A pre-trained language recognition model is used to determine the language type. This model employs a two-layer BiLSTM network structure, with an input of a 20ms frame-long MFCC feature sequence. First, 13-dimensional MFCC features and their first-order difference features are extracted from the audio, forming a 26-dimensional feature vector. Then, temporal features are extracted using a two-layer BiLSTM (256 hidden units per layer). Finally, a fully connected layer with 1024 nodes and a softmax classification layer are used to output the language probability distribution. When a non-target language is detected, a Transformer-structured neural translation model is called for real-time translation, and WaveNet speech synthesis technology is used to generate natural voiceovers. The system tracks the lip movements of the broadcaster and adjusts the dubbing rhythm according to the phoneme sequence to achieve lip-sound synchronization. In the image compositing stage, multiple video streams are combined according to a preset layout, and semi-transparent subtitles are overlaid in each broadcaster's area. Buffer queue management ensures audio-visual synchronization, and the overall processing latency is controlled within 500ms to meet the needs of real-time interaction.

[0095] Secondly, it provides a real-time optimization system for live streaming background replacement based on human face tracking, including: The first unit is used to acquire real-time frames from the live video stream; perform sparse optical flow estimation on the real-time frames to obtain a region motion intensity map; determine the sampling density distribution based on the region motion intensity map; construct a feature pyramid to perform human face detection to obtain an initial human face region; input the initial human face region into a dual-path tracking network, wherein the main path extracts target features based on an attention mechanism, and the auxiliary path predicts motion information based on historical frame features; and fuse them to obtain a target tracking box. The second unit is used to generate a corresponding depth map from an RGB image using a monocular depth estimation network. The RGB image and the corresponding depth map of the region corresponding to the target tracking box are input into a bimodal segmentation network to generate a foreground mask. The consistency value of the edge features of the RGB image and the edge features of the depth map is calculated, and the foreground mask is corrected to obtain the portrait segmentation result. The third unit is used to extract the illumination parameter vector of the corresponding region from the preset background image based on the portrait segmentation result, perform illumination mapping on the foreground portrait region, merge the illumination-mapped foreground portrait with the preset background, and perform temporal smoothing processing to obtain the background replacement result.

[0096] Thirdly, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

Claims

1. A real-time optimization method for live streaming background replacement based on human face tracking, characterized in that, include: Capture real-time frames from a live video stream; Sparse optical flow estimation is performed on the real-time frame to obtain a region motion intensity map. The sampling density distribution is determined based on the region motion intensity map, and a feature pyramid is constructed to perform human face detection to obtain an initial human face region. The initial human face region is input into a dual-path tracking network, where the main path extracts target features based on an attention mechanism, and the auxiliary path predicts motion information based on historical frame features. The two paths are then fused to obtain a target tracking box. A monocular depth estimation network is used to generate a corresponding depth map from an RGB image. The RGB image and the corresponding depth map of the region corresponding to the target tracking box are input into a bimodal segmentation network to generate a foreground mask. The consistency value of the edge features of the RGB image and the edge features of the depth map is calculated, and the foreground mask is corrected to obtain the portrait segmentation result. Based on the portrait segmentation results, the illumination parameter vector of the corresponding region is extracted from the preset background image, and illumination mapping is performed on the foreground portrait region; The foreground portrait after lighting mapping is blended with the preset background, and a time-series smoothing process is performed to obtain the background replacement result; Based on the regional motion intensity map, the sampling density distribution is determined, and a feature pyramid is constructed for human face detection to obtain the initial human face region, which includes: An adaptive sampling probability density function is constructed on the motion intensity map of the region. The probability density function includes a motion intensity term, a uniform distribution term, and a Gaussian prior term. The weight coefficients of each term are adaptively adjusted according to the degree of scene motion. A feature pyramid is constructed based on the number of sampling points with exponential decay, and the position of feature points is obtained by importance sampling based on the probability density function at each level. Multimodal feature vectors are extracted and normalized at the feature point locations. The detection confidence of each level is calculated. Based on the scale consistency and confidence distribution of the detection boxes at each level, inter-layer fusion weights are constructed. The detection boxes with different weights are weighted and fused to obtain the initial human image region.

2. The method according to claim 1, characterized in that, The initial human image region is input into a dual-path tracking network to obtain a target tracking bounding box, which includes: A convolution operation is performed on the initial human image region to obtain a feature map. The feature map is divided into a fixed grid. An average pooling operation is performed on the feature map in each grid to obtain a pooled feature. The pooled feature is then multiplied by the feature map to obtain a grid attention feature. Weight coefficients are assigned to the grid attention features of different grids according to the importance of the human body region. The weighted summation is then used to obtain the main path target feature. Fast optical flow estimation is performed on the historical frame sequence of the initial human image region to obtain the displacement field. Based on the amplitude of the displacement field, the feature update coefficient is calculated. The current frame feature and the previous frame feature are subjected to an exponential moving average through the feature update coefficient to obtain the auxiliary path motion feature. The magnitude of the displacement field is calculated and normalized to obtain the motion confidence. The main path target feature is convolved with the motion confidence to obtain the feature fusion weight. The main path target feature and the auxiliary path motion feature are linearly fused according to the feature fusion weight to obtain the final target feature. The position parameters and scale parameters of the target tracking box are obtained by regression based on the final target feature.

3. The method according to claim 1, characterized in that, Generating a corresponding depth map from an RGB image using a monocular depth estimation network, and inputting the RGB image and corresponding depth map of the region corresponding to the target tracking box into a bimodal segmentation network to generate a foreground mask includes: The monocular depth estimation network calculates depth attention coefficients using a query matrix and a key matrix, and generates the depth map by weighted combination of the depth feature map and the depth attention coefficients. The RGB image and depth map of the region corresponding to the target tracking box are input into a bimodal segmentation network to obtain RGB features and depth features. The RGB features and depth features are then aligned across modally using a complementary attention mechanism. The complementary attention mechanism uses the RGB features as a query vector and the depth features as key-value pairs to calculate cross-modal attention weights. The RGB features and depth features are then weighted and combined based on the cross-modal attention weights to obtain the aligned features. Extract edge features from the RGB image and depth map, calculate the edge difference between the RGB edge features and the depth edge features, and obtain the edge fusion weight based on the edge difference; The alignment features are input into a multi-level decoder to generate an RGB mask and a depth mask. The RGB mask and the depth mask are then weighted and fused according to the edge fusion weights to obtain a foreground mask.

4. The method according to claim 1, characterized in that, Calculate the consistency value between the edge features of the RGB image and the edge features of the depth map, correct the foreground mask, and obtain the portrait segmentation result, including: Within a sliding window, the local consistency value of the edge features of the RGB image and the edge features of the depth map is calculated, and the structural similarity between the edge features of the RGB image and the edge features of the depth map is calculated to obtain the global structural consistency value. The local consistency value is multiplied by the first weight and then added to the product of the global structural consistency value and the second weight to obtain the edge consistency value. The optimal threshold is calculated based on the histogram of the edge consistency values, and the pixels are divided into high consistency regions and low consistency regions. In the high consistency region, the edge features of the depth map are used to correct the foreground mask to obtain a first corrected mask, and in the low consistency region, the edge features of the RGB image are used to correct the foreground mask to obtain a second corrected mask. The first and second modified masks are subjected to total variational regularization to obtain the optimized mask. When the iteration difference between the optimized masks obtained in two adjacent iterations is greater than or equal to a preset difference threshold, the next round of iteration optimization is performed. When the iteration difference is less than the preset difference threshold, the final portrait segmentation result is output.

5. The method according to claim 4, characterized in that, Calculating the optimal threshold based on the histogram of the edge consistency values ​​includes: Construct a histogram of the edge consistency values, and generate multiple candidate thresholds based on the histogram; Calculate the structural similarity between the region partitioning results corresponding to each candidate threshold, and determine the candidate threshold with the maximum structural similarity as the optimal threshold for the current frame; Calculate the region overlap between the region segmentation result of the previous frame and the region segmentation result of the current frame, calculate the temporal weight based on the region overlap, multiply the temporal weight by the optimal threshold of the current frame to obtain a first product, multiply the complement of the temporal weight by the optimal threshold of the previous frame to obtain a second product, and add the first product and the second product to obtain the target threshold. Within a preset range of the region boundary corresponding to the target threshold, a smoothing weight is calculated based on the image gradient magnitude. The smoothing weight is multiplied by the target threshold to obtain a third product. The complement of the smoothing weight is multiplied by the local threshold of the region boundary to obtain a fourth product. The third product and the fourth product are added together to obtain the optimal threshold.

6. The method according to claim 1, characterized in that, Based on the portrait segmentation result, the illumination parameter vector of the corresponding region is extracted from the preset background image, and illumination mapping is performed on the foreground portrait region, including: The orientation features of a preset background image are calculated using a structure tensor. Anisotropy coefficients are calculated based on the orientation features. The filtering radius and regularization parameters of the guided filter are adaptively adjusted based on the anisotropy coefficients. An adaptive guided filter is then applied to the preset background image to obtain the illumination component. Construct a Gaussian pyramid of the preset background image, calculate the semantic segmentation score, gradient magnitude and brightness contrast of each layer of the Gaussian pyramid, and combine them in a weighted manner to obtain the region saliency weight; Based on the regional saliency weight and the illumination component, multiple illumination sampling points are selected from the preset background image. The illumination intensity, principal direction and color temperature of each illumination sampling point are calculated to obtain an illumination parameter vector. A local illumination model is constructed based on the illumination parameter vector. The diffuse reflection coefficient is obtained by calculating the ratio of the darkest area brightness value to the ambient light intensity of the foreground portrait area. The specular reflection coefficient is obtained by calculating the difference between the maximum brightness value and the median brightness value of the foreground portrait area. The glossiness parameter is calculated based on the gradient histogram characteristics of the foreground portrait area. The local lighting model is corrected using the diffuse reflection coefficient, specular reflection coefficient, and gloss parameters, and then the corrected local lighting model is applied to the foreground portrait area for lighting mapping.

7. A real-time optimization system for live streaming background replacement based on human face tracking, used to implement the method of any one of claims 1-6, characterized in that, include: The first unit is used to acquire real-time frame images from the live video stream. Sparse optical flow estimation is performed on the real-time frame to obtain a region motion intensity map. The sampling density distribution is determined based on the region motion intensity map, and a feature pyramid is constructed to perform human face detection to obtain an initial human face region. The initial human face region is input into a dual-path tracking network, where the main path extracts target features based on an attention mechanism, and the auxiliary path predicts motion information based on historical frame features. The two paths are then fused to obtain a target tracking box. The second unit is used to generate a corresponding depth map from an RGB image using a monocular depth estimation network. The RGB image and the corresponding depth map of the region corresponding to the target tracking box are input into a bimodal segmentation network to generate a foreground mask. The consistency value of the edge features of the RGB image and the edge features of the depth map is calculated, and the foreground mask is corrected to obtain the portrait segmentation result. The third unit is used to extract the illumination parameter vector of the corresponding region from the preset background image based on the portrait segmentation result, and to perform illumination mapping on the foreground portrait region. The foreground portrait after lighting mapping is blended with the preset background, and a time-series smoothing process is performed to obtain the background replacement result.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images