A video call background blurring and light supplement enhancement system for mobile phones

Abstract

The application discloses a video call background virtualization and light supplement enhancement system for a mobile phone, relates to the technical field of computer vision, receives a continuous video stream, calculates a time sequence motion vector of a current frame and a previous frame, judges the time sequence motion vector, and outputs a picture to be processed; a distance constraint core is used to demarcate a core space constraint region in the picture to be processed, and a double-flow depth separable network is used to extract key features in the core space constraint region, including first features and second features; a cross attention algorithm is used for routing calculation on the key features to generate double-modal fusion features; an intrinsic image decomposition algorithm is used to analyze the double-modal fusion features into reflectivity maps, and a core region enhanced picture is synthesized to generate a final video rendering frame in combination with the picture to be processed, so that the enhanced picture has a strong natural stereoscopic sense and the visual experience of a user is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a system for background blurring and light enhancement in video calls on mobile phones. Background Technology

[0002] With the widespread adoption of mobile internet and 5G communication technologies, video calls and mobile live streaming have become important methods of daily communication. In these scenarios, users are often in non-standard environments with complex lighting conditions (such as backlighting, low light, and interference from multiple light sources) and cluttered backgrounds. To improve the visual experience, existing video processing systems typically incorporate background blurring and portrait lighting techniques.

[0003] Existing deep learning vision models typically perform global, full-resolution convolutional computations on every frame of a video stream. In video call scenarios, users are often relatively still or in minimal motion, and the core focus of the image is concentrated on the face and upper body. This frame-by-frame intensive computation of the entire image generates significant spatiotemporal data redundancy, leading to rapid overheating, frequency throttling, and severe battery drain on mobile devices. Furthermore, image matting and lighting enhancement algorithms heavily rely on a single visible light modality. In low-light or backlight conditions, the edge features of RGB images are extremely weakened and accompanied by high-frequency noise, preventing the model from accurately separating the foreground image from the background. This results in severe artifacts, with the lighting effect overflowing into the background area, or the background blur incorrectly swallowing the edges of the subject. In addition, existing edge-side lighting algorithms often employ simple nonlinear mapping or global brightness adjustment, failing to follow the laws of physical optics, resulting in pale skin tones, poor transparency, and an inability to reproduce realistic light and shadow levels after enhancement. Summary of the Invention

[0004] The technical problem solved by this invention is that existing technologies have difficulty improving the rendering fidelity under complex lighting conditions, the inference delay of a single video frame is too long, the background area adjustment is rough, and artifacts are easily generated.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a video call background blurring and light enhancement system for mobile phones, comprising: The spatiotemporal dynamic determination module is used to receive continuous video streams, calculate the temporal motion vector between the current frame and the previous frame, determine the temporal motion vector, and output the image to be processed. The spatial feature extraction module is used to delineate a core spatial constraint region in the image to be processed using a distance constraint kernel, and to extract key features within the core spatial constraint region using a two-stream depth separable network to generate a corresponding scene depth map. The cross-attention fusion module is used to perform routing calculations on key features using the cross-attention algorithm to generate dual-modal fusion features. The physical decoupled rendering module is used to analyze the dual-modal fusion features into a reflectance map using an intrinsic image decomposition algorithm, and calculate the background light by combining the scene depth map. Based on the reflectance map and the background light, the core area is enhanced to obtain the final video rendering frame.

[0006] Preferably, the spatiotemporal dynamic determination module includes a motion vector estimation unit and a temporal overlap pruning unit; The motion vector estimation unit is used to receive a continuous video stream, extract the current frame and the previous frame from the continuous video stream, calculate the displacement change matrix between the current frame and the previous frame using an optical flow estimation algorithm, and generate a temporal motion vector. The temporal overlap pruning unit is used to receive the temporal motion vector, compare the amplitude of the temporal motion vector with a preset overlap threshold, and output the image to be processed.

[0007] Preferably, the specific process of comparison and determination by the temporal overlap pruning unit includes: If the magnitude of the temporal motion vector is less than or equal to the preset overlap threshold, it is determined that the content of the current frame and the previous frame are highly overlapping, the image rendering parameters of the previous frame are reused, and the subsequent feature extraction steps are skipped. If the magnitude of the temporal motion vector is greater than the preset overlap threshold, it is determined that there is a significant dynamic change in the current frame. The current frame is then extracted and marked as a scene to be processed, and output to the spatial feature extraction module. The preset overlap threshold is dynamically calculated in real time based on the current camera's sensitivity and the motion vector amplitude of the previous N frames. The calculation expression for the overlap threshold is: ; in, To preset the overlap threshold, and These are the historical mean and historical standard deviation of the motion vector magnitudes of the first N frames of video images, respectively. As a time-series adjustment factor, As the baseline sensitivity, This refers to the current light sensitivity of the camera. This is the noise compensation coefficient.

[0008] Preferably, the spatial feature extraction module includes an anchor point positioning and delineation unit and a dual-stream feature encoding unit; The anchor point positioning and delineation unit is used to receive the image to be processed, extract the target visual anchor point in the image to be processed, and construct a distance constraint kernel with the target visual anchor point as the center. The dual-stream feature encoding unit is used to receive the image to be processed after the constraints are defined, and to process the image stream in parallel through a dual-branch network architecture, extracting visible light texture features and depth structure features respectively, and generating a scene depth map.

[0009] Preferably, the specific process of constructing the distance constraint kernel by the anchor point positioning and delineation unit includes: The face detection algorithm is used to locate the face region in the image to be processed as the target visual anchor point; Based on the target visual anchor point, the core spatial constraint area is delineated by extending outward according to the preset spatial expansion coefficient; Pixels in the image to be processed that are outside the core spatial constraint region are marked as background buffers. Gaussian blur is applied directly to the background buffers, and only the core spatial constraint region is retained to enter the dual-stream feature coding unit.

[0010] Preferably, the dual-stream feature coding unit includes a first processing branch and a second processing branch; The first processing branch is used to receive visible light three-channel data within the core spatial constraint area, extract high-frequency color features and edge features through a depthwise separable convolutional network, and generate a first feature, which is a visible light texture feature. The second processing branch is used to receive the initial depth feature map of the single channel within the core spatial constraint area, extract low-frequency foreground contour and depth structure features through downsampling, generate a second feature, which is a depth structure feature, and save the first feature and the second feature as key features.

[0011] Preferably, the cross-attention fusion module includes a cross-modal mapping unit and a routing computation coordination unit; The cross-modal mapping unit is used to receive the key features and generate a query matrix, a key-value matrix, and a numerical matrix through linear projection transformation; The routing computation coordination unit is used to receive the query matrix, key matrix, and numerical matrix, calculate cross-modal response weights using a scaling dot product attention mechanism, and synthesize bimodal fusion features.

[0012] Preferably, the specific rules for the cross-modal mapping unit when generating the matrix are as follows: The deep structural features are mapped into a feature space to generate the query matrix; The visible light texture features are mapped into feature space to generate the key matrix and the numerical matrix; The calculation process of the routing calculation coordination unit includes: calculating the dot product of the query matrix and the transpose of the key matrix, dividing by the dimension scaling factor, and then obtaining the weight distribution matrix through the normalized exponential function; Multiplying the weight distribution matrix with the numerical matrix allows the edge features of the deep structure to guide the response of visible light pixels, thereby generating the dual-modal fusion feature.

[0013] Preferably, the physical decoupling rendering module includes an intrinsic map decoupling unit and a pixel-level synthesis unit; The intrinsic image decoupling unit is used to receive the dual-modal fusion features, and to perform intrinsic image decomposition on the dual-modal fusion features using a preset illumination decoupling decoder structure to extract a reflectance map that does not contain light source information. Receive the scene depth map and the original input image of the scene to be processed, and perform adaptive background light estimation based on the scene depth map: The top 0.1% of pixels with the largest depth value are selected from the scene depth map, and the average pixel value of each channel of the top 0.1% of pixels in the original input image is calculated as the first candidate background light. A quadtree hierarchical search is used. Each time, the sub-region with the smallest sum of pixel variances in each channel is selected. After n iterations, the average value of each channel in the final region is calculated as the second candidate background light. At the same time, the sum of the red-green difference and the red-blue difference of the pixels is used as the color difference measure. Each time, the sub-region with the largest average color difference is selected. After n iterations, the average value of each channel in the final region is calculated as the third candidate background light. The weights of the first candidate background light, the second candidate background light, and the third candidate background light are calculated and normalized respectively, and the final background light is obtained by weighted averaging. The depth values ​​of each pixel in the scene depth map are normalized to obtain normalized depth values. The illumination attenuation coefficient of each pixel is calculated based on the normalized depth values. The pixel-wise product of the original input image and the illumination attenuation coefficient is taken as the ambient illumination component. The pixel-wise ratio of the original input image minus the ambient illumination component to the illumination attenuation coefficient is taken as the scene reflection component. Guided filtering edge preservation optimization is performed on the ambient illumination component and the scene reflection component respectively. Finally, a lower limit threshold constraint is applied to the optimized transmittance map. The final background light is used as a global ambient light compensation term. The reflectance map is extracted and linearly fused with the final background light according to the light attenuation coefficient to generate a reflectance map.

[0014] Preferably, the pixel-level synthesis unit is used to receive the reflectance map, perform matrix calculation mapping, and generate an enhanced image of the core region: The reflectance map is subjected to contrast enhancement and detail sharpening processing to generate an enhanced image of the core area. Based on the background buffer after Gaussian blur preprocessing, the enhanced image of the core area is merged with the background buffer using a splicing and blending mechanism to generate the final video rendering frame, which is then output to the display terminal.

[0015] The beneficial effects of this invention are as follows: This invention innovatively performs dynamic judgment on video streams by extracting temporal motion vectors. For highly overlapping frames with minimal motion amplitude, the system directly puts the subsequent high-energy-consuming feature extraction and rendering networks to sleep, reusing the rendering parameters of the previous frame. This mechanism effectively eliminates temporal redundancy in video calls, reducing the NPU / CPU load of mobile devices by at least an order of magnitude and significantly improving the heat dissipation problem during long calls. By constructing a distance constraint kernel based on visual anchor points, most of the network computing power is precisely focused on the core area of ​​the human face, while the background buffer undergoes lightweight dimensionality reduction. This spatial dimension computation range clipping significantly shortens the inference latency of a single frame while ensuring the clarity of the human face. By utilizing low-frequency but stable geometric and physical contours in the depth map, the activation area of ​​the lighting network is forcibly constrained, effectively avoiding foreground and background confusion caused by complex lighting and solving the artifact problem commonly seen in video calls. Through this reconstruction method that strictly follows the optical prior principle, it is ensured that the network only brightens the light and shadow distribution without changing the original color of the human skin texture, resulting in a final output image with a rosy and natural skin tone and strong three-dimensionality, significantly improving the user's visual experience. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the basic process of a video call background blurring and light enhancement system for mobile phones, provided as an embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Example, refer to Figure 1 A system for enhancing background blur and light compensation in mobile video calls is provided, including: The spatiotemporal dynamic determination module is used to receive continuous video streams, calculate the temporal motion vector between the current frame and the previous frame, determine the temporal motion vector, and output the image to be processed. The spatial feature extraction module is used to delineate the core spatial constraint region in the image to be processed using a distance constraint kernel, and to extract key features within the core spatial constraint region using a two-stream deep separable network to generate the corresponding scene depth map. The cross-attention fusion module is used to perform routing calculations on key features using the cross-attention algorithm to generate dual-modal fusion features. The physical decoupled rendering module is used to analyze the dual-modal fusion features into a reflectance map using an intrinsic image decomposition algorithm, and calculate the background light by combining it with the scene depth map. Based on the reflectance map and the background light, the core area is synthesized to enhance the image and obtain the final video rendering frame.

[0019] This embodiment primarily runs on the processor of a mobile smart terminal, such as a smartphone or tablet. First, it captures a continuous video stream from the front-facing camera in real time. Inter-frame motion is evaluated through spatiotemporal dynamic judgment. If the current frame is almost stationary compared to the previous frame, the power-consuming deep learning inference is skipped, and the rendering parameters of the previous frame, i.e., image rendering parameters such as illumination map, segmentation mask, and blur coefficient, are directly reused, thereby achieving temporal overlap pruning and reducing power consumption. If significant motion is determined, the spatial feature extraction part is entered, using face localization to construct a distance constraint kernel, retaining only the human face. The core areas, such as the face and shoulders, are fed into a neural network, while the background area is directly split and subjected to low-computation Gaussian blur. The core area is then input into a dual-stream network to extract visible light and depth features. Through a cross-attention fusion module, the visible light texture is guided by the geometric depth boundary to solve the problem of blurry edges showing through the image. Then, based on intrinsic image decomposition theory and scene depth distribution, multiple candidate background lights are adaptively estimated and fused to perform noise-free physical-grade relighting enhancement on the image. The enhanced core portrait area and the Gaussian-blurred background layer are then smoothly feathered and stitched together to output the final video rendering frame to the mobile phone screen.

[0020] The spatiotemporal dynamic determination module includes a motion vector estimation unit and a temporal overlap pruning unit; The motion vector estimation unit is used to receive a continuous video stream, extract the current frame and the previous frame from the continuous video stream, calculate the displacement change matrix between the current frame and the previous frame using an optical flow estimation algorithm, and generate a temporal motion vector. The temporal overlap pruning unit is used to receive temporal motion vectors, compare the amplitude of the temporal motion vectors with a preset overlap threshold, and output the image to be processed.

[0021] This embodiment uses simple motion statistics instead of a complex frame difference network, resulting in minimal computational overhead and accurate differentiation between real-world image changes and noise jitter. In video call scenarios, users are mostly in a relatively static or slightly moving state. Content overlap pruning in the time dimension is the core strategy for controlling phone overheating and extending battery life. When a user only blinks or slightly moves their lips, the optical flow amplitude is usually less than 0.3 pixels. The system determines overlap and maintains the rendering of the previous frame. When the user turns their head or moves the phone, the optical flow amplitude can exceed 2 pixels, and the system immediately triggers a new frame processing to ensure response speed.

[0022] The specific process of comparing and determining time-overlapping pruned units includes: If the magnitude of the temporal motion vector is less than or equal to the preset overlap threshold, it is determined that the current frame and the previous frame are highly overlapping. The image rendering parameters of the previous frame are reused, the subsequent feature extraction steps are skipped, and the rendering matrix of the historical frame is directly called. The image rendering parameters include the core area enhancement image generated in the previous frame, the background Gaussian blur map and the Alpha fusion mask. After reuse, it is directly output without triggering new feature extraction and rendering calculations. If the magnitude of the temporal motion vector is greater than the preset overlap threshold, it is determined that there is a significant dynamic change in the current frame. The current frame is extracted and marked as the image to be processed, and output to the spatial feature extraction module to activate subsequent deep learning processing. The preset overlap threshold is dynamically calculated in real time based on the current camera's ISO and the motion vector magnitudes of the previous N frames. The expression for calculating the overlap threshold is: ; in, To preset the overlap threshold, and These are the historical mean and historical standard deviation of the motion vector magnitudes of the first N frames of video images, respectively. As a time-series adjustment factor, As the baseline sensitivity, This refers to the current light sensitivity of the camera. For the noise compensation factor, this embodiment preferably uses N=15, i.e., a sliding window of 0.5 seconds. The value range is [1.0, 2.0], used to control the sensitivity to minute movements. =100, The value ranges from [0.05, 0.15], representing the current camera's light sensitivity when the optical fiber is dim. As the noise level rises, the pseudo-motion amplitude calculated by the optical flow method increases, and the value in the formula changes accordingly. Automatically increase and raise the overlap threshold. This allows for the containment of pseudo-displacement caused by high noise, ensuring successful pruning in low light and preventing the phone from overheating.

[0023] Spatial constraints concentrate computing power on the subject of the portrait, with only about 40% of the pixels in the full frame entering the heavy network. The dual-stream branch uses depthwise separable convolution to compress the number of parameters, and the depthstream channel is extremely small, further reducing computation. This module achieves computational region focus in the spatial dimension, avoiding invalid edge segmentation and lighting analysis of the background. It forms a spatiotemporal joint pruning with the temporal pruning of the previous module, which greatly shortens the edge inference time.

[0024] The spatial feature extraction module includes an anchor point localization and delineation unit and a dual-stream feature coding unit; The anchor point positioning and delineation unit is used to receive the image to be processed, extract the target visual anchor points in the image to be processed, and construct a distance constraint kernel with the target visual anchor points as the center. The dual-stream feature coding unit receives the image to be processed after the constraints are defined, processes the image stream in parallel through a dual-branch network architecture, extracts visible light texture features and depth structure features respectively, and generates a scene depth map.

[0025] This embodiment primarily focuses on spatial dimensionality reduction of the image to be processed, thereby reducing the pixel area processed by the neural network. A lightweight face detection algorithm (based on the MobileNet face detector in this embodiment) is used to locate the bounding box of faces in the image, and the face center point is calculated as the target visual anchor point. Centered on the anchor point, a spatial expansion coefficient β (preferably β=2.0) is introduced to extend outward and construct a distance constraint kernel. This distance constraint kernel defines a core spatial constraint region in the image space that encompasses the face, shoulders, and chest area. Pixels outside this region are defined as background buffers. Instead of performing neural network feature extraction on the background buffer, we directly call the Gaussian kernel to perform one-dimensional Gaussian blur filtering on it: ; in, For the fuzzy radius, These are the coordinates corresponding to the pixels. The filtered background is replaced with an area; The image of the most cost-effective area in the core space is input into a dual-stream feature encoding network to obtain a scene depth map.

[0026] The specific process of constructing the distance constraint kernel by defining anchor point positioning units includes: The face detection algorithm is used to locate the face region in the image to be processed as the target visual anchor point; Based on the target visual anchor point, the core spatial constraint area is delineated by extending outward according to the preset spatial expansion coefficient; Pixels outside the core spatial constraint region in the image to be processed are marked as background buffers. Gaussian blur is applied directly to the background buffers, and only the core spatial constraint region is retained to enter the dual-stream feature coding unit.

[0027] If the face detection algorithm fails to return a valid detection box (e.g., side profile, occlusion, or failure in extremely low light), it automatically switches to a backup strategy: using the geometric center of the image to be processed as the target visual anchor point, and constructing a core spatial constraint region with a fixed radius of one-third of the shorter side length of the image, to ensure that the algorithm can still maintain the basic processing flow in extreme cases.

[0028] In low-light environments, RGB features often lose the edges between the subject and the background due to noise, while depth features are almost unaffected by lighting and have clear structural boundaries. By forcing the similarity of RGB features to the depth edges using a formula, the network learns to enhance RGB pixels only within the depth boundaries. This limits the lighting operation to the foreground subject area, fundamentally preventing artifacts caused by light halos spilling into the background.

[0029] Assuming the user is facing away from a window and their face is in the shadows, the parallax edge between the head and the background in the depth map is very sharp. The attention weights approach 0 at this edge, causing the background corresponding value matrix to be almost unselected by the enhanced illumination flow. This ensures that the background retains a realistic dark tone after illumination, while the brightness of the face is increased normally.

[0030] The dual-stream feature coding unit includes a first processing branch and a second processing branch; The first processing branch is used to receive visible light three-channel data within the core spatial constraint area, and extract high-frequency color features and edge features through a depthwise separable convolutional network to generate the first feature, which is a visible light texture feature. The second processing branch receives the initial depth feature map of a single channel within the core spatial constraint area, extracts low-frequency foreground contour and depth structure features through downsampling, generates a second feature, which is a depth structure feature, and saves the first and second features as key features.

[0031] The first processing branch takes a three-channel visible light image block as input. In this embodiment, the preferred configuration is... Separable convolutions at multiple depths Stacked encoders extract high-frequency color and edge features, and output visible light texture features. ; To significantly reduce computational load on mobile devices, depthwise separable convolutions are used instead of traditional 2D convolutions in the first processing branch. Let the input feature map be... The output feature map is The depthwise separable convolution process consists of two steps: the first step is channel-wise convolution, and the second step is point-wise convolution. The first step uses a number of channels... The first step involves using a filter with a kernel size of k×k (preferably 3×3) to independently perform convolution filtering on each channel of the input, generating intermediate features; the second step uses a filter with a kernel size of k×k (preferably 3×3) to generate intermediate features. A 1×1 kernel filter is used to perform linear combination and dimensionality enhancement of intermediate features between channels, outputting the first feature, which is the visible light texture feature. With the above design, the computational cost is only a fraction of that of standard convolution. To alleviate the real-time computing pressure on mobile phones, the convolutional layer and residual connection can be separated by four consecutive downsampling depths, and the first processing branch finally outputs visible light texture features. The second processing branch takes as input a single-channel initial depth patch. By employing a lighter asymmetric channel compression design (approximately 1 / 4 the number of channels of the first branch), low-frequency foreground contours and depth structure features are extracted, outputting depth structure features. ; Asymmetric channel compression design: Since the entropy of depth information is much lower than that of visible light RGB images, to avoid wasting computational resources and improve the cache hit rate of the edge NPU, the number of channels in the encoder structure of the second processing branch is limited to a fraction of the number of channels in the corresponding layer of the first processing branch. That is, if the number of feature channels in the l-th layer of the first processing branch is... Then the number of feature channels in the second processing branch corresponding to the l-th layer is... For example, in the feature output layer, when the number of channels for the first feature is 128, the number of channels for the second feature is only set to 32. The second processing branch uses a lightweight convolutional network to continuously downsample and smooth the initial depth feature map of a single channel. Due to the extreme compression of the number of channels, the floating-point computation of this branch is less than that of the first branch. This branch ultimately outputs the depth structure features (second features) that provide general outline and scene depth structure information, with a dimension of [missing information]. In this embodiment, the resolution is aligned with the first feature at 16×16, and the number of channels... =32; After parallel extraction is completed, the system aligns the first and second features in terms of spatial dimension (resolution) and packages them together to save them as key features.

[0032] The cross-attention fusion module includes a cross-modal mapping unit and a routing computation collaborative unit; The cross-modal mapping unit is used to receive key features and generate query matrix, key matrix and numerical matrix through linear projection transformation; The routing computation collaborative unit receives the query matrix, key matrix, and numerical matrix, and uses the scaling dot product attention mechanism to calculate cross-modal response weights and synthesize bimodal fusion features.

[0033] In low-light environments, RGB features often lose the edges of the subject and background due to noise, while depth features are almost unaffected by lighting and have clear structural boundaries. This can be achieved through the formula: ; Where Q is the query matrix, K is the key matrix, and V is the value matrix; By forcing the similarity of RGB features (K) to the depth edge (Q), the network learns to enhance RGB pixels only within the depth boundary, thus limiting the fill light operation to the foreground portrait area and preventing artifacts from spilling the halo into the background.

[0034] Assuming the user is facing away from a window, with their face in the shadows, the parallax edge between the head and background in the depth map is very sharp. Attention weights approach zero at this edge, causing the V-shaped area corresponding to the background to be almost entirely ignored by the enhanced illumination flow. This ensures that the background retains its true dark tone after fill lighting, while the face brightness is increased normally. The specific rules for cross-modal mapping units when generating matrices are as follows: The deep structural features are mapped to the feature space to generate a query matrix; The visible light texture features are mapped into the feature space to generate a key-value matrix and a numerical matrix. The computation process of the routing computation coordination unit includes: calculating the dot product of the query matrix and the transpose of the key matrix, dividing by the dimension scaling factor, and then obtaining the weight distribution matrix through the normalized exponential function; Multiplying the weight distribution matrix with the numerical matrix allows the edge features of the deep structure to guide the response of visible light pixels, generating dual-modal fusion features.

[0035] By forcibly mapping deep structural features to a query matrix (Q) and visible light texture features to a key matrix and a value matrix, a geometrically guided texture dependency is established. Since the depth map is unaffected by visible light noise in low light or backlight, it provides a clear portrait outline and background boundary. When calculating the softmax function, the attention weights decay rapidly along the depth edges, thus strictly limiting the lighting enhancement operation to only apply to the foreground portrait area. This mechanistically eliminates the edge-blowing artifacts of facial glow and background flooding commonly found in traditional lighting methods, resulting in a natural transition between the foreground and background. The scaling dot product and softmax normalization in the routing computation enable the network to adaptively focus on regions where deep structural signals are significant. Even in extremely low-light environments (1 lux illumination) where the RGB feature signal-to-noise ratio is extremely low, the deep branches can still provide stable spatial guidance, ensuring that attention weights reliably focus on effective face pixels, preventing incorrect lighting and image flicker caused by RGB noise, and significantly improving the image quality stability in complex lighting scenes; The number of feature channels output by the deep structured stream is only 1 / 4 that of the RGB stream, so the dimension of the generated Q matrix is ​​much smaller than that of the K and V matrices, significantly reducing the complexity of attention computation. At the same time, this rule avoids expensive self-attention computation within the RGB stream, and instead achieves multimodal feature fusion with minimal parameters and computational overhead through cross-modal asymmetric mapping, making it particularly suitable for parallel acceleration of mobile NPUs and INT8 quantization deployment.

[0036] The physics-decoupled rendering module includes an intrinsic map decoupling unit and a pixel-level compositing unit; The intrinsic image decoupling unit is used to receive the dual-modal fusion features and performs intrinsic image decomposition on the dual-modal fusion features using a pre-set illumination decoupling decoder structure to extract the reflectance map that does not contain light source information. Receive the scene depth map and the original input image of the scene to be processed, and perform adaptive background light estimation based on the scene depth map: Select the top 0.1% of pixels with the largest depth value from the scene depth map, calculate the average pixel value of each channel of the top 0.1% of pixels in the original input image, and use it as the first candidate background light; A quadtree hierarchical search is used. Each time, the sub-region with the smallest sum of pixel variances in each channel is selected. After n iterations, the average value of each channel in the final region is calculated as the second candidate background light. At the same time, the sum of the red-green difference and the red-blue difference of the pixels is used as the color difference measure. Each time, the sub-region with the largest average color difference is selected. After n iterations, the average value of each channel in the final region is calculated as the third candidate background light. The weights of the first, second, and third candidate background lights are calculated and normalized respectively, and the final background light is obtained by weighted averaging. The depth values ​​of each pixel in the scene depth map are normalized to obtain normalized depth values. The illumination attenuation coefficient of each pixel is calculated based on the normalized depth values. The pixel-wise product of the original input image and the illumination attenuation coefficient is taken as the ambient illumination component. The pixel-wise ratio of the original input image minus the ambient illumination component to the illumination attenuation coefficient is taken as the scene reflection component. Guided filtering edge preservation optimization is performed on the ambient illumination component and the scene reflection component respectively. Finally, a lower limit threshold constraint is applied to the optimized transmittance map. This embodiment uses an atmospheric scattering model to calculate the light attenuation coefficient, as shown in the following formula: ; in, This is the normalized depth value (range [0,1]). The attenuation factor is adaptively adjusted according to the ambient light intensity, with an optimal range of 0.8 to 1.2. A larger value is used in low-light environments to enhance foreground fill light, while a smaller value is used in bright environments to maintain a natural transition. The final background light is used as a global ambient light compensation term. The reflectance map is extracted and linearly fused with the final background light according to the light attenuation coefficient to generate a reflectance map.

[0037] The intrinsic map decoupling unit utilizes a lightweight illumination decoupling decoder pre-installed in the mobile phone's NPU. This decoder consists of several layers of transposed convolutions and residual convergence blocks, performing spatial resolution upsampling to restore the high-dimensional dual-modal fusion features, ultimately outputting an intrinsic reflectance map with the same resolution as the original input image. It receives the scene depth map and the original input image, and dynamically calculates a highly robust global background light vector through three independent and complementary estimation paths. The first candidate background light is obtained based on absolute spatial depth. In the scene depth map, the depth values ​​of all pixels are traversed, and the set of pixel coordinates with the largest depth values ​​in the top 0.1% is retrieved and locked. The pixels corresponding to the set of pixel coordinates in the original input image are located, and their three-channel pixel mean is calculated as the first candidate background light. The second candidate background light is obtained based on a noise-free flat region. The foreground dark image contains a large amount of high-frequency Gaussian noise, and directly calculating the mean is easily affected by noise outliers. It is necessary to find the flattest and most uniform noise-free region locally for background light calibration. Therefore, a quadtree hierarchical search mechanism is adopted: The original input image is divided into four sub-regions on an average basis. The sum of the variances of the pixels in each channel within each sub-region is calculated. The sub-region with the smallest sum of variances is the one with the flattest texture and the highest signal-to-noise ratio. Repeat the above steps iteratively for the sub-region, with the number of iterations being n. In this embodiment, n=4 is preferably set. Calculate the average value of each channel of all pixels in the final leaf node sub-region as the second candidate background light. The third candidate background light is based on the colored light source area with the greatest color difference. Since indoor and outdoor communication often includes biased light sources such as warm-colored desk lamps or colored neon lights, ordinary mean estimation will lead to severe color drift. It is necessary to actively locate the non-white light area with the most significant color difference in the image; Define arbitrary pixels Color difference measurement operator at the location : ; in The pixel value for channel R. The pixel value for channel G. For the pixel value of channel B, a quadtree hierarchical search mechanism is adopted. After each division, the average color difference in the sub-region is calculated. The sub-region with the largest average color difference is selected and further subdivided. After n iterations (preferably n=5 in this embodiment), the average value of each channel in the final converged sub-region is calculated as the third candidate background light. After obtaining the three candidate background lights, the corresponding confidence weights are calculated based on the feature distribution of each region. , and The background light is then normalized and finally synthesized by weighted averaging. : ; in, As the first candidate background light, As the second candidate background light, The third candidate background light; This embodiment was tested on a dataset containing 5000 frames of typical low-light video call footage from a mobile phone's front-facing camera. Three typical adverse scenarios were identified: a low-light, high-noise scenario; an indoor, multi-light source, warm-color-temperature scenario; and an outdoor late-night scenario. Taking the outdoor late-night scenario as an example, in this scenario, the user is outdoors, and the background has a very deep spatial structure, such as streets, distant buildings, and background streetlights. Spatial depth needs to be utilized to filter out local stray light interference. The experimental data is shown in Table 1. Table 1: Comparison of Image Restoration Metrics with Different Weight Combinations in Outdoor Night Scenes Referring to the data in the table above, in the late-night scene, group 2 performed best, with a structural similarity of 0.902. This is because the pixels at the maximum absolute depth can bypass the occlusion of people and noise in the foreground and objectively reflect the global ambient light base of the atmosphere at infinity. CIEDE2000 is the color difference formula value of the International Commission on Illumination. The smaller the value, the smaller the deviation between the facial skin color after relighting and the skin color under true white balance. The structural similarity value ranges from [0,1]. The closer it is to 1, the more complete the edges and structure of the image objects are preserved.

[0038] Using the final background light as a global ambient light compensation term, the pure intrinsic reflectance map parsed from the neural network is... According to the final background light according to the light attenuation coefficient Perform linear fusion to synthesize an enhanced reflectivity map. : ; in, For the gain adjustment coefficient of dark light supplementation, in this embodiment, it is preferred to be... =1.2.

[0039] The pixel-level synthesis unit receives the reflectance map, performs matrix calculation mapping, and generates enhanced images of the core region. The reflectance map is subjected to contrast enhancement and detail sharpening to generate an enhanced image of the core area. Based on the background buffer after Gaussian blur preprocessing, the enhanced image of the core area is merged with the background buffer using a splicing and blending mechanism to generate the final video rendering frame, which is then output to the display terminal.

[0040] The reflectance map's luminance channel undergoes Limit Contrast Adaptive Histogram Equalization (CLAHE), and a 3×3 Laplacian operator is used for high-frequency texture detail sharpening to generate a core region enhanced image. A background buffer layer that has undergone low-computational-power Gaussian blur preprocessing is received. A spatially gradient alpha weight mask is constructed using the face boundary extension area. At the intersection of the core enhanced image and the blurred background buffer, pixel-by-pixel alpha blending is performed. The weight mask has a value of 1.0 inside the core region and 0.0 inside the background buffer. A cosine function is applied to smoothly decay the weight mask within a 10-pixel area on each side of the boundary. This blending mechanism eliminates stitching gaps and transitions at the boundary between the portrait and the blurred background. The resulting final video rendering frame is output to a mobile display terminal for real-time preview and call transmission.

[0041] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0042] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0043] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0044] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this invention should be included within the scope of protection of this invention.

Claims

1. A system for background blurring and light enhancement in video calls on mobile phones, characterized in that, include: The spatiotemporal dynamic determination module is used to receive continuous video streams, calculate the temporal motion vector between the current frame and the previous frame, determine the temporal motion vector, and output the image to be processed. The spatial feature extraction module is used to delineate a core spatial constraint region in the image to be processed using a distance constraint kernel, and to extract key features within the core spatial constraint region using a two-stream depth separable network to generate a corresponding scene depth map. The cross-attention fusion module is used to perform routing calculations on key features using the cross-attention algorithm to generate dual-modal fusion features. The physical decoupled rendering module is used to analyze the dual-modal fusion features into a reflectance map using an intrinsic image decomposition algorithm, and calculate the background light by combining the scene depth map. Based on the reflectance map and the background light, the core area is enhanced to obtain the final video rendering frame.

2. The system for background blurring and light enhancement in video calls on mobile phones as described in claim 1, characterized in that, The spatiotemporal dynamic determination module includes a motion vector estimation unit and a temporal overlap pruning unit; The motion vector estimation unit is used to receive a continuous video stream, extract the current frame and the previous frame from the continuous video stream, calculate the displacement change matrix between the current frame and the previous frame using an optical flow estimation algorithm, and generate a temporal motion vector. The temporal overlap pruning unit is used to receive the temporal motion vector, compare the amplitude of the temporal motion vector with a preset overlap threshold, and output the image to be processed.

3. The system for background blurring and light enhancement in video calls on mobile phones as described in claim 2, characterized in that, The specific process of comparison and determination by the temporal overlap pruning unit includes: If the magnitude of the temporal motion vector is less than or equal to the preset overlap threshold, it is determined that the content of the current frame and the previous frame are highly overlapping, the image rendering parameters of the previous frame are reused, and the subsequent feature extraction steps are skipped. If the magnitude of the temporal motion vector is greater than the preset overlap threshold, it is determined that there is a significant dynamic change in the current frame. The current frame is then extracted and marked as a scene to be processed, and output to the spatial feature extraction module. The preset overlap threshold is dynamically calculated in real time based on the current camera's sensitivity and the motion vector amplitude of the previous N frames. The calculation expression for the overlap threshold is: ； in, To preset the overlap threshold, and These are the historical mean and historical standard deviation of the motion vector magnitudes of the first N frames of video images, respectively. As a time-series adjustment factor, As the baseline sensitivity, This refers to the current light sensitivity of the camera. This is the noise compensation coefficient.

4. The system for background blurring and light enhancement in video calls on mobile phones as described in claim 3, characterized in that, The spatial feature extraction module includes an anchor point positioning and delineation unit and a dual-stream feature encoding unit; The anchor point positioning and delineation unit is used to receive the image to be processed, extract the target visual anchor point in the image to be processed, and construct a distance constraint kernel with the target visual anchor point as the center. The dual-stream feature encoding unit is used to receive the image to be processed after the constraints are defined, and to process the image stream in parallel through a dual-branch network architecture, extracting visible light texture features and depth structure features respectively, and generating a scene depth map.

5. A video call background blurring and light enhancement system for mobile phones as described in claim 4, characterized in that, The specific process of constructing the distance constraint kernel by the anchor point positioning and delineation unit includes: The face detection algorithm is used to locate the face region in the image to be processed as the target visual anchor point; Based on the target visual anchor point, the core spatial constraint area is delineated by extending outward according to the preset spatial expansion coefficient; Pixels in the image to be processed that are outside the core spatial constraint region are marked as background buffers. Gaussian blur is applied directly to the background buffers, and only the core spatial constraint region is retained to enter the dual-stream feature coding unit.

6. A video call background blurring and light enhancement system for mobile phones as described in claim 5, characterized in that, The dual-stream feature coding unit includes a first processing branch and a second processing branch; The first processing branch is used to receive visible light three-channel data within the core spatial constraint area, extract high-frequency color features and edge features through a depthwise separable convolutional network, and generate a first feature, which is a visible light texture feature. The second processing branch is used to receive the initial depth feature map of the single channel within the core spatial constraint area, extract low-frequency foreground contour and depth structure features through downsampling, generate a second feature, which is a depth structure feature, and save the first feature and the second feature as key features.

7. A video call background blurring and light enhancement system for mobile phones as described in claim 6, characterized in that, The cross-attention fusion module includes a cross-modal mapping unit and a routing computation coordination unit; The cross-modal mapping unit is used to receive the key features and generate a query matrix, a key-value matrix, and a numerical matrix through linear projection transformation; The routing computation coordination unit is used to receive the query matrix, key matrix, and numerical matrix, calculate cross-modal response weights using a scaling dot product attention mechanism, and synthesize bimodal fusion features.

8. A video call background blurring and light enhancement system for mobile phones as described in claim 7, characterized in that, The specific rules for the cross-modal mapping unit when generating the matrix are as follows: The deep structural features are mapped into a feature space to generate the query matrix; The visible light texture features are mapped into feature space to generate the key matrix and the numerical matrix; The calculation process of the routing calculation coordination unit includes: calculating the dot product of the query matrix and the transpose of the key matrix, dividing by the dimension scaling factor, and then obtaining the weight distribution matrix through the normalized exponential function; Multiplying the weight distribution matrix with the numerical matrix allows the edge features of the deep structure to guide the response of visible light pixels, thereby generating the dual-modal fusion feature.

9. A video call background blurring and light enhancement system for mobile phones as described in claim 8, characterized in that, The physical decoupling rendering module includes an intrinsic map decoupling unit and a pixel-level synthesis unit; The intrinsic image decoupling unit is used to receive the dual-modal fusion features, and to perform intrinsic image decomposition on the dual-modal fusion features using a preset illumination decoupling decoder structure to extract a reflectance map that does not contain light source information. Receive the scene depth map and the original input image of the scene to be processed, and perform adaptive background light estimation based on the scene depth map: The top 0.1% of pixels with the largest depth value are selected from the scene depth map, and the average pixel value of each channel of the top 0.1% of pixels in the original input image is calculated as the first candidate background light. A quadtree hierarchical search is used. Each time, the sub-region with the smallest sum of pixel variances in each channel is selected. After n iterations, the average value of each channel in the final region is calculated as the second candidate background light. At the same time, the sum of the red-green difference and the red-blue difference of the pixels is used as the color difference measure. Each time, the sub-region with the largest average color difference is selected. After n iterations, the average value of each channel in the final region is calculated as the third candidate background light. The weights of the first candidate background light, the second candidate background light, and the third candidate background light are calculated and normalized respectively, and the final background light is obtained by weighted averaging. The depth values ​​of each pixel in the scene depth map are normalized to obtain normalized depth values. The illumination attenuation coefficient of each pixel is calculated based on the normalized depth values. The pixel-wise product of the original input image and the illumination attenuation coefficient is taken as the ambient illumination component. The pixel-wise ratio of the original input image minus the ambient illumination component to the illumination attenuation coefficient is taken as the scene reflection component. Guided filtering edge preservation optimization is performed on the ambient illumination component and the scene reflection component respectively. Finally, a lower limit threshold constraint is applied to the optimized transmittance map. The final background light is used as a global ambient light compensation term. The reflectance map is extracted and linearly fused with the final background light according to the light attenuation coefficient to generate a reflectance map.

10. A video call background blurring and light enhancement system for mobile phones as described in claim 9, characterized in that, The pixel-level synthesis unit is used to receive the reflectance map, perform matrix calculation mapping, and generate enhanced images of the core region. The reflectance map is subjected to contrast enhancement and detail sharpening processing to generate an enhanced image of the core area. Based on the background buffer after Gaussian blur preprocessing, the enhanced image of the core area is merged with the background buffer using a splicing and blending mechanism to generate the final video rendering frame, which is then output to the display terminal.

Images