An analysis method and device for video call virtual background privacy leakage
By employing multimodal detection and graph neural network reconstruction technologies, the problem of privacy leakage in virtual backgrounds during video calls has been solved. This enables accurate assessment and risk quantification of virtual backgrounds and provides visualization tools to help users understand their privacy protection status.
Patent Information
- Application Number
- CN202611019830.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-09
- Publication Date
- 2026-11-06
AI Technical Summary
Existing virtual background technology for video calls suffers from issues such as pixel blending of the person's edge and the boundary error between the real environment and the virtual background, segmentation delays caused by rapid movement, and a lack of methods to quantify the degree of privacy leakage, making it difficult for users to understand the actual risk of leakage.
We employ a method that combines multimodal leaked pixel detection, temporal aggregation environment reconstruction, and privacy risk quantification. By fusing RGB visual features, depth features, and motion features through a cross-modal attention mechanism, we utilize graph neural networks to aggregate leaked pixels from multiple frames to reconstruct the real background environment and quantify privacy risks.
It achieves robustness to diverse races, lighting conditions, and motion states, provides visual heatmaps and numerical scores to help users understand their privacy protection status, eliminates skin color detection bias, and uses graph neural networks to adapt to different scenarios without the need for manually designed heuristic rules.
Smart Images
Figure CN123052608A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and privacy security assessment technology, specifically to an analysis method and apparatus for privacy leaks in virtual backgrounds during video calls. This method achieves accurate assessment of privacy leaks by detecting leaked pixels in the virtual background, reconstructing the real environment through temporal aggregation, and quantifying privacy risks. It is applicable to scenarios such as video conferencing security auditing and privacy protection system testing. Background Technology
[0002] Video conferencing has become an indispensable tool in the daily operations of many companies and organizations, enabling remote connectivity, reducing commutes, and supporting working from home. Video conferencing for remote work is a valuable form of collaboration.
[0003] Video calling has become a core tool for remote work. However, video calls in a home environment pose serious privacy risks: personal items, documents, and photos in the background may reveal sensitive information such as the user's health status and financial situation.
[0004] Virtual background technology, as a mainstream privacy protection solution, separates a person from their real environment through semantic segmentation and replaces them with a preset background image. However, existing technologies have the following fundamental flaws: ① The pixel blending of the character's edge with the real environment and the virtual background results in segmentation boundary errors, and the cumulative privacy leakage can reach 15-25% of the environment area; ② Rapid movement causes segmentation delay, resulting in transient leakage. For example, a single gesture can leak 0.5-2 seconds of environmental information. ③ There is a lack of methods to quantify the extent of privacy breaches, making it difficult for users to understand the actual risk of breaches.
[0005] Therefore, there is an urgent need for a new generation of virtual background privacy leakage analysis method that does not rely on specific software, aggregates multiple frames in time, and has the ability to quantify risks. Summary of the Invention
[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing an analysis method and apparatus for privacy leakage of virtual backgrounds in video calls.
[0007] The objective of this invention is achieved through the following technical solution: Firstly, this invention provides an analysis method for privacy leaks caused by virtual backgrounds in video calls, comprising the following steps: Step 1: Multimodal Leaked Pixel Detection: Acquire video frame sequences, extract RGB visual features, depth features, and motion features, weighted fuse them to generate a leakage probability map, and process them to obtain a leakage mask; Step 2: Temporal Aggregation Environment Reconstruction: Using leaked pixels as nodes, a spatiotemporal graph is constructed by combining spatial proximity and temporal continuity. The real background environment is reconstructed by aggregating leaked pixels from multiple frames through a graph neural network. Step 3: Quantitative assessment of privacy risks: Calculate the leakage area based on the leakage mask, detect sensitive objects based on the reconstructed real background environment, and generate a privacy leakage risk score by weighted fusion combined with the exposure time.
[0008] Furthermore, in step one, a cross-modal attention mechanism is used to fuse RGB visual features, depth features, and motion features to generate pixel-level leakage probabilities. Figure 3 According to the method described in claim 1, in step one, the leakage probability map is thresholded and morphologically denoised to obtain a binary leakage mask.
[0009] Furthermore, in step two, the spatial coordinates and timestamps of the nodes are obtained, and the edge weights of the spatiotemporal graph structure are calculated based on the spatial scale parameters and the time scale parameters to represent the association strength between the two nodes.
[0010] Furthermore, in step two, the final node features aggregated through graph neural network features are decoded through a transposed convolutional network to generate a high-resolution complete background image and a confidence heatmap.
[0011] Furthermore, in step three, the risk score of privacy leakage is mapped to a color space to generate a risk heatmap superimposed on the original video. The risk heatmap uses a green-yellow-red gradient to map the risk level.
[0012] Secondly, the present invention also provides an analysis system for privacy leaks in virtual backgrounds during video calls, characterized in that the system comprises: The multimodal detection module is used to acquire video frame sequences, extract RGB visual features, depth features and motion features, weighted fuse them to generate a leakage probability map and process them to obtain a leakage mask; The temporal aggregation module is used to construct a spatiotemporal graph by treating leaked pixels as nodes and combining spatial proximity and temporal continuity. It then reconstructs the real background environment by aggregating leaked pixels from multiple frames through a graph neural network. The risk assessment module is used to calculate the leakage area based on the leakage mask, detect sensitive objects based on the reconstructed real background environment, and generate a privacy leakage risk score by weighted fusion combined with the exposure time.
[0013] Thirdly, the present invention also provides an analysis device for privacy leakage of virtual backgrounds in video calls, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the aforementioned analysis method for privacy leakage of virtual backgrounds in video calls.
[0014] Fourthly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for analyzing privacy leaks in virtual backgrounds during video calls.
[0015] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned analysis method for privacy leakage of virtual backgrounds in video calls.
[0016] This invention provides an analysis method for privacy leaks caused by virtual backgrounds in video calls, comprising the following three core steps: (1) Multimodal leak pixel detection: First, input video frame sequence and extract three complementary modal features in parallel: RGB visual features, depth features and motion features to overcome the limitations of single visual information; then, the above three features are adaptively weighted and fused through cross-modal attention mechanism to generate a pixel-level privacy leak probability map; finally, the probability map is thresholded and morphologically denoised to obtain the final binary leak mask.
[0017] (2) Temporal aggregation environment reconstruction: First, the leaked pixels detected in each frame are regarded as nodes of the graph, and edges are constructed according to spatial proximity and temporal continuity to form a spatiotemporal graph G; then, the graph neural network (Transformer) is used to perform message passing on the constructed spatiotemporal graph, aggregate long-range dependency information, and compensate for the lack of information in a single frame; finally, the aggregated high-dimensional features are decoded through a transposed convolutional network to generate a high-resolution complete background image and confidence heatmap.
[0018] (3) Quantitative assessment of privacy risks: First, for the reconstructed environmental images and leaked statistical information, basic risk factors are calculated from three dimensions: area, content and time. Then, the multi-dimensional factors are integrated into a comprehensive risk score of 0-100 by weighted summation. Finally, the risk score is mapped to a color space to generate a risk heat map superimposed on the original video.
[0019] The beneficial effects of this invention are: 1. Eliminates skin color detection bias and is highly robust to diverse races, lighting conditions, and movement states; 2. Graph neural networks adapt to different scenarios, eliminating the need for manually designing heuristic rules; 3. Provides visual heatmaps and numerical ratings to help users understand the status of privacy protection. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is a diagram illustrating the overall technical framework of the present invention; Figure 2 This is a schematic diagram of the multimodal leak pixel detection logic framework of the present invention; Figure 3 This is a diagram illustrating the logic framework for reconstructing the temporal aggregation environment of the present invention. Figure 4 This is a logical framework diagram for the quantitative assessment of privacy risks in this invention; Figure 5 This is a structural diagram of an analysis device for analyzing privacy leaks in virtual backgrounds during video calls, provided by the present invention. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the invention.
[0023] Reference Figure 1 This invention provides an analysis method for privacy leaks caused by virtual backgrounds in video calls, comprising the following steps: (1) Multimodal leak pixel detection: First, video frame sequences are acquired, and three complementary modal features, namely RGB visual features, depth features and motion features, are extracted in parallel to overcome the limitations of single visual information. Then, the above three complementary modal features are adaptively weighted and fused through a cross-modal attention mechanism to generate a pixel-level leak probability map. Finally, the probability map is thresholded and morphologically denoised to obtain the final binary leak mask.
[0024] (2) Temporal aggregation environment reconstruction: First, the leaked pixels detected in each frame are regarded as nodes of a graph. Edges are constructed based on spatial proximity and temporal continuity to form a spatiotemporal graph. G Then, a graph neural network (Transformer) is used to perform message passing on the constructed spatiotemporal graph, aggregating long-range dependency information and compensating for the lack of information in a single frame; finally, the aggregated high-dimensional features ( H aggThe transposed convolutional network is used to decode and reconstruct a high-resolution complete background image and a confidence heatmap.
[0025] (3) Quantitative assessment of privacy risks: First, for the reconstructed complete background image and the leakage statistics based on the leakage mask, the basic risk factors are calculated from three dimensions: area, content and time. Then, the multi-dimensional factors are integrated into a comprehensive risk score of 0-100 by weighted summation. Finally, the risk score is mapped to a color space to generate a risk heat map superimposed on the original video.
[0026] Reference Figure 2 The detailed steps of multimodal leak pixel detection in step (1) are as follows: (1-1) Multimodal Feature Extraction: Video frame sequences are acquired, and three complementary modal features are extracted in parallel to overcome the limitations of single visual information (such as illumination variations and skin color bias). Specifically, this includes: visual features. F v Uses a lightweight CNN to extract texture and color features from RGB images, capturing color anomalies in areas with blurred edges; depth features F d : Generate depth maps using monocular depth estimation networks (such as MiDaS) and extract geometric discontinuities (depth difference between people and background); motion features. F m : Calculate inter-frame motion vectors using an optical flow network (such as OF) to capture segmentation delay regions caused by rapid motion.
[0027] (1-2) Cross-modal attention fusion and probability generation: The three complementary modal features mentioned above are adaptively weighted and fused using a cross-modal attention mechanism to eliminate noise in specific modalities and generate a pixel-level leakage probability map. The privacy leakage probability is calculated using equations (1) and (2): (1) (2) in p These are pixel coordinates; For pixels p The probability range that falls within the privacy zone; It is the sigmoid activation function; Q , K , V These are query, key, and value matrices, respectively, composed of... F v , F d , F mThe importance weights of each modality are obtained through linear transformation and used for dynamic learning. d k This is a scaling factor, equal to the dimension of the key vector, used to prevent gradient vanishing; b This is a bias term.
[0028] (1-3) Binarization mask generation: Threshold segmentation is performed on the probability map, followed by morphological denoising to obtain the final binary leakage mask. B ).when Mask label M leak ( p =1, otherwise 0. The threshold can be dynamically adjusted based on the false alarm rate requirement.
[0029] Reference Figure 3 In step (2), the detailed steps for reconstructing the time-series aggregation environment are as follows: (2-1) Spatiotemporal graph structure construction: Treat the leaked pixels detected in each frame as pixel nodes of the graph ( V p Based on spatial proximity and temporal continuity, edges are constructed to form a spatiotemporal graph G, with spatial / temporal edges ( E s / E t ).node i and j Edge weights between The construction method is as shown in equation (3): (3) in, Represents a node i , j Spatial coordinate vectors in an image; t i Represents a node i The timestamp of the frame to which it belongs; A spatial scale parameter (e.g., 10 pixels) controls the spatial adjacency range. The time scale parameter (e.g., 5 frames) controls the temporal adjacency range. It is a Euclidean distance.
[0030] (2-2) Graph Neural Network Feature Aggregation: Utilizing a Graph Transformer to perform message passing on the constructed spatiotemporal graph, long-range dependency information is aggregated to compensate for missing information in single frames. l Layer nodes i eigenvectors As shown in equation (4): (4) in For the first l Layer nodes j eigenvectors, For nodes i The set of neighboring nodes, For the first l The learnable weight matrix of the layer, For nodes j For nodes i The attention aggregation weight coefficient is calculated by equation (5).
[0031] (5) in, For learnable attention vectors, This is a vector concatenation operation.
[0032] (2-3) Environment Decoding and Reconstruction: The aggregated high-dimensional features are decoded through a transposed convolutional network to generate a high-resolution complete background image and a confidence heatmap, as shown in Equation (6).
[0033] (6) in, For reconstructing a realistic environment image (RGB three channels); This is a confidence heatmap (single channel), representing the confidence level of the reconstructed region. For the decoder network, the parameters are: It typically includes upsampling layers and convolutional layers; These are the final node features after aggregation by a multi-layer graph neural network.
[0034] Reference Figure 4 The detailed steps for the quantitative assessment of privacy risks in step (3) are as follows: (3-1) Multidimensional risk privacy calculation: Calculate basic risk factors from three dimensions: area, content and time.
[0035] ① Leakage area ratio The proportion of leaked pixels to the entire background area is calculated based on the leakage mask statistics, using equation (7): (7) Where T is the total number of video frames. This represents the total number of pixels in the background area of a single frame. For the first t Frame pixels p The leaked mask value.
[0036] ②Score for sensitive objects The sum of confidence scores for detecting sensitive objects (documents, medicines, photos, etc.) in a reconstructed complete background image using an object detection model (such as YOLOv8).
[0037] ③ Exposure duration coefficient This calculates the persistent percentage of high-risk privacy regions within the total video duration. High-risk regions are typically the pixel-level representation of sensitive objects (presented as a heatmap). Specifically, it is obtained by summing the confidence heatmaps (single channels) to obtain the overall privacy risk confidence heatmap. For example, after summing two single-channel heatmaps with high confidence, the color will be darker in the subsequent visualization.
[0038] (3-2) Generation of comprehensive risk score: By using a weighted summation method, the multi-dimensional factors are integrated into a comprehensive risk score of 0-100. The scoring criteria are as shown in formula (8): (8) in, These are the weight coefficients of each factor, and their sum is 1; Here, is a normalization constant, representing the maximum possible sensitive object detection score, used to... Scale to the [0,1] interval.
[0039] (3-3) Visualized Heatmap Mapping: Risk scores are mapped to a color space to generate a risk heatmap overlaid on the original video, visually displaying high-risk areas. (pixels) p The display colors in the heatmap (such as a red-yellow-green gradient) are set as shown in equation (9): (9) in, For pixels p The local risk contribution value (based on its leakage frequency and the density of nearby sensitive objects), max / min are the maximum / minimum values of local risk values in the current video, and Colormap(⋅) is a predefined color mapping function.
[0040] Corresponding to the aforementioned embodiment of an analysis method for privacy leakage of virtual backgrounds in video calls, the present invention also provides an embodiment of an analysis apparatus for privacy leakage of virtual backgrounds in video calls.
[0041] See Figure 5 The present invention provides an analysis device for privacy leakage of virtual background in video calls, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an analysis method for privacy leakage of virtual background in video calls as described in the above embodiments.
[0042] An embodiment of the analysis device for privacy leakage of virtual backgrounds in video calls provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is used for the analysis device for privacy leakage of virtual backgrounds in video calls provided by the present invention. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0043] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0044] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0045] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements an analysis method for privacy leakage of virtual backgrounds in video calls as described in the above embodiments.
[0046] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0047] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned analysis method for privacy leakage of virtual backgrounds in video calls.
[0048] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.
Claims
1. A method for analyzing privacy leakage of a virtual background for a video call, characterized in that, Includes the following steps: Step 1: Multimodal Leaked Pixel Detection: Acquire video frame sequences, extract RGB visual features, depth features, and motion features, weighted fuse them to generate a leakage probability map, and process them to obtain a leakage mask; Step 2: Temporal Aggregation Environment Reconstruction: Using leaked pixels as nodes, a spatiotemporal graph is constructed by combining spatial proximity and temporal continuity. The real background environment is reconstructed by aggregating leaked pixels from multiple frames through a graph neural network. Step 3: Quantitative assessment of privacy risks: Calculate the leakage area based on the leakage mask, detect sensitive objects based on the reconstructed real background environment, and generate a privacy leakage risk score by weighted fusion combined with the exposure time.
2. The method of claim 1, wherein, In step one, a cross-modal attention mechanism is used to fuse RGB visual features, depth features, and motion features to generate a pixel-level leakage probability map.
3. The method of claim 1, wherein, In step one, the leakage probability map is thresholded and morphologically denoised to obtain a binary leakage mask.
4. The method of claim 1, wherein, In step two, the spatial coordinates and timestamps of the nodes are obtained, and the edge weights of the spatiotemporal graph structure are calculated based on the spatial scale parameters and the time scale parameters to represent the association strength between two nodes.
5. The method of claim 1, wherein, In step two, the final node features aggregated through graph neural network features are decoded through a transposed convolutional network to generate a high-resolution complete background image and a confidence heatmap.
6. The method of claim 1, wherein, In step three, the risk score of privacy leakage is mapped to a color space to generate a risk heatmap superimposed on the original video. The risk heatmap uses a green-yellow-red gradient to map the risk level.
7. An analysis system for implementing the analysis method for privacy leakage of virtual backgrounds in video calls as described in any one of claims 1-6, characterized in that, The system includes: The multimodal detection module is used to acquire video frame sequences, extract RGB visual features, depth features and motion features, weighted fuse them to generate a leakage probability map and process them to obtain a leakage mask; The temporal aggregation module is used to construct a spatiotemporal graph by treating leaked pixels as nodes and combining spatial proximity and temporal continuity. It then reconstructs the real background environment by aggregating leaked pixels from multiple frames through a graph neural network. The risk assessment module is used to calculate the leakage area based on the leakage mask, detect sensitive objects based on the reconstructed real background environment, and generate a privacy leakage risk score by weighted fusion combined with the exposure time.
8. An analysis apparatus for privacy leaks in virtual backgrounds during video calls, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an analysis method for privacy leakage of virtual backgrounds in video calls as described in any one of claims 1-6.
9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an analysis method for privacy leakage of virtual backgrounds in video calls as described in any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the analysis method for privacy leakage of virtual background in video calls as described in any one of claims 1-6.