Method and system for blurring background in video conference based on deep learning

Abstract

The present application relates to the field of intelligent image processing, and specifically discloses a method and system for background blurring in video conferencing based on deep learning, which first performs a preliminary positioning of the portrait area of ​​the current video frame to meet the needs of real-time processing. Subsequently, a unique edge refinement and stabilization mechanism is introduced: this mechanism does not process each frame independently, but intelligently uses the refined Alpha mask of the previous frame as an important reference, and combines it with the preliminary segmentation result of the current frame. By combining the image semantic understanding ability and the image edge refinement processing module of prior knowledge, the portrait mask of the current frame is deeply optimized, and finally an Alpha mask of the current frame with clearer edges, richer details and more temporal coherence is generated, and then this mask map is used to generate a background blur image of the current frame.

Description

Technical Field

[0001] The present application relates to the field of intelligent image processing, and more specifically, to a method and system for blurring the background of a video conference based on deep learning. Background Art

[0002] With the prevalence of remote work and online communication, video conferencing has become an indispensable part of daily life and work. During video conferencing, users often want to protect their personal privacy, maintain a professional image, or reduce background interference on the meeting, so the background blur function has become widely popular. By blurring the background behind the user, the visual focus can be placed on the speaker, effectively improving the video conferencing experience and concentration. Traditional background blur methods usually rely on specific hardware devices, such as depth cameras or green screen backgrounds, which are often not available in the daily use scenarios of ordinary users, limiting their wide application. Therefore, developing a background blur solution based on an ordinary camera and implemented through a software algorithm has important practical significance and application value.

[0003] Currently, several deep learning-based solutions for background blurring in video conferencing have been developed. These typically employ a semantic segmentation network (e.g., models based on U-Net and DeepLab architectures) to identify the subject region in a video frame, generate a binary face mask, and then blur the background region based on this mask. While these methods can achieve background blurring to a certain extent, they still face numerous challenges and drawbacks in practical applications. First, existing methods often suffer from suboptimal segmentation accuracy at the edges of subjects, particularly when dealing with complex conditions such as hair, clothing wrinkles, finger gaps, and when the subject and background have similar colors. This can easily lead to jagged edges, loss of detail, or mis-segmentation, resulting in a harsh and unnatural blurring effect. Second, video streams are time-sequential. If the segmentation results between consecutive frames lack stability, even minor jitter or inconsistency can cause the background blur to flicker or jump visually, severely impacting the user experience. Therefore, improving the fineness and temporal stability of background blurring while ensuring real-time performance remains a pressing challenge for current technologies.

[0004] Therefore, an optimized background blur solution for video conferencing is desired. Summary of the Invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a video conferencing background blur method and system based on deep learning, which first performs a preliminary positioning of the portrait area of ​​the current video frame to meet the needs of real-time processing. Subsequently, a unique edge refinement and stabilization mechanism is introduced: this mechanism does not process each frame independently, but intelligently uses the refined Alpha mask of the previous frame as an important reference, and combines it with the preliminary segmentation result of the current frame. By combining the image semantic understanding ability and the image edge refinement processing module of the prior knowledge, the portrait mask of the current frame is deeply optimized, and finally an Alpha mask of the current frame with clearer edges, richer details and more coherent in time is generated, and then this mask map is used to generate the background blur image of the current frame. This design can effectively smooth and correct the edges of the portrait, reduce the flickering and jumping of the background blur effect, thereby improving the visual realism and user experience while ensuring performance.

[0006] According to one aspect of the present application, a method for blurring background in a video conference based on deep learning is provided, which includes:

[0007] Collecting original video conference video stream data through a camera, and extracting a current frame image of the video conference from the original video conference video stream data;

[0008] Performing portrait semantic segmentation on the current frame image of the video conference to obtain a coarse segmentation mask image of the portrait of the current frame of the video conference;

[0009] Acquire a frame image preceding the current frame image of the video conference to obtain a previous frame image of the video conference;

[0010] Performing alpha masking on the previous frame image of the video conference to obtain a previous frame alpha mask image;

[0011] The coarse segmentation mask image of the portrait in the current frame of the video conference and the Alpha mask image in the previous frame are processed by an image edge refinement processing module based on prior information to obtain the Alpha mask image in the current frame;

[0012] Generate a blurred background image based on the current frame image of the video conference;

[0013] The current frame Alpha mask image is used as a weight, the current frame image of the video conference is used as a foreground, and the blurred background image is used as a background, and Alpha blending is performed to obtain a background blurred image of the current frame of the video conference.

[0014] According to another aspect of the present application, a video conferencing background blurring system based on deep learning is provided, comprising:

[0015] A video conference current frame acquisition module is used to collect original video conference video stream data through a camera and extract the video conference current frame image from the original video conference video stream data;

[0016] A portrait semantic segmentation module is used to perform portrait semantic segmentation on the current frame image of the video conference to obtain a coarse segmentation mask image of the portrait of the current frame of the video conference;

[0017] A module for acquiring an image frame before a conference, configured to acquire an image frame before a current image frame of the video conference to obtain an image frame before the video conference;

[0018] A previous frame Alpha mask module, configured to perform Alpha masking on the previous frame image of the video conference to obtain a previous frame Alpha mask image;

[0019] An image edge refinement processing module, configured to process the coarse segmentation mask image of the portrait of the current frame of the video conference and the Alpha mask image of the previous frame through the image edge refinement processing module based on prior information to obtain the Alpha mask image of the current frame;

[0020] A blurred background image generating module, configured to generate a blurred background image based on the current frame image of the video conference;

[0021] The conference background blurred image generation module is used to use the current frame Alpha mask image as a weight, the video conference current frame image as a foreground and the blurred background image as a background, and perform Alpha blending to obtain the video conference current frame background blurred image.

[0022] Compared with the existing technology, the present application provides a method and system for video conferencing background blur based on deep learning, which first performs a preliminary positioning of the portrait area of ​​the current video frame to meet the needs of real-time processing. Subsequently, a unique edge refinement and stabilization mechanism is introduced: this mechanism does not process each frame independently, but intelligently uses the refined Alpha mask of the previous frame as an important reference, and combines it with the preliminary segmentation result of the current frame. By combining the image semantic understanding ability and the image edge refinement processing module of prior knowledge, the portrait mask of the current frame is deeply optimized, and finally an Alpha mask of the current frame with clearer edges, richer details and more temporal coherence is generated, and then this mask map is used to generate the background blur image of the current frame. This design can effectively smooth and correct the edges of the portrait, reduce the flickering and jumping of the background blur effect, thereby improving the visual realism and user experience while ensuring performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 Flowchart of a method for background blurring in video conferencing based on deep learning according to an embodiment of the present application;

[0025] Figure 2 Schematic diagram of data flow of a method for background blurring in video conferencing based on deep learning according to an embodiment of the present application;

[0026] Figure 3 A flowchart of a method for blurring background in a video conference based on deep learning according to an embodiment of the present application, wherein the coarse segmentation mask image of the portrait in the current frame of the video conference and the alpha mask image in the previous frame are processed by an image edge refinement processing module based on prior information to obtain the alpha mask image in the current frame;

[0027] Figure 4 A flowchart of a method for blurring background in a video conference based on deep learning according to an embodiment of the present application, wherein the semantic feature map of the previous frame image of the video conference and the semantic feature vector of the alpha mask of the previous frame are processed by an image semantic joint encoder based on prior information auxiliary optimization to obtain a semantic optimized representation vector of the portrait of the current frame of the video conference;

[0028] Figure 5 4 is a block diagram of a deep learning-based video conferencing background blur system according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0030] As used in this application and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0031] Although the present application makes various references to certain modules in the system according to embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are illustrative only, and different aspects of the system and method can use different modules.

[0032] Flowcharts are used in this application to illustrate the operations performed by the systems according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps may be processed in reverse order or simultaneously, as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0033] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0034] This technical solution aims to build an efficient and natural-looking background blurring solution for video conferencing through deep learning, addressing existing issues such as rough edge processing, loss of detail, and unstable blurring effects across video sequences. Its core approach is to employ a strategy that combines rapid initial segmentation with refined temporal optimization. Specifically, it first performs a rapid preliminary localization of the portrait region within the current video frame to meet real-time processing requirements. The key technical innovation then lies in the introduction of a unique edge refinement and stabilization mechanism: rather than processing each frame independently, this mechanism intelligently utilizes the refined alpha mask of the previous frame as a key reference, combined with the initial segmentation results of the current frame. An "image edge refinement module," incorporating image semantic understanding capabilities and prior knowledge, deeply optimizes the portrait mask of the current frame. This module fuses and learns high-level semantic features from the coarse information of the current frame and the fine information of the previous frame, ultimately generating an alpha mask for the current frame with sharper edges, richer details, and greater temporal coherence. The system then performs a comprehensive video conference background blurring process based on the current frame's alpha mask, the current frame image, and the blurred background image. This design effectively smooths and corrects portrait edges, reducing flickering and jumps in the background blur effect, thereby improving visual realism and user experience while maintaining performance.

[0035] In the technical solution of this application, a video conferencing background blurring method based on deep learning is proposed. Figure 1 Flowchart of a method for blurring background in video conferencing based on deep learning according to an embodiment of the present application. Figure 2Schematic diagram of data flow of the method for blurring the background of a video conference based on deep learning according to an embodiment of the present application. Figure 1 and Figure 2 As shown, the video conference background blurring method based on deep learning according to the embodiment of the present application includes the following steps: S100, collecting original video conference video stream data through a camera, and extracting the current frame image of the video conference from the original video conference video stream data; S200, performing portrait semantic segmentation on the current frame image of the video conference to obtain a coarse segmentation mask image of the current frame portrait of the video conference; S300, obtaining the previous frame image of the current frame image of the video conference to obtain the previous frame image of the video conference; S400, performing Alpha masking on the previous frame image of the video conference to obtain the previous frame image of the video conference; Obtain the previous frame Alpha mask image; S500, pass the coarse segmentation mask image of the portrait of the current frame of the video conference and the previous frame Alpha mask image through the image edge refinement processing module based on prior information to obtain the current frame Alpha mask image; S600, generate a blurred background image based on the current frame image of the video conference; S700, use the current frame Alpha mask image as a weight, the current frame image of the video conference as the foreground and the blurred background image as the background, and perform Alpha blending to obtain the background blurred image of the current frame of the video conference.

[0036] Specifically, in step S100, the original video conference video stream data is collected by the camera, and the current frame image of the video conference is extracted from the original video conference video stream data. It should be understood that the background blur technology is essentially to separate the foreground portrait and background of a single frame image and blur the background area, and the input of the video conference is a continuous dynamic video stream. Therefore, it is necessary to first obtain discrete static image frames one by one from this continuous video stream as the basic unit for subsequent deep learning model processing. Each frame of the image carries the user image and background information at a specific moment, and is the direct object of portrait segmentation, feature extraction and background blur operations.

[0037] More specifically, in a specific example of this application, the camera device is first initialized and accessed. This is typically accomplished through an operating system-provided application programming interface (API) or a standardized multimedia framework to establish a communication link with the physical camera. Secondly, once the camera is activated, it begins continuously outputting raw video data, forming a video stream; the system needs to capture this data stream in real time. Next, because the raw video stream data is often encoded and compressed (e.g., in formats such as H.264 and VP9), it needs to be decoded using a corresponding decoder to restore it to a series of uncompressed or low-compressed image frame data, each frame typically represented as a pixel matrix. Finally, from the decoded continuous image frame sequence, an image frame at a specific moment is selected as the "current video conference frame image" based on the video frame rate (e.g., 25 or 30 frames per second) or processing requirements for subsequent deep learning processing.

[0038] Specifically, in step S200, the portrait semantic segmentation is performed on the current frame image of the video conference to obtain a coarse segmentation mask image of the portrait of the current frame of the video conference. It should be understood that in order to accurately apply a blurring effect to the background without affecting the clarity of the foreground portrait, the system must first understand the image content at the pixel level to clarify which pixels belong to the portrait and which belong to the background. Therefore, in the technical solution of the present application, the portrait semantic segmentation is performed on the current frame image of the video conference to obtain a coarse segmentation mask image of the portrait of the current frame of the video conference. Portrait semantic segmentation can assign a category label (such as "person" or "background") to each pixel in the image, thereby generating a coarse segmentation mask image of the portrait of the current frame of the video conference that identifies the portrait area. This mask image directly guides the subsequent background blurring and alpha blending operations, and is the "blueprint" for distinguishing the foreground from the background.

[0039] More specifically, in an embodiment of the present application, the portrait semantic segmentation is performed on the current frame image of the video conference to obtain a rough segmentation mask image of the portrait of the current frame of the video conference, including: passing the current frame image of the video conference through a portrait semantic segmentation module based on MobileNet to obtain a rough segmentation mask image of the portrait of the current frame of the video conference. Specifically, the obtained "current frame image of the video conference" is first preprocessed, which may include adjusting its size to the input size expected by the MobileNet model (for example, 224x224 or a size with a specific aspect ratio), and performing pixel value normalization operations (for example, scaling the pixel value from the range of 0-255 to the range of 0-1 or -1 to 1). Subsequently, the preprocessed image data is fed into a semantic segmentation network based on the MobileNet architecture. As a lightweight convolutional neural network, MobileNet's core lies in the use of depthwise separable convolution, which can significantly reduce the computational parameters and amount of computation while maintaining good feature extraction capabilities, making it very suitable for portrait segmentation in resource-constrained devices or video conferencing scenarios that require real-time response. The MobileNet backbone extracts deep semantic features from the image. A decoder structure (e.g., a U-Net-like upsampling path or a module that maintains and expands feature map resolution using dilated convolutions, such as the ASPP module in the DeepLab series) is typically connected to it to gradually restore spatial resolution and perform pixel-level classification. The network ultimately outputs a segmentation map of the same size as the input image (or a resized version of the input image). The value of each pixel in the map indicates the probability of belonging to a person or a direct class label (e.g., 1 for person, 0 for background). This output map, known as the "coarse segmentation mask image for the current frame of the video conference," defines the general outline and area of ​​the person, providing a foundation for subsequent refined processing and background blurring. This initial segmentation result, generated using a lightweight model, provides a good starting point for more refined edge processing while meeting real-time requirements.

[0040] Specifically, in step S300 and step S400, the previous frame image of the current frame image of the video conference is obtained to obtain the previous frame image of the video conference, and the previous frame image of the video conference is Alpha masked to obtain the previous frame Alpha mask image. It should be understood that since the content changes between adjacent frames in the video are usually gradual, the processing result of the previous frame, especially its refined portrait mask, contains valuable prior information about the edge, details and shape of the portrait. By introducing the previous frame Alpha mask of the previous frame image of the video conference, it can provide strong guidance for the refinement of the edge of the portrait segmentation of the current frame, which helps to correct the jitter, discontinuity or detail loss problems that may exist in the coarse segmentation of the current frame, thereby generating a background blur effect that is smoother in the time dimension and more visually natural. If the previous frame information is not used, each frame is segmented and blurred independently, and the accumulation of small segmentation errors may cause unnatural flickering or jumping at the edge of the background blur area in the final output video, affecting the user experience.

[0041] More specifically, in a specific example of the present application, first, regarding "obtaining the previous frame image of the current frame image of the video conference to obtain the previous frame image of the video conference", the system needs to maintain a buffer containing at least the previous frame image when processing a continuous video stream. When the system processes the Nth frame (i.e., the current frame), the N-1th frame (i.e., the previous frame image) has been captured and stored in the previous processing cycle. This process is usually performed after the video acquisition module, and a simple data structure (such as a queue or pointer) is used to manage the frame sequence to ensure that when the current frame is processed, the image data of the immediately previous frame is accessible. Secondly, regarding "performing an Alpha mask on the previous frame image of the video conference to obtain an Alpha mask image of the previous frame", the "performing an Alpha mask" here means obtaining or generating a refined Alpha mask corresponding to the previous frame image. Throughout the continuous processing flow, when the "previous frame" (i.e., frame N-1) is processed as the "current frame," the system has already performed a series of operations, including portrait segmentation, edge refinement, and alpha blending, ultimately generating the "current frame alpha mask image" for that frame. This refined "current frame alpha mask image" generated for frame N-1 is then used as the "previous frame alpha mask image" when processing frame N. Therefore, "performing alpha masking" here does not involve re-performing the complete segmentation and mask generation process on the previous frame. Instead, it refers to directly utilizing or retrieving the highly optimized alpha mask generated by the system during the previous frame's processing. This step is essentially a state preservation and transfer mechanism: at the end of each processing cycle, the generated "current frame alpha mask image" is temporarily stored so that it can be used as the "previous frame alpha mask image" in the next processing cycle (when processing the next frame) by the subsequent "prior information-based image edge refinement processing module." If the system is first started or processing the first frame of a video stream, and there is no "previous frame alpha mask image," a default fully transparent or fully opaque mask can be used, or refinement based on the previous frame can be skipped, relying solely on the coarse segmentation results of the current frame. This approach of using the previous frame's refined alpha mask introduces a high-quality temporal prior for edge refinement in the current frame. This prior, itself a product of the previous round of deep learning model processing and optimization, includes a more accurate description of the portrait's edges.

[0042] Specifically, in step S500, the coarse segmentation mask image of the portrait in the current frame of the video conference and the alpha mask image in the previous frame are processed through an image edge refinement processing module based on prior information to obtain an alpha mask image for the current frame. It should be understood that relying solely on the coarse segmentation mask image of the portrait in the current frame of the video conference (e.g., the result quickly generated by a lightweight network such as MobileNet) often fails to strike a balance between speed and accuracy. In particular, jagged edges, such as hair and clothing outlines, are prone to appearing, blurring, or incorrect segmentation, directly affecting the naturalness and realism of the final blur effect. Furthermore, independent segmentation of each frame in the video stream may result in a lack of temporal continuity in the mask, manifesting as jitter or flickering of the blurred edges, severely degrading the user experience. Therefore, to compensate for the lack of coarse segmentation accuracy and introduce temporal information to enhance segmentation stability, in the technical solution of the present application, the coarse segmentation mask image of the portrait in the current frame of the video conference and the alpha mask image in the previous frame are further processed through an image edge refinement processing module based on prior information to obtain an alpha mask image for the current frame. The previous frame's alpha mask image carries the refined edge information of the portrait from the previous moment, constituting valuable "prior information." This step is performed precisely to address the issues of insufficient coarse segmentation mask accuracy and temporal discontinuity. A dedicated module is used to fuse the initial judgment of the current frame with the high-quality results of the previous frame. Specifically, the image edge refinement processing module, based on prior information, intelligently determines which areas of the "current frame's coarse portrait segmentation mask image" need to be "attracted" and corrected by the more precise edge information in the "previous frame's alpha mask image." Rather than simply superimposing or averaging the two masks, this process selectively incorporates and fuses the fine edge details from the previous frame based on prior knowledge (such as edge continuity and smoothness assumptions, as well as the inherent characteristics of human structure), while suppressing noise and errors that may be introduced by the coarse segmentation of the current frame. This process can be seen as a feature decoupling and targeted enhancement of the rough mask of the current frame. The goal is to make the final generated alpha mask of the current frame not only pixel-level accurate within the current frame, but also show good neighborhood correlation universality in the transition with the previous frame, thereby promoting the harmonious and unified extraction of the contextual joint semantics between the character edge and the background blur during subsequent alpha blending.

[0043] Figure 3 The flowchart of the method for blurring the background of a video conference based on deep learning according to an embodiment of the present application is to obtain the current frame Alpha mask image by passing the coarse segmentation mask image of the current frame of the video conference and the Alpha mask image of the previous frame through the image edge refinement processing module based on prior information. Figure 3As shown, according to the deep learning-based video conferencing background blurring method of an embodiment of the present application, step S500 includes: S510, extracting video conferencing semantic features from the coarse segmentation mask image of the current frame portrait of the video conferencing to obtain a semantic feature map of the previous frame image of the video conferencing; S520, extracting the semantic features of the previous frame Alpha mask image from the previous frame Alpha mask image to obtain a previous frame Alpha mask semantic feature vector; S530, passing the semantic feature map of the previous frame image of the video conferencing and the previous frame Alpha mask semantic feature vector through an image semantic joint encoder based on prior information auxiliary optimization to obtain a semantic optimized representation vector of the current frame portrait of the video conferencing; S540, generating the current frame Alpha mask image based on the semantic optimized representation vector of the current frame portrait of the video conferencing.

[0044] Specifically, in step S510, the semantic features of the video conference are extracted from the coarse segmentation mask image of the portrait of the current frame of the video conference to obtain the semantic feature map of the previous frame image of the video conference. It should be understood that when the pixel-level "coarse segmentation mask image of the portrait of the current frame of the video conference" and the "previous frame alpha mask image" are directly used for fusion and refinement, the dimension of the information representation is low, and it is difficult to capture the deep structural and content associations between the two, especially when dealing with complex situations such as inaccurate edges of the coarse segmentation mask, the presence of noise, or large differences from the previous frame fine mask. In order to achieve smarter and more robust edge refinement, it is necessary to elevate the coarse segmentation mask to the semantic feature level so that it contains higher-level abstract information, such as the outline, posture, key parts and other general features of the portrait, rather than just the attribution of pixels. Therefore, in the technical solution of the present application, it is necessary to extract the semantic features of the video conference from the coarse segmentation mask image of the portrait of the current frame of the video conference to obtain the semantic feature map of the previous frame image of the video conference. More specifically, in an embodiment of the present application, the coarse segmentation mask image of the portrait of the current frame of the video conference is passed through a video conference semantic feature extractor based on a deep separable convolutional neural network model to obtain a semantic feature map of the previous frame image of the video conference.

[0045] It's worth noting that the depthwise separable convolutional neural network model used here is suitable for real-time video processing scenarios due to its computational efficiency. It can reduce model complexity while maintaining a certain level of feature extraction capability. The feature extractor learns to map the input coarse mask image into a multi-channel feature map, where each channel corresponds to a different semantic attribute or spatial structural feature of the portrait. This semantic feature map of the previous videoconference frame serves as a key input to the subsequent image-semantic joint encoder, interacting and fusing with the semantic features extracted from the previous frame's alpha mask image at a higher level. This allows for a deeper understanding of the connections and differences between the coarse segmentation results of the current frame and the fine-grained results of the previous frame. This system no longer relies solely on pixel-level matching or simple filtering operations for edge refinement, but instead integrates information at the semantic level, enabling more intelligent utilization of prior information from the previous frame to guide the optimization of the current frame's mask. For example, even if an edge in the coarse segmentation of the current frame has significant deviation, its semantic features may still indicate that the region belongs to a human body part (such as an arm). Combined with the fine-grained semantic features of the corresponding region in the previous frame's alpha mask, the joint encoder can more accurately determine and correct the edges in the current frame. This helps overcome the uncertainty caused by coarse segmentation, thereby improving the stability and realism of the background blur effect.

[0046] Specifically, in step S520, the semantic features of the previous frame Alpha mask image are extracted from the previous frame Alpha mask image to obtain a previous frame Alpha mask semantic feature vector. It should be understood that although the "previous frame Alpha mask image" itself is a refined high-quality result, in order to perform a deeper and more effective fusion and comparison with the semantic features of the "video conference current frame portrait coarse segmentation mask image" in the subsequent "image semantic joint encoder", it is necessary to convert this pixel-level fine mask into an abstract, global semantic representation. In other words, directly comparing or fusing two masks from different sources (one coarse, one fine; one current, one past) at the pixel level may make it difficult to capture their similarities and differences in macroscopic structure and contour morphology, while semantic features can provide this higher-dimensional understanding. Therefore, in the technical solution of the present application, the semantic features of the previous frame Alpha mask image are further extracted from the previous frame Alpha mask image to obtain a previous frame Alpha mask semantic feature vector. More specifically, in this embodiment of the present application, the previous frame alpha mask image is passed through a previous frame alpha mask image semantic feature extractor based on the ViT model to obtain the previous frame alpha mask semantic feature vector. This is achieved by employing a previous frame alpha mask image semantic feature extractor based on the ViT (Vision Transformer) model, leveraging the ViT model's powerful ability to capture global image dependencies and long-range features. ViT views the image as a series of image patches and learns the relationships between these patches through a self-attention mechanism. This effectively extracts the overall morphology and key semantic information from the entire "previous frame alpha mask image" and compresses it into a fixed-length previous frame alpha mask semantic feature vector. This feature vector serves as a "summary" of the previous frame's high-quality segmentation results, carrying strong prior knowledge about the desired morphology of the ideal portrait mask, ready to intelligently interact with information from the current frame.

[0047] Specifically, in step S530, the semantic feature map of the previous video conference frame image and the semantic feature vector of the previous frame alpha mask are passed through an image semantic joint encoder based on prior information-assisted optimization to obtain a semantically optimized representation vector of the portrait of the current video conference frame. It should be understood that the previous steps have already extracted preliminary semantic features from the coarse segmentation mask of the current frame (i.e., the "semantic feature map of the previous video conference frame image," representing the current frame information) and high-quality semantic features from the refined alpha mask of the previous frame (i.e., the "semantic feature vector of the previous frame alpha mask," representing the historical high-quality prior). However, these two features come from different sources and have different accuracies. Simply concatenating or averaging them together will not fully leverage their respective advantages and effectively suppress noise. Therefore, the fusion of these two different information sources requires an intelligent, selective interaction mechanism rather than a blind merger. Based on this, in the technical solution of the present application, the semantic feature map of the previous video conference frame image and the semantic feature vector of the previous frame alpha mask are further passed through an image semantic joint encoder based on prior information-assisted optimization to obtain a semantically optimized representation vector of the portrait of the current video conference frame. This optimized semantic representation vector for the current frame portrait must not only accurately reflect the true semantic boundaries of the portrait in the current video frame, but also leverage the advantages of the previous frame segmentation to maintain temporal continuity and stability. Through processing using a joint image semantic encoder optimized with prior information, it performs complex nonlinear transformations and deeply models the interaction between the two input semantic features. The "previous frame alpha mask semantic feature vector" exerts an "attractive" or "corrective" effect on the local semantics in the "previous frame image semantic feature map," enhancing current frame features that are more consistent and reliable with the previous frame's high-quality priors while suppressing inconsistent or noisy features. In this way, the encoder learns to extract the most essential and reliable combined information from the preliminary semantics of the current frame and the refined semantics of the previous frame. Subsequently, the "current frame alpha mask image" generated based on this optimized semantic representation vector demonstrates superior performance in terms of edge refinement, preservation of details such as hair, and temporal stability during dynamic motion (reducing flicker and jitter). This results in a more natural and professional-looking background blur effect for video conferencing, significantly enhancing the user experience.

[0048] Figure 4 The flowchart of the method for blurring the background of a video conference based on deep learning according to an embodiment of the present application is to obtain the semantic optimization representation vector of the portrait of the current frame of the video conference by passing the semantic feature map of the previous frame image of the video conference and the semantic feature vector of the alpha mask of the previous frame through an image semantic joint encoder based on prior information auxiliary optimization. Figure 4As shown, according to the video conferencing background blurring method based on deep learning according to the embodiment of the present application, step S530 includes: S531, performing feature decoupling on the semantic feature map of the video conferencing front frame image to obtain a set of local semantic feature coding vectors of the video conferencing front frame image; S532, determining a fast interactive set of local semantic feature coding vectors of the video conferencing front frame image based on the feature-like magnetic effect between the semantic feature vector of the front frame alpha mask and each local semantic feature coding vector of the video conferencing front frame image in the set of local semantic feature coding vectors of the video conferencing front frame image; S533, combining the front frame alpha mask semantic feature map with the local semantic feature coding vector of the video conferencing front frame image. The fast interactive set of the frame Alpha mask semantic feature vector and the local semantic feature coding vector of the previous frame image of the video conference is input into a multimodal multi-scale joint encoder based on the LSTM-Transformer hybrid architecture to obtain the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector; S534, the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector are fused to obtain the video conference current frame portrait semantic optimization representation vector.

[0049] More specifically, step S531 performs feature decoupling on the semantic feature map of the frame image before the video conference to obtain a set of local semantic feature encoding vectors of the frame image before the video conference, which is expressed as follows:

[0050]

[0051]

[0052] in, is the semantic feature map of the frame image before the video conference, represents a one-dimensional convolutional layer or a fully connected layer, Indicates the block position index, Indicates feature decoupling, Indicates feature flattening processing, is the set of local semantic feature encoding vectors of the frame image before the video conference, are respectively the first, second, and third in the set of local semantic feature coding vectors of the frame image before the video conference. and The local semantic feature encoding vector of the frame image before the video conference.

[0053] It should be understood that the semantic feature map of the previous videoconference frame image is a relatively holistic feature representation that includes spatial dimension information. However, in order to achieve more detailed and selective interaction and fusion with the global "previous frame alpha mask semantic feature vector" representing the fine segmentation result of the previous frame, this holistic semantic feature map of the previous videoconference frame image needs to be decomposed into smaller, more representative local units. Without decoupling, only a rough global-to-global comparison can be performed, making it difficult to achieve differentiated processing and fine-tuning of different regions of the current frame's coarse segmentation mask. Each local semantic feature encoding vector of the previous videoconference frame image carries the semantic information of the "previous videoconference frame image semantic feature map" (i.e., the semantic representation of the current frame's coarse segmentation mask) in a specific local region. Through this decoupling, for example, by dividing and encoding along the channel dimension or spatial dimension, a complex spatial structure feature map can be converted into a set of discrete, fixed-dimensional local feature vectors. This is done to allow the global "previous frame alpha mask semantic feature vector" to interact with the feature vectors representing different local regions of the current frame one by one, thereby quantifying the "feature magnetic attraction value" or the strength of the correlation between them. This decoupling process is the prerequisite for subsequent selective information fusion and fine-tuning, laying the foundation for determining which local regions of the current frame's coarse segmentation are more "attractive" or consistent with the high-quality prior of the previous frame.

[0054] Accordingly, according to an embodiment of the present application, step S532, based on the characteristic class magnetic effect between the previous frame Alpha mask semantic feature vector and each video conference previous frame image local semantic feature coding vector in the set of the video conference previous frame image local semantic feature coding vector, determines the fast interactive set of the video conference previous frame image local semantic feature coding vector, including: calculating the class magnetic characteristic parameters between the previous frame Alpha mask semantic feature vector and each video conference previous frame image local semantic feature coding vector in the set of the video conference previous frame image local semantic feature coding vector to obtain a set of video conference current frame-previous frame class magnetic characteristic parameters; based on the comparison between the set of the video conference current frame-previous frame class magnetic characteristic parameters and a preset threshold, extracting the fast interactive set of the video conference previous frame image local semantic feature coding vector from the set of the video conference previous frame image local semantic feature coding vector.

[0055] More specifically, the magnetic attraction characteristic parameters between the previous frame Alpha mask semantic feature vector and each of the local semantic feature coding vectors of the previous frame image of the video conference are calculated to obtain a set of magnetic attraction characteristic parameters of the current frame of the video conference - the previous frame, which is expressed as follows:

[0056]

[0057] ;

[0058]

[0059] in, represents the two-norm of the vector, is the previous frame Alpha mask semantic feature vector, is the trainable permutation weight matrix, is the trainable modulation weight vector, for and The magnetic attraction characteristic parameters of the current frame and the previous frame of the video conference between It means extracting the maximum value in the vector. It means extracting the minimum value in the vector. It means to calculate the average value of the vector. for The autocorrelation characteristic parameter, for The autocorrelation characteristic parameters.

[0060] It should be understood that the previous frame Alpha mask semantic feature vector represents the global semantic fingerprint of the previous frame fine segmentation result, and the set of local semantic feature encoding vectors of the previous frame image of the video conference represents the semantic description of the current frame coarse segmentation result in different local areas. In order to achieve truly intelligent fusion and optimization, rather than simply imposing the previous frame information on the current frame, the system must first evaluate the correlation strength or similarity between each local semantic feature of the current frame and the global semantic prior of the previous frame. Without this quantitative evaluation, the subsequent fusion optimization will be blind and may not be able to effectively distinguish which local information in the coarse segmentation of the current frame is reliable and which needs to be focused on by the fine information of the previous frame. Therefore, in the technical solution of the present application, the magnetic attraction characteristic parameter between the previous frame Alpha mask semantic feature vector and each local semantic feature encoding vector of the previous frame image of the video conference in the set of local semantic feature encoding vectors of the previous frame image of the video conference is further calculated. This magnetic attraction characteristic parameter is intended to quantify the "attraction" or semantic relevance between the semantic information of each local area of ​​the coarse segmentation mask of the current frame and the "previous frame Alpha mask semantic feature vector" representing the high-quality segmentation result. In this scenario, a high "magnetic attraction characteristic parameter" means that the coarse segmentation result of the local area in the current frame is highly consistent with the fine segmentation result of the previous frame, indicating high credibility. Conversely, a low parameter value may indicate that the coarse segmentation of the local area deviates significantly or does not meet the ideal state, requiring further adjustment based on information from the previous frame. Therefore, generating this set of parameters is a key prerequisite for subsequent selective fusion and weighted optimization, providing a basis for intelligent decision-making.

[0061] More specifically, based on the comparison between the set of magnetic attraction characteristic parameters of the current frame and the previous frame of the video conference and a preset threshold, a fast interactive set of local semantic feature coding vectors of the previous frame image of the video conference is extracted from the set of local semantic feature coding vectors of the previous frame image of the video conference, which is expressed as follows:

[0062] ;

[0063] ;

[0064] in, is the predetermined threshold, A fast interactive collection of local semantic feature encoding vectors for the preceding frame image of a video conference. They are the first, second and third in the fast interaction set of local semantic feature encoding vectors of the previous frame image of the video conference. The local semantic feature encoding vector of the frame image before the video conference.

[0065] It should be understood that after calculating the "magnetic-like characteristic parameters" between each local semantic area of ​​the current frame and the high-quality global semantics of the previous frame, it is clear that not all local areas of the current frame coarse segmentation mask are equally important or have the same credibility for subsequent optimization. Some areas may be highly consistent with the previous frame and have higher quality, while other areas may be very different from the previous frame or contain significant noise due to motion, occlusion or the limitations of the coarse segmentation itself. If all local semantic features are indiscriminately sent to the subsequent complex "image semantic joint encoder" for processing, it will not only greatly increase the computational burden and affect real-time performance, but may also interfere with the optimization process by introducing too much low-quality or irrelevant information, and even reduce the final segmentation accuracy. Therefore, in the technical solution of the present application, based on the comparison between the set of magnetic-like characteristic parameters of the current frame-previous frame of the video conference and a preset threshold, a fast interactive set of local semantic feature coding vectors of the previous frame image of the video conference is extracted from the set of local semantic feature coding vectors of the previous frame image of the video conference. This fast interactive set only includes local semantic feature vectors whose corresponding "magnetic-like characteristic parameters" are above a preset threshold. These are local regions of the current frame that are preliminarily judged to be semantically "attractive" or highly correlated with the high-quality mask of the previous frame. This screening process, performed by setting a threshold, aims to remove local semantic information that has a weak prior correlation with the previous frame and may contain noise or errors. This allows the subsequent optimization module to focus its attention on the regions preliminarily judged to be most relevant, ensuring that the joint encoder focuses on the features most likely to benefit from the previous frame's information and contribute to high-quality segmentation results.

[0066] Accordingly, according to an embodiment of the present application, step S533 inputs the fast interaction set of the previous frame Alpha mask semantic feature vector and the local semantic feature coding vector of the previous frame image of the video conference into a multimodal multi-scale joint encoder based on the LSTM-Transformer hybrid architecture to obtain the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector, including: performing universal correlation correction on each video conference previous frame image local semantic feature coding vector in the fast interaction set of the local semantic feature coding vector of the previous frame image of the video conference to obtain a fast interaction set of the corrected local semantic feature coding vector of the previous frame image of the video conference; inputs the fast interaction set of the previous frame Alpha mask semantic feature vector and the corrected local semantic feature coding vector of the previous frame image of the video conference into a multimodal multi-scale joint encoder based on the LSTM-Transformer hybrid architecture to obtain the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector.

[0067] More specifically, universal correlation correction is performed on each local semantic feature coding vector of the video conferencing frame image before the video conferencing in the fast interaction set of local semantic feature coding vectors of the video conferencing frame image before the video conferencing to obtain a fast interaction set of local semantic feature coding vectors of the video conferencing frame image before the video conferencing. It should be understood that the local semantic feature coding vector of the video conferencing frame image before the video conferencing in the fast interaction set of local semantic feature coding vectors of the video conferencing frame image before the video conferencing In the global cross-modal interaction field, it is essentially based on the semantic feature vector of the previous frame Alpha mask As the core of the neighborhood area, then, to ensure that the video conference before the local semantic feature encoding vector of the frame image and the semantic feature vector of the previous frame Alpha mask While the neighbor association characteristics are also expected, the local semantic feature encoding vector of the frame image before the video conference is also expected to be They can have the universality of proximity association, thus promoting the extraction of subsequent context fusion semantics.

[0068] The first step is to Corresponding magnetic characteristic parameters , calculate its difference with the threshold The difference , and in On the interval of Integrate so that each vector The boundary properties of the whole vector The global system property of It has universality in the global sense, free from interference from micro factors:

[0069]

[0070] Then, the universal correlation structure of the polynomial solution of the single perturbation is used to modify the local semantic feature encoding vector of each frame image before the video conference. :

[0071]

[0072] in, Indicates the The local semantic feature encoding vector of the previous frame image of the modified video conference.

[0073] That is, before maintaining each video conference, the local semantic feature encoding vector of the frame image As an autonomous distribution, the universal correlation structure is determined by algebraic equations, so as to realize the local semantic feature encoding vector of the frame image before the video conference with the weak correlation stability of the association metric. The universality of the neighbor association between them will help to extract the subsequent context fusion semantics.

[0074] More specifically, the fast interactive set of the previous frame Alpha mask semantic feature vector and the corrected video conference previous frame image local semantic feature coding vector is input into a multimodal multi-scale joint encoder based on an LSTM-Transformer hybrid architecture to obtain the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector, which can be expressed as follows:

[0075]

[0076]

[0077] in, is the local context semantic joint encoding vector of the current frame and the previous frame of the video conference, is the global context semantic joint encoding vector of the current frame and the previous frame of the video conference, It is a multi-head attention mechanism.

[0078] It should be understood that after the aforementioned steps, we have obtained a global fingerprint representing the high-quality segmentation result of the previous frame (i.e., the "previous frame alpha mask semantic feature vector") and a set of semantic information representing key local regions of the current frame, which has been filtered and corrected for common relevance (i.e., the "rapid interactive set of corrected local semantic feature encoding vectors of the previous video conference frame image"). At this point, although the local features of the current frame have undergone preliminary optimization and screening, achieving the most accurate and robust portrait segmentation in the current frame requires deep, multi-dimensional interaction and fusion of these two information streams—global information representing accurate past experience and local information representing current optimization but still requiring further integration. Simply concatenating or shallowly fusing them fails to fully exploit the complex dependencies and complementarities between them, and thus fails to maximize the quality of the segmentation mask for the current frame. Therefore, the technical solution of this application utilizes a "multimodal and multi-scale joint encoder based on a hybrid LSTM-Transformer architecture" to deeply process the two input features to generate two key outputs: a "joint semantic encoding vector of the current frame and the previous frame's local context," which precisely captures the relationship between the local region of the current frame and its context and the priors of the previous frame; and a "joint semantic encoding vector of the current frame and the previous frame's global context," which integrates all information from a more macroscopic perspective to form a comprehensive understanding of the overall morphology and temporal consistency of the portrait in the current frame. LSTM (Long Short-Term Memory) networks excel at capturing local, ordered contextual dependencies in sequential data. Even if the input "rapid interaction set" itself may not be strictly ordered, LSTMs can learn short-term associations between its local features. The core self-attention mechanism of the Transformer effectively captures long-range dependencies between any elements in the sequence, achieving global context awareness and powerfully modeling the complex interactions between the global prior of the "previous frame alpha mask semantic feature vector" and each "corrected local semantic feature encoding vector." Through this hybrid architecture, the encoder aims to leverage the local, fine-grained modeling capabilities of LSTM and the global, cross-modal (here, information about the current frame and previous frames) interaction modeling capabilities of the Transformer. It performs complex nonlinear transformations and interaction modeling on the filtered and corrected key information, learning deep semantic associations between modalities (or between previous and next frames in the temporal dimension). The encoder is designed to simultaneously output encoding results reflecting interactive information at different granularities. These vectors serve as the direct basis for generating the final "current frame alpha mask image," improving mask accuracy, reducing edge jaggedness and artifacts, and effectively suppressing flickering and jumps in the background blur effect. This results in a natural, soft, stable, and smooth background blur in video conferences, significantly enhancing the user's visual experience and professional image.

[0079] More specifically, according to an embodiment of the present application, step S534 is to fuse the local context semantic joint coding vector of the current frame of the video conference and the previous frame and the global context semantic joint coding vector of the current frame of the video conference and the previous frame to obtain the semantic optimization representation vector of the portrait of the current frame of the video conference, which is expressed as follows:

[0080]

[0081] in, and are the fusion weight matrix and fusion bias vector respectively, represents vector concatenation, Represents the semantically optimized representation vector of the portrait in the current frame of the video conference.

[0082] It should be understood that while the previous step successfully extracted and generated two valuable but differently focused semantic representations from different perspectives—one focusing on local details and context (local context semantic joint encoding vector), the other on overall structure and global consistency (global context semantic joint encoding vector)—while each of these two vectors carries rich information, effectively integrating these two different scales of information is necessary to form the most comprehensive and accurate final semantic judgment of the current frame portrait and, accordingly, generate a high-quality alpha mask. Therefore, the video conference current frame-previous frame local context semantic joint encoding vector and the video conference current frame-previous frame global context semantic joint encoding vector are further fused to obtain the video conference current frame portrait semantic optimized representation vector. This final video conference current frame portrait semantic optimized representation vector aims to achieve the most accurate and robust depiction of the current frame portrait semantic boundary by organically combining the accuracy of local details with the stability of global structure. This fusion is not a simple splicing or averaging, but a more intelligent weighted combination, attention mechanism fusion or other advanced feature fusion techniques. The purpose is to maximize the advantages of the two input vectors while suppressing their possible individual noise or deviations, thereby obtaining a highly condensed semantically optimized representation that is most instructive for subsequent Alpha mask generation.

[0083] Specifically, in step S540, the current frame Alpha mask image is generated based on the semantic optimization representation vector of the portrait of the current frame of the video conference. It should be understood that although the "semantic optimization representation vector of the portrait of the current frame of the video conference" contains all the high-level abstract understandings about the portrait contour, details, and correlation with the previous frame, it itself is not a pixel-level mask that can be directly used for image processing. It needs to be "decoded" or "translated" into an image with the same size as the original video frame, and each pixel indicates the probability (i.e., Alpha value) of belonging to the foreground (portrait) or background. Therefore, in the technical solution of the present application, the current frame Alpha mask image is further generated. In particular, in an embodiment of the present application, based on the semantic optimization representation vector of the portrait of the current frame of the video conference, the current frame Alpha mask image is generated, including: passing the semantic optimization representation vector of the portrait of the current frame of the video conference through an AIGC-based Alpha mask image generator to obtain the current frame Alpha mask image. The "AIGC" (AI Generated Content) here utilizes advanced generative AI techniques, such as complex decoder networks, the generator portion of generative adversarial networks (GANs), and the reverse process of diffusion models. Its purpose is not simply to convert vectors into images, but to leverage the AIGC model's powerful generation and detail restoration capabilities to ensure that the generated alpha mask not only aligns with the semantic vector in terms of macroscopic contours, but also achieves extremely high fidelity and precision in microscopic details (such as hair edges, semi-transparent areas, and the edges of complex clothing textures). This aims to overcome the problems of rough edges and loss of detail that can occur with traditional segmentation methods, generating an alpha channel map that closely matches real-world occlusion relationships, with smooth transitions and fine boundaries.

[0084] Specifically, in step S600, a blurred background image is generated based on the current frame image of the video conference. It should be understood that if a preset static blurred image or an image unrelated to the current video content is directly used as the background, it is very easy to cause problems such as color imbalance and lighting mismatch, resulting in a stiff and unnatural synthesis effect, which seriously affects the user experience. By blurring the current frame image of the video conference itself to create the background, it can be ensured that the blurred background remains consistent with the foreground character in terms of color tone, brightness and ambient light reflection. This is the key prerequisite for achieving a seamless and natural background blur effect. Even if the foreground character is accurately segmented, a blurred background that is inconsistent with the current scene will greatly reduce the overall visual effect.

[0085] More specifically, in one example of this application, image filtering technology is employed. First, a complete "current video conference frame image" is obtained as input. Next, one or more blur filters are applied to this image. The most commonly used is Gaussian Blur. In practice, this involves performing a two-dimensional convolution operation on the image with a Gaussian kernel. The kernel's parameters (such as kernel size and standard deviation σ) determine the degree and range of the blur. A larger standard deviation results in a stronger blur effect, but also results in greater loss of image detail. Gaussian Blur produces a smooth and natural blur effect, consistent with the human eye's visual perception of out-of-focus scenes. In addition to Gaussian Blur, other methods such as Mean Blur (Box Blur) or Median Blur can also be used. The former blurs by calculating the average of neighboring pixels, while the latter takes the median of neighboring pixels. While Gaussian Blur effectively suppresses salt-and-pepper noise, it is preferred for background blurring due to its smooth and natural appearance. These filtering operations can be efficiently implemented using built-in functions in mature computer vision libraries (such as OpenCV and PIL / Pillow). Developers only need to specify the input image and blur parameters to obtain the blurred image.

[0086] Specifically, in step S700, the current frame Alpha mask image is used as a weight, the current frame image of the video conference is used as the foreground, and the blurred background image is used as the background, and Alpha blending is performed to obtain a blurred background image of the current frame of the video conference. It should be understood that Alpha blending uses the pixel-by-pixel weight information provided by the Alpha mask to determine to what extent each pixel in the final image should display the foreground content and to what extent the background content should be displayed, so as to generate a seamlessly integrated "current frame blurred background image of the video conference" with a sense of depth and visual focus (i.e., foreground characters). Specifically, its goal is to use the Alpha mask as a guide to maintain the original clarity and details of the area identified as the foreground in the original video frame (mainly the participants), and to smoothly and naturally superimpose it on the previously generated blurred background image. The change in grayscale value in the alpha mask (usually from 0 representing full background to 1 representing full foreground) allows for a soft transition at the edge of the foreground and background, which is crucial for dealing with complex boundaries such as hair and translucent objects, thereby avoiding a harsh "cut and paste" feeling, making the final synthesized image look more realistic and credible, and improving the overall visual quality and professionalism.

[0087] More specifically, in one example of this application, an "over" operation based on the classic Porter-Duff compositing equation or direct linear interpolation is employed. First, the system ensures that the "current frame alpha mask image" (whose pixel value is α, normalized to the interval [0, 1]), the "current video conference frame image" (serving as the foreground image F), and the "blurred background image" (serving as the background image B) are spatially aligned, meaning they have the same size and pixel correspondence. Next, for each pixel (x, y) in the output image (i.e., the "current video conference frame blurred background image" O), its color value O(x, y) is calculated using the following formula: O(x, y) = α(x, y) * F(x, y) + (1 - α(x, y)) * B(x, y). This calculation is performed independently for each color channel of the image (e.g., R, G, and B channels). When α(x,y) is 1, the output pixel is taken entirely from the foreground image F(x,y); when α(x,y) is 0, the output pixel is taken entirely from the background image B(x,y); when α(x,y) is between 0 and 1, the output pixel is a weighted average of the foreground and background pixel colors, achieving a smooth transition. This operation can be efficiently performed using functions provided by standard image processing libraries (such as OpenCV and Pillow), which have already optimized this type of pixel-level operation.

[0088] In summary, according to the embodiment of the present application, a method for background blurring of a video conference based on deep learning is explained, which first performs a preliminary positioning of the portrait area of ​​the current video frame quickly to meet the needs of real-time processing. Subsequently, a unique edge refinement and stabilization mechanism is introduced: this mechanism does not process each frame independently, but intelligently uses the refined Alpha mask of the previous frame as an important reference, and combines it with the preliminary segmentation result of the current frame. By combining the image semantic understanding ability and the image edge refinement processing module of the prior knowledge, the portrait mask of the current frame is deeply optimized, and finally an Alpha mask of the current frame with clearer edges, richer details and more temporal coherence is generated, and then this mask map is used to generate the background blur image of the current frame. This design can effectively smooth and correct the edges of the portrait, reduce the flickering and jumping of the background blur effect, thereby improving the visual realism and user experience while ensuring performance.

[0089] Furthermore, a video conferencing background blur system based on deep learning is also provided.

[0090] Figure 5 FIG is a block diagram of a video conferencing background blurring system based on deep learning according to an embodiment of the present application. Figure 5As shown, according to the embodiment of the present application, the video conference background blurring system 500 based on deep learning includes: a video conference current frame acquisition module 510, which always collects the original video conference video stream data through the camera and extracts the video conference current frame image from the original video conference video stream data; a portrait semantic segmentation module 520, which is used to perform portrait semantic segmentation on the video conference current frame image to obtain a video conference current frame portrait coarse segmentation mask image; a conference previous frame image acquisition module 530, which is used to obtain the previous frame image of the video conference current frame image to obtain the video conference previous frame image; a previous frame alpha mask module 540, which is used to perform alpha masking on the video conference previous frame image ha mask to obtain the previous frame Alpha mask image; an image edge refinement processing module 550, used to pass the video conference current frame portrait coarse segmentation mask image and the previous frame Alpha mask image through the image edge refinement processing module based on prior information to obtain the current frame Alpha mask image; a blurred background image generation module 560, used to generate a blurred background image based on the video conference current frame image; a conference background blurred image generation module 570, always uses the current frame Alpha mask image as a weight, takes the video conference current frame image as the foreground and the blurred background image as the background, and performs Alpha blending to obtain the video conference current frame background blurred image.

[0091] As described above, the video conferencing background blur system 500 based on deep learning according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a video conferencing background blur algorithm based on deep learning. In one possible implementation, the video conferencing background blur system 500 based on deep learning according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the video conferencing background blur system 500 based on deep learning can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the video conferencing background blur system 500 based on deep learning can also be one of the many hardware modules of the wireless terminal.

[0092] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A video conferencing background blurring method based on deep learning, characterized in that: include: Collecting original video conference video stream data through a camera, and extracting a current frame image of the video conference from the original video conference video stream data; Performing portrait semantic segmentation on the current frame image of the video conference to obtain a coarse segmentation mask image of the portrait of the current frame of the video conference; Acquire a frame image preceding the current frame image of the video conference to obtain a previous frame image of the video conference; Performing alpha masking on the previous frame image of the video conference to obtain a previous frame alpha mask image; The coarse segmentation mask image of the portrait of the current frame of the video conference and the Alpha mask image of the previous frame are processed through an image edge refinement processing module based on prior information to obtain the Alpha mask image of the current frame of the video conference, including: extracting the semantic features of the video conference from the coarse segmentation mask image of the portrait of the current frame of the video conference to obtain the semantic feature map of the previous frame image of the video conference; extracting the semantic features of the previous frame Alpha mask image from the previous frame Alpha mask image to obtain the semantic feature vector of the previous frame Alpha mask; performing feature decoupling on the semantic feature map of the previous frame image of the video conference to obtain a set of local semantic feature coding vectors of the previous frame image of the video conference; based on the relationship between the local semantic feature coding vectors of the previous frame Alpha mask semantic feature vector and the local semantic feature coding vectors of the previous frame image of the video conference The characteristic class magnetic effect is used to determine the fast interactive set of local semantic feature coding vectors of the previous frame image of the video conference; the previous frame Alpha mask semantic feature vector and the fast interactive set of local semantic feature coding vectors of the previous frame image of the video conference are input into the multimodal multi-scale joint encoder based on the LSTM-Transformer hybrid architecture to obtain the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector; the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector are fused to obtain the video conference current frame portrait semantic optimization representation vector; based on the video conference current frame portrait semantic optimization representation vector, the current frame Alpha mask image is generated; Generate a blurred background image based on the current frame image of the video conference; The current frame Alpha mask image is used as a weight, the current frame image of the video conference is used as a foreground, and the blurred background image is used as a background, and Alpha blending is performed to obtain a background blurred image of the current frame of the video conference.

2. The method for blurring background in video conferencing based on deep learning according to claim 1, characterized in that: Performing portrait semantic segmentation on the current frame image of the video conference to obtain a coarse segmentation mask image of the portrait of the current frame of the video conference, including: passing the current frame image of the video conference through a portrait semantic segmentation module based on MobileNet to obtain the coarse segmentation mask image of the portrait of the current frame of the video conference.

3. The method for blurring background in video conferencing based on deep learning according to claim 2, characterized in that: Extracting video conference semantic features from the coarse segmentation mask image of the current frame portrait of the video conference to obtain a semantic feature map of the previous frame image of the video conference, including: passing the coarse segmentation mask image of the current frame portrait of the video conference through a video conference semantic feature extractor based on a deep separable convolutional neural network model to obtain the semantic feature map of the previous frame image of the video conference.

4. The method for blurring background in video conferencing based on deep learning according to claim 3, characterized in that: Extracting the semantic features of the previous frame Alpha mask image from the previous frame Alpha mask image to obtain a previous frame Alpha mask semantic feature vector, including: passing the previous frame Alpha mask image through a previous frame Alpha mask image semantic feature extractor based on a ViT model to obtain the previous frame Alpha mask semantic feature vector.

5. The method for blurring background in video conferencing based on deep learning according to claim 4, characterized in that: Determining a fast interactive set of local semantic feature coding vectors of the video conference front frame image based on a feature-like magnetic attraction effect between the previous frame alpha mask semantic feature vector and each local semantic feature coding vector of the video conference front frame image in the set of local semantic feature coding vectors of the video conference front frame image includes: Calculating the class magnetic attraction characteristic parameters between the previous frame Alpha mask semantic feature vector and each video conference previous frame image local semantic feature coding vector in the set of the video conference previous frame image local semantic feature coding vector to obtain a set of video conference current frame-previous frame class magnetic attraction characteristic parameters; Based on the comparison between the set of magnetic attraction characteristic parameters of the current frame-previous frame of the video conference and a preset threshold, a fast interactive set of local semantic feature coding vectors of the previous frame image of the video conference is extracted from the set of local semantic feature coding vectors of the previous frame image of the video conference.

6. The method for blurring background in video conferencing based on deep learning according to claim 5, characterized in that: Inputting the fast interactive set of the previous frame Alpha mask semantic feature vector and the local semantic feature coding vector of the previous frame image of the video conference into a multimodal multi-scale joint encoder based on an LSTM-Transformer hybrid architecture to obtain a video conference current frame-previous frame local context semantic joint coding vector and a video conference current frame-previous frame global context semantic joint coding vector, including: Performing universal correlation correction on each local semantic feature coding vector of the pre-video conference frame image in the fast interactive set of local semantic feature coding vectors of the pre-video conference frame image to obtain a corrected fast interactive set of local semantic feature coding vectors of the pre-video conference frame image; The fast interactive set of the previous frame Alpha mask semantic feature vector and the corrected video conference previous frame image local semantic feature coding vector is input into a multimodal multi-scale joint encoder based on the LSTM-Transformer hybrid architecture to obtain the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector.

7. The method for blurring background in video conferencing based on deep learning according to claim 6, characterized in that: Based on the semantic optimization representation vector of the portrait of the current frame of the video conference, the current frame Alpha mask image is generated, including: passing the semantic optimization representation vector of the portrait of the current frame of the video conference through an Alpha mask image generator based on AIGC to obtain the current frame Alpha mask image.

8. A video conferencing background blur system based on deep learning, characterized by: include: A video conference current frame acquisition module is used to collect original video conference video stream data through a camera and extract the video conference current frame image from the original video conference video stream data; A portrait semantic segmentation module is used to perform portrait semantic segmentation on the current frame image of the video conference to obtain a coarse segmentation mask image of the portrait of the current frame of the video conference; A module for acquiring an image frame before a conference, configured to acquire an image frame before a current image frame of the video conference to obtain an image frame before the video conference; A previous frame Alpha mask module, configured to perform Alpha masking on the previous frame image of the video conference to obtain a previous frame Alpha mask image; The image edge refinement processing module is used to process the coarse segmentation mask image of the current frame portrait of the video conference and the alpha mask image of the previous frame through the image edge refinement processing module based on prior information to obtain the current frame alpha mask image, including: extracting the semantic features of the video conference from the coarse segmentation mask image of the current frame portrait of the video conference to obtain the semantic feature map of the previous frame image of the video conference; extracting the semantic features of the previous frame alpha mask image from the previous frame alpha mask image to obtain the semantic feature vector of the previous frame alpha mask; performing feature decoupling on the semantic feature map of the previous frame image of the video conference to obtain a set of local semantic feature coding vectors of the local semantic feature coding vectors of the previous frame image of the video conference; and extracting the semantic features of the previous frame image of the video conference based on the local semantic feature coding vectors of the previous frame alpha mask and the previous frame image of the video conference. The feature-like magnetic effect between the feature coding vectors is used to determine the fast interactive set of local semantic feature coding vectors of the previous frame image of the video conference; the previous frame Alpha mask semantic feature vector and the fast interactive set of local semantic feature coding vectors of the previous frame image of the video conference are input into the multimodal multi-scale joint encoder based on the LSTM-Transformer hybrid architecture to obtain the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector; the video conference current frame-previous frame local context semantic joint coding vector and the video conference current frame-previous frame global context semantic joint coding vector are fused to obtain the video conference current frame portrait semantic optimization representation vector; based on the video conference current frame portrait semantic optimization representation vector, the current frame Alpha mask image is generated; A blurred background image generating module, configured to generate a blurred background image based on the current frame image of the video conference; The conference background blurred image generation module is used to use the current frame Alpha mask image as a weight, the video conference current frame image as a foreground and the blurred background image as a background, and perform Alpha blending to obtain the video conference current frame background blurred image.

Images