A GPU accelerated green screen real-time matting and spill suppression method for XR live streaming

Abstract

The application discloses a GPU accelerated green screen real-time matting and spill suppression method for XR live streaming, and belongs to the technical field of XR live streaming, digital matting and real-time virtual-real synthesis. The method collects a green screen video frame and a background sample, generates chroma Alpha in a GPU, and generates RVM Alpha with time sequence continuity through an RVM robust video matting branch; finally, the method obtains a final Alpha through the fusion of chroma confidence, RVM confidence, edge gradient, motion intensity and frame budget, and performs de-spill, corrosion, expansion, feathering and GPU real-time synthesis. The application can improve the stability of hair, semi-transparent edges and motion blur areas while retaining the real-time performance of traditional green screen keying, reduce green edge, hole and edge hardening problems, and support live streaming quality review and parameter archiving.

Description

Technical Field

[0001] This invention belongs to the fields of XR live streaming, digital keying, GPU image processing, foreground / background separation, and real-time virtual-real compositing. Specifically, it relates to a GPU-accelerated green screen real-time keying and color bleeding suppression method for XR live streaming, and more particularly to an engineering solution that combines Chroma Key, background model foreground / background separation, GPU Shader, color bleeding removal, black and white clamping, erosion, dilation, and feathering with XR virtual shooting, real-time live streaming, virtual-real fusion, or virtual pre-visualization scenes. Background Technology

[0002] With the development of digital technology and new digital products, image processing, computer graphics, and real-time interactive technologies are widely used in film and television special effects, live streaming of galas, e-commerce live streaming, and virtual press conferences. Real-time rendering and fusion of virtual and real scenes requires that the real scene and the virtual scene maintain consistency in spatial location, temporal synchronization, lighting style, and output frame rate.

[0003] In practical applications, the system is based on digital display technology and develops an AR live streaming platform system based on binocular vision. It focuses on common technologies such as real-time rendering and fusion of virtual and real images, real-time camera tracking, real-time foreground keying under green screen background, dynamic rendering, motion capture, automatic scene setting for virtual pre-show, virtual and real interaction, real-time character generation, and construction of a 3D model asset library, forming a complete solution for real-time live streaming and virtual pre-show.

[0004] In XR live streaming green screen shooting, actors' hair, clothing edges, semi-transparent objects, curtain wrinkles, uneven lighting, and green reflections can cause rough keying edges, foreground green overflow, background residue, and insufficient real-time performance. Traditional CPU pixel-by-pixel processing is difficult to meet the real-time requirements of 2K or 4K live streaming.

[0005] Existing systems often only address a single aspect, such as camera tracking, keying, or rendering, lacking a complete technical chain connecting real-world shooting, algorithmic processing, virtual scenes, and application cases. Therefore, it is necessary to refine systems, methods, and implementation examples based on the actual engineering needs of XR virtual filming.

[0006] In this invention, AI stands for Artificial Intelligence; XR stands for Extended Reality; AR stands for Augmented Reality; GPU stands for Graphics Processing Unit; CPU stands for Central Processing Unit; RGB stands for Red, Green, and Blue; RVM stands for Robust Video Matting; MSE stands for Mean Squared Error; and BCE stands for Binary Cross Entropy. Summary of the Invention

[0007] The purpose of this invention is to provide a GPU-accelerated real-time green screen keying and color overflow suppression method for XR live streaming, which enables the foreground of the person, the alpha mask, the virtual background and the live output to be synthesized in real time with stable edges, natural colors and verifiable results under high frame rate conditions.

[0008] To achieve the above objectives, this invention takes green screen background sampling, RGB channels, motion information, and historical frame states as inputs, and employs GPU chroma keying, RVM robust video keying, Alpha confidence fusion, color correction, morphological edge cleanup, and GPU shader real-time synthesis to form a keying chain.

[0009] Here, Alpha is a transparency parameter used in this invention to uniformly express the weights of chroma keying mask, core mask, junk mask, and foreground / background compositing. Its preferred value range is 0 to 1. When Alpha is close to 1, the pixel participates in compositing as foreground; when Alpha is close to 0, the pixel is treated as background or a virtual background area; when Alpha is in the middle value, the system forms a semi-transparent transition area based on feathering radius, edge gradient, and local motion state. Therefore, Alpha is not a standalone display effect, but rather a unified engineering variable connecting chroma keying, edge cleansing, color correction, and real-time GPU compositing.

[0010] The RVM robust video keying branch is a temporal edge enhancement module added to the GPU chroma keying link in this invention. The input parameters of this branch include the current green screen video frame. Hidden state of the previous frame or several frames Reasoning scale Output frame number Frame budget And optional main area prompts Output parameters include RVMAlpha and hidden state. RVM confidence level and reasoning time Among them, the reasoning scale The scale can be set to original, half-scale, or quarter-scale depending on the live stream resolution and GPU load, and hidden. Used to preserve the main edge information of the previous time slice, so that the current frame has better continuity in hair, clothing edges and motion-blurred areas.

[0011] The RVM branch does not replace the main chroma keying link; instead, it only increases weights in regions with complex edges or insufficient chroma keying confidence. The system calculates weights based on edge gradient, motion intensity, and chroma distance. Green screen lighting uniformity (Shadow) and RVM inference time and historical Alpha change Calculate the RVM confidence score; when the RVM confidence score is lower than a preset threshold... ,or When the time limit allocated to RVM in the current frame budget is exceeded, the system reduces the RVM fusion weight, retains the hidden state of the previous frame, or reverts to GPU chroma keying enhancement mode.

[0012] The main parameters of the Alpha confidence fusion module include chroma Alpha. RVM Alpha Color confidence RVM confidence level fusion weight and Low confidence threshold Edge feathering radius Corrosion radius Expansion radius Color bleeding coefficient and quality score threshold In areas with a uniform green screen background and simple edges, the system improves performance. To ensure real-time performance; in areas with fine hairs, semi-transparent edges, rapid motion, or noticeable green reflections, the system improves... To enhance mask continuity; when both confidence levels are low, the system outputs a verification mark and prioritizes conservative feathering and core masking to protect the subject.

[0013] The frame budget parameter is used to define the project boundaries of the RVM branch. The system will calculate the total time per frame. Breaking down the time consumption into chroma keying RVM inference time Alpha fusion time Removing excess color takes time Morphological processing is time-consuming Synthesis time The system requires that the sum of all items not exceed the single frame time corresponding to the target frame rate. When the live output approaches the frame budget limit, the system prioritizes reducing the RVM inference scale, reducing the RVM call frequency, or freezing non-critical edge enhancements, without changing the main workflow of chroma keying, de-overflow, and final compositing.

[0014] The GPU chroma keying module is used to quickly generate basic alpha based on green dominance, background color distance, and high / low thresholds. This module has low computational cost and is suitable for clean background areas and low-latency live streaming requirements.

[0015] The RVM robust video keying module is used to generate temporal alpha using the current frame and historical hidden states. It is suitable for handling areas that are difficult to stably process with chroma keying, such as hair strands, semi-transparent edges, motion blur, and local green screen pollution.

[0016] The Alpha confidence fusion module generates the final Alpha based on chroma Alpha, RVM Alpha, edge gradient, motion intensity, color confidence, and frame budget. Clean background areas rely more on chroma keying, while complex edge areas rely more on RVM. When RVM takes too long, the system reduces the frequency of RVM calls or reuses historical states.

[0017] The de-blooming and edge cleansing modules are used to eliminate foreground green reflections, holes, breaks, and jagged edges. The system performs black-and-white clamping, erosion, dilation, feathering, de-blooming, and core masking correction, incorporating processing time into the frame budget.

[0018] The input interface of this invention includes green screen video frames, background sampling area, RVM history status, GPU frame budget, virtual background, and manual verification parameters; the output interface includes final alpha, de-overflowing foreground, composite frame, quality score, abnormal frame record, and parameter archive.

[0019] Through the above technical solutions, this invention solves the technical problems of traditional Chroma Key in producing green edges, holes or edge hardening in scenes with hair strands, semi-transparent edges, motion blur and uneven lighting, as well as the difficulty of CPU pixel-by-pixel processing in meeting the real-time requirements of XR live streaming.

[0020] The quality scoring module is the output verification module set up in this invention. The system combines edge cleanliness, color overflow ratio, alpha temporal stability, normalized frame rate, RVM confidence, and manual verification results into a quality score, which is used to determine whether the current parameter set can be directly used for live output. If the quality score is lower than the threshold, the system will prioritize prompting for resampling the background, adjusting the fusion weights, reducing the RVM inference scale, adjusting the feathering radius, or initiating manual verification.

[0021] This invention further limits RVM and Alpha fusion to only enhance the chroma keying link, rather than replacing the original GPU real-time keying. GPU chroma keying is still prioritized for clean background areas to ensure real-time performance; RVM weights are increased for hair strands, semi-transparent edges, fast-moving areas, and green screen contamination areas to ensure edge continuity; when frame budget is insufficient, the system reduces the RVM call frequency or retains historical hidden states, while preserving the main workflow of chroma keying, de-overflow, and final compositing. Attached Figure Description

[0022] Figure 1 This is a diagram showing the overall structure of the real-time keying system of this invention.

[0023] Figure 2 This is a flowchart of the dual alpha generation and confidence fusion process of the present invention.

[0024] Figure 3 This invention relates to GPU Shader parallel processing and frame budget graph.

[0025] Figure 4 This is a flowchart of the color removal and edge correction process of the present invention.

[0026] Figure 5 This is the live-stream synthesis quality evaluation and verification diagram of the present invention.

[0027] Figure 6 This is a flowchart of the green screen keying test for a dance troupe according to Embodiment 1 of the present invention.

[0028] Figure 7 This is a flowchart comparing the consistency and efficiency of GPU and CPU in Embodiment 2 of the present invention.

[0029] Figure 8 This is a flowchart of the real-time synthesis process for the XR live streaming room according to Embodiment 3 of the present invention.

[0030] Figure 9 This is a schematic diagram of the green screen foreground and background compositing effect of the present invention. (a) Custom background image (b) Result after compositing. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection; without departing from the technical concept of combining GPU chroma keying, RVM robust video keying, alpha fusion, color overflow suppression and real-time compositing, similar substitutions can be made for camera models, RVM network versions, shader languages ​​or output interfaces to implement the present invention.

[0032] Combination Figures 1 to 3This implementation integrates the input image, green screen background sampling, RVM history memory state, foreground mask, GPU texture, virtual background and output frame number into a unified link. Figure 2 The process of generating chroma alpha using GPU chroma keying, generating RVM alpha using RVM temporal keying, and determining the fusion weights together using chroma confidence and RVM confidence are presented. Figure 3 Furthermore, frame budget judgments are provided for texture uploading, shader parallel processing, RVM variable-scale inference, alpha fusion, de-overflow, morphological processing, and synthetic output.

[0033] The reason for introducing RVM is that traditional Chroma Keying is sensitive to hair strands, semi-transparent edges, motion blur, and uneven lighting; RVM can utilize the memory of adjacent frames to maintain the continuity of the foreground contour, but using RVM alone may be affected by the training domain and inference time. Therefore, this invention adopts a dual-branch fusion of Chroma Keying and RVM to balance real-time performance, green screen prior knowledge, and AI edge quality.

[0034] In this embodiment, the GPU chroma keying, RVM video keying, and Alpha confidence fusion algorithms are not run as isolated software modules, but are embedded in the acquisition, modeling, prediction, control, and verification chain of XR live streaming and studio green screen real-time keying compositing scenes. The system first encodes the physical boundaries, device status, scene labels, and manual verification results of the engineering object into a computable state. Then, the algorithm module outputs predicted values, confidence levels, constraint residuals, or control candidate quantities. Finally, the engineering rules transform the algorithm results into executable adjustment, early warning, or re-inspection actions.

[0035] The reason for choosing GPU chroma keying, RVM video keying, and Alpha confidence fusion algorithms is that green screen live streaming requires low-latency output, but human hair, semi-transparent edges, motion blur, and uneven lighting can cause traditional chroma keying to produce green edges or holes. If only static thresholding or a single depth keying model is used, the system will find it difficult to simultaneously balance the speed of clean background areas, the quality of complex edges, and the live frame budget.

[0036] Furthermore, GPU chroma keying is suitable for providing real-time base alpha, RVM is suitable for utilizing video memory to maintain the continuity of subject edges, and alpha confidence fusion is suitable for achieving a balance between speed and edge quality. Therefore, this invention does not directly equate the algorithm output with the final processing result in either the training or inference phases. Instead, it sets data fitting terms, physical consistency terms, temporal stability terms, risk constraint terms, and manual verification terms, so that the model can achieve high predictive ability while maintaining interpretability of engineering boundaries.

[0037] In terms of algorithm configuration, the output of this implementation scenario will be directly displayed on the live stream. The training objectives must simultaneously cover Alpha accuracy, edge stabilization, color overflow suppression, and frame budget. The training record retains the sample source, background sampling, loss components, thresholds, and human verification opinions. After changes in lighting, clothing, or background conditions, technicians can use this information to determine whether to update the RVM weights, adjust the chroma key threshold, freeze the inference frequency, or simply perform a manual review.

[0038] Before the samples were entered into the database, the system labeled uneven green screen brightness, hair edges, semi-transparent objects, motion blur, and near-green areas of clothing. Chromatic distance, edge gradient, motion intensity, RVM memory state, and manually refined alpha were normalized according to scene scale, and abnormal frames and manually revised results were used as high-confidence supervision signals in training.

[0039] Model training is conducted in three steps: offline material pre-training, live-stream small-sample calibration, and conservative updates during broadcast. Pre-training acquires foreground segmentation capabilities, live calibration incorporates differences in green screen lighting, camera color, and anchor movements, and during broadcast, only the fusion weights, thresholds, and memory state parameters are updated to avoid sudden drift in the live stream.

[0040] The background color model can be represented as:

[0041] (1)

[0042] In the formula, The average color of the green screen background. Let N be the background color covariance, and N be the number of sampled pixels. For green screen sampling area, Let be the color vector of pixel p, and the superscript T indicates transpose.

[0043] The chromaticity alpha can be represented as:

[0044] (2)

[0045] In the formula, `alpha` is the chroma keying function, and `clip` is the clipping function. and The threshold values ​​are high and low, and R, G, and B are the red, green, and blue channel values ​​of the pixel.

[0046] The RVM temporal matting branch can be represented as:

[0047] (3)

[0048] In the formula, Output Alpha for RVM, In the current hidden state, RVM is a robust video keying network. For the current frame image, -1 represents the memory state of the previous frame. This is the RVM inference scale.

[0049] The confidence level of RVM can be expressed as:

[0050] (4)

[0051] In the formula, Here, σ represents the RVM confidence level, γ0 to γ4 are the weighting coefficients, Edge represents edge integrity, and Temp represents temporal consistency. Let Tr represent the degree of motion blur, and Tr represent the RVM inference time. This is RVM Alpha.

[0052] Color confidence can be expressed as:

[0053] (5)

[0054] In the formula, Here, σ represents the chroma keying confidence level, σ is the Sigmoid function, and δ0 to δ3 are the weighting coefficients. For green advantages, The distance is the background color distance, and Shadow is the penalty for uneven lighting or shadows.

[0055] The fusion of Alpha is:

[0056] (6)

[0057] In the formula, To integrate Alpha, and These are the confidence scores for chroma keying and RVM, respectively. For chroma keying Alpha, Let be RVM Alpha, and ε be a constant to prevent the denominator from being zero.

[0058] Morphological edge correction is as follows:

[0059] (7)

[0060] In the formula, This represents the morphologically processed alpha, where F is the feathering operator, Dilate is the dilation operator, and Erode is the erosion operator. , and These are the feathering, expansion, and corrosion radii, respectively. To merge Alpha.

[0061] Remove excess color and synthesize as follows:

[0062] (8)

[0063] In the formula, ′ represents the foreground pixel after removing color bleeding. The original foreground pixels, For the color correction factor, R, G, and B are the three channels of the pixel. To synthesize the output color, For the final Alpha, Foreground color, Used as the virtual background color.

[0064] The real-time frame budget constraint is:

[0065] (9)

[0066] In the formula, This represents the total time consumed per frame. Chroma key control time consumption The time required for RVM inference, The fusion process takes time. Removing excess color takes time. Morphological processing is time-consuming. Due to the time required for synthesis, The minimum target frame rate.

[0067] The quality score can be expressed as:

[0068] (10)

[0069] In the formula, Q represents the image quality score. to These are the weighting coefficients. For edge cleanliness, Spill represents the percentage of spillage, and Temp represents temporal consistency. To normalize the frame rate, The confidence level of RVM.

[0070] Combination Figure 4 and Figure 5 The system calculates the alpha gradient, green reflection, and overflow ratio in the edge region, and corrects the hair strands, semi-transparent edges, and motion-blurred areas by using the de-overflow coefficient, erosion expansion radius, feathering radius, and RVM weights. Figure 5 The low-confidence kernel branch is given, formed by edge integrity, spillover ratio, temporal stability, and quality score threshold. Combined with... Figures 6 to 8 The examples illustrate the dance troupe green screen test, the comparison of GPU / CPU and RVM fusion, and the foreground compositing for XR live streaming.

[0071] To ensure that the green screen keying process goes beyond empirical thresholds, the system further describes the GPU's pixel-by-pixel processing using color distance, soft thresholding, temporal stability, and spillover constraints. Mahalanobis color distance is used to describe the degree to which the current pixel deviates from the green screen background model.

[0072] (11)

[0073] In the formula, Let p be the Mahalanobis distance relative to the green screen background. For pixel color vectors, and These represent the mean and covariance of the green screen background, respectively. This distance is used to handle uneven green screen brightness and localized shadows.

[0074] The soft threshold for chroma alpha can be achieved using a smooth step function, so that the foreground edges do not show obvious jaggedness due to the hard threshold.

[0075] (12)

[0076] In the formula, smoothstep is the smooth step function, and S(p) is the comprehensive chromaticity discriminant. As a green-dominant weight, G, R, and B are the pixel's green, red, and blue channel values, respectively. and The thresholds are low and high. The system computes this formula in parallel for each pixel in the GPU shader.

[0077] For live video, correct Alpha in a single frame is not enough to guarantee image stability. The system also constrains the Alpha changes in adjacent frames to be consistent with the results after motion compensation.

[0078] (13)

[0079] In the formula, For time-stability loss, Let be the current frame's alpha, and Warp be the compensation function executed based on the motion vector vt. This formula is used to suppress flickering in hair strands, clothing edges, and motion-blurred areas.

[0080] Color bleeding suppression also needs to limit the green residue at the foreground edges, rather than removing all the real green clothing.

[0081] (14)

[0082] In the formula, To suppress color bleeding loss, p is the pixel number. The final alpha is represented by the edge weight. , and These are the green, red, and blue channel values ​​of pixel p, respectively. This represents the permissible green residue threshold. This formula only plays a major role at the foreground edges and in high-alpha regions.

[0083] During the training phase, the system primarily focuses on foreground alpha, de-overflowing foreground, synthesized frame quality, and real-time frame budget, but does not use a single mean squared error as the overall optimization objective. This is because a single error term only indicates the distance between the model output and historical labels; it cannot constrain whether the prediction process conforms to engineering principles, nor can it prevent the model from making unstable judgments on boundary conditions, low-confidence samples, or stage-switching samples.

[0084] The loss function is set around Alpha supervision, synthetic reconstruction, edge gradient, and temporal color bleeding suppression. Alpha supervision ensures that the foreground transparency is close to the label, synthetic reconstruction constrains the final image color, edge gradient protects hair and clothing boundaries, and temporal color bleeding suppression limits green fringing and flickering.

[0085] The weights of each loss component are adaptively configured according to the screen area and the live broadcast status. The internal area of ​​the main subject emphasizes alpha stability, hair strands and semi-transparent edges increase the gradient loss weight, fast-moving segments increase the temporal loss weight, and green screen reflections or near-green clothing segments increase the color overflow suppression weight. When the frame budget is tight, the system reduces the frequency of non-critical edge enhancement calls.

[0086] When a certain loss component continues to rise, the system does not immediately interpret the anomaly as an algorithm failure. Instead, it first cross-checks the data fit component with the original input, device logs, and manual review records. If the data fit component rises while the mechanistic component remains normal, it usually indicates a bias in the label or local input. If the mechanistic component rises while the data fit component remains normal, it suggests that the model may have learned a random correlation. If the safety boundary component rises, a conservative handling strategy and manual review process are triggered first.

[0087] (15)

[0088] In the formula, For Alpha supervision loss, To predict Alpha, For manual or high-quality offline image matting labels, BCE is the binary cross-entropy.

[0089] (16)

[0090] In the formula, For the synthetic reconstruction loss, For Alpha prediction, F is the foreground image, B is the virtual background image, and C* is the reference composite frame.

[0091] (17)

[0092] In the formula, The marginal gradient loss is denoted by ∇, where ∇ is the gradient operator. and For prediction and label alpha, C* is the output composite frame, and C* is the reference composite frame.

[0093] (18)

[0094] In the formula, For the total loss function, For Alpha supervision loss, For the synthetic reconstruction loss, For edge gradient loss, For time-stability loss, To suppress color bleeding and loss, to These are the weighting coefficients.

[0095] Alpha supervision loss ensures correct basic classification of the subject and background regions, synthesis and reconstruction loss constrains the final image's appearance, and edge gradient loss enhances hair strands, clothing edges, and semi-transparent areas. These three factors work together to prevent the model from performing well only on a binary mask while the synthesized image still has hard edges.

[0096] RVM is suitable for this scenario because the outline of the main subject in the live video has temporal continuity. The hidden state of RVM can pass the edge information of the previous frame to the current frame, reducing holes and broken edges during motion; however, RVM inference is time-consuming, so this invention retains GPU chroma keying as the real-time basic branch.

[0097] Alpha confidence fusion enables the system to select different criteria in different regions: pure background regions rely more on chroma keying, hair strands and motion edges rely more on RVM, and low confidence regions use feathering, morphological correction, or manual verification marking.

[0098] To ensure that GPU chroma keying, RVM video keying, and Alpha confidence fusion algorithms can stably adapt to XR live streaming and studio green screen real-time keying compositing scenarios, our system does not treat all historical data equally at the sample level. Instead, it stratifies samples into five categories: normal operation, rehearsal transition, asset loading disturbance, manual intervention, and verification / confirmation. Normal operation samples are used to learn basic mapping relationships; rehearsal-to-live broadcast transition samples are used to learn parameter transition patterns; sudden lighting changes, rapid movements, and near-green clothing samples are used to identify keying risk lead times; manual intervention samples are used to constrain the system from outputting difficult-to-execute actions; and verification / confirmation samples are used to calibrate model confidence.

[0099] Regarding the sample time window setting, the system determines the window length based on the physical or operational latency of state changes in XR live streaming and studio green screen real-time keying compositing scenes. For variables with fast responses, short windows are used to capture instantaneous changes; for variables with lag effects, long windows are used to preserve historical impacts. Overlapping windows are allowed, but the same anomalous event will not be repeatedly counted as multiple independent high-value samples to avoid exaggerating the weight of a small number of anomalous events during the training phase.

[0100] During label construction, the system records the final results of foreground alpha, de-overflowing foreground, composite frame quality, and real-time frame budget, while simultaneously retaining intermediate interpretation labels. These intermediate interpretation labels include key input sources, trigger thresholds, human-judged reasons, executed actions, and verification conclusions. After model training is complete, the system can simultaneously output prediction results and their main supporting evidence, facilitating technical personnel to verify the consistency between model conclusions and engineering mechanisms.

[0101] The loss weights are adjusted according to the image region. Alpha stability is emphasized within the main subject, gradient weights are increased for hair and semi-transparent edges, temporal weights are increased for fast-moving segments, and color bleeding suppression weights are increased for green screen reflections or near-green clothing segments. This processing ensures the model serves the quality of the live stream, avoiding the formation of scores based solely on single frames.

[0102] Model validation does not employ a single random partitioning method, but rather grouped validation by time, stage, and scenario. If random validation performs well but cross-stage validation results decline, it indicates that the model may have learned accidental correlations; if cross-scenario validation results decline, it indicates that the model needs on-site fine-tuning; if the error on manually reviewed samples is large, the loss weights and sample labels are corrected first, and then the model size is evaluated.

[0103] A cold start period is set up before the system is deployed on-site. During the cold start period, the GPU chroma keying, RVM video keying, and Alpha confidence fusion algorithms only output prediction results, risk components, and suggested basis, without triggering automatic execution. Only after the prediction results, loss components, and manual review results continuously meet the stability conditions will the system allow the output of semi-automatic or automatic control suggestions. This setting ensures that the algorithm results are first verified on-site and manually confirmed before entering the semi-automatic or automatic control stage.

[0104] During runtime, the system performs drift detection on the input distribution. If the mean, variance, missing rate, or scene label proportion of the input variables significantly deviates from the training distribution, the system increases the confidence penalty and marks the corresponding samples as samples to be reviewed. If the drift occurs only on a few variables, the system prioritizes adjusting the input scaling parameters; if the drift occurs on multiple core variables, it triggers small-sample fine-tuning or manual recalibration.

[0105] The model output records alpha error, edge integrity, color overflow ratio, frame latency, and fusion weights. Technicians can use this information to determine whether the problem stems from background sampling, RVM inference, edge processing, or insufficient GPU frame budget.

[0106] Low-confidence keying does not directly enter the final output, but instead enters a degradation or verification branch. The system can resample the background, increase the chroma key weights, reduce the RVM inference frequency, freeze the previous frame's alpha, or prompt for relighting.

[0107] In utilizing anomalous samples, the system distinguishes between genuine anomalies and collected anomalies. Genuine anomalies retain their risk labels when added to the training set, while collected anomalies retain their quality labels. This prevents the model from mislearning sensor malfunctions, blurry images, missing logs, or human errors as engineering patterns, and also prevents it from losing its ability to identify genuine risks by deleting all anomalous samples.

[0108] For parameter updates, the system employs versioned model management. Each model version saves the training data range, main loss weights, threshold configuration, manual review pass rate, and deployment time. If a new version experiences output jumps or unstable interpretations during field operation, the system can revert to the previous version and retain the failed samples from the new version for subsequent analysis.

[0109] The project execution is divided into three levels: alerts, suggestions, and real-time compositing. The alerts level marks abnormal frames, the suggestions level provides parameter adjustment schemes, and the real-time compositing level is only enabled when the alpha confidence, overflow ratio, and frame budget meet the requirements.

[0110] In the flowchart corresponding to the attached figure, the data input node is used for sample construction and quality labeling; the algorithm node is used to execute GPU chroma keying, RVM video matting, and Alpha confidence fusion algorithms and calculate the loss components; the decision node is used to check thresholds, confidence levels, and safety boundaries; the output node is used to generate engineering actions or review tasks; and the archiving node is used to save the model evidence chain. Therefore, each flowchart in the attached figure corresponds to the steps of on-site data processing, model calculation, manual review, and archiving.

[0111] Manual review records include acceptance, trimming, rejection, and explanations. Acceptance records reinforce similar lighting conditions, trimming records are used to correct edge weights, rejection records form negative samples, and explanations are included in the quality archive.

[0112] At the implementation boundary, GPU chroma keying, RVM video matting, and Alpha confidence fusion algorithms can be deployed on local servers, edge computing nodes, or cloud training platforms. However, on-site inference should retain conservative strategies for situations involving network outages or high latency. When the cloud is unavailable, the system uses the most recently approved model version and local thresholds for execution; once local data is restored, the training logs and approval results are synchronized.

[0113] In terms of safety, the system will not automatically relax engineering limits due to short-term improvements in model predictions. All limits are derived from constraints set by process, equipment, image quality, or manually, and the algorithm can only find better solutions within these limits. If the model suggests exceeding the limits, the system marks the suggestion as unexecutable and outputs the loss component that generated the suggestion for manual evaluation to determine if there is an input or model anomaly.

[0114] During on-site implementation, the keying parameters are not set once and remain unchanged. Before broadcasting, the system first collects samples of the empty background, the host's position, rapid gestures, and edge hair, and checks the chroma key, RVM alpha, de-overflow, and feathering results respectively. During broadcasting, if the lighting is adjusted, the clothing color changes, or the foreground movement speed changes significantly, the system only allows minor updates to the blending weights; operations involving large threshold changes must be verified in the preview channel first to confirm that there are no obvious green edges, holes, or flickering at the edges of the image before switching to the main output.

[0115] During quality review, the system saves the green edge ratio, edge integrity, alpha stability, and parameter changes for each shot segment. Technicians can review the background sampling, fusion weights, and RVM state before and after the abnormal frames to determine whether the problem stems from lighting, clothing color, motion blur, or insufficient model confidence. For confirmed parameter combinations, the system saves them as initial templates for similar live streams.

[0116] For multi-camera live streaming, the system also saves background samples and fusion parameters separately for each camera position. The lighting angles and edge states of people are different for the main camera, side camera, and moving camera, so the parameters should not be simply shared. If an anomaly occurs in a certain camera position, the system only updates the weight and threshold of that camera position to avoid affecting the stable output of other cameras.

[0117] Multi-camera parameter archiving also records lens number, white balance, exposure, and background sampling time. When reusing the same live broadcast room later, technicians can first call up a similar camera template, and then make minor adjustments based on the current lighting and clothing conditions, reducing the need for repeated parameter testing before the broadcast.

[0118] For preview tests that do not enter the main output, the system also saves the parameters and screenshots as a reference for subsequent debugging.

[0119] This record is also used to define the initial threshold range before the next broadcast.

[0120] Example 1: Real-time keying of green screen footage from a university dance troupe in Beijing

[0121] Combination Figure 6 This embodiment uses green screen photos and video test materials from a dance troupe as the subject. Figure 6 Using green screen footage of a dance troupe and empty background samples as input, the system sequentially performs background alignment, chroma alpha and RVM alpha fusion; it judges the verification effect of node verification edges, color overflow and motion areas. The Yes branch outputs the test screen and archives it, while the No branch adjusts the chroma threshold, RVM weight or removes the color overflow coefficient before recording the sample.

[0122] The system first samples the color distribution from the green screen background area, calculates the green dominance threshold and softening coefficient, and then generates a chroma alpha for each pixel in the GPU shader. Simultaneously, the RVM robust video keying branch generates an AI alpha based on the current frame and historical hidden states. For clothing edges, hair strands, and motion-blurred areas, the system reduces hard edges, holes, and jagged edges by fusing the confidence of the chroma alpha and RVM alpha.

[0123] The edge cleanup node sequentially performs black and white clamping, de-bleeding, erosion, dilation, and feathering. The de-bleeding process adjusts the pixel color based on the proportion of green residue at the foreground edge, so that no obvious green edge appears when the character outline blends with the virtual background.

[0124] The effect comparison node can simultaneously display the original image, initial mask, corrected mask, and composite result. If edge residue or foreground damage exceeds the threshold, the system prompts for resampling the background or adjusting the threshold, instead of directly outputting the final live stream.

[0125] Background sampling prioritizes green screen areas that are unobstructed, shadowless, and free of reflections. If there is a significant gradient in the green screen lighting, the system can establish background color models for different regions, making the keying thresholds at different locations more consistent with the ambient lighting.

[0126] After alpha generation, the system performs a continuity check on the foreground edges. If holes or breaks appear at the edges, the softening coefficient and morphological radius are adjusted first; if the foreground is eroded as a whole, the system suggests reducing the green dominance threshold or adding a core mask.

[0127] The test archive saves the original footage, parameter groups, mask results, and composite frames. Subsequent similar live streams can read the archived parameters as the initial configuration and then fine-tune them according to the on-site clothing colors, lighting, and camera settings.

[0128] Combination Figure 6In this embodiment, the dance troupe green screen test is used to illustrate the processing order of GPU chroma keying, RVM, and alpha fusion. The system first acquires a dance troupe green screen video and an empty background sample, and then generates chroma alpha and RVM alpha. The fusion module outputs the final alpha based on edge, motion, and color confidence, and generates the composite result through edge cleanup and de-overflow.

[0129] During the data preparation phase, the system collects dance troupe movement videos and background sampling areas. This step aims to organize multi-source data under corresponding working conditions into unified features, preventing the model from making judgments based on a single indicator. If missing, conflicting, or low-confidence inputs are found in this step, the system retains quality markers in the feature vectors and incorporates them into subsequent confidence level assessments.

[0130] During the model computation phase, the system performs parallel computation of chromatic alpha and RVM alpha. This step simultaneously outputs the primary predictor and the auxiliary diagnostic; the primary predictor is used to generate control, recommendation, tracking, or early warning results, while the auxiliary diagnostic is used to explain which variables, time slices, spatial nodes, or image regions the results originate from.

[0131] During the engineering constraint phase, the focus of verification is on merging the alpha and performing de-overflow and feathering. When the calculation results are within the threshold margin, they are recorded as executable suggestions; when the results are close to the safety boundary, inconsistent with the live production rules, or conflict with manual recordings, a review task and low-risk candidate solutions are first generated, and then entered into the execution interface after confirmation by technical personnel. Thus, the model output remains a basis for on-site decision-making, rather than an offline score detached from engineering conditions.

[0132] During the archiving phase, the system performs archiving edge quality checks, color bleeding ratio checks, and abnormal frames. Archiving content includes at least the input window, model version, loss components, key thresholds, manual review comments, and final handling results. When the same or similar scenarios occur again, the system can retrieve this record as a priori sample to shorten model calibration time and improve handling consistency.

[0133] Through the above process Figure 6 Each process node corresponds to a specific implementation action: input nodes correspond to data preparation and quality verification, algorithm nodes correspond to model calculation and constraint verification, and output nodes correspond to handling suggestions and archiving review. Therefore, this embodiment can illustrate the complete implementation path of GPU chroma keying, RVM video keying, and Alpha confidence fusion algorithms in a dance troupe green screen test, and make the algorithm steps correspond to the on-site implementation actions.

[0134] Combination Figure 6Example 1 records the dance troupe's green screen footage, empty background samples, chroma alpha, RVM alpha, blending weights, effect review conclusions, and Yes / No branch results. Results that pass review are output as test images along the Yes branch and form test records. Results that fail review are adjusted along the No branch, adjusting the chroma threshold, RVM weights, or de-overflow coefficients, and the parameters before and after adjustment are archived.

[0135] Figure 6 The abnormal branches in the algorithm are handled according to their source: for input anomalies, re-samples, retests, or weight reduction are performed first; for model anomalies, parameters are frozen and a backup model is called; for boundary condition anomalies, a conservative solution is output; and when review opinions are inconsistent, dispute records and final handling results are retained. This approach ensures that the algorithm module retains an auditable operating trajectory even under on-site disturbances.

[0136] After this embodiment is completed, the system binds the live stream room number, camera position number, background sampling time, chroma threshold, RVM status, alpha fusion weight, de-overflow parameters, quality score, and manual confirmation record into the same archive unit. When similar lighting, clothing, or camera position conditions occur subsequently, the system can call this archive unit as the initial parameters for keying.

[0137] Example 2: Comparison of consistency and efficiency between GPU and CPU versions

[0138] Combination Figure 7 This embodiment focuses on a comparative test of the same green screen material across CPU pixel-by-pixel processing, GPU parallel processing, and RVM edge enhancement. Figure 7 Generate CPU reference results and GPU real-time results respectively, and then compare MSE, edge integrity, color overflow ratio, frame time and FPS using consistency judgment nodes; the Yes branch records efficiency indicators and outputs a comparison report, and the No branch adjusts GPU parameters or RVM scale and archives parameter groups.

[0139] During testing, the CPU and GPU versions used the same green advantage threshold, de-overflow parameters, erosion dilation radius, and feathering radius. The system output alpha masks and composite frames respectively, and the consistency of the two results was compared using MSE or equivalent difference metrics.

[0140] The frame rate statistics node records the time spent on keying, de-overflow, morphological processing, and compositing. If the GPU version meets the target frame rate while maintaining consistency, the system will use this parameter group as a candidate configuration for live streaming; if the time spent on a certain step is too high, the shader implementation or texture access method of that step will be adjusted first.

[0141] Figure 7The GPU acceleration used to illustrate this invention is not simply a hardware replacement, but rather a transformation of pixel-by-pixel calculation into a parallel processing chain suitable for real-time live streaming, while maintaining the consistency of the keying logic.

[0142] The CPU version is used to verify the basic chroma keying logic, the GPU version is used to verify real-time deployment, and the RVM branch is used to verify the temporal edge enhancement effect. All three use the same input materials and corresponding parameter sets, making the comparison of efficiency, edge quality, and temporal stability more meaningful for engineering purposes.

[0143] In addition to MSE comparison, the system can also record the percentage of edge residue, the percentage of foreground damage, and the percentage of color bleeding. If the MSE is low but the edge appearance is poor, the system will still prompt for manual review, because the quality of live broadcast cannot be determined by a single numerical indicator.

[0144] Frame rate statistics are broken down by processing stage to facilitate bottleneck identification. If decolorization is time-consuming, prioritize optimizing color transformation; if morphology is time-consuming, prioritize optimizing texture reading and convolution radius.

[0145] Combination Figure 7 In this embodiment, the comparison between GPU and CPU / RVM is used to illustrate consistency and efficiency verification. The system runs CPU pixel-by-pixel processing, GPU parallel processing, and RVM branch under the same threshold, the same background sampling, and the same material conditions, respectively, and records MSE, edge integrity, color overflow ratio, frame time, and parameter set.

[0146] During the data preparation phase, the system records the execution time of GPU Shader, CPU pixel-by-pixel processing, and RVM branch processing. This step aims to organize multi-source data under the corresponding operating conditions into unified features, preventing the model from making judgments based on a single metric. If missing, conflicting, or low-confidence inputs are found in this step, the system retains quality markers in the feature vectors and incorporates them into subsequent confidence level assessments.

[0147] During the model computation phase, the system performs a comparison of the synthesis quality and edge integrity within the frame budget. This step simultaneously outputs the primary predictor and the auxiliary diagnostic; the primary predictor is used to generate control, recommendation, tracking, or warning results, while the auxiliary diagnostic is used to explain which variables, time slices, spatial nodes, or image regions the results originate from.

[0148] During the engineering constraint phase, the focus of verification is switching the inference frequency based on the latency threshold. When the calculation results are within the threshold margin, they are recorded as executable suggestions; when the results are close to the safety boundary, inconsistent with the live broadcast production rules, or conflict with manual recording, a review task and low-risk candidate solutions are first generated, and then entered into the execution interface after confirmation by technical personnel. In this way, the model output remains a basis for on-site decision-making, rather than an offline score detached from engineering conditions.

[0149] During the archiving phase, the system saves the parameter combinations and screen review results. The archived content includes at least the input window, model version, loss components, key thresholds, manual review comments, and final handling results. When the same or similar scenarios occur again, the system can retrieve this record as a priori sample to shorten model calibration time and improve handling consistency.

[0150] Through the above process Figure 7 Each process node corresponds to a specific implementation action: input nodes correspond to data preparation and quality verification, algorithm nodes correspond to model calculation and constraint verification, and output nodes correspond to handling suggestions and archiving review. Therefore, this embodiment can illustrate the complete implementation path of GPU chroma keying, RVM video keying, and Alpha confidence fusion algorithms in the comparison between GPU and CPU / RVM, and make the algorithm steps correspond to the on-site implementation actions.

[0151] Combination Figure 7 Example 2 focuses on recording the comparison of images before and after the keying parameters are adjusted. The system saves the chroma threshold, RVM weight, alpha fusion weight, de-spec coefficient, morphological radius, MSE, edge integrity, and on-site confirmation opinions to distinguish the differences caused by chroma keying errors, RVM inference errors, and insufficient GPU frame budget.

[0152] Figure 7 The abnormal branches in the algorithm are handled according to their source: for input anomalies, re-samples, retests, or weight reduction are performed first; for model anomalies, parameters are frozen and a backup model is called; for boundary condition anomalies, a conservative solution is output; and when review opinions are inconsistent, dispute records and final handling results are retained. This approach ensures that the algorithm module retains an auditable operating trajectory even under on-site disturbances.

[0153] After this embodiment is completed, the system binds the live stream room number, camera position number, background sampling time, chroma threshold, RVM status, alpha fusion weight, de-overflow parameters, quality score, and manual confirmation record into the same archive unit. When similar lighting, clothing, or camera position conditions occur subsequently, the system can call this archive unit as the initial parameters for keying.

[0154] Example 3: Real-time Foreground Compositing in XR Live Streaming Room

[0155] Combination Figure 8 This embodiment focuses on the foreground composition of anchors in Taobao live streaming, studio live streaming, or press conferences. Figure 8 Using the live stream frame as input, the final alpha is calculated and a virtual background is synthesized; the judgment node is used to determine whether the quality is below the threshold. The Yes branch enters the preview parameter adjustment and re-verification, and the No branch pushes the main path screen along the main process. Finally, the broadcast record is output and the live stream sample is archived.

[0156] During the live stream, the system prioritizes maintaining the continuity of the footage. When the local alpha confidence decreases, it can maintain the parameters from the previous frame or perform conservative feathering. When the green screen lighting changes significantly, the system prompts for background resampling and updates the parameters without interrupting the live stream.

[0157] The output streaming node sends the composited footage to the live or recording interface, while saving keyframes, parameters, and anomaly markers. The parameter verification node is used to analyze edge residue, color overflow, and frame rate fluctuations after the live stream, establishing a stable configuration for subsequent similar live streams.

[0158] This invention is applicable to XR live streaming, e-commerce live streaming, studio programs, and press conference foreground compositing scenarios, and can be implemented in conjunction with camera capture, virtual background rendering, and live streaming systems.

[0159] During real-time compositing, the foreground mask, virtual live streaming background, and output frame must use the same frame number. If the foreground and background are out of sync, the system will prioritize discarding the delayed frame or using the previous background frame to avoid misalignment between the streamer's silhouette and the background.

[0160] Parameter verification can be completed without interrupting the live stream. Technicians compare the current composite frame with the simulated frame after parameter adjustments through the preview window, and switch to live output only after confirming that there is no obvious damage.

[0161] The streaming archive saves keyframes and anomaly markers. If green edges or localized flickering are observed after the live stream, the archive parameters can be used to pinpoint whether the issue is related to sampling, thresholding, de-overflow, or frame synchronization.

[0162] Combination Figure 8 In this embodiment, the real-time compositing of the live broadcast room is used to illustrate the broadcast link. The system aligns the anchor foreground, final alpha, virtual background, and composite frame according to the same frame number; when changes in lighting, clothing, or motion state cause a drop in quality score, the system first adjusts the chroma threshold, RVM weight, or de-overflow coefficient in the preview channel before switching to the main output.

[0163] During the data preparation phase, the system reads the anchor foreground, virtual background, and streaming frame number. This step aims to organize multi-source data under the corresponding operating conditions into unified features, preventing the model from making judgments based on a single indicator. If missing, conflicting, or low-confidence inputs are found in this step, the system retains quality markers in the feature vectors and incorporates them into subsequent confidence level assessments.

[0164] During the model computation phase, the system outputs synthesized frames and calculates quality scores in real time. This step simultaneously outputs primary predictive values ​​and auxiliary diagnostic values; the primary predictive values ​​are used to generate control, recommendation, tracking, or early warning results, while the auxiliary diagnostic values ​​are used to explain which variables, time slices, spatial nodes, or image regions the results originate from.

[0165] During the engineering constraint phase, the focus of review is on prompting resampling of the background or adjusting the fusion weights when the quality is low. When the calculation results are within the threshold margin, they are recorded as executable suggestions; when the results are close to the safety boundary, inconsistent with the live production rules, or conflict with manual recording, a review task and low-risk candidate solutions are first generated, and then entered into the execution interface after confirmation by technical personnel. In this way, the model output remains the basis for on-site decision-making, rather than an offline score detached from engineering conditions.

[0166] During the archiving phase, the system archives the live broadcast parameters and the results of manual verification. The archived content includes at least the input window, model version, loss components, key thresholds, manual review comments, and the final handling result. When the same or similar scenarios occur again, the system can retrieve this record as a priori sample to shorten model calibration time and improve handling consistency.

[0167] Through the above process Figure 8 Each process node corresponds to a specific implementation action: input nodes correspond to data preparation and quality verification, algorithm nodes correspond to model calculation and constraint verification, and output nodes correspond to handling suggestions and archiving review. Therefore, this embodiment can illustrate the complete implementation path of GPU chroma keying, RVM video keying, and Alpha confidence fusion algorithms in real-time compositing in a live broadcast room, and make the algorithm steps correspond to the on-site implementation actions.

[0168] Combination Figure 8 In Example 3, the live broadcast frames, final alpha, virtual background, quality threshold judgment, preview parameter tuning records, and main stream status are archived together. Frames with quality below the threshold enter the preview parameter tuning via the Yes branch, while frames with quality above the threshold are pushed normally via the No branch on the main stream. The system saves the broadcast records and live samples.

[0169] Figure 8 The abnormal branches in the algorithm are handled according to their source: for input anomalies, re-samples, retests, or weight reduction are performed first; for model anomalies, parameters are frozen and a backup model is called; for boundary condition anomalies, a conservative solution is output; and when review opinions are inconsistent, dispute records and final handling results are retained. This approach ensures that the algorithm module retains an auditable operating trajectory even under on-site disturbances.

[0170] After this embodiment is completed, the system binds the live stream room number, camera position number, background sampling time, chroma threshold, RVM status, alpha fusion weight, de-overflow parameters, quality score, and manual confirmation record into the same archive unit. When similar lighting, clothing, or camera position conditions occur subsequently, the system can call this archive unit as the initial parameters for keying.

[0171] Combination Figure 9After chroma keying, RVM alpha enhancement, edge cleansing, and de-overflow processing, the green screen foreground can be combined with the outdoor scene or virtual background to form a real-time composite image.

Claims

1. A GPU-accelerated real-time green screen keying and color overflow suppression method for XR live streaming, characterized in that, The method uses green screen video frames. Green screen background sampling area Hidden state of the previous frame Virtual background Frame number and frame budget As input, the method outputs a blended alpha mask, a final alpha mask, a de-bleeded foreground, a composite frame, a frame rate, and a quality score; the method includes the following steps: S1. Sample the green screen background area that is unobstructed, has no obvious shadows, and has no reflections to obtain sampled pixels. And establish the average color of the green screen background. Covariance : （1） S2. In the GPU shader, calculate the color distance and green advantage relative to the green screen background model for each pixel of the current green screen video frame in parallel, and generate chroma alpha based on high and low thresholds, where the color distance and chroma alpha satisfy: （2） S3, Move the current frame Hidden state of the previous frame and RVM inference scale Input the RVM robust video keying branch and generate the RVM Alpha. and the current hidden state The RVM confidence score is calculated based on edge integrity, temporal consistency, motion ambiguity, and RVM inference time. : （3） （4） S4. Based on green advantages Background color distance Uneven lighting penalty Calculate chroma keying confidence The chroma alpha and RVM alpha are then fused according to confidence level to obtain the fused alpha. : （5） （6） S5. Perform erosion, dilation, and feathering on the merged Alpha to obtain the final Alpha. ,in , and These are the corrosion radius, expansion radius, and feathering radius, respectively. （7） S6. Perform green residue suppression on the foreground edge region corresponding to the final Alpha to obtain the foreground pixels after de-spaming. and in the GPU, the foreground color Ultimate Alpha With virtual background color Combined into output color : （8） S7, Reduce Chroma Keying Time RVM inference time Alpha fusion time Removing excess color takes time Morphological processing is time-consuming Synthesis time Included in the single-frame budget constraint, and satisfying: （9） S8. While outputting the composite frame, record edge integrity, color overflow ratio, temporal stability, normalized frame rate, RVM confidence, abnormal frames and parameter groups for previewing and adjusting parameters before live streaming, degradation processing during live streaming, and quality archiving after live streaming.

2. The method according to claim 1, characterized in that, The green screen background sampling area includes an empty background area, the background area around the main body's position, and areas with uneven local lighting. When a green screen brightness gradient, shadow, or reflection is detected, the system updates the green screen background color model according to the image area, and then updates the model accordingly. , High threshold and low threshold As input parameters for GPU chroma keying.

3. The method according to claim 1, characterized in that, The GPU chroma keying performs pixel-level parallel computation of green advantage, background color distance, and chroma alpha, and combines core masking, garbage masking, black and white clamping, and soft thresholding to generate a basic foreground mask; among them, pure background areas are preferentially used for compositing with chroma alpha, while low-confidence edge areas are entered into RVM enhancement or manual verification branches.

4. The method according to claim 1, characterized in that, The RVM robust video keying branch takes the current frame, historical hidden state, inference scale, and optional subject region cues as input, and only increases the fusion weights in areas such as hair strands, semi-transparent edges, fast motion, motion blur, or localized green screen contamination; when the RVM confidence is below a threshold... At this time, the system reduces the RVM fusion weights or uses the hidden state of the previous frame to maintain the continuity of the foreground edges.

5. The method according to claim 1, characterized in that, The Alpha fusion is based on the chroma Alpha. RVMAlpha Color confidence RVM confidence level The fusion weights are determined by edge gradient, motion intensity, and color confidence. Increase chroma keying weights in areas with uniform background and simple edges, increase RVM weights in areas with hair strands, semi-transparent edges, fast motion, or obvious green reflections, and output kernel markers in areas with low confidence in both categories, while using conservative feathering and core masking to protect the subject.

6. The method according to claim 1, characterized in that, The decolorization and edge cleanup include green residue detection and decolorization coefficient. Adjustments, black and white clamping, erosion, dilation, feathering, and core / junk mask blending are applied. Among these, the de-overflow processing mainly works on the foreground edges and high-alpha areas corresponding to the final alpha, in order to reduce green edges, holes, breaks, and jagged edges, while preserving the true foreground color.

7. The method according to claim 1, characterized in that, When RVM inference takes time When the total time for a single frame approaches the frame budget limit, the system performs degradation processing in the following order: reducing RVM inference scale, reducing RVM call frequency, freezing non-critical edge enhancement, retaining historical hidden states, and reverting to GPU chroma keying enhancement mode, while ensuring that chroma keying, de-overflow, and the final GPU compositing main process are not interrupted.

8. The method according to claim 1, characterized in that, The quality score is based on edge cleanliness. Color overflow ratio Timing consistency Normalized frame rate and RVM confidence calculate: （10） When the quality score Q is below the threshold When prompted, the system will prompt you to resample the background, adjust the RVM fusion weights, adjust the feathering radius, reduce the RVM inference scale, or proceed to manual review.

9. The method according to claim 1, characterized in that, The method supports test shooting, rehearsal, and live streaming modes. In test shooting mode, it increases the archiving granularity to record original footage, masks, and parameter groups. In rehearsal mode, it calibrates the chromaticity threshold, RVM weight, and de-overflow coefficient based on changes in lighting, costumes, and movements. In live streaming mode, it prioritizes maintaining frame rate and image continuity and is suitable for dance troupe green screen testing, live broadcasts, product launches, and XR studio programs.

10. A real-time green screen compositing system for XR live streaming, characterized in that, The system is used to perform the method according to any one of claims 1 to 9, the system comprising: a green screen background sampling unit for establishing a green screen background color model; a GPU chroma keying unit for generating chroma alpha; an RVM robust video keying unit for generating RVM alpha and hidden states; an alpha confidence fusion unit for generating fused alpha and final alpha; a de-overflow and edge cleanup unit for suppressing green residue and correcting foreground edges; a real-time compositing unit for completing the compositing of foreground, final alpha and virtual background in the GPU; and a quality review and archiving unit for recording frame rate, quality score, abnormal frames, parameter groups and manual review results.

Images