XR real-time group photo and virtual celebration system for marriage commemoration

Abstract

The invention discloses an XR real-time group photo and virtual celebration system for marriage commemoration, and the system comprises a data collection and preprocessing module which is used for collecting and preprocessing a color image, a depth map, a human body key point, and an actual measurement optical flow; the ConvGRU dynamic filtering input module is used for generating sampling offset and convolution weight based on the actually measured optical flow; the ConvGRU time sequence reasoning module is used for outputting a figure posture change prediction result and a prediction optical flow diagram; the Alpha-Matting preposition alignment module is used for generating an initial migration mask image and a three-valued area image; the Alpha-Matting reasoning and correcting module is used for outputting an optimized transparent mask image and a transparent mask confidence coefficient image; and the rendering and output module is used for completing illumination rendering and fusion output based on physics. According to the method, ConvGRU dynamic filtering and Alpha-Matting time sequence depth perception matting are combined, so that high-precision fusion of a character and a virtual scene is realized, and the stability and naturalness of XR real-time rendering are improved.

Description

Technical Field

[0001] This invention relates to the field of extended reality technology, and in particular to an XR real-time group photo and virtual celebration system for a golden wedding anniversary. Background Technology

[0002] With the widespread application of XR technology in virtual imaging and immersive interaction, real-time character synthesis and virtual scene fusion have become research focuses. Existing XR group photo and virtual celebration systems mostly rely on human pose detection, image matting, and rendering techniques, using keypoint estimation and alpha-matting to combine characters with virtual backgrounds. However, under complex lighting or dynamic motion conditions, the real-time performance and fusion stability of these systems are still significantly insufficient, and the generated images are prone to problems such as unstable edges, jitter, or occlusion errors.

[0003] Traditional ConvGRU often uses fixed convolutional kernels, which cannot dynamically adjust the sampling position according to changes in optical flow. This results in insufficient capture of continuous changes in human pose and a tendency for inter-frame prediction jumps. Meanwhile, Alpha-Matting algorithms are mostly static matting methods, lacking temporal consistency and utilization of depth information. This leads to flickering and unstable transparency in multi-frame video matting, and distorted blending of human boundaries with the background.

[0004] Furthermore, existing rendering methods generally use fixed thresholds to control the relationship between lighting and occlusion, which cannot adaptively adjust according to scene changes, resulting in unnatural transitions at the boundaries of the blended image and inconsistent lighting and shadows. In summary, existing technologies are insufficient in terms of temporal stability of pose prediction, depth consistency of matting, and adaptability of lighting and occlusion, making it difficult to meet the needs of XR virtual group photo scenes with high realism and emotional expression requirements, such as golden wedding anniversaries.

[0005] Therefore, how to provide an XR real-time group photo and virtual celebration system for golden wedding anniversaries is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an XR real-time group photo and virtual celebration system for golden wedding anniversaries. This invention combines the ConvGRU temporal prediction model and the Alpha-Matting depth-aware matting algorithm to achieve dynamic prediction of human poses and high-precision generation of transparent masks. It has the advantages of strong temporal stability, natural virtual-real fusion, and high rendering realism.

[0007] An XR real-time group photo and virtual celebration system for a golden wedding anniversary, according to an embodiment of the present invention, includes the following steps:

[0008] The data acquisition and preprocessing module is used to acquire and preprocess data, and output color images, depth maps, human key points and measured optical flow.

[0009] The ConvGRU dynamic filtering input module is used to form a temporal input feature sequence by sliding a fixed-length window for human key points, replace the convolution kernel with a variable convolution kernel, and receive the measured optical flow through the dynamic filtering network to generate sampling offset and convolution weights.

[0010] The ConvGRU temporal inference module is used to recursively calculate the temporal input feature sequence under the action of a variable convolution kernel, and output the prediction results of human pose change and the predicted optical flow map based on the sampling offset and convolution weight.

[0011] The Alpha-Matting pre-alignment module is used to perform optical flow alignment on the previous frame's transparent mask map using the predicted optical flow map to generate an initial migration mask map, and to generate a ternary region map based on the color image and depth map.

[0012] The Alpha-Matting inference and correction module takes a color image, a ternary region map, and an initial migration mask map as input, generates a current frame transparency mask map and a transparency mask confidence map, performs depth-guided smoothing and temporal consistency correction by combining the depth map and the predicted optical flow map, performs pixel-level weighted fusion based on the transparency mask confidence map, and outputs an optimized transparency mask map.

[0013] The rendering and output module is used to update the pose of the virtual 3D scene character based on the prediction results of the character pose change, optimize the transparency mask map and depth map to perform occlusion detection and physically based lighting rendering, and set feathering weights and occlusion thresholds in the uncertain boundary areas according to the transparency mask confidence map to generate a blended rendering image.

[0014] Optionally, modules can be integrated using the following methods:

[0015] S1, Data acquisition and preprocessing, to obtain color images, depth maps, human key points, and measured optical flow;

[0016] S2, ConvGRU dynamic filtering input and configuration, forms a temporal input feature sequence by sliding the human body key points in a fixed-length window, replaces the convolution kernel with a variable convolution kernel, and the dynamic filtering network receives the measured optical flow and generates sampling offset and convolution weight;

[0017] S3, ConvGRU temporal inference, recursively calculates the temporal input feature sequence under the action of a variable convolution kernel, and outputs the prediction results of human pose change and the predicted optical flow map based on the sampling offset and convolution weight;

[0018] S4, Alpha-Matting pre-alignment, uses the predicted optical flow map to align the previous frame's transparent mask map to generate the initial migration mask map, and generates a ternary region map based on the color image and depth map;

[0019] S5, Alpha-Matting Inference and Correction: Input color image, ternary region map, and initial migration mask map to obtain current frame transparent mask map and transparent mask confidence map. Use depth map and predicted optical flow map to perform depth-guided smoothing and temporal consistency correction. Based on the transparent mask confidence map, perform pixel-level weighted fusion of initial migration mask map and current frame transparent mask map to output optimized transparent mask map.

[0020] S6, rendering and output, updates the virtual 3D scene character pose based on the prediction results of character pose changes, uses optimized transparent mask map and depth map to complete occlusion detection and physically based lighting rendering, sets feathering weight and occlusion judgment threshold in the uncertain boundary area using transparent mask confidence map, and generates a blended rendering image.

[0021] Optionally, S2 specifically includes:

[0022] S21. Arrange the human body key point sequence in chronological order and divide it into several continuous frame segments with a fixed-length sliding window. The key point position coordinates in each frame segment are normalized to form a temporal input feature sequence.

[0023] S22. Initialize the dynamic filtering network by replacing the original fixed convolution kernel in the standard convolution gated recurrent unit with a space-time variable convolution kernel. The weights of the variable convolution kernel are generated in real time by the dynamic filtering network.

[0024] S23. The dynamic filtering network receives the measured optical flow as input, extracts the spatial gradient and temporal variation features of the optical flow, and calculates and generates a sampling offset matrix as the dynamic displacement of the convolution kernel in spatial position, and a convolution weight matrix as the adaptive weighting coefficient of the convolution kernel in the channel and time dimensions.

[0025] S24. Within each convolution window, the sampling coordinates of the input feature map are translated in the direction of optical flow according to the sampling offset matrix, and the feature values ​​of each sampling point are weighted and summed according to the convolution weight matrix to obtain dynamic convolution in the spatial dimension.

[0026] S25. Apply the sampling offset matrix and convolution weight matrix to the key point feature frames within the sliding window, and perform an exponential weighted average on each key point feature frame to generate a temporal feature representation.

[0027] Optionally, the dynamic filtering network in S22 specifically includes:

[0028] S221. Construct the input tensor of the dynamic filtering network, which is composed of measured optical flow map, depth map and key point feature map spliced ​​by channel, and extracts spatial and temporal features through convolutional layer and nonlinear activation layer.

[0029] S222, The dynamic filtering network outputs a sampling offset matrix and a convolution weight matrix. The sampling offset matrix is ​​limited to a set spatial displacement range, and the convolution weight matrix is ​​normalized by softmax.

[0030] S223. During training, L2 regularization and gradient smoothing constraints are introduced to impose sparsity constraints on the convolution weight matrix.

[0031] Optionally, S3 specifically includes:

[0032] S31. Using temporal feature representation as input, construct the recursive structure of ConvGRU (Convolutional Gated Recurrent Unit), which includes update gate, reset gate and candidate hidden state calculation unit.

[0033] S32. In each time step, the input temporal feature representation and the hidden state of the previous time step are convolved. The spatial-temporal variable convolution kernel is used in the convolution operation to obtain the input feature convolution result and the previous hidden state convolution result.

[0034] S33. The update gate sets the retention ratio of the current hidden state by weighted summing of the input feature convolution result and the previous hidden state convolution result and mapping through the Sigmoid function.

[0035] S34. The reset gate sets the influence range of historical information in the current calculation by weighted summation of input features and the previous hidden state and mapping through the Sigmoid function.

[0036] S35. The candidate hidden state is obtained by the input features and the reset hidden state participating in convolution and nonlinear transformation. The sampling offset matrix and the convolution weight matrix also act on the convolution process.

[0037] S36. The previous hidden state and the candidate hidden states are weighted and summed according to the update gate output to generate the current hidden state;

[0038] S37. The current hidden state is linearly transformed and the predicted character posture change result and the predicted optical flow map are output.

[0039] Optionally, S4 specifically includes:

[0040] S41. Receive the predicted optical flow map and the previous frame transparent mask map, perform smooth interpolation and noise suppression processing on the predicted optical flow map to form a smooth optical flow field. If it is the first frame input, set the zero optical flow field as the smooth optical flow field. Generate the initial transparent mask map as the previous frame transparent mask map by human body region segmentation based on the current color image and depth map.

[0041] S42. Calculate the motion vector of each pixel in the time series based on the smooth optical flow field, and map the corresponding pixels in the previous frame's transparent mask image along the direction of the motion vector to generate an initial migration mask image that is aligned with the spatial position of the current frame.

[0042] S43. Based on the current frame color image and depth map, extract color gradient features and depth change features, and combine gradient magnitude and depth difference threshold to divide the foreground region, background region and uncertain transition region to generate a three-value region map.

[0043] S44. The color gradient features and depth change features are fused to form a three-value region map. By setting thresholds for the foreground, background and uncertain regions, the region is divided into foreground region, background region and transition region.

[0044] Optionally, S5 specifically includes:

[0045] S51. Input a color image, depth map, initial migration mask map and ternary region map, and input a predicted optical flow map; in the ternary region map, use the foreground region as the foreground constraint, the background region as the background constraint, and the uncertain transition region as the transparency region to be estimated.

[0046] S52. Based on the predicted optical flow map, reverse coordinate mapping is performed on the initial migration mask map, and bilinear interpolation is used to calculate the initial value of the aligned transparent mask.

[0047] S53. Within the area of ​​transparency to be estimated, a color linear blending model is established using closed matting technology. The pixel transparency is solved based on foreground and background constraints, and the current frame transparency mask image is output.

[0048] S54. The temporal consistency error is calculated as the pixel-by-pixel difference between the current frame's transparent mask image and the initial value of the aligned transparent mask. An exponentially weighted average is used for temporal fusion to generate a temporally smoothed transparent mask image. The temporal consistency error is defined as the L2 norm difference between the current transparent mask image and the initial value of the aligned mask.

[0049] S55. Guided by the depth map, a depth-guided bilateral filter is used to smooth the temporal smooth transparent mask map, and a depth-smooth transparent mask map is output.

[0050] S56. Construct a confidence map of the transparent mask. The confidence of the transparent mask is obtained by weighted summation of the alignment residual, color reconstruction residual and depth gradient and then mapped by the Sigmoid function.

[0051] S57. Based on the confidence map of the transparent mask, perform pixel-level weighted fusion of the depth smooth transparent mask map and the initial value of the aligned transparent mask to output the optimized transparent mask map.

[0052] Optionally, S6 specifically includes:

[0053] S61. Receive the prediction results of the character's pose change, optimize the transparent mask map and the depth map, and use the prediction results of the character's pose change to update the pose of the character model in the virtual 3D scene.

[0054] S62. Perform occlusion detection based on the optimized transparent mask map and depth map, and use Z-Buffer occlusion removal technology to determine the visibility of foreground and background pixels;

[0055] S63. In the virtual scene rendering stage, PBR-based rendering technology is used to calculate lighting, reflection and shadow to achieve the fusion of virtual characters and scenes under real lighting conditions.

[0056] S64. In the rendering output stage, feathering weights and occlusion judgment thresholds are set in the uncertain boundary areas using the transparency mask confidence map, and multi-layer alpha blending rendering is performed to generate a blended rendering image.

[0057] The beneficial effects of this invention are:

[0058] (1) The present invention adopts a ConvGRU dynamic filtering structure with spatial-temporal variable convolution kernel, which can adaptively adjust the convolution sampling position according to optical flow and spatial features, realize smooth prediction of human pose in time series, effectively reduce inter-frame jumps and deformations, and improve the temporal continuity and stability of pose prediction.

[0059] (2) In the Alpha-Matting process, the present invention introduces a temporal depth perception mechanism. By combining optical flow alignment, temporal smoothing and depth-guided bilateral filtering, the consistency and boundary accuracy of the transparent mask in multiple frames of video are improved, and the problem of flickering and jittering in complex lighting and occlusion scenes of traditional matting algorithms is solved.

[0060] (3) This invention utilizes a transparent mask confidence map to achieve dynamic weighted fusion, and adaptively adjusts the feathering weight and occlusion threshold according to the confidence during the rendering stage, so that the virtual character and the real background are more naturally integrated in terms of lighting, shadow and edge transition, and the overall visual realism and immersive experience of XR real-time group photos and virtual celebrations are improved. Attached Figure Description

[0061] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0062] Figure 1 This is a schematic diagram of the overall structure of an XR real-time group photo and virtual celebration system for a golden wedding anniversary proposed in this invention;

[0063] Figure 2This is a schematic diagram of the ConvGRU dynamic filtering input structure of an XR real-time group photo and virtual celebration system for golden wedding anniversaries proposed in this invention.

[0064] Figure 3 This is a schematic diagram of the Alpha-Matting inference and correction process for an XR real-time group photo and virtual celebration system for a golden wedding anniversary proposed in this invention. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0066] refer to Figure 1-3 An XR real-time photo and virtual celebration system for a golden wedding anniversary includes:

[0067] The data acquisition and preprocessing module is used to acquire and preprocess data, and output color images, depth maps, human key points and measured optical flow.

[0068] The ConvGRU dynamic filtering input module is used to form a temporal input feature sequence by sliding a fixed-length window for human key points, replace the convolution kernel with a variable convolution kernel, and receive the measured optical flow through the dynamic filtering network to generate sampling offset and convolution weights.

[0069] The ConvGRU temporal inference module is used to recursively calculate the temporal input feature sequence under the action of a variable convolution kernel, and output the prediction results of human pose change and the predicted optical flow map based on the sampling offset and convolution weight.

[0070] The Alpha-Matting pre-alignment module is used to perform optical flow alignment on the previous frame's transparent mask map using the predicted optical flow map to generate an initial migration mask map, and to generate a ternary region map based on the color image and depth map.

[0071] The Alpha-Matting inference and correction module takes a color image, a ternary region map, and an initial migration mask map as input, generates a current frame transparency mask map and a transparency mask confidence map, performs depth-guided smoothing and temporal consistency correction by combining the depth map and the predicted optical flow map, performs pixel-level weighted fusion based on the transparency mask confidence map, and outputs an optimized transparency mask map.

[0072] The rendering and output module is used to update the pose of the virtual 3D scene character based on the prediction results of the character pose change, optimize the transparency mask map and depth map to perform occlusion detection and physically based lighting rendering, and set feathering weights and occlusion thresholds in the uncertain boundary areas according to the transparency mask confidence map to generate a blended rendering image.

[0073] In this embodiment, the modules are interconnected using the following method:

[0074] S1, Data acquisition and preprocessing, to obtain color images, depth maps, human key points, and measured optical flow;

[0075] S2, ConvGRU dynamic filtering input and configuration, forms a temporal input feature sequence by sliding the human body key points in a fixed-length window, replaces the convolution kernel with a variable convolution kernel, and the dynamic filtering network receives the measured optical flow and generates sampling offset and convolution weight;

[0076] S3, ConvGRU temporal inference, recursively calculates the temporal input feature sequence under the action of a variable convolution kernel, and outputs the prediction results of human pose change and the predicted optical flow map based on the sampling offset and convolution weight;

[0077] S4, Alpha-Matting pre-alignment, uses the predicted optical flow map to align the previous frame's transparent mask map to generate the initial migration mask map, and generates a ternary region map based on the color image and depth map;

[0078] S5, Alpha-Matting Inference and Correction: Input color image, ternary region map, and initial migration mask map to obtain current frame transparent mask map and transparent mask confidence map. Use depth map and predicted optical flow map to perform depth-guided smoothing and temporal consistency correction. Based on the transparent mask confidence map, perform pixel-level weighted fusion of initial migration mask map and current frame transparent mask map to output optimized transparent mask map.

[0079] S6, rendering and output, updates the virtual 3D scene character pose based on the prediction results of character pose changes, uses optimized transparent mask map and depth map to complete occlusion detection and physically based lighting rendering, sets feathering weight and occlusion judgment threshold in the uncertain boundary area using transparent mask confidence map, and generates a blended rendering image.

[0080] In this embodiment, S2 specifically includes:

[0081] S21. Arrange the human body key point sequence in chronological order and divide it into several continuous frame segments with a fixed-length sliding window. The key point position coordinates in each frame segment are normalized to form a temporal input feature sequence.

[0082] S22. Initialize the dynamic filtering network by replacing the original fixed convolution kernel in the standard convolution gated recurrent unit with a space-time variable convolution kernel. The weights of the variable convolution kernel are generated in real time by the dynamic filtering network.

[0083] S23. The dynamic filtering network receives the measured optical flow as input, extracts the spatial gradient and temporal variation features of the optical flow, and calculates and generates a sampling offset matrix as the dynamic displacement of the convolution kernel in spatial position, and a convolution weight matrix as the adaptive weighting coefficient of the convolution kernel in the channel and time dimensions.

[0084] S24. Within each convolution window, the sampling coordinates of the input feature map are translated in the direction of optical flow according to the sampling offset matrix, and the feature values ​​of each sampling point are weighted and summed according to the convolution weight matrix to obtain dynamic convolution in the spatial dimension.

[0085] S25. Apply the sampling offset matrix and convolution weight matrix to the key point feature frames within the sliding window, and perform an exponential weighted average on each key point feature frame to generate a temporal feature representation. In this embodiment, the time smoothing coefficient is set to 0.5, thus obtaining the following weights: current frame weight is 0.571; previous frame weight is 0.286; and second frame weight is 0.143.

[0086] In this embodiment, the dynamic filtering network in S22 specifically includes:

[0087] S221. Construct the input tensor of the dynamic filtering network, which is composed of measured optical flow map, depth map and key point feature map spliced ​​by channel, and extracts spatial and temporal features through convolutional layer and nonlinear activation layer.

[0088] S222, The dynamic filtering network outputs a sampling offset matrix and a convolution weight matrix. The sampling offset matrix is ​​limited to a set spatial displacement range, and the convolution weight matrix is ​​normalized by softmax.

[0089] S223. During training, L2 regularization and gradient smoothing constraints are introduced to impose sparsity constraints on the convolution weight matrix.

[0090] In this embodiment, the spatial-temporal variable convolution kernel calculation process in S22 specifically includes:

[0091] S224, sampling offset matrix and convolution weight matrix. Each position in the sampling offset matrix contains two parameters, representing the displacement in the horizontal and vertical directions respectively, with values ​​limited to between -2 pixels and +2 pixels, used to determine new sampling coordinates on the input feature map; each weight in the convolution weight matrix ranges from 0 to 1 and is normalized so that the sum of weights in the same convolution region is equal to 1, used to control the relative contribution of the feature values ​​of each sampling point in the convolution calculation;

[0092] S225. In the spatial computation process, the system takes a 3×3 convolution window as an example and performs 9 offset samplings for each output pixel position. Each sampling determines the offset coordinates according to the sampling offset matrix and performs bilinear interpolation on the four closest pixel values ​​in the input feature map. The feature values ​​of the 9 sampling points are multiplied by the corresponding weight coefficients in the convolution weight matrix, and the results are summed to obtain the spatial convolution output at that position.

[0093] S226. During the time calculation process, the system performs time fusion within three consecutive frames. Assuming the current frame is frame t, and the previous two frames are frames t-1 and t-2, the output features are smoothly fused in the time dimension through exponential weighted averaging. Specifically, the weight of the convolution result in frame t is 0.6, the weight of frame t-1 is 0.3, and the weight of frame t-2 is 0.1. The results of the three frames are weighted and summed according to the corresponding coefficients and then normalized to form the final time fusion output.

[0094] In this embodiment, S3 specifically includes:

[0095] S31. Using temporal feature representation as input, construct the recursive structure of ConvGRU (Convolutional Gated Recurrent Unit), which includes update gate, reset gate and candidate hidden state calculation unit.

[0096] S32. In each time step, the input temporal feature representation and the hidden state of the previous time step are convolved. The spatial-temporal variable convolution kernel is used in the convolution operation to obtain the input feature convolution result and the previous hidden state convolution result.

[0097] S33. The update gate sets the retention ratio of the current hidden state by weighted summing of the input feature convolution result and the previous hidden state convolution result and mapping through the Sigmoid function.

[0098] S34. The reset gate sets the influence range of historical information in the current calculation by weighted summation of input features and the previous hidden state and mapping through the Sigmoid function.

[0099] S35. The candidate hidden state is obtained by the input features and the reset hidden state participating in convolution and nonlinear transformation. The sampling offset matrix and the convolution weight matrix also act on the convolution process. In this embodiment, the nonlinear transformation adopts the hyperbolic tangent function.

[0100] S36. The previous hidden state and the candidate hidden states are weighted and summed according to the update gate output to generate the current hidden state; the weight of the weighted sum is the retention ratio of the current hidden state in S33.

[0101] S37. The current hidden state is linearly transformed and the predicted character posture change result and the predicted optical flow map are output. In this embodiment, the linear transformation is implemented using a 1×1 convolutional layer.

[0102] In this embodiment, S4 specifically includes:

[0103] S41. Receive the predicted optical flow map and the previous frame transparent mask map, perform smooth interpolation and noise suppression processing on the predicted optical flow map to form a smooth optical flow field. If it is the first frame input, set the zero optical flow field as the smooth optical flow field. Generate the initial transparent mask map as the previous frame transparent mask map by human body region segmentation based on the current color image and depth map.

[0104] S42. Calculate the motion vector of each pixel in the time series based on the smooth optical flow field, and map the corresponding pixels in the previous frame's transparent mask image along the direction of the motion vector to generate an initial migration mask image that is aligned with the spatial position of the current frame.

[0105] S43. Based on the current frame color image and depth map, extract color gradient features and depth change features, and combine gradient magnitude and depth difference threshold to divide the foreground region, background region and uncertain transition region to generate a three-value region map.

[0106] S44. The color gradient features and depth change features are fused to form a three-value region map. By setting thresholds for the foreground, background and uncertain regions, the region is divided into foreground region, background region and transition region.

[0107] In this embodiment, S44 specifically includes:

[0108] S441. Extract color gradient features from the color image. Use the Sobel operator in the existing technology to calculate the brightness gradient of each pixel in the horizontal and vertical directions, and obtain the gradient magnitude to get the color gradient feature map.

[0109] S442. Perform depth change feature extraction on the depth map. Use the neighborhood difference calculation method in the existing technology to calculate the depth difference between adjacent pixels and take the absolute sum to form a depth change feature map to reflect the degree of depth jump on the object surface.

[0110] S443. Weighted fusion of color gradient features and depth change features, and calculation of fusion confidence value. The fusion calculation adopts the linear weighting method designed in this invention:

[0111] Fusion confidence = 0.6 × (color gradient intensity / 255) + 0.4 × depth variation value;

[0112] The color gradient intensity and depth variation values ​​were both normalized to the range of 0 to 1.

[0113] S444, the color gradient threshold is set to 25, the depth change threshold is set to 0.15, pixels with a fusion confidence higher than 0.7 are classified as foreground regions, pixels with a confidence lower than 0.3 are classified as background regions, and pixels between 0.3 and 0.7 are classified as uncertain transition regions.

[0114] S445. Generate a three-value region map by labeling the foreground region, background region, and transition region with different masks, where the foreground region value is set to 1, the background region value is set to 0, and the transition region value is set to 0.5.

[0115] In this embodiment, S5 specifically includes:

[0116] S51. Input a color image, depth map, initial migration mask map and ternary region map, and input a predicted optical flow map; in the ternary region map, use the foreground region as the foreground constraint, the background region as the background constraint, and the uncertain transition region as the transparency region to be estimated.

[0117] S52. Based on the predicted optical flow map, reverse coordinate mapping is performed on the initial migration mask map, and bilinear interpolation is used to calculate the initial value of the aligned transparent mask.

[0118] S53. Within the area of ​​transparency to be estimated, a color linear blending model is established using closed matting technology. The pixel transparency is solved based on foreground and background constraints, and the current frame transparency mask image is output.

[0119] S54. The temporal consistency error is calculated as the pixel-by-pixel difference between the current frame transparent mask image and the initial value of the aligned transparent mask. An exponentially weighted average is used for temporal fusion to generate a temporally smooth transparent mask image. The temporal consistency error is defined as the L2 norm difference between the current transparent mask image and the initial value of the aligned mask. In this embodiment, the exponentially weighted average is the same as in S226.

[0120] S55. Guided by the depth map, a depth-guided bilateral filter is used to smooth the temporal smooth transparent mask map, and a depth-smooth transparent mask map is output.

[0121] S56. Construct a confidence map for the transparent mask. The system calculates three indicators: alignment residual, color reconstruction residual, and depth gradient. The alignment residual adopts the temporal mask difference method, the color reconstruction residual adopts the closed matting color linear mixing model, and the depth gradient adopts the Sobel operator. After normalization, the three indicators are weighted and summed with fixed weights of 0.4, 0.4, and 0.2, and the confidence map for the transparent mask is obtained by mapping through the Sigmoid function.

[0122] S57. Based on the confidence map of the transparent mask, perform pixel-level weighted fusion of the depth smooth transparent mask map and the initial value of the aligned transparent mask to output the optimized transparent mask map.

[0123] In this embodiment, S6 specifically includes:

[0124] S61. Receive the prediction results of the character's pose change, optimize the transparent mask map and the depth map, and use the prediction results of the character's pose change to update the pose of the character model in the virtual 3D scene.

[0125] S62. Perform occlusion detection based on the optimized transparent mask map and depth map, and use Z-Buffer occlusion removal technology to determine the visibility of foreground and background pixels;

[0126] S63. In the virtual scene rendering stage, PBR-based rendering technology is used to calculate lighting, reflection and shadow to achieve the fusion of virtual characters and scenes under real lighting conditions.

[0127] S64. In the rendering output stage, feathering weights and occlusion judgment thresholds are set in the uncertain boundary areas using the transparency mask confidence map, and multi-layer alpha blending rendering is performed to generate a blended rendering image.

[0128] Example 1:

[0129] To verify the feasibility of this invention in practice, it was applied to a virtual celebration scene for a golden wedding anniversary. This scene simulates a real anniversary ceremony, with participants including two elders and virtual relatives. Real images were captured by a camera and then merged into a virtual image in real time. The experimental environment was a standard indoor scene, equipped with RGB-D sensors and a high-performance computing terminal for acquiring color images, depth maps, human key points, and measured optical flow. The purpose of the experiment was to verify the system's stability, accuracy, and rendering consistency under complex lighting, character movement, and occlusion conditions.

[0130] During implementation, the system first acquires multi-source input data through a data acquisition and preprocessing module, performing image denoising, depth normalization, and keypoint localization. In the ConvGRU dynamic filtering input module, the human keypoint sequence is grouped chronologically to form a sliding window feature, and a spatial-temporal variable convolution kernel is used to calculate the temporal features. The dynamic filtering network receives measured optical flow in real time, generating a sampling offset matrix and a convolution weight matrix to achieve adaptive perception of dynamic pose changes. This process enables the network to adjust the sampling position according to changes in optical flow, improving the stability and accuracy of keypoint prediction.

[0131] The ConvGRU temporal inference module performs pose prediction and optical flow estimation within a temporally recursive structure. Through update and reset gate mechanisms, the system balances the weights of historical information and the current input across consecutive frames, dynamically adjusting the hidden state retention ratio and outputting a smoother temporal feature representation. This module effectively avoids the temporal drift problem caused by traditional fixed convolutional kernels under non-uniform motion. The predicted optical flow map and pose results then proceed to the next stage of matting and fusion.

[0132] In the Alpha-Matting pre-alignment module, the system performs optical flow alignment on the previous frame's transparent mask image based on the predicted optical flow, generates an initial transition mask image, and combines the depth map and color gradient to generate a ternary region map, clearly distinguishing the foreground, background, and transition regions.

[0133] The Alpha-Matting inference and correction module takes a color image, a depth map, and an initial migration mask image as input. It uses a closed-loop matting algorithm to establish a linear color blending model and calculates the transparency distribution. Subsequently, the system utilizes depth-guided bilateral filtering and temporal consistency correction algorithms for smoothing optimization, ultimately outputting an optimized transparent mask image and constructing a transparent mask confidence map to guide pixel-level fusion. In experiments, confidence calculation combines alignment residuals, color reconstruction residuals, and depth gradients. After mapping with the Sigmoid function, a confidence value in the 0-1 range is obtained, allowing for dynamic adjustment of feathering weights in uncertain boundary regions.

[0134] The rendering and output module updates the 3D pose of the virtual character based on the predicted pose, adopts PBR-based lighting rendering technology, and performs occlusion detection and fusion rendering by combining optimized transparent mask map and depth map to achieve a natural photo of the real character and the virtual scene.

[0135] Experimental results show that even under dynamic lighting, occlusion, and significant changes in motion, the system can still achieve stable real-time rendering output, with natural visual boundary transitions and high overall blending. To verify the technical effectiveness of this invention, the system was compared with the traditional KNN Matting algorithm and the standard ConvGRU model. Test metrics included pose prediction error, mask smoothness, mask jitter amplitude, average frame rate, and subjective visual consistency score.

[0136] Table 1: Performance Results of Each Model

[0137] Indicator Name Results of traditional methods Results of the Invention Increase (%) Measurement method Average error of pose prediction (pixels) 4.85 2.91 39.9 Key point Euclidean distance Transparent mask edge smoothness index 0.72 0.88 22.2 Gradient consistency coefficient Mask jitter amplitude (standard deviation) 0.061 0.028 54.1 Time series standard deviation Average processing frame rate (FPS) 24.6 30.2 22.8 Real-time rendering test Subjective consistency score of fused images 3.9 4.7 20.5 Manual assessment Occlusion detection accuracy (%) 83.4 91.7 9.9 Occlusion pixel detection Boundary transparency consistency 0.67 0.85 26.9 Boundary transparency similarity

[0138] As shown in Table 1, this invention significantly outperforms traditional methods in multiple performance dimensions. The pose prediction error is reduced by approximately 39.9%, indicating that the ConvGRU dynamic filtering module effectively improves the temporal feature modeling capability and reduces prediction offset caused by motion blur. The smoothness of the transparent mask edges is improved by 22.2%, proving that the Alpha-Matting temporal depth perception algorithm can achieve more natural transitions at complex boundaries. The mask jitter amplitude is reduced by 54.1%, indicating that the temporal consistency correction mechanism significantly reduces flickering and abrupt changes in transparent areas. The average frame rate is increased to 30.2 FPS, meeting the requirements for real-time group photo rendering. The subjective score is improved by 20.5%, verifying the naturalness and immersiveness of the visual fusion effect. The occlusion detection accuracy is improved to 91.7%, indicating that the depth guidance and confidence fusion strategy has higher accuracy in occlusion area detection. In summary, this invention effectively solves the problems of pose prediction delay, boundary spikes and lighting inconsistencies in traditional XR real-time group photography by combining ConvGRU dynamic filtering with Alpha-Matting depth matting. It achieves high-quality virtual fusion output in uncontrolled lighting and multi-person scenes, with high robustness and real-time performance. This verifies the feasibility and promotional value of this invention in golden wedding anniversary celebrations and other human-computer fusion scenarios.

[0139] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A system for Golden Wedding Anniversary XR Real Time Group Photo and Virtual Celebration, characterized in that, Comprise: a data acquisition and preprocessing module for acquiring and preprocessing data, outputting color images, depth maps, human key points and measured optical flow; a ConvGRU dynamic filtering input module for forming a time sequence input feature sequence by sliding a fixed length window on the human key points, replacing the convolution kernel with a variable convolution kernel, and generating sampling offsets and convolution weights by receiving the measured optical flow through a dynamic filtering network; a ConvGRU time sequence inference module for recursively calculating the time sequence input feature sequence under the action of the variable convolution kernel, and outputting a human pose change prediction result and a predicted optical flow map based on the sampling offsets and the convolution weights; an Alpha-Matting pre-alignment module for generating an initial migration mask map by aligning the previous frame transparent mask map using the predicted optical flow map, and generating a three-value region map based on the color image and the depth map; an Alpha-Matting inference and correction module for inputting the color image, the three-value region map and the initial migration mask map, generating a current frame transparent mask map and a transparent mask confidence map, performing depth guided smoothing and temporal consistency correction using the depth map and the predicted optical flow map, and performing pixel-level weighted fusion based on the transparent mask confidence map to output an optimized transparent mask map; a rendering and output module for updating the virtual three-dimensional scene character pose according to the human pose change prediction result, performing occlusion detection and physically-based lighting rendering using the optimized transparent mask map and the depth map, setting feathering weights and occlusion thresholds in the boundary uncertain area according to the transparent mask confidence map, and generating a fusion rendering image.

2. The XR real-time photo and virtual celebration system for golden wedding anniversary as claimed in claim 1 wherein, The modules are realized by the following methods: S1, data acquisition and preprocessing, to obtain color images, depth maps, human key points and measured optical flow; S2, ConvGRU dynamic filtering input and configuration, forming a time sequence input feature sequence by sliding a fixed length window on the human key points, replacing the convolution kernel with a variable convolution kernel, and generating sampling offsets and convolution weights by receiving the measured optical flow through a dynamic filtering network; S3, ConvGRU time sequence inference, recursively calculating the time sequence input feature sequence under the action of the variable convolution kernel, and outputting a human pose change prediction result and a predicted optical flow map based on the sampling offsets and the convolution weights; S4, Alpha-Matting pre-alignment, generating an initial migration mask map by aligning the previous frame transparent mask map using the predicted optical flow map, and generating a three-value region map based on the color image and the depth map; S5, Alpha-Matting inference and correction, inputting the color image, the three-value region map and the initial migration mask map, obtaining a current frame transparent mask map and a transparent mask confidence map, performing depth guided smoothing and temporal consistency correction using the depth map and the predicted optical flow map, and performing pixel-level weighted fusion based on the transparent mask confidence map to output an optimized transparent mask map; S6, rendering and outputting, updating the virtual three-dimensional scene character pose according to the character pose change prediction result, using the optimized transparent mask and depth map to complete the occlusion detection and physically-based lighting rendering, setting the feathering weight and occlusion judgment threshold in the boundary uncertain area with the transparent mask confidence map, and generating a fusion rendering image.

3. The XR real-time photo and virtual celebration system for golden wedding anniversary as claimed in claim 2 wherein, The S2 specifically comprises: S21, arranging the human body key point sequence in time sequence, and dividing into a plurality of continuous frame segments by a fixed length sliding window, the key point position coordinates in each frame segment being composed into a time sequence input feature sequence after normalization; S22, initializing a dynamic filtering network, replacing the original fixed convolution kernel in the standard convolution gated recurrent unit with a space-time variable convolution kernel, the weight of the variable convolution kernel being generated in real time by the dynamic filtering network; S23, the dynamic filtering network receiving the measured optical flow as input, extracting the spatial gradient and time change feature of the optical flow, calculating a sampling offset matrix as the dynamic displacement of the convolution kernel in the spatial position, and a convolution weight matrix as the adaptive weighting coefficient of the convolution kernel in the channel and time dimension; S24, in each convolution window, the sampling coordinates of the input feature map are translated in the optical flow direction according to the sampling offset matrix, and the feature values of each sampling point are weighted and summed according to the convolution weight matrix, to obtain dynamic convolution in the spatial dimension; S25, the sampling offset matrix and the convolution weight matrix are applied to the key point feature frames in the sliding window, and the key point feature frames are exponentially weighted and averaged to generate a time sequence feature representation.

4. The XR real-time photo and virtual celebration system for golden wedding anniversary as claimed in claim 3 wherein, The dynamic filtering network in the S22 specifically comprises: S221, constructing an input tensor of the dynamic filtering network, composed of the measured optical flow map, the depth map and the key point feature map by channel splicing, extracting spatial and temporal features through convolution layers and nonlinear activation layers; S222, the dynamic filtering network outputs a sampling offset matrix and a convolution weight matrix, the sampling offset matrix being limited within a set spatial displacement range, and the convolution weight matrix being normalized by softmax; S223, introducing L2 regularization constraint and gradient smoothing constraint in the training process, and imposing sparsity constraint on the convolution weight matrix.

5. The XR real-time photo and virtual celebration system for golden wedding anniversary as claimed in claim 3 wherein, The S3 specifically comprises: S31, constructing a recursive structure of the convolution gated recurrent unit ConvGRU with the time sequence feature representation as input, including an update gate, a reset gate and a candidate hidden state calculation unit; S32, in each time step, the input time sequence feature representation and the hidden state at the last time are respectively subjected to convolution operation, and a space-time variable convolution kernel is used in the convolution operation to obtain the input feature convolution result and the last hidden state convolution result; S33, the update gate sets the retention proportion of the current hidden state by weighted summation of the input feature convolution result and the last hidden state convolution result and mapping through the Sigmoid function; S34, the reset gate sets the influence range of historical information in the current calculation by weighted summation of the input feature and the last hidden state and mapping through the Sigmoid function; S35, the candidate hidden state is calculated by convolution and nonlinear transformation of the input feature and the reset hidden state, and the sampling offset matrix and the convolution weight matrix also act on the convolution process; S36, the previous hidden state and the candidate hidden state are weighted and summed according to the update gate output to generate the current hidden state; S37, the current hidden state is linearly transformed to output the character pose change prediction result and the predicted optical flow map.

6. The XR real-time photo and virtual celebration system for golden wedding anniversary, as claimed in claim 5, wherein The S4 specifically comprises: S41, receiving the predicted optical flow map and the previous frame transparent mask map, performing smoothing interpolation and noise suppression processing on the predicted optical flow map to form a smooth optical flow field, setting a zero optical flow field as the smooth optical flow field if it is the first frame input, and generating an initial transparent mask map as the previous frame transparent mask map according to the current color image and the depth map through human region segmentation; S42, calculating the motion vector of each pixel in the time sequence according to the smooth optical flow field, and mapping the corresponding pixels in the previous frame transparent mask map along the motion vector direction to generate an initial migration mask map aligned with the spatial position of the current frame; S43, extracting color gradient features and depth change features according to the current frame color image and the depth map, combining gradient amplitude and depth difference threshold to divide foreground region, background region and uncertain transition region, and generating a three-value region map; S44, fusing the color gradient features and the depth change features to form a three-value region map, and dividing the foreground region, the background region and the transition region by setting the threshold values of the foreground, the background and the uncertain region.

7. The XR real-time photo and virtual celebration system for golden wedding anniversary, as claimed in claim 6, wherein The S5 specifically comprises: S51, inputting the color image, the depth map, the initial migration mask map and the three-value region map, and inputting the predicted optical flow map; in the three-value region map, the foreground region is regarded as a foreground constraint, the background region is regarded as a background constraint, and the uncertain transition region is regarded as a to-be-estimated transparency region; S52, performing reverse coordinate mapping on the initial migration mask map based on the predicted optical flow map, and calculating the aligned transparent mask initial value by using bilinear interpolation; S53, in the to-be-estimated transparency region, a color linear mixing model is established by using a closed-loop matting technology, the pixel transparency is solved according to the foreground and background constraints, and the current frame transparent mask map is output; S54, calculating the time sequence consistency error as the pixel-by-pixel difference between the current frame transparent mask map and the aligned transparent mask initial value, performing time dimension fusion by using exponential weighted average to generate a time sequence smooth transparent mask map; the time sequence consistency error is defined as the L2 norm difference between the current transparent mask map and the aligned mask initial value; S55, using the depth map as a guide, performing smoothing processing on the time sequence smooth transparent mask map by using depth-guided bilateral filtering to output a depth smooth transparent mask map; S56, constructing a transparent mask confidence map, and the transparent mask confidence is obtained by weighted summing the aligned residual error, the color reconstruction residual error and the depth gradient and then mapping by a Sigmoid function; S57, performing pixel-level weighted fusion on the depth smooth transparent mask map and the aligned transparent mask initial value according to the transparent mask confidence map to output an optimized transparent mask map.

8. The XR real-time photo and virtual celebration system for golden wedding anniversary, according to claim 7, wherein, The S6 specifically comprises: S61, receive the character pose change prediction result, optimize the transparent mask map and the depth map, and use the character pose change prediction result to update the pose of the character model in the virtual three-dimensional scene; S62, perform occlusion detection according to the optimized transparent mask map and the depth map, and use Z-Buffer occlusion blanking technology to judge the visibility of foreground and background pixels; S63, in the virtual scene rendering stage, use PBR rendering technology to calculate light, reflection and shadow, realize the fusion of virtual characters and scenes in real light environment; S64, in the rendering output stage, set the feathering weight and the occlusion judgment threshold in the boundary uncertain area by using the transparent mask confidence map, perform multi-layer Alpha blending rendering, and generate a fusion rendering image.

Images