A virtual foreground and background fusion method based on dense fusion and occlusion relationship reasoning

Abstract

This invention discloses a virtual foreground and background fusion method based on dense fusion and occlusion relationship inference, belonging to the field of XR virtual filming and computer vision fusion control technology. The method acquires binocular or multi-view images, lens parameters, camera pose, virtual scene depth layers, and timestamps. It uses SAM2 for foreground instance segmentation, RAFTStereo to estimate dense parallax, and uses Depth Anything V2 to perform monocular depth filling for low-texture, reflective, transparent edges, and occluded areas. Subsequently, a fusion depth map is generated based on residuals, edge gradients, instance categories, and historical stability. An occlusion relationship map is constructed, and boundary refinement, contact consistency verification, and low-confidence backoff are performed. This improves the spatial hierarchy consistency between people, vehicles, props, and virtual elements in XR studios, virtual press conferences, and live television broadcasts.

Description

Technical Field

[0001] This invention belongs to the fields of XR virtual filming, computer vision dense depth estimation, occlusion relationship reasoning, and virtual-real fusion control technology. Specifically, it relates to a foreground-background fusion method and system that links foreground instance segmentation, binocular or multi-view dense depth restoration, monocular depth filling, depth confidence fusion, occlusion relationship graph reasoning, boundary refinement, contact consistency verification, and low-confidence backoff control into the same virtual filming link. Background Technology

[0002] XR virtual scene creation typically requires simultaneously obtaining the real camera pose, foreground mask, and virtual scene rendering results, and then compositing the real foreground with the virtual background, virtual mid-scene objects, or virtual foreground objects. Existing technologies focus more on camera pose estimation, green screen keying, and virtual scene rendering output, while the processing of pixel-level depth relationships between people, vehicles, handheld props, and virtual objects is still not precise enough.

[0003] In real-world filming projects, common scenarios include a presenter's arm extending into front of a virtual screen, a vehicle's front wheels resting on a virtual platform, a microphone or prop obscuring virtual subtitles, an actor's feet touching virtual steps, and transparent props passing over virtual foreground elements. If virtual scenes are directly overlaid using only a two-dimensional foreground mask, it's easy to mistakenly place virtual elements that should be in the background in front, or to fully display parts that should be obscured by the foreground, resulting in issues such as overlapping, floating, blurred edges, and distorted depth of field.

[0004] Binocular vision, digital keying, motion capture, camera motion control, and virtual scene rendering have formed a relatively complete basic chain for XR virtual filming, providing camera pose, lens parameters, and foreground / background separation results. However, sparse features or single keying results are difficult to directly use for the stable sorting of densely occluded details such as presenter's hair, clothing hem, car window outline, microphone pole, low-texture green screen areas, and reflective areas.

[0005] While green screen environments are beneficial for foreground separation, the simple texture of the green screen itself, along with reflections, wrinkles, localized shadows, and near-green clothing, increases the difficulty of depth estimation and boundary judgment. Highlights on car bodies, screen reflections, transparent elements, and rapid foreground movement, common in press conferences and studios, further amplify errors in occlusion. Relying solely on manual layer adjustments or static rules often requires extensive on-site trial and error, making it difficult to meet the rapid production requirements of live broadcasts, real-time studios, and virtual press conferences.

[0006] Therefore, it is necessary to propose an XR virtual shooting foreground and background fusion scheme that integrates visual dense depth estimation, foreground instance understanding, occlusion relationship reasoning, boundary refinement, contact consistency verification, and low confidence backoff control, so that the occlusion sequence, contact boundary, and virtual-real overlay relationship can be automatically adjusted according to image content, depth confidence, and temporal state. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for foreground and background fusion in XR virtual shooting based on dense depth confidence fusion and occlusion relationship inference, so as to solve the problems of existing virtual shooting systems such as rough occlusion judgment, unstable boundary contact, floating feet or tires, clipping hands, unnatural semi-transparent edge overlay, and high cost of manual layer adjustment when the foreground and virtual objects have complex spatial interactions.

[0008] To achieve the above objectives, this invention constructs a complete technical chain consisting of image acquisition and time synchronization, foreground instance segmentation, dense disparity estimation, monocular depth completion, depth confidence fusion, occlusion relationship graph inference, boundary refinement and contact consistency verification, virtual-real layered compositing and rollback control, and archiving verification. This chain, based on existing camera pose, foreground segmentation, and virtual scene rendering, introduces pixel depth, instance occlusion, boundary states, and contact relationships, and uniformly converts them into engineering control parameters that can be directly applied to the compositing process.

[0009] The method of this invention includes: acquiring binocular or multi-view images, lens parameters, camera pose, virtual scene depth layer, and timestamps under the same time reference; obtaining real foreground instance masks, alpha boundaries, and propagation confidence using SAM2 temporal instance segmentation; performing dense disparity estimation on left and right view images or main and auxiliary camera images using RAFTStereo to obtain binocular depth maps; performing monocular depth filling on low-texture, reflective, transparent edges, and occluded monocular visible areas of green screen using Depth Anything V2; generating fusion weights based on disparity residuals, left-right consistency, edge gradients, instance categories, monocular confidence, pose consistency, and historical stability; constructing an occlusion relationship graph based on the fused depth map, and obtaining occlusion levels through depth difference, projection overlap, relative motion, contact prior, and historical stability; performing gated conditional random field refinement on the boundaries and performing contact consistency verification; and finally performing virtual-real layered synthesis or low-confidence backoff based on the quality score, and saving the verification record.

[0010] Compared with existing technologies, the present invention has at least the following beneficial effects: First, it fuses the binocular depth and monocular depth-filling results according to pixel confidence, so that low-texture, reflective, and locally occluded areas can still obtain continuous depth information that can be used for occlusion judgment; Second, it constructs an occlusion relationship graph by combining real instances, virtual instances, and contact areas, avoiding the error in the order of the entire layer when bending an arm, opening a car door, or holding a prop due to a single layer sorting; Third, it processes flexible fine edges and rigid contact areas respectively through gated conditional random fields and contact consistency checks, reducing the risk of edge blurring, tire suspension, and foot clipping; Fourth, through low-confidence backoff and archiving verification mechanisms, it prevents algorithm anomalies from directly entering the main output screen, improving the broadcast stability of XR live streaming and virtual press conferences. Attached Figure Description

[0011] Figure 1 Overall architecture diagram of XR virtual shooting foreground and background fusion system.

[0012] Figure 2 A schematic diagram showing the relationship between on-site data collection and the virtual / real depth layer.

[0013] Figure 3 Flowchart of foreground instance segmentation and dense disparity estimation.

[0014] Figure 4 Flowchart of monocular depth completion and depth confidence fusion.

[0015] Figure 5 Construction of occlusion relationship graph and occlusion hierarchy reasoning graph.

[0016] Figure 6 Flowchart for boundary refinement and contact consistency verification.

[0017] Figure 7 Virtual-real layered synthesis and low-confidence backoff control chart.

[0018] Figure 8 Diagram illustrating a closed-loop application across multiple scenarios. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the present invention is not limited to a specific binocular camera, motion capture device, rendering engine, mechanical tracking device, or gimbal device; as long as the same time reference can be provided for the real foreground image, lens parameters, camera pose, and virtual scene depth layer, the present invention can be implemented.

[0020] Combination Figure 1The system of this invention operates in the following sequence: "acquisition and synchronization—instance segmentation—dense disparity estimation—monocular depth completion—depth confidence fusion—occlusion relationship graph reasoning—boundary refinement and contact consistency verification—virtual-real layered synthesis and rollback control—archiving and verification." System inputs include binocular or multi-view images, lens parameters, camera pose, virtual scene depth layers, and timestamps. Outputs include instance masks, fused depth maps, occlusion levels, boundary confidence, synthesized frames, and anomaly verification records.

[0021] The image acquisition and time synchronization module receives synchronization frames from the main binocular camera, auxiliary camera, or main / auxiliary camera positions, and simultaneously receives lens focal length, distortion correction results, focus status, camera pose, and virtual scene depth layer. Camera pose can be provided by visual odometry, mechanical tracking, inertial measurement, or an external tracking link. The focus of this invention is not on repeatedly solving for the pose, but on jointly interpreting the image layer depth and virtual layer depth as occlusion control rules that can be used for foreground and background fusion.

[0022] Combination Figure 2 The on-site acquisition system can use either a main binocular camera or an auxiliary side camera added to the main camera. The presenter, actors, vehicles, handheld props, microphones, tabletops, feet, tires, and virtual ground contact lines are registered as key observation objects. The virtual engine provides depth information for virtual background layers, virtual object or subtitle layers, virtual ground or step layers, and other virtual element layers. The system establishes a correspondence between the real foreground image, synchronization relationships, and virtual scene depth layers within the same frame number.

[0023] In camera projection relationships, the system projects real or virtual spatial points onto the current image plane. The projection relationship can be represented as follows:

[0024]

[0025] In the formula, It is a pixel coordinate vector. For perspective projection operators, For the camera intrinsic parameter matrix, For rotation matrix Translation vector The camera extrinsic matrix is ​​composed of... This represents the three-dimensional coordinates of a point in space. This relationship is used to unify the depth layers of real space, camera images, and virtual scenes.

[0026] In the dense depth calculation stage, the system uses binocular or multi-view parallax to recover the true spatial depth. For pixel p, its binocular depth can be expressed as:

[0027]

[0028] In the formula, For pixels The corresponding spatial depth, The calibrated focal length. For binocular baseline, For pixels The disparity value.

[0029] The RAFTStereo branch iterates over the disparity field using a cyclic update mechanism, and the recursive relationship can be expressed as:

[0030]

[0031] In the formula, For the first The disparity field of the next iteration To iteratively update the disparity increment of the network output, This is the parallax field for the next iteration. This mechanism enables more continuous parallax propagation between neighboring pixels, making it suitable for objects such as close-up portraits, stage edges, and vehicle outlines.

[0032] Combination Figure 3 The foreground instance segmentation branch uses the SAM2 temporal segmentation network to obtain the instance mask of the real foreground object. Its processing relationship can be represented as follows:

[0033]

[0034] In the formula, The mask for the current frame instance. For the current frame image, For the previous frame or historical propagation mask, Provides a cue mask or interactive cue for the current frame. SAM2 outputs instance class, binary subject mask, consecutive alpha boundaries, and propagation confidence to preserve details such as hair strands, fingers, clothing hems, microphone booms, car door edges, and vehicle outlines, and to suppress outline flickering in consecutive frames.

[0035] The dense disparity estimation branch receives epipolar-corrected left and right eye images or master / slave camera images and outputs the disparity field, iterative residual, left-right consistency mask, and binocular depth map. During training or calibration, left and right eye repetition projection constraints can be used.

[0036]

[0037] In the formula, For photometric reprojection loss, and Left eye pixels The brightness or color of the reprojected pixel corresponding to the right eye. Foreground instance mask stability can be constrained by the following loss condition:

[0038]

[0039] In the formula, For instance mask consistency loss, Foreground pixel count For the current frame label or high-confidence mask, This is the network prediction mask. The loss is used to constrain the stability of the boundary and instance regions.

[0040] Combination Figure 4 The monocular depth estimation branch uses Depth Anything V2 to estimate the relative depth of the current main view and performs scale alignment using a calibration plane, a high-confidence binocular region, or a virtual scene reference plane. For low-texture green screens, reflective areas, edges of transparent objects, and unstable areas caused by occlusion, the system increases monocular depth involvement; for areas such as the ground, wheels, rigid props, and stage steps, binocular geometry is prioritized. The binocular depth weight at pixel p can be expressed as:

[0041]

[0042] In the formula, For binocular depth weights, For parallax residuals, For residual scaling parameters, For binocular depth confidence, For monocular depth confidence, and For the binocular and monocular participation factors corresponding to the instance categories, For historical stability, To prevent positive numbers with a denominator of zero, the corresponding monocular depth weight is used. satisfy The fusion depth can be expressed as:

[0043]

[0044] In the formula, For pixels The depth of fusion at that point For binocular depth, This is the result of monocular depth completion. After fusion, isolated jump points are suppressed by temporal median and motion compensation filtering, but smoothing is performed without crossing the confirmed instance boundaries.

[0045] Combination Figure 5The occlusion relationship graph reasoning module places real instances, virtual instances, and scene contact areas into a unified relationship graph. Graph nodes include the host, actors, vehicles, handheld props, microphones, virtual subtitles, virtual desktops, virtual platforms, virtual props, and ground contact areas; graph edges include blending depth difference, projection overlap ratio, relative motion trend, contact prior, and historical occlusion stability. The directed occlusion relationship score of instance i relative to instance j can be represented as:

[0046]

[0047] In the formula, For example relative instance The occlusion relationship score, The difference in fusion depth within the overlapping region, The projection overlap rate is... For the relative motion trend term, In order to access prior terms, For historical stability, to Here are the weighting coefficients. The occlusion ranking result can be expressed as:

[0048]

[0049] In the formula, For example Relative to instance The sorting results before and after. For positive representation instances Located in instance Previously, negative values ​​represented instances. Located in instance Afterwards, the occlusion graph satisfies antisymmetry and local transitivity: if instance A is before instance B, instance B cannot be simultaneously determined to be before instance A; in a locally connected subgraph without transparent layers, if A is before B and B is before C, the order of A and C remains consistent.

[0050] When both a foreground and a background relationship exist within the overlapping area of ​​the projections of a real foreground instance and a virtual asset instance, the system splits the instance into multiple visible partitions and generates occlusion masks for each. This mechanism is applicable to partial interweaving scenarios such as a bent arm, an open car door, a transparent car window, a handheld prop passing through a virtual screen, and a vehicle outline occluding a virtual light strip.

[0051] In depth smoothing, the system employs edge-aware smoothing, ensuring depth continuity within homogeneous regions while allowing for abrupt changes in actual depth at boundaries:

[0052]

[0053] In the formula, For edge-aware depth smoothing loss, and These are the partial derivatives of the fusion depth in the horizontal and vertical directions, respectively. and This represents the image gradient.

[0054] Combination Figure 6 The boundary refinement and contact consistency verification module is located after occlusion inference. This is because only by first determining whether the local occlusion belongs to the real foreground leading edge or the virtual foreground leading edge can we determine which side the boundary should contract, expand, or retain transparency towards. The boundary correction energy of a gated conditional random field can be expressed as:

[0055]

[0056] In the formula, Energy is used to correct the boundary. For the matting boundary term, For depth boundary terms, For image edge items, to Apply boundary weights. Reduce geometric shrinkage while maintaining continuous transparency in areas such as character hair, veils, clothing hems, and microphone cables; increase depth boundary constraints in areas such as vehicle tires, stage steps, tabletops, physical props, and virtual ground contact lines to prevent flexible edges from being cut off or rigid edges from becoming blurry.

[0057] Contact consistency checking is used to examine the spatial relationship between the soles of feet, tires, props, or platforms and the virtual plane. Its loss can be expressed as:

[0058]

[0059] In the formula, To mitigate consistency loss, For the confidence weight of the contact point, The depth of contact point between the foot, tire, or prop. This corresponds to the virtual plane depth. When the contact depth difference exceeds the threshold, the system does not output the current occlusion result directly. Instead, it reduces the weight of the relevant virtual front layer, performs local boundary correction or contact area reordering, and writes the abnormal area into the review record.

[0060] Combination Figure 7 In the compositing stage, the final image is broken down into a real foreground layer, a virtual background layer, a virtual midground layer, a virtual foreground layer, and a shadow layer. An element is only allowed to enter the virtual foreground layer if the occlusion graph confirms that the virtual element should be in front of the real foreground and the quality score meets the threshold. The final composite color can be represented as:

[0061]

[0062] In the formula, For the final synthesized color, The true foreground color. To satisfy the virtual foreground color requirement of front occlusion, Virtual background color, to The is the synthesis coefficient determined by the occlusion level and Alpha, and whose sum is 1.

[0063] The occlusion blending quality score is generated based on reprojection error, temporal stability, contact consistency, and boundary integrity, and can be expressed as:

[0064]

[0065] In the formula, To assess the quality of occlusion blending, This is the normalized value of the reprojection error. For time stability, For contact consistency score, For boundary integrity, to , where is the weighting coefficient. When the score is higher than the passing threshold, the system enables a virtual foreground layer based on the graph inference results; when the score is in the review interval, the most recent high-confidence occlusion relationship is maintained and the prediction validity period is shortened; when the score is lower than the safety threshold, high-risk virtual foreground objects are frozen, the real foreground and virtual background are preserved, and the coordinates of the abnormal area, the triggering reason, and the suggested review frame are output.

[0066] During the training or joint optimization phase, the total loss of the system can be expressed as:

[0067]

[0068] In the formula, For total training loss, , , and These are photometric reprojection, instance masking, edge-aware smoothing, and contact consistency loss, respectively. to The corresponding weights are used. This total loss enables the collaborative optimization of segmentation boundaries, depth continuity, occlusion ordering, and contact relationships under the same engineering objective.

[0069] When archiving, the system not only saves the composite results, but also the instance mask, stereo depth map, monocular depth map, fusion weights, fusion depth map, occlusion relationship diagram, boundary correction parameters, contact anomaly areas, low-confidence area coordinates, trigger reasons, manual review results, and scene labels. This allows for the use of similar occlusion rule templates when shooting again in the same studio, on the same press conference stage, or at the same green screen base, reducing redundant debugging.

[0070] Example 1: The presenter is integrated with virtual steps in an immersive studio.

[0071] Combination Figure 8 This embodiment of the immersive studio application focuses on an XR studio where a real presenter, virtual steps, virtual ground extension elements, and a screen background coexist. A main binocular camera captures video of the presenter standing and moving, an existing pose tracking system provides camera pose and lens parameters, and the virtual engine provides depth buffering for the virtual steps and ground layer.

[0072] The system first extracts instance masks for the presenter, microphone, and temporary props using SAM2, and then generates initial dense parallax using RAFTStereo. Depth Anything V2 supplements the depth prior for the green screen edges and ground reflections. When the presenter approaches virtual steps or crosses virtual lines, the occlusion graph uses the presenter's feet, the edges of the virtual steps, and the virtual shadow area as key nodes, and checks the contact consistency between the feet and the virtual plane.

[0073] When the confidence level of the foot contact area is insufficient, the system maintains the presenter's complete foreground and temporarily cancels the occlusion of the feet by the leading edge of the step; if the depth and contact conditions are met again for several consecutive frames, the virtual foreground layer is restored in a gradual manner. After each shot, the system archives the binocular synchronization status, presenter instance trajectory, fusion depth, occlusion map, contact correction amount, and manual confirmation results for that shot.

[0074] Example 2: Hierarchical Correction of Host and Vehicle in Product Launch Events

[0075] This example uses an XR launch event scenario where a presenter explains a vehicle. In such scenarios, the front of the car, windows, door edges, tires, and the presenter's body, arms, and gestures interact frequently. If the occlusion is not accurately judged, the presenter's hand may accidentally pass through the vehicle or the virtual light strip may cover the real wheels.

[0076] During the rehearsal phase, the system captures video frames of the host moving around the vehicle, opening doors, and pointing. The system also identifies the vehicle's outline, doors, windows, tires, and the edges of the virtual platform as key instances. For areas with highly reflective paint, the system reduces the weight of the photometric reprojection term and increases the monocular depth completion participation. For the contact line between the tires and the virtual platform, the system prioritizes binocular geometry to prevent the vehicle from appearing to float.

[0077] As the presenter's arm quickly sweeps across the front of the car, SAM2 provides continuous instance boundaries. The deep fusion module maintains a high monocular structural prior for the hand's edges, and the occlusion graph, combined with the direction of motion and the projection overlap ratio, determines whether the arm is located in front of the car body or within the door space. Only when multiple indicators simultaneously indicate that the virtual element should indeed be in front of the foreground, does the system allow the virtual element to enter the front layer; otherwise, a conservative compositing strategy is maintained.

[0078] Example 3: Integration of TV presenters and virtual subtitles during live broadcasts

[0079] This example focuses on a live news broadcast or program studio setting where the presenter stands and broadcasts. This scenario typically requires overlaying virtual subtitles, virtual screens, extended backgrounds, and partial virtual foreground decorative elements before and after the presenter, and demands high levels of low latency and stability.

[0080] The system adopts a conservative occlusion mode as the initial configuration: first, it ensures the integrity of the real foreground of the presenter and microphone, and then gradually releases virtual foreground elements according to the fusion depth and occlusion relationship diagram. When the presenter holds the script, turns around, or points to the screen, the system prioritizes protecting the hand and script outline, and does not allow virtual text bars to directly cover the real foreground at low confidence boundaries.

[0081] When the occlusion quality score falls below a threshold, the system freezes the high-risk virtual front layer, retaining only the background and mid-range layer outputs, and writes the abnormal segment to the review queue. After the program ends, the review unit generates records according to the anomaly type, including binocular asynchrony, low texture depth holes, instance identity switching, boundary refinement failure, and virtual front layer freezing. Results verified manually can be used to update the template for the same program.

Claims

1. A virtual foreground / background fusion method based on dense fusion and occlusion relationship reasoning, characterized in that, Includes the following steps: S1. Acquire binocular or multi-view images, lens parameters, camera pose, virtual scene depth layer and scene timestamp under the same time reference, and synchronize the frame number of the binocular or multi-view images with the virtual scene depth layer. S2. Use a temporal instance segmentation network to segment the real foreground objects in the current frame image to obtain the instance category, instance mask, alpha boundary and propagation confidence of the real foreground objects. The real foreground objects include at least one of the following: host, actor, vehicle, handheld prop, microphone, stage foreground. S3. A dense stereo matching network is used to perform pixel-level disparity estimation on the left and right eye images or the main and auxiliary camera images to obtain the disparity field, iterative residual, left and right consistency mask and binocular depth confidence of the current frame, and the disparity field is converted into a binocular depth map based on lens parameters and camera baseline. S4. Use a monocular depth estimation network to perform monocular depth filling on the current frame image to obtain a monocular depth map and monocular depth confidence. Then use the monocular depth map to fill in the depth of low-texture areas in green screen, local reflective areas, transparent edge areas, monocular visible occlusion areas, and binocular parallax unstable areas. S5. Based on the iterative residual, left-right consistency mask, edge gradient, instance category, monocular depth confidence, camera pose consistency, and historical frame stability, generate binocular depth weights and monocular depth weights, and perform confidence fusion on the binocular depth map and the monocular depth map to obtain a fused depth map for virtual-real occlusion inference. S6. Construct an occlusion relationship graph by combining real foreground instances, virtual asset instances, and scene contact areas. The real foreground instances, virtual asset instances, and scene contact areas are used as graph nodes. The depth difference, projection overlap ratio, relative motion trend, contact prior, and historical occlusion stability are fused as graph edge features to generate occlusion relationship scores and occlusion levels between instances. S7. Based on the occlusion level, instance mask, Alpha boundary and fusion depth map, refine the boundaries of the real foreground boundary and the virtual front boundary, and perform contact consistency verification based on the depth difference between the foot, tire, prop or table and the virtual plane. S8. Based on the instance mask, fusion depth map, occlusion level, alpha boundary and virtual scene depth layer, perform layered synthesis of real foreground layer, virtual background layer, virtual midground layer, virtual front layer and shadow layer, and output virtual and real composite frame. S9. Generate an occlusion fusion quality score based on reprojection error, temporal stability, contact consistency score, and boundary integrity. When the occlusion fusion quality score is lower than a preset threshold or the area of ​​the low-confidence region exceeds a preset area threshold, freeze the high-risk virtual front layer or switch to a conservative occlusion mode that prioritizes the real foreground, and record the abnormal review results.

2. The virtual foreground / background fusion method based on dense fusion and occlusion relationship reasoning according to claim 1, characterized in that, In step S2, the temporal instance segmentation network is a Segment Anything Model 2 temporal segmentation network. The Segment Anything Model 2 temporal segmentation network takes the current frame image, historical frame mask, and cue mask as input, and outputs the instance category, binary subject mask, continuous alpha boundary, and propagation confidence. It also uses the propagation results of the historical mask to suppress the flickering of the host's hair, fingers, clothing hem, microphone stick, car door edge, and vehicle outline between consecutive frames.

3. The virtual foreground / background fusion method based on dense fusion and occlusion relationship reasoning according to claim 1, characterized in that, In step S3, the dense stereo matching network is a RAFTStereo network; the RAFTStereo network performs cyclic disparity updates on the epipolar corrected left and right eye images or main and auxiliary camera images, outputs the disparity field, disparity increment, iterative residual and left and right consistency mask, and determines the pixel-level binocular depth confidence based on the iterative residual and the left and right consistency mask.

4. The virtual foreground / background fusion method based on dense fusion and occlusion relationship reasoning according to claim 1, characterized in that, In step S4, the monocular depth estimation network is the Depth Anything V2 monocular depth estimation network. The Depth Anything V2 monocular depth estimation network outputs a relative depth map and structural confidence, and performs scale alignment on the relative depth map through a calibration plane, a high-confidence binocular region, or a virtual scene reference plane, so that it participates in depth fusion in the low-confidence binocular region, rather than replacing the high-confidence absolute depth in the binocular depth map.

5. The virtual foreground / background fusion method based on dense fusion and occlusion relationship reasoning according to claim 1, characterized in that, In step S5, the binocular depth weight and monocular depth weight are determined based on the pixels. parallax residual Binocular Depth Confidence Monocular depth confidence The monocular depth weight is determined jointly by instance category weight and historical stability. With the binocular depth weights satisfy The fusion depth is obtained according to the following relationship. : ； in, For pixels Binocular depth, For pixels Monocular depth, For pixels The fusion depth is increased; the binocular depth weight is increased for areas such as the ground, wheels, rigid props, and stage steps, while the monocular depth weight is increased for areas such as hair strands, veils, transparent edges, low-texture areas of green screen, and monocular visible occlusion areas.

6. The virtual foreground / background fusion method based on dense fusion and occlusion relationship reasoning according to claim 1, characterized in that, In step S6, each directed edge in the occlusion relationship graph represents a preceding, following, contact, or uncertain relationship between one instance and another instance; instance Relative to instance Occlusion relationship score The occlusion relationship is determined based on the difference in fusion depth, projection overlap rate, relative motion trend, contact prior, and historical occlusion stability of the overlapping regions of the two instances. The occlusion relationship graph satisfies antisymmetry constraints and local transitivity constraints to avoid the simultaneous occurrence of contradictory ordering results in the same local region.

7. The virtual foreground / background fusion method based on dense fusion and occlusion relationship reasoning according to claim 6, characterized in that, When both a foreground relationship and a background relationship exist simultaneously in the overlapping area of ​​the projections of a real foreground instance and a virtual asset instance, the corresponding real foreground instance or virtual asset instance is split into multiple visible partitions, and occlusion masks are generated for each of the multiple visible partitions; the visible partitions are used to handle at least one of the following local interweaving scenarios: a bent arm, an open car door, a transparent car window, a handheld prop passing through a virtual screen, and a vehicle outline occluding a virtual light strip.

8. The virtual foreground / background fusion method based on dense fusion and occlusion relationship reasoning according to claim 1, characterized in that, In step S7, a gated conditional random field is used to perform boundary refinement. The gate value of the gated conditional random field is jointly determined by the instance class, alpha gradient, depth gradient, and occlusion confidence. For areas such as hair strands, veils, hemlines, and microphone cables, the geometric shrinkage is reduced while maintaining continuous transparency. For areas such as vehicle tires, stage steps, tabletops, physical props, and virtual ground contact lines, the depth boundary constraint is increased to suppress the shearing of flexible edges or the blurring of rigid edges.

9. The virtual foreground / background fusion method based on dense fusion and occlusion relationship reasoning according to claim 1, characterized in that, In step S9, the anomaly verification results include instance mask, binocular depth map, monocular depth map, fused depth map, occlusion relationship map, boundary correction parameters, contact anomaly region, low confidence region coordinates, triggering cause, manual verification results, and scene label; when binocular synchronization is lost for a short time in the current frame, monocular depth and historical motion compensation are used to maintain the occlusion sorting within a limited duration. When the duration exceeds the allowed time, the addition of new virtual foreground occlusion is stopped and the conservative occlusion mode is entered.

10. A virtual foreground / background fusion system based on dense fusion and occlusion relationship reasoning, implementing the method as described in any one of claims 1-9, characterized in that, include: The image acquisition and time synchronization unit is used to acquire binocular or multi-view images, lens parameters, camera pose, virtual scene depth layer and scene timestamp under the same time reference; The system comprises the following units: a foreground instance segmentation unit for temporally segmenting real foreground objects to obtain instance category, instance mask, alpha boundary, and propagation confidence; a dense disparity estimation unit for estimating dense disparity between left and right camera images or main / auxiliary camera images to obtain disparity field, iteration residual, left / right consistency mask, and binocular depth confidence; a monocular depth completion unit for generating monocular depth maps and monocular depth confidence for low-texture, reflective, transparent edge, occluded monocular visible, and binocular unstable regions; a depth confidence fusion unit for weighted fusion of binocular and monocular depth maps based on disparity residual, left / right consistency, edge gradient, instance category, monocular depth confidence, camera pose consistency, and historical frame stability to obtain a fused depth map; and an occlusion relationship graph inference unit for integrating real foreground instances, virtual asset instances, and the scene. The contact area is constructed into an occlusion relationship map, and an occlusion level is generated based on the fusion depth difference, projection overlap ratio, relative motion trend, contact prior, and historical occlusion stability. The boundary refinement and contact consistency verification unit is used to refine the boundaries of the real foreground boundary and the virtual front boundary based on the occlusion level and the fusion depth map, and to verify the contact consistency between the depth difference between the feet, tires, props or platforms and the virtual plane. The virtual-real compositing and rollback control unit is used to perform layered compositing based on the instance mask, fusion depth map, occlusion level, alpha boundary, and virtual scene depth layer, and to freeze the virtual front layer or switch to conservative occlusion mode when the occlusion fusion quality score is lower than a preset threshold. The archiving and review unit is used to record the instance mask, depth confidence, occlusion judgment result, boundary correction parameters, contact anomaly areas, and manual review results.

Images