A Method and System for Optimizing Dynamic Rendering of Virtual Scenes Based on Light Field Reconstruction

Abstract

This invention discloses a method and system for optimizing dynamic rendering of virtual scenes based on light field reconstruction, specifically relating to the field of image data processing technology in computational imaging. First, it completes the access of multi-source data, geometric / optical calibration, and affine alignment of timestamps, constructing a unified time base and evidence chain within a sliding window of fixed step size. Then, it establishes a multi-scale spatial parameterized structure within the scene bounding box, using path tracing as the forward model to jointly solve geometry, radiation, and material properties, completing static / dynamic decomposition and fusing color, depth, polarization, and spectral residuals to generate an optical path consistency map. Based on this, it constructs a priority scalar field fused from uncertainty and importance, performing area-level budget arbitration and degradation control, and mapping it to executable control quantities such as sampling density, bounce depth, geometric level, and indirect light order. Finally, it improves cross-frame stability through temporal robust fusion and differential updates, and feeds quality metrics and operational statistics back to the next frame's scheduling to form a closed loop.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing of computed imaging, and more particularly, to a virtual scene dynamic rendering optimization method and system based on light field reconstruction. BACKGROUND

[0002] With the wide application of virtual reality / augmented reality, immersive interaction and film-level digital assets, the real-time rendering demand of complex scenes on mobile or desktop is rapidly growing. In the prior art, one type of method relies on traditional rasterization or screen space techniques to reduce cost, but it is difficult to accurately reconstruct complex optical effects such as transparent bodies, strong mirrors and thick media, and it is easy to produce flicker, artifacts and physical inconsistencies in occlusion boundaries and highlight areas. Another type of method is based on volume rendering / light field reconstruction and path tracing, which can better depict multiple reflections and refractions, but it is difficult to run stably under dynamic content and strict time delay constraints.

[0003] At the same time, the existing scheduling strategies are mainly fixed sampling or coarse-grained region priority, lacking a priority scalar field that unifies rendering uncertainty and task importance / gaze information / optically sensitive area into the same decision framework, making it difficult to perform explainable arbitration and degradation within the power / memory and time delay budget; also lacking a time domain robust fusion and differential update mechanism across frames, making it difficult to ensure picture stability and model continuous convergence in long-term operation. In view of this, there is an urgent need for a dynamic rendering optimization scheme that closes the loop of constructible renderable light field, physical consistency measurement, priority-driven budget arbitration, and time domain robust fusion under the premise of unified time base and engineering traceability, in order to balance physical correctness, visual quality and real-time performance. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a virtual scene dynamic rendering optimization method and system based on light field reconstruction to solve the problems raised in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] The virtual scene dynamic rendering optimization method based on light field reconstruction comprises the following steps:

[0007] Access multi-view images, depth and inertial measurement unit data, perform geometric and optical calibration, complete distortion correction and color unification; align the timestamps of each data source to a unified time base through affine mapping, organize the data within a fixed step sliding window, and establish a versioned optical property prior knowledge base;

[0008] A multi-scale spatial parameterized structure is established within the scene bounding box. Path tracing is used as the forward model for inverse rendering optimization. Geometric, radiation, and material parameters are jointly solved to decompose static backgrounds and dynamic objects, and output the soft mask and deformation field of dynamic objects. Based on path tracing, optical path prediction and multi-view realistic observation are used to fuse color reprojection residuals, parallax or depth residuals, polarization residuals, and spectral residuals at the pixel or voxel scale to generate an optical path consistency map.

[0009] An uncertainty field is constructed based on the optical path consistency map, and an importance field is generated by combining the task interest region, soft mask and deformation boundary information. The two are then fused to generate a priority scalar field. The current field of view is divided into spatial regions, and budget arbitration is performed under the constraints of computing power budget and time delay. The region priority is mapped to path sampling density, maximum bounce depth, geometric detail level and indirect light evaluation order.

[0010] The rendering estimate of the previous frame is reprojected to the current frame based on the deformation field or optical flow prior and temporal fusion is performed. When the optical path consistency map continuously reaches a threshold in a local area within a preset time window, the corresponding area is triggered to be solved incrementally to update the optical field model, and the quality metric and operation statistics are fed back to the next frame scheduling to form a closed loop.

[0011] In a preferred embodiment, the sliding window is configured with a late data buffering strategy and an abnormal frame suppression mechanism.

[0012] In a preferred embodiment, the multi-scale spatial parameterization structure is one of a sparse voxel octree, a hash-coded grid, or a planar factorized tensor representation, and is superimposed with an occupation acceleration and early stopping mechanism.

[0013] In a preferred embodiment, the process of decomposing static backgrounds and dynamic objects includes: within a sliding time window, identifying dynamic regions based on parallax reprojection residuals, optical flow consistency, and occlusion confidence, and combining material and lighting estimates obtained from inverse rendering to eliminate pseudo-motion responses caused by specular highlights and refraction; after decomposition, for each dynamic subfield, the system defines a soft mask to characterize the spatiotemporal existence probability and boundary ambiguity of the object, and defines a deformation field containing explicit optical deformation components to express its non-rigid deformation.

[0014] In a preferred embodiment, the soft mask and deformation field are subjected to smoothing and sparse priors in time and space; the soft mask, deformation field and material parameters are versioned and archived after each update, and an evidence chain is established with the original data pointer.

[0015] In a preferred embodiment, the process of generating the optical path consistency map includes: fusing color reprojection residuals, parallax or depth residuals, polarization residuals and spectral residuals at the pixel or voxel scale; normalizing each residual into a confidence component by a robust mapping function, then performing weighted fusion, and introducing an occlusion gating term to suppress misjudgment of occluded areas.

[0016] In a preferred embodiment, the priority scalar field is formed by weighted fusion of the uncertainty field and the importance field, wherein the fusion weight coefficient is adaptively adjusted at the frame level or sub-region level according to the scene optical complexity or task attention; the importance field integrates the optically sensitive area and motion deformation boundary from the observation side, as well as the gaze point and task interest area from the application side.

[0017] In a preferred embodiment, temporal fusion employs temporal fusion coefficients and limiting amplitudes, and adaptively adjusts the temporal fusion coefficients and limiting amplitudes according to the physical material type.

[0018] In a preferred embodiment, when the confidence level of optical path consistency remains below a threshold within a preset time window, incremental re-solving initiates local parameter fine-tuning within the spatiotemporal range defined by the soft mask, prioritizing the updating of materials and deformation fields; and encapsulates the model changes into differential packets, transmits and applies them in the order of priority of the regions, and uses atomic operations to avoid state tearing.

[0019] In a preferred embodiment, the following modules are included:

[0020] The access and caliber unification module is used to access multi-view images, depth and inertial measurement unit data, perform geometric and optical calibration, and complete distortion correction and color unification; it aligns the timestamps of each data source to a unified time base through affine mapping, organizes the data within a sliding window with a fixed step size, and establishes a versioned optical attribute prior knowledge base.

[0021] The reconstruction and consistency assessment module is used to establish a multi-scale spatial parameterized structure within the scene bounding box, perform inverse rendering optimization using path tracing as the forward model, jointly solve geometric, radiation, and material parameters, realize the decomposition of static background and dynamic objects, and output the soft mask and deformation field of dynamic objects; based on path tracing optical path prediction and multi-view realistic observation, it fuses color reprojection residuals, parallax or depth residuals, polarization residuals, and spectral residuals at the pixel or voxel scale to generate an optical path consistency map;

[0022] The scheduling and execution module is used to construct an uncertainty field based on the optical path consistency map, and generate an importance field by combining the task interest region, soft mask and deformation boundary information. The two are then fused to generate a priority scalar field. The current field of view is divided into spatial regions, and budget arbitration is performed under the constraints of computing power budget and time delay. The region priority is mapped to path sampling density, maximum bounce depth, geometric detail level and indirect light evaluation order.

[0023] The temporal fusion and closed-loop module is used to reproject the rendering estimate of the previous frame to the current frame based on the deformation field or optical flow prior and perform temporal fusion. When the optical path consistency map continuously reaches the threshold in a local area within a preset time window, the corresponding area is triggered to be solved incrementally to update the optical field model, and the quality measurement and operation statistics are fed back to the next frame scheduling to form a closed loop.

[0024] The technical effects and advantages of this invention are as follows:

[0025] First, in terms of physical consistency and complex optical adaptation, this invention uses path tracing as the forward model and performs inverse rendering optimization to jointly solve geometric, radiation, and material parameters. At the same time, it introduces multi-view observation and integrates multi-source residuals such as color reprojection, parallax / depth, polarization, and spectrum to generate an optical path consistency map, which is used to constrain the reconstruction and rendering of complex optical phenomena such as mirrors, high reflectivity, transparent bodies, and thick media, thereby reducing the occurrence of specular flicker, refraction ghosting, and occlusion boundary artifacts from the root.

[0026] Secondly, regarding temporal stability and real-time performance, this invention organizes data through a unified time base and a sliding window with a fixed step size, and configures late arrival buffering and abnormal frame suppression to stabilize input quality. In the rendering stage, a temporal robust fusion is adopted, which reprojects the previous frame estimate to the current frame based on the deformation field or optical flow prior, and superimposes strategies such as asymmetric rate limiting, hysteresis, and minimum dwell time to significantly suppress cross-frame jitter and texture flicker. When the local consistency continuously reaches a threshold of abnormality within a preset time window, only the corresponding area is triggered for incremental re-solution, so as to only repair the areas that need to be repaired, restore local quality with lower overhead, and stabilize end-to-end latency.

[0027] Furthermore, regarding the adaptability and interpretability of resource scheduling, this invention uses an optical path consistency map to drive uncertainty field modeling and fuses it with information such as task interest region, soft mask, and deformation boundary to generate an importance field, further forming a unified priority scalar field. Based on this, the current view is divided into regions and budget arbitration is performed, directly mapping region priority to executable rendering control quantities (including path sampling density, maximum bounce depth, geometric detail level, and indirect lighting evaluation order). When the budget is under pressure, low-weight regions are rolled back according to a predetermined degradation sequence, thereby prioritizing the structure and details of high uncertainty and high importance regions under limited computing power and strict latency constraints, while taking into account both image quality and frame rate.

[0028] Furthermore, in terms of quality closed loop and continuous convergence, this invention injects the quality metrics and operational statistics at the region and frame levels back into the scheduling logic of the next frame, forming a closed loop of evaluation, scheduling, rendering, and re-evaluation; combined with differential update and differential packet atomic application mechanisms, the model parameters continuously approach the real feasible domain in long-term operation, while avoiding the fluctuations and costs caused by global recalculation.

[0029] Furthermore, due to the systematic design for engineering feasibility and traceability, this invention implements versioning and evidence chain management for key objects such as input aperture, calibration parameters, time alignment coefficients, optical priors, light field parameters, and scheduling logs, recording scheduling metadata such as triggering causes, boundary strength, and budget impact, which facilitates problem review, version rollback, and cross-platform collaboration. Combined with engineering strategies such as occupation acceleration and early stopping mechanisms, occlusion gating, and sentinel sampling, redundant computation in low-contribution areas is further reduced, improving overall throughput and stability in complex scenarios. Attached Figure Description

[0030] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0031] Figure 1 This is a flowchart illustrating the virtual scene dynamic rendering optimization method based on light field reconstruction of the present invention.

[0032] Figure 2 This is a schematic diagram of the structure of the virtual scene dynamic rendering optimization system based on light field reconstruction of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1

[0035] This invention relates to a method for optimizing dynamic rendering of virtual scenes based on light field reconstruction, such as... Figure 1 As shown, it includes the following steps:

[0036] Step 1: Multi-source spatiotemporal alignment and optical property enhancement

[0037] In step one, the system performs multi-source data access, calibration, and unified time base construction to establish a spatiotemporally consistent observation foundation for subsequent processing. The data sources accessed by the system include multi-view color cameras, depth cameras, inertial measurement units, and eye trackers. Furthermore, polarization cameras or spectral cameras are introduced as optical property sensing sensors to capture the polarization state or spectral information of the scene, thereby providing key data support for subsequent optical path analysis and inverse rendering optimization.

[0038] During the calibration phase, the system performs intrinsic and extrinsic parameter calibration for each camera to ensure the accuracy of geometric projection relationships. For polarization cameras, the system additionally calibrates the polarization angle response; for spectral cameras, it performs spectral sensitivity calibration to ensure that all imaging devices have a uniform response aperture in terms of radiation and optical properties. By implementing distortion correction and color consistency correction, the system maps image data from different sources to a common radiation reference. Simultaneously, the system uses physical measurement results provided by optical sensors to initially identify and mark specular reflection and transmission areas in the scene, thereby establishing a priori knowledge base of optical properties to provide prior information for subsequent processing steps.

[0039] System construction with unified time base Through affine mapping relations Local timestamps of each sensor Synchronize to this unified time base, where Represents a unified time base. Used to compensate for the Clock scaling and drift in data sources Used to align its fixed time offset. The system uses a fixed sampling step size. All image streams and sensor measurement data are resampled and aligned. Image data is mapped to discrete-time grids using nearest neighbor preservation or interpolation methods.

[0040] A sliding window W(t) centered on the current time is set, and a late data buffering strategy is configured within it. To further improve input data quality, a quality screening and preprocessing process is executed on a unified time base. This preprocessing process includes not only anomaly frame suppression based on exposure histogram analysis and motion blur detection, but also a reflection suppression algorithm based on polarization analysis, effectively mitigating the interference of high reflectivity and transparent surfaces on imaging. All calibration parameters, time alignment coefficients, and generated optical prior knowledge are version-managed and linked to a complete, traceable chain of evidence with the original data pointer.

[0041] Step 2: Renderable Light Field Construction and Static / Dynamic Decomposition

[0042] Based on the unified time base and calibration results provided in step one, the system constructs a renderable light field representation using the scene bounding box as the domain. It also completes the effective decomposition of static background and dynamic objects within the same spatiotemporal framework, and generates a light path consistency map for measuring physical consistency; the scene bounding box is a minimal simple geometry that can tightly wrap all objects in the entire virtual scene.

[0043] First, the system establishes a multi-scale spatial parameterized structure within the bounding box to store volume density, view-dependent radiative color, and material-related physical parameters. This structure can be implemented using a sparse voxel octree, a hash-coded grid, or a planar factorized tensor representation, and uses the internal and external participating radiative apertures output from step one as a unified benchmark to ensure consistency in geometry and radiometric metrics across different data sources. To improve tracking efficiency, the system superimposes occupation acceleration and early stopping mechanisms on this structure, allowing ray integration to terminate quickly in low-contribution regions and maintain sufficient sampling in high-frequency details and optically complex regions. Using this structure, the system simulates the entire process of light rays originating from a virtual light source, undergoing several scattering, reflection, and refraction events on scene surfaces and media before reaching the imaging plane using path tracing; direct illumination, indirect illumination, and the interactive effects caused by specular surfaces and transparent media are explicitly calculated in this simulation.

[0044] Subsequently, the system uses multi-view observations with good time proximity and parallax coverage within a sliding window W(t) as supervision signals to initiate incremental inverse rendering optimization, jointly solving for deep physical properties such as geometry, radiation, and materials. The system represents this optimization objective as the sum of data consistency terms and physical prior terms:

[0045] ;in, The image is a true observation image at viewpoint v and time t; To represent in the current light field With physical parameter set Synthetic images obtained by path tracing under conditions including roughness, metallicity, refractive index, etc.; This is a robust loss function used to suppress the effects of occlusion boundaries and abnormal frames. The robust loss function can be the Cauchy loss function, defined as follows: ,in Here, r is the scaling parameter, and r is the residual, representing the error between the model prediction and the actual data. , , For example, its weighting coefficients , , It can adaptively adjust based on the data quality within the window and the optical complexity of the scene.

[0046] Material regularization term The material parameters constrained by this constraint (such as roughness and metallicity) should change smoothly and not deviate from a physically reasonable range. ;in The prior values ​​for the material parameters can be obtained from the prior knowledge base of optical properties established in step one; This is the total variation of the material parameter in space, used to promote its spatial smoothness; This represents the weight of the total variation term, with a typical value range of [0.001, 0.1].

[0047] Geometric regularization term The geometry of this constrained scene (determined by the volume density field) The representation should have sparsity and smoothness; ;in The L1 norm of the volume density field is used to promote the sparsity of space, making most regions empty; The L2 norm of the volume density field gradient is used to promote the smoothness of geometric surfaces; , These are the corresponding weighting coefficients, with typical values ​​ranging from [0.01, 0.5].

[0048] Physical consistency regularization term This method utilizes observation data of multiple physical quantities such as polarization and spectroscopy to constrain the material parameters obtained by the solution to conform to physical laws. ;

[0049] in The predicted degree of polarization refers to the polarization based on the current optical field representation. and the set of physical parameters to be optimized (Including surface normals, roughness, refractive index, etc.), the polarization degree of light rays at a point on the imaging plane is simulated and calculated by the path tracing forward model. This predicted value reflects the polarization state that the virtual scene model should exhibit under given parameters; The polarization angle is the light polarization angle calculated by simulating the path tracing model; the corresponding polarization angle is... and These are the observations obtained from actual images captured by the polarization camera; The predicted spectral response refers to the response based on the current optical field representation. and the set of physical parameters to be optimized The spectral radiance or spectral reflectance is simulated and calculated at a specific wavelength using a path-tracing forward model. This predicted value reflects the spectral characteristics that the virtual scene model should exhibit under given physical parameters. The observed spectral response refers to the actual spectral radiance or reflectance data at a specific wavelength, which is actually acquired by a spectral camera or multispectral imaging system. and This is a weighting coefficient; its value is set to zero when the corresponding sensor data is unavailable. For example... , .

[0050] During the joint optimization process, the system simultaneously completes static / dynamic component decomposition. Specifically, within W(t), the system identifies regions that change significantly over time based on parallax reprojection residuals, optical flow consistency, and occlusion confidence. It also combines material and lighting estimates obtained during the inverse rendering stage to eliminate pseudo-motion responses caused by specular highlights and refraction. Ultimately, the static background is... This indicates that the dynamic part is represented as several dynamic subfields. The set. For each dynamic subfield The system defines a soft mask. To characterize the object's existence probability and boundary ambiguity in space and time, and to define the deformation field. Used to express its non-rigid deformation relative to a reference attitude; to accurately cover the apparent displacement caused by refraction. It includes explicit optical deformation components. and Smoothing and sparse priors are applied in time and space to avoid overfitting and boundary flickering; parameters such as mask, deformation and material are versioned and archived after each update and linked to the original data pointer to establish an evidence chain.

[0051] Based on the joint solution results described above, the system generates an optical path consistency map to quantify the degree of consistency between the optical path prediction obtained through path tracing based on the current optical field representation and physical parameters at each pixel or voxel location and the actual observations from multiple viewpoints. To take into account geometric, radiometric, and sensor-side physical measurements, this consistency is locally obtained by fusing multi-source residual information: its core includes color reprojection residuals. Parallax or depth residual Polarization residual With spectral residuals And controlled by the shielding gate item Suppress misjudgments of occluded areas. It indicates that it is obscured. The system indicates that the residuals are visible. The system uses robust mapping to represent each residual. The confidence components are normalized to [0,1], and then a weighted fusion is used to obtain the consistency measure. : ;in, , , , These are non-negative weights, and their sum is 1; their values ​​are adaptively determined by the scene's optical properties and sensor availability: when the scene contains transparent / high-reflectivity regions and polarization and spectral measurements are available, the weights are increased. and This strengthens the constraint on physical consistency; if a certain type of measurement is missing, its corresponding weight is reset to zero without affecting the overall calculation. A monotonically decreasing robust function can be chosen to enhance the suppression of outliers; specifically, an exponentially decaying function can be used. This indicates the normalization of the range of the weighted sum, keeping the consistency measure within [0,1]. It uses a 0-1 gating system, determined based on multi-view occlusion consistency and depth relationship. At the implementation level... Spatiotemporal bilateral smoothing is applied to remove isolated noise and preserve structural edges. To meet the scheduling requirements of subsequent steps, the system simultaneously calculates the hierarchical statistics of the optical path consistency map and archives them in conjunction with the optical field parameter version number. The hierarchical statistics of the optical path consistency map include the region mean, variance, and lower quantile threshold.

[0052] Through the above process, the system obtains an enhanced optical field representation after optical path correction. And provide the corresponding static background. Dynamic subfield Soft mask With deformation field The spatiotemporal solution, and the optical path consistency diagram with both physical and observational basis, are obtained by specifically calculating the consistency metric within the spatiotemporal range.

[0053] These outputs together constitute the input for subsequent steps of adaptive rendering scheduling and quality closure. The optical path consistency map will directly participate in uncertainty modeling, and the static / dynamic decomposition results provide structured support for differential updates and temporal stability. All key parameters, weights, and evidence pointers are versioned on a unified time base to ensure that the system behavior is interpretable, reproducible, and engineering feasible.

[0054] Step 3: Construction of Priority Scalar Field and Budget Arbitration

[0055] The enhanced optical field representation generated in step two Based on its associated priors, this step implements adaptive rendering scheduling under fixed frame timing and computational / memory budget constraints, aiming at final visual quality and specific task performance. The associated priors include static backgrounds. Dynamic subfield Soft mask Deformation field Pixel / voxel level consistency measurement With obstruction gate control The system constructs the uncertainty field in parallel within a uniform sliding time window W(t). With importance field The former is used as a reverse indicator to quantify the credibility of local rendering, while the latter is used to identify areas that are more critical to the observer or application target. To maintain clarity of definition, the uncertainty field is... Give, and constrain in implementation The uncertainty field increases when there are issues such as insufficient material convergence, geometrical occlusion boundaries, multiple interactions between transparent / mirror surfaces, or sensor malfunctions. Importance field. Combining clues from both the observation and application sides: the observation side includes the optically sensitive area obtained through inverse rendering in step two, and the data derived from... and Induced motion / deformation boundaries; on the application side, this includes real-time gaze points from the eye tracker and task interest regions dynamically assigned by the business module. Optically sensitive areas include high-roughness mirrors, strong refractive bands, and multiple scattering hotspots.

[0056] The aforementioned clues are first projected onto the current virtual camera's imaging plane and then gating based on visibility and occlusion relationships, whereby... Suppression is applied to temporarily invisible areas, retaining only minimal sentinel sampling to detect the risk of exposure in advance; subsequently, robust normalization and spatiotemporal smoothing are performed to form .

[0057] To unify the two types of heterogeneous information into the same decision-making framework, the system generates a priority scalar field at the pixel / voxel scale. Its calculation is .in To achieve fusion weighting coefficients, adaptive adjustment can be performed at the frame or sub-region level: when the scene contains regions with high optical complexity, such as transparent bodies, strong specular highlights, or thick media, the system improves... The value of makes scheduling more biased towards reducing uncertainty; when the application side pays more attention to a specific target or the eye tracker provides a stable gaze, the system reduces ... This allows the scheduling to be more biased towards areas related to the support mission.

[0058] Based on a priority scalar field, the system divides the imaging plane of the current field of view into spatial regions. This division can be achieved using a regular grid or a hierarchical quadtree / octree partitioning method. For each region j, the system performs a weighted aggregation of the priority scalar field within its coverage area to obtain a comprehensive weight. The weight calculation considers both occlusion gating and dynamic boundary reinforcement: for And close to and Samples in the indicated transition zone are assigned a higher weighting factor to prioritize protection of flicker-prone boundary regions; for The samples are aggregated using only the sentinel proportion to avoid false positives. At the beginning of each frame, based on the frame-level computing power budget... With delay threshold ,right Normalize and obtain the budget ratio , making At the same time, ensure To avoid over-concentration or dilution, if the predicted runtime execution cost may exceed τ, the system will first roll back proportionally in low-weight regions. This triggers a degradation sequence, adjusting the execution level in the order of priority for geometric visibility and boundary fidelity, followed by material evaluation, and then noise reduction and post-processing, thereby satisfying the delay constraint without sacrificing the quality of critical areas.

[0059] Once the budget proportions are determined, the system will abstract... Mapped to directly executable rendering controls, including path sampling density per pixel within the region. Maximum allowable bounce depth Adapted geometric level of detail and material / indirect light evaluation order The mapping strategy follows the principle of increasing priority in the high-priority area. and The principle of preserving the lowest steady state in low-priority regions is followed. At dynamic subfield boundaries or in high-deformation regions, the system tends to increase [efforts] under the same budget. Instead of simply raising prices Prioritize the suppression of temporal noise and boundary flicker; in areas with mirror crosstalk or thick dielectrics, moderately enhance... To realistically reproduce multiple reflections / refractions. , , and The update uses asymmetric rate limiting and hysteresis / dwell control to avoid frequent up-and-down jitter; when the state of the area changes but the dwell time requirement has not yet been met, the system only allows fine-grained fine-tuning and prohibits abrupt level changes.

[0060] During rendering, the system monitors computing power and video memory usage in real time and uses a budget arbitrator to adjust resource flow between regions: when the actual rendering time of a region deviates significantly from the estimate, causing pressure on the global budget, the path sampling density and maximum allowable bounce depth per pixel are proportionally rolled back in regions far from the area of ​​interest and with low uncertainty and importance fields; when a persistent anomaly with local high uncertainty fields unrelated to occlusion occurs within a region, the system generates a local optimization task within the same rendering loop to optimize that region. The subset, along with the most recent observations and constraints, is encapsulated and submitted to the backend or parallel channel for incremental re-solution. Task priority, triggering reason, boundary strength, estimated benefit, and budget impact are all written into the evidence chain along with the frame log. At the end of each frame, the system generates a scheduling log, including frame time and information for each region. It maps corresponding control variables, budget utilization, affected area maps, and threshold / resident status snapshots; the log is bound and archived with the light field version number and observation data pointer to ensure that scheduling behavior is interpretable, replayable, and rollbackable.

[0061] Through the above process, this step achieves refined resource allocation driven by the fusion of uncertainty and importance under strict computing power and latency constraints. This prioritizes regions with high uncertainty and high importance, while regions with low uncertainty and low importance automatically revert to steady-state overhead. Combined with asymmetric rate limiting, hysteresis and minimum residency control, as well as budget arbitration and evidence chain archiving, the output includes... , , , A stable control field and complete scheduling logs provide structured input for subsequent quality assessment, release and rollback, and ensure the physical correctness, visual stability and engineering feasibility of rendering results under complex optical scenes and dynamic content conditions.

[0062] Step 4: Time-domain robust fusion and incremental re-solution

[0063] After completing the adaptive rendering scheduling in step three and generating the preliminary rendering result of the current frame, the system executes step four. Its core task is to improve the temporal stability of the output image, perform efficient differential updates of the scene model, and establish a quality-driven closed-loop archiving mechanism to ensure the rendering quality and efficiency of the system in long-term operation.

[0064] To achieve temporal stability in the output image, the system, based on the dynamic object deformation field and optical flow prior established in step two, reprojects the density and radiation estimates of the previous frame onto the coordinate system of the current frame, generating temporal prediction reference values. The composite result obtained through rendering calculations for the current frame is then used as the basis for these values. With reprojection reference value The system performs temporal fusion in areas without significant occlusion or depth conflict. This fusion operation is described as follows: ,in This is the time-domain fusion coefficient, used to control the mixing ratio of historical information; To limit the amplitude parameter and prevent significant motion blur or ghosting when historical samples differ greatly from current samples, the system adaptively adjusts the physical material type obtained from the inverse rendering optimization in step two. and The value of is taken; in particular, for transparent or high refractive index surfaces, the system uses a relatively small value. The value is adjusted to effectively avoid the ghosting phenomenon caused by rapid changes in the light path.

[0065] While maintaining a stable output image, the system also enhances the light field representation constructed in step two. The system employs a differential update strategy for the model parameters of its components, including a static background and various dynamic subfields. Based on continuous quality assessment and optical path consistency analysis within a sliding window, the system determines the need for parameter updates in each region of the scene. When a significant increase in reconstruction residuals occurs in a specific spatiotemporal region, or the optical path consistency confidence level remains below a set threshold, the system does not globally update the model. Instead, it initiates local parameter fine-tuning only within the spatiotemporal range defined by the soft mask corresponding to that region. This fine-tuning process focuses on optimizing the physical material properties of that region and the deformation field parameters describing its motion, thereby achieving model consistency with the current observation data with minimal computational overhead; physical material properties include roughness and refractive index.

[0066] In cloud-based collaborative deployment scenarios, the system encapsulates the changes in model parameters after differential updates into compact differential packets. Based on the region priority calculated in step three, the system maintains an update packet transmission queue to ensure that differential packets corresponding to high-weight regions are sent to the client first. After receiving the differential packets, the client applies these updates atomically, thereby avoiding screen tearing or inconsistencies during the update process. When network bandwidth is limited, the system automatically delays differential updates for low-priority regions, prioritizing timely updates of core viewport content and stable rendering frame rates.

[0067] To create a continuously optimized quality loop, the system calculates multiple quality metrics and resource consumption statistics at the frame and region levels after each frame rendering task is completed. Among these, objective quality metrics based on optical path consistency form the core basis for evaluating rendering effects. The system performs exponential smoothing on these metrics and dynamically feeds the smoothed results back to the uncertainty estimation module and region weight mapping logic in step three to correct the resource allocation strategy for the next and subsequent frames. All key hyperparameters, their dynamic adjustment history, and versioned data generated in each step are fully recorded by the system and linked to the original data evidence chain established in step one, thereby ensuring that the entire system operating environment and decision-making path have complete traceability and reproducibility.

[0068] Example 2

[0069] The design of the virtual scene dynamic rendering optimization system based on light field reconstruction in this invention is based on the method in Example 1, specifically as follows: Figure 2 The following modules are shown:

[0070] The access and aperture unification module is used to perform geometric and optical calibration, distortion and color consistency correction after the multi-view color / depth camera, inertial measurement unit, eye tracker, and polarization / spectral camera are connected. It also affinely aligns the timestamps of each source to a unified time base and organizes the running data in a sliding window W(t) with a fixed sampling step size. To handle jitter and delays in the real-time stream, a buffer with limited waiting time and anomaly frame suppression strategy are configured within the window. At the same time, optical property masks are generated based on priors such as specularity, transparency, and strong reflection and are versioned. All input apertures, calibration parameters, and data snapshots are mapped using evidence pointers, which serve as the basis for subsequent physical consistency constraints and resource scheduling.

[0071] The reconstruction and consistency assessment module is used to construct a multi-scale spatial parameterized structure within the scene bounding box. Path tracing is used as the forward model to jointly solve for geometric, radiative, and material parameters. Polarization / spectral measurements are incorporated as robust objectives to constrain specular roughness and refraction-related parameters. During the solution process, soft masks and deformation fields for the static background and several dynamic subfields are simultaneously generated and archived using a unified time base. Based on parallax reprojection residuals, optical flow consistency, and occlusion confidence, combined with inverse rendering residuals fused at the pixel / voxel scale, an optical path consistency map is generated. Through occlusion gating and spatiotemporal bilateral smoothing, a robust consistency metric is obtained, providing a quantitative basis for uncertainty modeling and subsequent scheduling.

[0072] The scheduling and execution module defines an uncertainty field within W(t) and integrates optically sensitive areas, soft mask / deformation boundaries, eye-tracking gaze points, and task interest regions to generate an importance field. These two fields are then fused at the pixel / voxel scale to obtain a priority scalar field. This priority scalar field is projected onto the current field of view and divided into regions. After aggregating the comprehensive weights of these regions, arbitration is performed under a given frame-level computing power / memory budget and latency threshold. Path tracing rendering is driven by this control variable. Occupancy acceleration and early stopping are used to reduce the integration steps in low-contribution areas, and sentinel sampling is deployed in temporarily invisible areas to detect occlusion risks. Online monitoring of time consumption and memory usage is performed. If local uncertainty remains abnormal and is not caused by occlusion, a local incremental re-solution is triggered, and the triggering cause and budget impact are recorded.

[0073] The temporal fusion and closed-loop module is used to reproject the density / radiance estimate of the previous frame to the current frame based on the deformation field and optical flow prior, and to smooth out regions without significant occlusion or depth conflict with a smoothing coefficient. With noise suppression coefficient Perform robust temporal fusion to suppress flickering and ghosting; when local reconstruction residuals increase or When the values ​​are below a threshold, parameter fine-tuning is performed only within the spatiotemporal range of the corresponding soft mask, prioritizing updates to the material and deformation field. Model changes are packaged into differential packets and distributed according to region priority, employing atomic operations to avoid state fragmentation. The system performs frame-level and region-level calculations of quality indicators and resource statistics centered on optical path consistency, applies exponential smoothing, and feeds back to uncertainty estimation and weight mapping logic to drive resource reallocation in the next frame. Simultaneously, key hyperparameters, dynamic adjustment history, model versions, and observation data pointers are archived across the entire chain, forming a traceable and reproducible chain of evidence.

[0074] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0076] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0078] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for optimizing dynamic rendering of a virtual scene based on light field reconstruction, characterized in that, Comprising the following steps: Access multi-view images, depth and inertial measurement unit data, conduct geometric and optical calibration, complete distortion correction and color unification; Align the timestamps of each data source to a unified time base through affine mapping, organize the data within a fixed step sliding window, and establish a versioned optical attribute priori knowledge base; Establish a multi-scale spatial parameterization structure within the scene bounding box, use path tracing as a forward model for inverse rendering optimization, jointly solve geometric, radiometric and material parameters, achieve the decomposition of static background and dynamic objects, and output the soft mask and deformation field of dynamic objects; Based on path tracing light path prediction and multi-view real observation, fuse color reprojection residual, disparity or depth residual, polarization residual and spectral residual at pixel or voxel scale to generate light path consistency map; According to the light path consistency map, construct the uncertainty field, and combine the task interest area, soft mask and deformation boundary information to generate the importance field, and fuse the two to generate the priority scalar field; Divide the current view into spatial patches, perform budget arbitration under the constraints of computing power budget and time delay, and map the patch priority to path sampling density, maximum bounce depth, geometric detail level and indirect light evaluation order; Project the rendering estimate of the last frame to the current frame according to the deformation field or optical flow priori and perform temporal fusion; When the light path consistency map continuously abnormally reaches a threshold in a local area within a preset time window, trigger incremental re-solution for the corresponding patch to update the light field model, and feed back the quality metrics and running statistics to the next frame scheduling to form a closed loop.

2. The method of claim 1, wherein: The sliding window is configured with a late data buffer strategy and an abnormal frame suppression mechanism.

3. The method of claim 1, wherein: The multi-scale spatial parameterization structure is one of sparse voxel octree, hash coding grid or plane factorization tensor representation, and superimposes occupancy acceleration and early termination mechanism.

4. The method of claim 1, wherein: The process of decomposition of static background and dynamic objects includes: within the sliding time window, based on the disparity reprojection residual, optical flow consistency and occlusion confidence, identify the dynamic area, and combined with the material and illumination estimate obtained by inverse rendering, remove the pseudo motion response caused by highlights and refraction; After decomposition, for each dynamic subfield, the system defines a soft mask to depict the existence probability and boundary ambiguity of the dynamic subfield in space-time, and defines a deformation field containing an explicit optical deformation component to express the non-rigid deformation of the dynamic subfield.

5. The method of claim 4, wherein: The soft mask and deformation field are applied with smoothing and sparsity priori in time and space; The soft mask, deformation field and material parameters are versioned and archived after each update, and are associated with the evidence chain of the original data pointer.

6. The method of claim 1, wherein: The generation process of the light path consistency map includes: at the pixel or voxel scale, fuse color reprojection residual, disparity or depth residual, polarization residual and spectral residual; After each residual is normalized to a confidence component by a robust mapping function, it is fused by weighting, and an occlusion gating term is introduced to suppress the misjudgment of the occluded area.

7. The method of claim 1, wherein: The priority scalar field is obtained by weighted fusion of the uncertainty field and the importance field, and the fusion weight coefficient is adjusted adaptively according to the scene optical complexity or task attention at frame level or sub-area level; The importance field integrates the optical sensitive region, motion deformation boundary from the observation side, and the gaze point and task interest region from the application side.

8. The method of claim 1, wherein: The time domain fusion adopts a time domain fusion coefficient and a limit amplitude, and adaptively adjusts the time domain fusion coefficient and the limit amplitude according to a physical material type. 9.The method of claim 1, wherein: When the light path consistency confidence is continuously lower than a threshold value within a preset time window, the incremental re-solving is started in the spatio-temporal range defined by the soft mask, the local parameter fine tuning is started, the material and the deformation field are preferentially updated, and the model change is encapsulated as a differential package, which is transmitted and applied according to the slice area priority order, and the atomic operation is adopted to avoid state tearing.

10. A system for dynamic rendering optimization of a virtual scene based on light field reconstruction, characterized in that, The optimization system is used to implement the method in any one of claims 1-9, and comprises the following modules: The access and caliber unification module is used for accessing multi-view images, depth and inertial measurement unit data, performing geometric and optical calibration, completing distortion correction and color unification, aligning timestamps of each data source to a unified time base through affine mapping, organizing data in a fixed step sliding window, and establishing a versioned optical property prior knowledge base; The reconstruction and consistency evaluation module is used for establishing a multi-scale spatial parameterization structure in a scene bounding box, performing inverse rendering optimization by path tracing as a forward model, jointly solving geometric, radiometric and material parameters, realizing decomposition of static background and dynamic objects, and outputting a soft mask and a deformation field of the dynamic objects; based on path tracing light path prediction and multi-view real observation, color reprojection residuals, disparity or depth residuals, polarization residuals and spectral residuals are fused at a pixel or voxel scale to generate a light path consistency map; The scheduling and execution module is used for constructing an uncertainty field according to the light path consistency map, generating an importance field combined with a task interest region, a soft mask and a deformation boundary information, and fusing the two to generate a priority scalar field; the current view is divided into spatial slice areas, budget arbitration is performed under the constraints of computing power budget and time delay, and the slice area priority is mapped to path sampling density, maximum bounce depth, geometric detail level and indirect light evaluation order; The time domain fusion and closed loop module is used for reprojecting the rendering estimation of the last frame to the current frame according to the deformation field or the optical flow prior, and performing time domain fusion; when the light path consistency map is continuously abnormal in a local area within a preset time window and reaches a threshold value, the incremental re-solving is triggered to update the light field model, and quality metrics and running statistics are fed back to the next frame scheduling to form a closed loop.

Images