A method and system for rendering multi-stereoscopic volumes based on 3D Gaussian sputtering

Abstract

This application belongs to the field of image reconstruction, specifically disclosing a multi-view volume rendering method and system based on 3D Gaussian sputtering. This application proposes a novel adaptive control method. Specifically, it classifies Gaussian spheres in the 3D Gaussian voxel model according to scene features, performs voxel division operations for regions with high detail requirements, and performs voxel merging operations for regions with low detail requirements, dynamically loading voxels. This application uses extracted scene features as the basis for region classification, accurately identifying high / low detail requirement regions across all views. Furthermore, the voxel adjustment strategy is strongly bound to region features, and combined with a dynamic loading mechanism of view frustum culling to filter invalid voxels, it achieves a balance between scene detail restoration and rendering efficiency across multiple views. Compared to traditional algorithms, it improves the frame rate of complex scene rendering by 20%-40%, increases object edge sharpness by more than 30%, and reduces GPU memory usage by 30%-50%, meeting the rendering needs of multiple scenes.

Description

Technical Field

[0001] This application belongs to the field of image reconstruction, and more specifically, relates to a method and system for rendering multi-dimensional solid views based on 3D Gaussian sputtering. Background Technology

[0002] With the rapid popularization of immersive technologies such as VR / AR (Virtual Reality / Augmented Reality) and Metaverse, the demand for both "high quality" and "high efficiency" in 3D scene rendering is becoming increasingly urgent. Among traditional 3D rendering techniques, rasterization rendering, while highly efficient, has limitations in depicting complex lighting and details; ray tracing rendering, while capable of reproducing realistic physical lighting, suffers from poor real-time performance due to its high computational cost, making it difficult to meet the interactive needs of immersive scenes. 3D Gaussian sputtering technology, as an emerging 3D scene representation and rendering solution, has the core advantage of directly reconstructing 3D scenes from multi-view images without the need to build complex mesh models, and achieving real-time rendering through hardware acceleration (such as GPUs). This technology has already demonstrated application potential in areas such as dynamic scene reconstruction and virtual scene roaming.

[0003] The complete process of the classic 3D Gaussian sputtering algorithm is as follows: Figure 1 As shown: Based on sparse SFM point clouds, a set of three-dimensional Gaussian distributions is generated; the three-dimensional Gaussian distributions are projected to obtain their two-dimensional projected coordinates; a loss function of a differentiable Gaussian rasterization algorithm is constructed based on the two-dimensional projected coordinates; the parameters are optimized using a differentiable Gaussian rasterization algorithm to correct parameter deviations and then rendered to form a 3D Gaussian voxel model; based on the 3D Gaussian voxel model, the voxel density of the three-dimensional Gaussian distribution is dynamically adjusted through an adaptive control algorithm; the differentiable Gaussian rasterization algorithm uses gradient descent to optimize parameters, and after multiple iterations until convergence, the rendered three-dimensional model is obtained.

[0004] However, the classic adaptive control method of 3D Gaussian sputtering first adjusts the global density of Gaussian voxels based on image features (such as pixel gradients) from a single viewpoint, then increases or decreases the number of voxels by using a fixed voxel splitting / merging threshold, and finally uses full voxel loading to participate in rendering calculations. The whole method uses local features from a single viewpoint as the sole basis for adjusting the voxel distribution, which has the following defects and shortcomings: First, the region division lacks the support of scene features from multiple viewpoints. Relying solely on single-viewpoint features can easily lead to insufficient voxel density in high-detail areas and voxel redundancy in low-detail areas, failing to balance detail restoration and computational efficiency under all viewpoints. Second, the fixed voxel adjustment threshold cannot adapt to the differences in structural complexity of different scenes. For complex industrial parts, historical sites, and other scenes, it is easy to cause problems such as loss of detail or excessive GPU load. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application aims to provide a multi-view volume rendering method and system based on 3D Gaussian sputtering, which solves the problems that existing technologies cannot balance detail restoration and computational efficiency under all viewpoints, and are prone to detail loss or excessive GPU load in complex scenes.

[0006] To achieve the above objectives, in a first aspect, this application provides a multi-stereoscopic volume rendering method based on 3D Gaussian sputtering, comprising:

[0007] A set of three-dimensional Gaussian distributions is generated based on sparse SFM point cloud combined with depth map data.

[0008] A three-dimensional Gaussian distribution is projected to obtain its two-dimensional projected coordinates. Based on the two-dimensional projected coordinates, a loss function for a differentiable Gaussian rasterization algorithm is constructed. The differentiable Gaussian rasterization algorithm is used to optimize parameters to correct parameter deviations and then render the model, forming a 3D Gaussian voxel model. Based on the 3D Gaussian voxel model, the voxel density of the three-dimensional Gaussian distribution is dynamically adjusted through an improved adaptive control algorithm. The differentiable Gaussian rasterization algorithm uses gradient descent to optimize parameters, and after multiple iterations until convergence, the rendered three-dimensional model is obtained.

[0009] The improved adaptive control algorithm includes the following sub-steps:

[0010] S1. Extract scene features: Extract scene features from the 3D Gaussian voxel model after multi-view collaborative training, and classify the Gaussian spheres in the 3D Gaussian voxel model according to the scene features to obtain high detail requirement regions, low detail requirement regions and normal regions.

[0011] S2. Dynamic adjustment of Gaussian volume element density: Perform volume element division operation for each high detail demand region, and perform volume element merging operation for each low detail demand region.

[0012] S3. Dynamic loading of Gaussian voxels: Combining the view frustum culling algorithm, Gaussian voxels within the visible range of the current rendering view are selected. All selected Gaussian voxels participate in real-time calculation, while Gaussian voxels in the invisible area are temporarily stored in the video memory cache and called as needed when the view changes.

[0013] Preferably, step S1 specifically includes:

[0014] Edge detection, regional texture grayscale variance calculation, and depth gradient calculation of adjacent Gaussian voxels are performed on the 3D Gaussian voxel model after multi-view collaborative training.

[0015] If the edge gradient value is greater than 20, the region texture grayscale variance is greater than 50, and the depth gradient of adjacent Gaussian voxels is greater than 0.2mm, it is marked as a region with high detail requirements. If the edge gradient value is less than 10, the region texture grayscale variance is less than 20, and the depth gradient of adjacent Gaussian voxels is less than 0.1mm, it is marked as a region with low detail requirements. All others are marked as normal regions.

[0016] Preferably, the method employs an improved differentiable Gaussian rasterization algorithm, which includes the following sub-steps:

[0017] T1. Multi-view projection: Perform full-view projection on all three-dimensional Gaussian volume elements to obtain two-dimensional projected coordinates from all viewpoints;

[0018] T2. Differentiable rendering and composite loss calculation: Differentiable Gaussian rasterization algorithm is used to generate multi-view rendering images in parallel. Using two-dimensional projection coordinates, the theoretical value of disparity calculated based on camera pose and epipolar equation parameters, the composite total loss of differentiable Gaussian rasterization is constructed. The composite total loss includes single-view reconstruction loss, disparity consistency loss and epipolar constraint loss.

[0019] T3. Gradient Descent Iterative Optimization: Through a differentiable path, the gradient of the composite total loss is backpropagated to the geometric and appearance properties of the Gaussian voxel. The gradient descent algorithm is used to iteratively update the parameters until the composite loss converges, and finally a visually consistent Gaussian voxel model from multiple perspectives is obtained.

[0020] Preferably, the method of constructing a composite total loss for differentiable Gaussian rasterization using two-dimensional projected coordinates, theoretical values ​​of disparity calculated based on camera pose, and epipolar equation parameters specifically involves:

[0021] Calculate Gaussian volume elements In the camera projection point In the camera The corresponding polar line ,in, Dual perspective The fundamental matrix is ​​used to further calculate the Gaussian volume elements. camera Projection point on to the poles Square distance ;

[0022] Calculate Gaussian volume elements respectively In dual perspectives Theoretical disparity value The actual parallax of Gaussian volume projection ;

[0023] Summing the parallax bias of all Gaussian voxels and camera pairs As a parallax consistency loss, the deviation of all Gaussian voxels and camera pairs is summed. As the epipolar constraint loss, where, This represents the set of all valid dual-view camera pairs.

[0024] Preferably, the formula for calculating the total composite loss is as follows:

[0025]

[0026] in,

[0027]

[0028] In the formula, Represents the total compound loss. Indicates single-view reconstruction loss. Parallax consistency loss Indicates epipolar constraint loss. The weighting coefficients representing the parallax consistency loss. The weighting coefficients represent the epipolar constraint loss; Single-view The number of pixels in the width direction of the image; Single-view The number of pixels in the height direction of the image. For single-view Below, the image rendered by the 3D Gaussian volumetric model is in pixel coordinates. RGB color vector at the location; For the corresponding perspective Below, the actual images captured by the industrial camera in pixel coordinates RGB color vector at that location, The number of viewpoints.

[0029] Preferably, after gradient descent, the method optimizes the distribution of Gaussian volume elements in three-dimensional space in the following way:

[0030] (1) Based on the Gaussian volume element parameters after gradient descent, the deviation volume element regions in the three-dimensional Gaussian volume element model are selected;

[0031] (2) Perform Gaussian volume element position fine-tuning and density adaptation on the deviation volume element region to make it conform to the parallax law of "near is larger and far is smaller" and the geometric constraint that "the projection point must be on the epipolar line".

[0032] (3) Verify the matching degree between the optimized 3D Gaussian voxel distribution and the multi-view projection. If the matching degree deviation is greater than the preset threshold, proceed to step (2). If the 3D Gaussian voxel distribution and the multi-view projection are consistent, then end the Gaussian voxel distribution optimization.

[0033] Preferably, the step of performing Gaussian voxel position fine-tuning and density adaptation on the deviation voxel regions specifically includes: fine-tuning the spatial coordinates along the depth direction for near-view deviation Gaussian voxels; correcting the projection position along the horizontal direction for far-view deviation Gaussian voxels; increasing the Gaussian voxel density by 1-2 times for high detail deviation regions; and reducing the Gaussian voxel density by 50%-70% for low detail deviation regions.

[0034] Preferably, the method obtains a sparse SFM-based point cloud in the following manner:

[0035] Multiple industrial cameras are arranged in a ring array to acquire multi-view images without blind spots in the 0°-360° circumferential and ±30° pitch range at equal intervals. The spatiotemporal consistency of the multi-view images is ensured by hardware synchronous control, and the intrinsic and extrinsic parameters of each camera are recorded synchronously.

[0036] By simulating and reproducing camera pose errors through random rotation transformation and perspective distortion, and combining dynamic Gamma correction and local brightness modulation to simulate illumination changes, and then generating intermediate view virtual images based on binocular parallax to fill the view gaps, the effective sample size is increased by 2-3 times.

[0037] First, the intrinsic and extrinsic parameters of each industrial camera are calibrated. Then, based on the calibrated camera parameters, the SFM algorithm is executed on the multi-view images to reconstruct the 3D structure of the scene, generating sparse SFM point clouds during the reconstruction process.

[0038] Preferably, after obtaining the preliminary 3D Gaussian voxel model through gradient descent iterative optimization and before performing Gaussian voxel spatial distribution optimization, the method further includes: multi-view fusion noise suppression and adaptive rendering parameter adjustment processing; wherein,

[0039] The multi-view fusion noise suppression is specifically as follows: for multi-view rendered images, firstly, the projection coordinates are precisely aligned by SIFT feature point matching, then the illumination noise is filtered by neighborhood mean filtering and the projection ghosting is suppressed by weighted fusion algorithm. At the same time, abnormal Gaussian voxels are identified by feature clustering, and then the missing area after the abnormal voxels are removed is filled by interpolation replacement method.

[0040] The adaptive rendering parameter adjustment specifically involves dynamically adjusting the Gaussian voxel sampling density and calculation accuracy based on the scene type and hardware computing power.

[0041] To achieve the above objectives, in a second aspect, this application provides a multi-stereoscopic volume rendering system based on 3D Gaussian sputtering, including a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions; the one or more processors call the computer instructions to cause the system to execute the multi-stereoscopic volume rendering method as described in the first aspect.

[0042] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0043] This application provides a multi-view volume rendering method and system based on 3D Gaussian sputtering. It improves upon the classic adaptive control algorithm of 3D Gaussian sputtering by proposing a novel adaptive control method. Specifically, it classifies Gaussian spheres in the 3D Gaussian voxel model according to scene features, then performs voxel division operations on high-detail-requirement regions and voxel merging operations on low-detail-requirement regions, followed by dynamic voxel loading. This application uses extracted scene features as the basis for region classification, accurately identifying high / low detail-requirement regions across all views. Furthermore, the voxel adjustment strategy is strongly bound to region features, and a dynamic loading mechanism using frustum culling filters invalid voxels. Therefore, it achieves a balance between scene detail reproduction and rendering efficiency across multiple views. Compared to traditional algorithms, it improves the frame rate of complex scene rendering by 20%-40%, enhances detail indicators such as object edge sharpness by over 30%, and reduces GPU memory usage by 30%-50%, meeting the rendering needs of various scenarios such as VR immersive interaction and high-precision industrial displays. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the classic 3D Gaussian sputtering algorithm provided by existing technology.

[0045] Figure 2 This is a flowchart illustrating a multi-dimensional volume rendering method based on 3D Gaussian sputtering provided in an embodiment of this application.

[0046] Figure 3 This is a flowchart illustrating the improved adaptive control algorithm provided in the embodiments of this application.

[0047] Figure 4 This is a flowchart illustrating the improved differentiable Gaussian rasterization algorithm provided in the embodiments of this application.

[0048] Figure 5 This is a schematic diagram of the distribution of the ring-shaped multi-view image acquisition device provided in the embodiments of this application.

[0049] Figure 6This is a schematic diagram illustrating the rendering effect of the multi-view reconstruction and volume rendering of the model under different viewpoints provided in the embodiments of this application.

[0050] Figure 7 This is a comparison chart of the rendering effects of the classic 3D Gaussian sputtering algorithm provided in the embodiments of this application and the improved 3D Gaussian sputtering algorithm proposed in this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] In this application, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A existing alone, A and B existing simultaneously, and B existing alone. In this application, the symbol " / " indicates that the related objects are in an "or" relationship, for example, A / B means A or B.

[0053] In this application, the terms "first" and "second," etc., are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0054] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0056] The embodiments of this application are described below with reference to the accompanying drawings.

[0057] like Figure 2 As shown, this application provides a multi-stereoscopic volume rendering method based on 3D Gaussian sputtering, including:

[0058] A set of three-dimensional Gaussian distributions is generated based on sparse SFM (Structure from Motion) point clouds combined with depth map data;

[0059] A three-dimensional Gaussian distribution is projected to obtain its two-dimensional projected coordinates. Based on the two-dimensional projected coordinates, a loss function for a differentiable Gaussian rasterization algorithm is constructed. The differentiable Gaussian rasterization algorithm is used to optimize parameters to correct parameter deviations and then render the model, forming a 3D Gaussian voxel model. Based on the 3D Gaussian voxel model, the voxel density of the three-dimensional Gaussian distribution is dynamically adjusted through an improved adaptive control algorithm. The differentiable Gaussian rasterization algorithm uses gradient descent to optimize parameters, and after multiple iterations until convergence, the rendered three-dimensional model is obtained.

[0060] like Figure 3 As shown, the improved adaptive control algorithm includes the following sub-steps:

[0061] S1. Extract scene features: Extract scene features from the 3D Gaussian voxel model after multi-view collaborative training, and classify the Gaussian spheres in the 3D Gaussian voxel model according to the scene features to obtain high detail requirement regions, low detail requirement regions and normal regions.

[0062] S2. Dynamic adjustment of Gaussian volume element density: Perform volume element division operation for each high detail demand region, and perform volume element merging operation for each low detail demand region.

[0063] S3. Dynamic loading of Gaussian voxels: Combining the view frustum culling algorithm, Gaussian voxels within the visible range of the current rendering view are selected. All selected Gaussian voxels participate in real-time calculation, while Gaussian voxels in the invisible area are temporarily stored in the video memory cache and called as needed when the view changes.

[0064] Preferably, step S1 specifically includes:

[0065] Edge detection, regional texture grayscale variance calculation, and depth gradient calculation of adjacent Gaussian voxels are performed on the 3D Gaussian voxel model after multi-view collaborative training.

[0066] If the edge gradient value is greater than 20, the region texture grayscale variance is greater than 50, and the depth gradient of adjacent Gaussian voxels is greater than 0.2mm, it is marked as a region with high detail requirements. If the edge gradient value is less than 10, the region texture grayscale variance is less than 20, and the depth gradient of adjacent Gaussian voxels is less than 0.1mm, it is marked as a region with low detail requirements (such as a flat surface or a solid color background). All others are marked as normal regions.

[0067] It should be noted that these three types of information, as scene features, each have clear representativeness. Among them, edge gradient corresponds to the contours and structurally abrupt regions of objects (such as the edges of parts and engravings on artifacts), and is the core carrier of visual detail sharpness; regional texture grayscale variance corresponds to the complexity of surface texture (such as fabric texture and brushed metal surfaces), and is the key to distinguishing high texture density areas from solid color smooth areas; adjacent voxel depth gradient corresponds to the undulations of spatial dimensions (such as relief protrusions and grooves in parts), and is the core basis for identifying three-dimensional detail areas. Compared with scene features using a single feature (such as texture variance alone) or other indicators (such as average brightness), combining these three types of features can cover "two-dimensional image features + three-dimensional spatial features," avoiding misjudgments caused by a single feature (for example, using texture variance alone may miss the edges of metal parts with no texture but sharp edges). Through multi-dimensional cross-validation, detailed regions are accurately divided, ensuring voxel density in high-detail areas to restore fine structures, while reducing voxel redundancy in low-detail areas to save computing power, ultimately achieving dual optimization of detail restoration and rendering efficiency under multiple perspectives.

[0068] In one illustrated embodiment, the Canny operator is used to extract edges.

[0069] In one illustrated embodiment, for marked high-detail areas, voxels with a scale greater than or equal to 0.3 mm are split into smaller voxels of 0.05-0.1 mm, increasing the number of voxels by 2-3 times to ensure detail restoration. For low-detail areas, voxel merging is performed, merging voxels with a scale less than 0.01 mm into larger voxels of 0.3-0.5 mm, reducing the number of voxels by 50%-70% and lowering the computational load.

[0070] like Figure 4 As shown, preferably, the method employs an improved differentiable Gaussian rasterization algorithm, which includes the following sub-steps:

[0071] T1. Multi-view projection: Perform full-view projection on all three-dimensional Gaussian volume elements to obtain two-dimensional projected coordinates from all viewpoints;

[0072] T2. Differentiable rendering and composite loss calculation: Differentiable Gaussian rasterization algorithm is used to generate multi-view rendering images in parallel. Using two-dimensional projection coordinates, the theoretical value of disparity calculated based on camera pose and epipolar equation parameters, the composite total loss of differentiable Gaussian rasterization is constructed. The composite total loss includes single-view reconstruction loss, disparity consistency loss and epipolar constraint loss.

[0073] T3. Gradient Descent Iterative Optimization: Through a differentiable path, the gradient of the composite total loss is backpropagated to the geometric and appearance properties of the Gaussian voxel. The gradient descent algorithm is used to iteratively update the parameters until the composite loss converges, and finally a visually consistent Gaussian voxel model from multiple perspectives is obtained.

[0074] It should be noted that the classic 3D Gaussian sputtering differentiable Gaussian rasterization algorithm calculates the reconstruction loss based solely on single-view projection coordinates, updates voxel parameters through gradient descent, and directly outputs the single-view rendering result. However, this differentiable Gaussian rasterization algorithm has the following defects and shortcomings: relying solely on single-view loss, it is prone to multi-view spatial misalignment and color inconsistency; it does not introduce epipolar constraints, resulting in large disparity deviations in the rendering results; and it lacks composite optimization of multi-dimensional losses, leading to poor multi-view consistency of the voxel model. To address these issues, this application proposes an improved differentiable Gaussian rasterization algorithm based on the classic 3D Gaussian sputtering algorithm. The main improvement lies in transforming the voxel spatial position, theoretical disparity value, and epipolar equation parameters into multi-dimensional geometric constraint losses (epochal constraint loss and disparity consistency loss), which are then combined with the single-view reconstruction loss to construct a composite total loss. The composite total loss includes single-view reconstruction loss, disparity consistency loss, and epipolar constraint loss. This multi-dimensional loss can simultaneously complete the calibration of voxel geometric position and the single-view rendering accuracy constraint. It strengthens the multi-view consistency of the voxel model from three dimensions: "pixel color restoration", "dual-view parallax matching" and "epochal geometric alignment". Therefore, it can achieve a spatial alignment error of less than 1 pixel and a 50% reduction in parallax deviation in the rendering results under multiple views. The multi-view consistency of the voxel model at the geometric projection and pixel restoration levels is significantly improved.

[0075] Preferably, the method of constructing a composite total loss for differentiable Gaussian rasterization using two-dimensional projected coordinates, theoretical values ​​of disparity calculated based on camera pose, and epipolar equation parameters specifically involves:

[0076] Calculate Gaussian volume elements In the camera projection point In the camera The corresponding polar line ,in, Dual perspective The fundamental matrix is ​​used to further calculate the Gaussian volume elements. camera Projection point on to the poles Square distance ;

[0077] Calculate Gaussian volume elements respectively In dual perspectives Theoretical disparity value The actual parallax of Gaussian volume projection ;

[0078] Summing the parallax bias of all Gaussian voxels and camera pairs As a parallax consistency loss, the deviation of all Gaussian voxels and camera pairs is summed. As the epipolar constraint loss, where, This represents the set of all valid dual-view camera pairs.

[0079] In one illustrated embodiment, the actual parallax is calculated using the projected coordinates. Gaussian element In dual perspectives The theoretical parallax is Dual perspective baseline (Simplified to translation vector magnitude, assuming a horizontal baseline), calculated from extrinsic parameters. ,in, , Polar equation: , Each is a perspective and The translation vector of the lower camera. From the perspective To view The relative translation vector, Each is a perspective and The rotation matrix of the camera below, From the perspective To view The relative rotation matrix, From the perspective Next Gaussian element The depth value, Here are the horizontal pixel coordinates of the Gaussian volumetric projection points, corresponding to the horizontal axis position of the image plane. represents the vertical pixel coordinates of the Gaussian volume element projection point, corresponding to the vertical axis position of the image plane.

[0080] Preferably, the formula for calculating the total composite loss is as follows:

[0081]

[0082] in,

[0083]

[0084] In the formula, Represents the total compound loss. Indicates single-view reconstruction loss. Parallax consistency loss Indicates epipolar constraint loss. The weighting coefficients representing the parallax consistency loss. The weighting coefficients represent the epipolar constraint loss; Single-view The number of pixels in the width direction of the image; Single-view The number of pixels in the height direction of the image. For single-view Below, the image rendered by the 3D Gaussian volumetric model is in pixel coordinates. RGB color vector at the location; For the corresponding perspective Below, the actual images captured by the industrial camera in pixel coordinates RGB color vector at that location, The number of viewpoints.

[0085] Preferably, after gradient descent, the method optimizes the distribution of Gaussian volume elements in three-dimensional space in the following way:

[0086] (1) Based on the Gaussian volume element parameters after gradient descent, the deviation volume element regions in the three-dimensional Gaussian volume element model are selected;

[0087] (2) Perform Gaussian volume element position fine-tuning and density adaptation on the deviation volume element region to make it conform to the parallax law of "near is larger and far is smaller" and the geometric constraint that "the projection point must be on the epipolar line".

[0088] (3) Verify the matching degree between the optimized 3D Gaussian voxel distribution and the multi-view projection. If the matching degree deviation is greater than the preset threshold, proceed to step (2). If the 3D Gaussian voxel distribution and the multi-view projection are consistent, then end the Gaussian voxel distribution optimization.

[0089] It should be noted that the classic 3D Gaussian sputtering algorithm processes voxel parameters directly after gradient descent, without specific regional deviation verification or voxel adjustment steps, and the voxels participate in subsequent rendering according to their initial distribution. However, this approach has the following drawbacks: local deviations in voxel spatial distribution are prone to occur (such as overlapping voxels in the foreground and sparse voxels in the background), and the voxel density does not match the scene detail requirements, leading to unbalanced rendering details and wasted computational power. To address this issue, this application proposes an optimized voxel distribution method in three-dimensional space based on the classic 3D Gaussian sputtering algorithm. After gradient descent, a "deviation region identification - voxel position fine-tuning - density adaptation" process is added. Deviation regions are screened through parallax / epochal deviation, and the spatial position and density of voxels are adjusted in combination with the regional detail requirements. This can accurately match the scene's detail distribution characteristics, thus improving the rationality of voxel spatial distribution by 40% and achieving a two-way optimization of rendering detail fidelity and computational efficiency.

[0090] Preferably, the step of performing Gaussian voxel position fine-tuning and density adaptation on the deviation voxel regions specifically includes: fine-tuning the spatial coordinates along the depth direction for near-view deviation Gaussian voxels; correcting the projection position along the horizontal direction for far-view deviation Gaussian voxels; increasing the Gaussian voxel density by 1-2 times for high detail deviation regions; and reducing the Gaussian voxel density by 50%-70% for low detail deviation regions.

[0091] In one illustrated embodiment, for near-field deviation voxels, the spatial coordinates are finely adjusted along the depth direction (shifted 0.05-0.1mm away from the camera when the deviation value is >0.1mm); for far-field deviation voxels, the projection position is corrected along the horizontal direction (alignment epipolar deviation <0.5 pixels); the voxel density in high detail deviation areas is increased by 1-2 times (e.g., toothed or scribing areas), and the voxel density in low detail deviation areas is reduced by 50%-70% (e.g., solid color plane areas). After adjustment, the epipolar alignment of the voxels is verified (error <1 pixel).

[0092] Preferably, the method obtains a sparse SFM-based point cloud in the following manner:

[0093] Multiple industrial cameras are arranged in a ring array to acquire multi-view images without blind spots in the 0°-360° circumferential and ±30° pitch range at equal intervals. The spatiotemporal consistency of the multi-view images is ensured by hardware synchronous control, and the intrinsic and extrinsic parameters of each camera are recorded synchronously.

[0094] By simulating and reproducing camera pose errors through random rotation transformation and perspective distortion, and combining dynamic Gamma correction and local brightness modulation to simulate illumination changes, and then generating intermediate view virtual images based on binocular parallax to fill the view gaps, the effective sample size is increased by 2-3 times.

[0095] First, the intrinsic and extrinsic parameters of each industrial camera are calibrated. Then, based on the calibrated camera parameters, the SFM algorithm is executed on the multi-view images to reconstruct the 3D structure of the scene, generating sparse SFM point clouds during the reconstruction process.

[0096] It should be noted that this application constructs a stereo image dataset covering the entire field of view through the aforementioned preferred method. Data augmentation expands the sample size, increasing the effective sample size by 2-3 times, allowing the model to learn complete scene features from all perspectives. In one illustrated embodiment, a circular array of 6-12 industrial cameras is used to achieve 0°-360° circumferential and ±30° pitch range acquisition without blind spots at 30°-60° intervals. Figure 5 As shown.

[0097] Preferably, after obtaining the preliminary 3D Gaussian voxel model through gradient descent iterative optimization and before performing Gaussian voxel spatial distribution optimization, the method further includes: multi-view fusion noise suppression and adaptive rendering parameter adjustment processing; wherein,

[0098] The multi-view fusion noise suppression is specifically as follows: For multi-view rendered images, firstly, the projection coordinates are precisely aligned by SIFT (Scale Invariant Feature Transform) feature point matching, with an alignment error of <1 pixel. Then, the neighborhood mean filter is used to filter illumination noise, and the weighted fusion algorithm is used to suppress projection ghosting. At the same time, abnormal Gaussian voxels are identified by feature clustering, and then the interpolation replacement method is used to fill in the missing areas after the abnormal voxels are removed.

[0099] The adaptive rendering parameter adjustment specifically involves dynamically adjusting the Gaussian voxel sampling density and calculation accuracy based on the scene type and hardware computing power.

[0100] In one illustrated embodiment, the following steps are performed: Illumination noise filtering: An ambient lighting baseline value is calculated using the average brightness of multi-view images. Pixels deviating from the baseline value by more than ±15% are corrected using a "neighborhood mean replacement" algorithm to eliminate brightness noise caused by illumination fluctuations. Projection ghosting suppression: Overlapping pixel regions at the same physical point are detected in multi-view projected images. The pixel grayscale variance of the overlapping regions is calculated. If the variance is greater than a threshold (e.g., 30), it is identified as a "ghosting region." A "weighted fusion" algorithm (with viewpoint sharpness as the weight) is used to merge overlapping pixels, eliminating ghosting. Gaussian voxel outlier filtering: For trained Gaussian voxels, the deviations of their color and scale parameters from those of neighboring voxels are calculated. If the deviation is greater than 3 times the standard deviation, it is identified as an "outlier voxel." Interpolation is used to replace the outlier voxels with the average parameters of the neighboring voxels to avoid rendering defects caused by outlier voxels.

[0101] In one illustrated embodiment, real-time interactive scenarios use single-precision floating-point calculations, while high-precision modeling scenarios use double-precision floating-point calculations.

[0102] This application also provides a multi-view volume rendering system based on 3D Gaussian sputtering, including a memory and one or more processors; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions; the one or more processors call the computer instructions to cause the system to execute the multi-view volume rendering method described above.

[0103] Example

[0104] In this embodiment, when the target scene is large, a drone is used with a resolution of 1920*1080 and a frame rate of 60fps to collect approximately 1000 images of buildings from different perspectives, covering a 0°-360° circumferential view and a -45° to +45° pitch view, ensuring comprehensive coverage of the overall structure, facade details, and spatial layout of the large scene. When the target is a small to medium-sized scene, industrial cameras can be used, such as six industrial cameras with a resolution of 720*540 and a frame rate of 30fps, arranged in a circular array (radius 1.5m), covering a 0°-360° view. During data acquisition, the six cameras are controlled to shoot simultaneously, collecting 1000 sets of multi-view images, each set containing six room images from different perspectives. Preprocessing operations are then performed: Zhang Zhengyou camera calibration is implemented using the OpenCV library to correct lens distortion; histogram equalization is used to normalize brightness and eliminate lighting differences between different perspectives.

[0105] The training process of the 3DGS (3D Gaussian Sputtering) voxel model is as follows: Initialization: The initial depth map is calculated based on the binocular matching algorithm (SGBM). Gaussian voxels with a density of 500 voxels / mm² are deployed in high detail regions, and voxels with a density of 100 voxels / mm² are deployed in low detail regions, with a total initial voxel count of approximately 1 million. Loss function settings: The single-view reconstruction loss weight is 1, the disparity consistency loss weight is 30%, and the epipolar constraint loss weight is 20%. Training environment: The GPU is an RTX4090 with 24GB of VRAM, the framework is PyTorch2.0, the number of iterations is 1000, the initial learning rate is 0.001, and it decays to 1 / 10 of the original rate every 200 iterations, finally obtaining the trained 3DGS voxel model.

[0106] Scene feature extraction: The Canny algorithm is used to extract scene edges, calculate texture grayscale variance, and calculate depth gradient. Regions with edge gradient values ​​> 20, region texture grayscale variance > 50, and adjacent Gaussian voxel depth gradients > 0.2mm are marked as high detail regions, while regions with edge gradient values ​​< 10, region texture grayscale variance < 20, and adjacent Gaussian voxel depth gradients < 0.1mm are marked as low detail regions.

[0107] Voxel adjustment: The marked area is subdivided into voxels, with the original 0.2mm scale voxels subdivided into 0.08mm scale voxels, increasing the number of voxels to 1.5 million; the flat area of ​​the scene (variance 20, depth gradient 0.1mm) is merged into voxels, with the 0.1mm scale voxels merged into 0.4mm scale voxels, reducing the number of voxels to 800,000.

[0108] Multi-view fusion: Through SIFT feature point matching, the corresponding tooth feature points in 12 view images are found, the coordinate offset is calculated (average deviation 0.8 pixels), and the voxel projection coordinates are corrected; the neighborhood mean replacement algorithm is used to filter illumination noise (the proportion of pixels with brightness deviation exceeding 15% is reduced from 12% to 3%); ghosting is eliminated through weighted fusion, and the gray-level variance of the overlapping area is reduced from 45 to 15.

[0109] Adaptive rendering parameter adjustment test: High-performance hardware (24GB VRAM) + static display scene: 100% sampling density, double-precision floating-point calculation, TAA (temporal anti-aliasing) anti-aliasing, 4K rendering resolution, stable frame rate at 60fps, and edge sharpness (edge ​​gradient value) of 250; Medium-performance hardware (12GB VRAM) + VR interaction scene: 70% sampling density, single-precision floating-point calculation, FXAA (fast approximate anti-aliasing) anti-aliasing, 2K rendering resolution (VR binocular resolution), stable frame rate at 90fps, meeting the low latency requirements of VR interaction; Low-performance hardware (6GB VRAM) + mobile interaction scene: 40% sampling density, single-precision floating-point calculation, FXAA anti-aliasing, 1080P rendering resolution, stable frame rate at 60fps, can run smoothly on VR all-in-one devices.

[0110] The rendering effects of the model under different viewpoints after multi-view reconstruction and volume rendering are as follows: Figure 6 As shown. Multi-view consistency verification results show that when rendering room images from four perspectives (0°, 90°, 180°, and 270°), the spatial deviation calculated using a pixel comparison tool is a maximum of 0.9 pixels, and the maximum color deviation is 4.2%, meeting the multi-view consistency requirements. Detail detection verification results show that the accuracy rate for identifying 0.1mm-level fine details such as door frame lines in the rendered image reaches 98%, with no false positives due to noise. Efficiency verification results show that compared to traditional uniform voxel distribution (1 million voxels), this invention improves the frame rate from 32fps to 58fps on medium-power hardware, representing an 81% efficiency improvement.

[0111] A comparison of the rendering effects using the classic 3D Gaussian sputtering algorithm and the improved 3D Gaussian sputtering algorithm proposed in this application is shown. Figure 7 As shown, the new method on the right significantly reduces the area of ​​artifacts compared to the classic algorithm on the left.

[0112] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0113] Based on the methods in the above embodiments, this application provides an electronic device that may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor may invoke logical instructions stored in the memory to execute the methods in the above embodiments.

[0114] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0115] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0116] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0117] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0118] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0119] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0120] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0121] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for rendering multiple stereoscopic views based on 3D Gaussian sputtering, characterized in that, include: A set of three-dimensional Gaussian distributions is generated based on sparse SFM point cloud combined with depth map data. A three-dimensional Gaussian distribution is projected to obtain its two-dimensional projected coordinates. A loss function for a differentiable Gaussian rasterization algorithm is constructed based on the two-dimensional projected coordinates. The parameters are optimized using a differentiable Gaussian rasterization algorithm to correct parameter deviations and then rendered to form a 3D Gaussian voxel model. Based on the 3D Gaussian voxel model, the voxel density of the three-dimensional Gaussian distribution is dynamically adjusted through an improved adaptive control algorithm. The differentiable Gaussian rasterization algorithm uses gradient descent to optimize parameters and iterates multiple times until convergence to obtain the rendered three-dimensional model. The improved adaptive control algorithm includes the following sub-steps: S1. Extract scene features: Extract scene features from the 3D Gaussian voxel model after multi-view collaborative training, and classify the Gaussian spheres in the 3D Gaussian voxel model according to the scene features to obtain high detail requirement regions, low detail requirement regions and normal regions. S2. Dynamic adjustment of Gaussian volume element density: Perform volume element division operation for each high detail demand region, and perform volume element merging operation for each low detail demand region. S3. Dynamic loading of Gaussian voxels: Combining the view frustum culling algorithm, Gaussian voxels within the visible range of the current rendering view are selected. All selected Gaussian voxels participate in real-time calculation, while Gaussian voxels in the invisible area are temporarily stored in the video memory cache and called as needed when the view changes. The method employs an improved differentiable Gaussian rasterization algorithm, which includes the following sub-steps: T1. Multi-view projection: Perform full-view projection on all three-dimensional Gaussian volume elements to obtain two-dimensional projected coordinates from all viewpoints; T2. Differentiable rendering and composite loss calculation: Differentiable Gaussian rasterization algorithm is used to generate multi-view rendering images in parallel. Using two-dimensional projection coordinates, the theoretical value of disparity calculated based on camera pose and epipolar equation parameters, the composite total loss of differentiable Gaussian rasterization is constructed. The composite total loss includes single-view reconstruction loss, disparity consistency loss and epipolar constraint loss. T3. Gradient Descent Iterative Optimization: Through a differentiable path, the gradient of the total composite loss is backpropagated to the geometric and appearance properties of the Gaussian voxel. The gradient descent algorithm is used to iteratively update the parameters until the composite loss converges, and finally a Gaussian voxel model with consistent visual characteristics from multiple perspectives is obtained. The method utilizes two-dimensional projected coordinates, theoretical disparity values ​​calculated based on camera pose, and epipolar equation parameters to construct a differentiable Gaussian rasterization composite total loss, specifically as follows: Calculate Gaussian volume elements In the camera projection point In the camera The corresponding polar line ,in, Dual perspective The fundamental matrix is ​​used to further calculate the Gaussian volume elements. camera Projection point on to the poles Square distance ; Calculate Gaussian volume elements respectively In dual perspectives Theoretical disparity value The actual parallax of Gaussian volume projection ; Summing the parallax bias of all Gaussian voxels and camera pairs As a parallax consistency loss, the deviation of all Gaussian voxels and camera pairs is summed. As the epipolar constraint loss, where, This represents the set of all valid dual-view camera pairs.

2. The multi-stereoscopic view volume rendering method as described in claim 1, characterized in that, Step S1 is as follows: Edge detection, regional texture grayscale variance calculation, and depth gradient calculation of adjacent Gaussian voxels are performed on the 3D Gaussian voxel model after multi-view collaborative training. If the edge gradient value is greater than 20, the region texture grayscale variance is greater than 50, and the depth gradient of adjacent Gaussian voxels is greater than 0.2mm, it is marked as a region with high detail requirements. If the edge gradient value is less than 10, the region texture grayscale variance is less than 20, and the depth gradient of adjacent Gaussian voxels is less than 0.1mm, it is marked as a region with low detail requirements. All others are marked as normal regions.

3. The multi-stereoscopic view volume rendering method as described in claim 1, characterized in that, The formula for calculating the total composite loss is as follows: in, In the formula, Represents the total compound loss. Indicates single-view reconstruction loss. Parallax consistency loss Indicates epipolar constraint loss. The weighting coefficients representing the parallax consistency loss. The weighting coefficients represent the epipolar constraint loss; Single-view The number of pixels in the width direction of the image; Single-view The number of pixels in the height direction of the image. For single-view Below, the image rendered by the 3D Gaussian volumetric model is in pixel coordinates. RGB color vector at the location; For the corresponding perspective Below, the actual images captured by the industrial camera in pixel coordinates RGB color vector at that location, The number of viewpoints.

4. The multi-stereoscopic view volume rendering method as described in claim 1, characterized in that, After gradient descent, the method optimizes the distribution of Gaussian volume elements in three-dimensional space in the following way: (1) Based on the Gaussian volume element parameters after gradient descent, the deviation volume element regions in the three-dimensional Gaussian volume element model are selected; (2) Perform Gaussian volume element position fine-tuning and density adaptation on the deviation volume element region to make it conform to the parallax law of "near is larger and far is smaller" and the geometric constraint that "the projection point must be on the epipolar line"; (3) Verify the matching degree between the optimized 3D Gaussian voxel distribution and the multi-view projection. If the matching degree deviation is greater than the preset threshold, proceed to step (2). If the 3D Gaussian voxel distribution and the multi-view projection are consistent, then end the Gaussian voxel distribution optimization.

5. The multi-stereoscopic view volume rendering method as described in claim 4, characterized in that, The process of performing Gaussian voxel position fine-tuning and density adaptation on the deviation voxel regions specifically includes: fine-tuning the spatial coordinates along the depth direction for near-view deviation Gaussian voxels; correcting the projection position along the horizontal direction for far-view deviation Gaussian voxels; increasing the Gaussian voxel density by 1-2 times for high detail deviation regions; and reducing the Gaussian voxel density by 50%-70% for low detail deviation regions.

6. The multi-stereoscopic view volume rendering method as described in claim 1, characterized in that, This method obtains sparse SFM-based point clouds in the following way: Multiple industrial cameras are arranged in a ring array to acquire multi-view images without blind spots in the 0°-360° circumferential and ±30° pitch range at equal intervals. The spatiotemporal consistency of the multi-view images is ensured by hardware synchronous control, and the intrinsic and extrinsic parameters of each camera are recorded synchronously. By simulating and reproducing camera pose errors through random rotation transformation and perspective distortion, and combining dynamic Gamma correction and local brightness modulation to simulate illumination changes, and then generating intermediate view virtual images based on binocular parallax to fill the view gaps, the effective sample size is increased by 2-3 times. First, the intrinsic and extrinsic parameters of each industrial camera are calibrated. Then, based on the calibrated camera parameters, the SFM algorithm is executed on the multi-view images to reconstruct the 3D structure of the scene, generating sparse SFM point clouds during the reconstruction process.

7. The multi-stereoscopic view volume rendering method according to any one of claims 1 to 6, characterized in that, After obtaining the initial 3D Gaussian volumetric model through gradient descent iterative optimization and before performing Gaussian volumetric spatial distribution optimization, this method also includes: multi-view fusion noise suppression and adaptive rendering parameter adjustment processing; wherein... The multi-view fusion noise suppression is specifically as follows: for multi-view rendered images, the projection coordinates are first aligned by SIFT feature point matching, then the illumination noise is filtered by neighborhood mean filtering and the projection ghosting is suppressed by weighted fusion algorithm. At the same time, abnormal Gaussian voxels are identified by feature clustering, and then the missing area after the abnormal voxels are removed is filled by interpolation replacement method. The adaptive rendering parameter adjustment specifically involves dynamically adjusting the Gaussian voxel sampling density and calculation accuracy based on the scene type and hardware computing power.

8. A multi-stereoscopic volume rendering system based on 3D Gaussian sputtering, characterized in that, Includes memory and one or more processors; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions; The one or more processors invoke the computer instructions to cause the system to perform the multi-stereoscopic volume rendering method as described in any one of claims 1-7.

Images