Virtual space rapid construction method based on 3D Gaussian Splitting rendering

Abstract

The invention discloses a rapid virtual space construction method based on 3D Gaussian Splitting rendering, and relates to the technical field of 3D rendering, and the method comprises the steps: S1, data collection: collecting traditional cartoon content information through collection equipment, and carrying out the automatic classification of the collected data according to the cartoon content information; according to the rapid virtual space construction method based on 3D Gaussian Splicing rendering, virtual space construction based on a 3D Gaussian Splicing (3DGS) rendering technology is adopted, for dynamic scenes, large city scenes, human body tracking and the like appearing in cartoons, the 3DGS rapidly generates high-quality three-dimensional point clouds and surface models from images of multiple visual angles, and the three-dimensional point clouds and the surface models are subjected to three-dimensional point cloud and surface modeling. According to the method, point cloud data in a three-dimensional space is represented and rendered by using 3D Gaussian distribution, rendering quality is improved by optimizing parameters of the distribution, so that real-time rendering of generating a 3D scene from a group of 2D images is completed, and developers can utilize efficient rendering and vivid visual effects provided by 3DGS by integrating a 3D Gaussian Splitting technology into a Unity 3D engine.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D rendering, and in particular to a method for quickly constructing a virtual space based on 3D Gaussian Splatting rendering. Background Art

[0002] Virtual space generally refers to a simulated reality or a completely fictional digital space constructed through computer technology, sensors, and networks, where users can interact, explore, socialize, work, and engage in other activities. It ranges from simple two-dimensional virtual environments to highly immersive three-dimensional virtual worlds and is widely used in gaming, education, industry, healthcare, social interaction, and other fields. In recent years, the popularity of VR headsets in China has also increased rapidly, but the lack of integration between hardware technology and content applications remains a key constraint on the development of the entire industry. With the recent rise of VR devices, people have begun to study how to combine VR technology with digital comics, allowing readers to experience comics with three-dimensional virtual reality effects. Thus, VR comics came into being. The foundation of VR comics is the construction of virtual space.

[0003] However, the existing three-dimensional virtual space is not realistic enough, which easily causes readers to feel a lack of immersion and cannot perfectly combine the three-dimensional scenes with the two-dimensional comic images, which greatly interferes with the readers' browsing experience and the overall aesthetics of VR comics. At the same time, the inefficient virtual space construction efficiency also greatly interferes with the user's VR comic production efficiency. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for quickly constructing a virtual space based on 3D Gaussian Splatting rendering to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering, comprising the following steps:

[0006] S1. Data collection: using collection equipment to collect traditional comic content information, and automatically classifying the collected data according to the comic content information;

[0007] S2. Data preprocessing: The collected data is preprocessed to facilitate subsequent processing and rendering. Feature point matching (such as SIFT and ORB) and structure from motion (SfM) algorithms are then used to estimate the pose of the animated character. A sparse point cloud is also generated as the starting point for the initial Gaussian distribution.

[0008] S3, 3D Gaussian Splatting processing:

[0009] S31, point cloud generation: convert the pre-processed sparse point cloud into an initial Gaussian sphere. The parameters (such as covariance) can be set to uniform distribution or adjusted according to the point cloud density.

[0010] S32, Gaussian function fitting: Fit the point cloud data to a Gaussian function, where each Gaussian function represents the position and shape of a 3D point;

[0011] S33, Rendering Optimization: Achieve fast 3D rendering through Gaussian function rendering optimization algorithm;

[0012] S4, rendering and detail enhancement:

[0013] S41, Real-time rendering: Utilizes 3D Gaussian Splatting technology to quickly calculate the color of each pixel without explicitly building a grid. Rendering speed increases linearly with the number of Gaussians.

[0014] S42, Multi-scale Gaussian: Use Gaussian spheres of different sizes to represent scene details (large Gaussian represents macrostructure, small Gaussian depicts texture details) to improve rendering quality;

[0015] S43, Anti-aliasing and blending: Reduce rendering noise and achieve smooth transitions by adjusting Gaussian transparency and overlapping area blending weights;

[0016] S5. Optimization and debugging: Optimize rendering algorithms and data structures to improve rendering speed and efficiency, and conduct comprehensive debugging and testing to ensure the stability and reliability of the virtual space;

[0017] S6. Application and deployment: Save the optimized Gaussian set in a lightweight format (such as a custom binary file), or convert it into a traditional mesh model (such as PLY, OBJ) for use in other rendering engines, and then integrate it into a virtual space platform (such as Unity, Unreal Engine) to implement interactive roaming, virtual display and other functions, so that subsequent users can browse, interact and explore the comic content immersively through different VR headsets.

[0018] Furthermore, in step S2, the original data is preprocessed using a point cloud processing library or an image processing library, including the following steps:

[0019] Denoising: remove noise from data and improve data quality;

[0020] Registration: Register data from different sources to ensure data consistency;

[0021] Segmentation: Segment the data to extract the required features and objects.

[0022] Furthermore, the key features extracted in step S2 include image feature extraction, text feature extraction, and storyboard and narrative structure feature extraction, wherein the image features include dynamic scenes, large urban scenes, human body tracking and other aspects.

[0023] Furthermore, in step S2, after the initial Gaussian is generated, invalid Gaussian samples are automatically eliminated, that is, if the weight (such as transparency) of a Gaussian sample is lower than the threshold, or the projected area under the viewpoint is smaller than the pixel, it is automatically marked as "invisible" and the rendering is skipped. At the same time, the screen space error (Screen-Space Error, SSE) is combined to determine whether the Gaussian needs to be rendered.

[0024] Furthermore, each Gaussian sphere in step S31 is described by a parameter group (xi, ci, Σi), where: xi is the coordinates of the Gaussian center (spatial position), ci is the color value (RGB or RGBA), and Σi is the covariance matrix (controlling the shape and size of the Gaussian). When the Gaussian spheres overlap in space, the colors are mixed by weighted summation to form a continuous surface rendering effect.

[0025] Furthermore, in step S33, Gaussian parameters (such as position, color, and covariance) are optimized to match the rendering result with the input image (such as a multi-view photo), thereby achieving reconstruction from the two-dimensional image to the three-dimensional scene.

[0026] Furthermore, in step S33 , a neural network or a traditional optimization algorithm (such as gradient descent) is used to iteratively adjust parameters, so as to minimize the error between the rendered image and the real image.

[0027] The present invention provides a method for quickly constructing a virtual space based on 3D Gaussian Splatting rendering, which has the following beneficial effects:

[0028] 1. The present invention is based on the construction of virtual space using 3D Gaussian Splatting (3DGS) rendering technology. For dynamic scenes, large urban scenes, human body tracking, etc. that appear in comics, 3DGS quickly generates high-quality three-dimensional point clouds and surface models from images of multiple perspectives, uses 3D Gaussian distribution to represent and render point cloud data in three-dimensional space, and improves rendering quality by optimizing the parameters of these distributions, thereby completing real-time rendering of 3D scenes from a set of 2D images. By integrating 3D Gaussian Splatting technology into the Unity3D engine, developers can take advantage of the efficient rendering and realistic visual effects provided by 3DGS to quickly reconstruct the original comic virtual space.

[0029] 2. The present invention can automatically convert the collected image data into a sparse point cloud as the starting point of the initial Gaussian distribution through data preprocessing operations, and can effectively remove invalid data in the data, thereby providing convenience for subsequent space rendering work. Moreover, through the rendering optimization and real-time rendering settings, the virtual space constructed by this method can be rendered by a natural lighting system, so that the three-dimensional scene and the two-dimensional comic image are perfectly integrated in a virtual space environment, allowing users to naturally integrate into the picture world, creating a stronger sense of immersion, and allowing comics to be presented in a more vivid and three-dimensional interactive way. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic flow chart of the steps of a method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering according to the present invention;

[0031] Figure 2 A schematic flow chart of 3D Gaussian Splatting processing steps of a method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering according to the present invention;

[0032] Figure 3 A schematic flow chart of the rendering and detail enhancement steps of a method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering according to the present invention;

[0033] Figure 4 This is an example diagram of Unity3D virtual space construction for a method of quickly constructing a virtual space based on 3D Gaussian Splatting rendering in the present invention. DETAILED DESCRIPTION

[0034] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0035] like Figure 1-Figure 4 As shown, a method for quickly constructing a virtual space based on 3D Gaussian Splatting rendering includes the following steps:

[0036] S1. Data collection: using collection equipment to collect traditional comic content information, and automatically classifying the collected data according to the comic content information;

[0037] S2, data preprocessing: preprocess the collected data to facilitate subsequent processing and rendering, then use feature point matching (such as SIFT, ORB) and motion recovery structure (SfM) algorithm to estimate the pose of the cartoon character, and generate a sparse point cloud as the starting point of the initial Gaussian distribution. In step S2, use the point cloud processing library or image processing library to preprocess the original data, including the following steps: denoising: remove noise in the data and improve data quality, registration: align data from different sources to ensure data consistency, segmentation: segment the data and extract the required features and objects. The key features extracted in step S2 include image feature extraction, text feature extraction, and storyboard and narrative structure feature extraction, among which image features include dynamic scenes, large urban scenes, human tracking and other aspects. In step S2, when the initial Gaussian is generated, invalid Gaussian samples are automatically eliminated, that is, the weight (such as transparency) of a Gaussian sample is lower than the threshold, or the projected area under the viewpoint is smaller than the pixel, and it is automatically marked as "invisible" and skipped for rendering. At the same time, combined with the screen space error (Screen-Space Error, SSE) determines whether the Gaussian needs to be rendered;

[0038] S3, 3D Gaussian Splatting processing:

[0039] S31, point cloud generation: convert the preprocessed sparse point cloud into an initial Gaussian sphere. The parameters (such as covariance) can be set to uniform distribution or adjusted according to the point cloud density. In step S31, each Gaussian sphere is described by a parameter group (xi,ci,Σi), where: xi is the coordinates of the Gaussian center (spatial position), ci is the color value (RGB or RGBA), and Σi is the covariance matrix (controlling the shape and size of the Gaussian). When the Gaussian spheres overlap in space, the colors are mixed by weighted summation to form a continuous surface rendering effect.

[0040] S32, Gaussian function fitting: Fit the point cloud data to a Gaussian function, where each Gaussian function represents the position and shape of a 3D point;

[0041] S33, rendering optimization: A rendering optimization algorithm based on a Gaussian function is used to achieve fast 3D rendering. In step S33, the Gaussian parameters (such as position, color, and covariance) are optimized to match the rendering result with the input image (such as a multi-view photo), thereby achieving reconstruction from the two-dimensional image to the three-dimensional scene. In step S33, the parameters are iteratively adjusted using a neural network or a traditional optimization algorithm (such as gradient descent) to minimize the error between the rendered image and the real image.

[0042] S4, rendering and detail enhancement:

[0043] S41, Real-time rendering: Utilizes 3D Gaussian Splatting technology to quickly calculate the color of each pixel without explicitly building a grid. Rendering speed increases linearly with the number of Gaussians.

[0044] S42, Multi-scale Gaussian: Use Gaussian spheres of different sizes to represent scene details (large Gaussian represents macrostructure, small Gaussian depicts texture details) to improve rendering quality;

[0045] S43, Anti-aliasing and blending: Reduce rendering noise and achieve smooth transitions by adjusting Gaussian transparency and overlapping area blending weights;

[0046] S5. Optimization and debugging: Optimize rendering algorithms and data structures to improve rendering speed and efficiency, and conduct comprehensive debugging and testing to ensure the stability and reliability of the virtual space;

[0047] S6. Application and deployment: Save the optimized Gaussian set in a lightweight format (such as a custom binary file), or convert it into a traditional mesh model (such as PLY, OBJ) for use in other rendering engines, and then integrate it into a virtual space platform (such as Unity, Unreal Engine) to implement interactive roaming, virtual display and other functions, so that subsequent users can browse, interact and explore the comic content immersively through different VR headsets.

[0048] The embodiments of the present invention are presented for purposes of illustration and description and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments are chosen and described in order to better illustrate the principles of the invention and its practical application and to enable those skilled in the art to understand the invention and design various embodiments with various modifications as suited for specific applications.

Claims

1. A method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering, characterized in that: The following steps are involved: S1. Data collection: using collection equipment to collect traditional comic content information, and automatically classifying the collected data according to the comic content information; S2. Data preprocessing: The collected data is preprocessed to facilitate subsequent processing and rendering. Feature point matching and structure-from-motion algorithms are then used to estimate the pose of the animated character. A sparse point cloud is generated as the starting point for the initial Gaussian distribution. S3, 3D Gaussian Splatting processing: S31, point cloud generation: convert the pre-processed sparse point cloud into an initial Gaussian sphere. The parameters (such as covariance) can be set to uniform distribution or adjusted according to the point cloud density. S32, Gaussian function fitting: Fit the point cloud data to a Gaussian function, where each Gaussian function represents the position and shape of a 3D point; S33, Rendering Optimization: Achieve fast 3D rendering through Gaussian function rendering optimization algorithm; S4, rendering and detail enhancement: S41, Real-time rendering: Utilizes 3D Gaussian Splatting technology to quickly calculate the color of each pixel without explicitly building a grid. Rendering speed increases linearly with the number of Gaussians. S42, Multi-scale Gaussian: Use Gaussian spheres of different sizes to represent scene details. Large Gaussian represents macrostructure, while small Gaussian depicts texture details, thereby improving rendering quality. S43, Anti-aliasing and blending: Reduce rendering noise and achieve smooth transitions by adjusting Gaussian transparency and overlapping area blending weights; S5. Optimization and debugging: Optimize rendering algorithms and data structures to improve rendering speed and efficiency, and conduct comprehensive debugging and testing to ensure the stability and reliability of the virtual space; S6. Application and Deployment: Save the optimized Gaussian set in a lightweight format, or convert it into a traditional mesh model for use in other rendering engines, and then integrate it into the virtual space platform to implement interactive roaming, virtual display and other functions, so that subsequent users can browse, interact and explore the comic content immersively through different VR headsets.

2. The method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering according to claim 1, characterized in that: In step S2, the original data is preprocessed using a point cloud processing library or an image processing library, including the following steps: Denoising: remove noise from data and improve data quality; Registration: Register data from different sources to ensure data consistency; Segmentation: Segment the data to extract the required features and objects.

3. The method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering according to claim 2, characterized in that: The key features extracted in step S2 include image feature extraction, text feature extraction, and storyboard and narrative structure feature extraction, wherein the image features include dynamic scenes, large urban scenes, human body tracking and other aspects.

4. The method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering according to claim 3, characterized in that: In step S2, after the initial Gaussian is generated, invalid Gaussian samples are automatically eliminated, that is, if the weight of a Gaussian sample is lower than the threshold, or the projected area under the viewpoint is smaller than the pixel, it is automatically marked as "invisible" and the rendering is skipped. At the same time, the screen space error is combined to determine whether the Gaussian needs to be rendered.

5. The method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering according to claim 1, characterized in that: In step S31, each Gaussian sphere is described by a parameter group xi, ci, Σi, where: xi is the coordinate spatial position of the Gaussian center, ci is the color value RGB or RGBA, and Σi is the covariance matrix controlling the shape and size of the Gaussian. When the Gaussian spheres overlap in space, the colors are mixed by weighted summation to form a continuous surface rendering effect.

6. The method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering according to claim 1, characterized in that: In step S33, Gaussian parameters such as position, color, and covariance are optimized to match the rendering result with the input image, thereby achieving reconstruction from the two-dimensional image to the three-dimensional scene.

7. The method for rapidly constructing a virtual space based on 3D Gaussian Splatting rendering according to claim 6, characterized in that: In step S33, a neural network or a traditional optimization algorithm is used to iteratively adjust parameters to minimize the error between the rendered image and the real image.

Images