A three-dimensional immersive content interaction method and system based on 3D Gaussian splash

Abstract

The present invention belongs to the field of three-dimensional scene interaction technology, and in particular relates to a three-dimensional immersive content interaction method and system based on 3D Gaussian splash. The method includes: collecting multi-perspective images and performing motion structure recovery to obtain a camera pose matrix; performing three-dimensional reconstruction processing on the image data through a 3D Gaussian splash algorithm according to the camera pose matrix to obtain a reconstruction model; performing behavioral analysis on the user's historical viewport trajectory to obtain a perspective prediction result; based on the perspective prediction result and the reconstruction result of the reconstruction model, a real-time rendering engine is used to generate video frames on the client to obtain a transmission result. The present invention realizes the application of 3DGS in three-dimensional expression technology, improves rendering quality, and enhances user participation and immersion.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional scene interaction technology, and in particular relates to a three-dimensional immersive content interaction method and system based on 3D Gaussian splashing. Background Art

[0002] With the development of immersive technologies such as virtual reality (VR) and augmented reality (AR), users are placing higher demands on high-quality, highly realistic 3D content. However, traditional 3D reconstruction techniques, such as mesh, point cloud, and voxel-based modeling, are gradually showing limitations in efficiency and quality in current immersive application scenarios.

[0003] The emerging 3D Gaussian Splash (3DGS) technology offers a new approach to immersive content generation. 3DGS overcomes the limitations of traditional geometric modeling by representing a 3D scene as a set of Gaussian distributions with spatial position, orientation, color, and opacity attributes, and rendering them using rasterization.

[0004] Existing immersive content delivery systems, such as edge rendering solutions, attempt to perform the majority of rendering work on edge servers close to the user. Based on user perspective prediction, these systems then stream rendered 2D images or video clips to the terminal. While this approach offers advantages in reducing terminal pressure and conserving bandwidth, it places extremely high demands on the accuracy of user perspective prediction. Any discrepancies in prediction can result in content that doesn't meet user expectations, impacting the immersive experience. Furthermore, most existing delivery algorithms are not optimized for novel representations like 3DGS, hindering their full potential for lightweight and flexible performance. Summary of the Invention

[0005] In view of the above deficiencies in the prior art, the object of the present invention is to provide a three-dimensional immersive content interaction method and system based on 3D Gaussian splashing.

[0006] The present invention provides a three-dimensional immersive content interaction method based on 3D Gaussian splashing, comprising:

[0007] S1: Collect multi-view images and perform motion structure recovery to obtain the camera pose matrix;

[0008] S2: performing three-dimensional reconstruction processing on the image data using a 3D Gaussian splash algorithm according to the camera pose matrix to obtain a reconstructed model;

[0009] S3: Perform behavioral analysis on the user's historical viewport trajectory to obtain the viewpoint prediction result;

[0010] S4: Based on the view prediction result and the reconstruction result of the reconstruction model, a video frame is generated on the client through a real-time rendering engine to obtain a transmission result.

[0011] According to a three-dimensional immersive content interaction method based on 3D Gaussian splashing provided by the present invention, step S1 further includes:

[0012] S11: Shoot the target content from multiple angles to obtain multi-view images;

[0013] S12: extracting feature points from the multi-view image using a feature extraction algorithm to obtain image feature points;

[0014] S13: performing geometric consistency verification using an image pairing algorithm based on the image feature points to obtain a valid image pair;

[0015] S14: Estimating an extrinsic parameter matrix based on the valid image pair to obtain the camera pose matrix.

[0016] According to a three-dimensional immersive content interaction method based on 3D Gaussian splashing provided by the present invention, step S2 further includes:

[0017] S21: Generate an initial representation of the multi-view image according to the camera pose matrix to obtain sparse point cloud data;

[0018] S22: Continuously modeling the sparse point cloud data through Gaussian distribution parameter learning to obtain Gaussian distribution parameters;

[0019] S23: Reconstruct the scene of the multi-view image according to the Gaussian distribution parameters to obtain a reconstruction model and a reconstruction result.

[0020] According to a three-dimensional immersive content interaction method based on 3D Gaussian splashing provided by the present invention, step S3 further includes:

[0021] S31: Extract motion features from the user's historical viewport trajectory to obtain user motion features;

[0022] S32: performing a perspective time series fitting on the user's historical viewport trajectory using a timestamp linear fitting algorithm according to the user's motion characteristics to obtain prediction window parameters;

[0023] S33: Perform future viewing angle inference according to the prediction window parameters to obtain the viewing angle prediction result.

[0024] According to a three-dimensional immersive content interaction method based on 3D Gaussian splashing provided by the present invention, step S4 further includes:

[0025] S41: Calculating rendering parameters of the reconstructed model according to the viewing angle prediction result;

[0026] S42: performing image rendering on the rendering parameters using a 3DGS rendering algorithm to obtain an original image frame;

[0027] S43: Generating depth information through a depth extraction algorithm according to the original image frame to obtain a transmission result including an RGB video frame and depth map data.

[0028] A three-dimensional immersive content interaction method based on 3D Gaussian splashing provided by the present invention further includes:

[0029] S5: performing prediction error correction on the transmission result to obtain a correction result including a viewport prediction error correction result and a rotation error correction result.

[0030] According to a three-dimensional immersive content interaction method based on 3D Gaussian splashing provided by the present invention, step S5 specifically includes:

[0031] Calculating a viewport prediction error, and transforming an image drawing area for displaying a transmission result according to the viewport prediction error to obtain a viewport prediction error correction result;

[0032] According to the depth map data, the rotation error is remapped at the pixel level by a ray tracing remapping algorithm to obtain the rotation error correction result.

[0033] According to a three-dimensional immersive content interaction method based on 3D Gaussian splashing provided by the present invention, step S4 further includes:

[0034] S44: Dynamically adjust the video frame rate and the video bit rate according to the network status of the client receiving the transmission result.

[0035] According to a three-dimensional immersive content interaction method based on 3D Gaussian splashing provided by the present invention, step S44 further includes:

[0036] S441: Quantify the user experience based on the frame quality and interaction delay using a QoE evaluation model to obtain a quality of experience indicator result.

[0037] S442: According to the experience quality indicator result, parameter adjustment processing is performed through a frame-level optimization algorithm to obtain an optimal frame rate and an optimal bit rate.

[0038] The present invention further provides a three-dimensional immersive content interaction system based on 3D Gaussian splashing, which is used to execute any of the three-dimensional immersive content interaction methods based on 3D Gaussian splashing described above, comprising:

[0039] An image acquisition module, a 3D reconstruction module, a real-time rendering module, and a video transmission module are provided on the edge server, and a view prediction module and a frame matching module are provided on the client; wherein the client and the edge server establish a real-time communication connection based on the WebRTC protocol;

[0040] Image acquisition module, used to collect multi-view images and perform motion structure recovery to obtain the camera pose matrix;

[0041] A three-dimensional reconstruction module is used to perform three-dimensional reconstruction processing on the image data using a 3D Gaussian splash algorithm according to the camera pose matrix to obtain a reconstructed model, and store the reconstructed model in a model database;

[0042] The view prediction module is used to perform behavioral analysis on the user's historical viewport trajectory, obtain the view prediction result, and send the view prediction result as a pre-fetch request to the edge server through the SCTP data channel;

[0043] A real-time rendering module, configured to generate video frames through a real-time rendering engine based on the view prediction result received from the client and the reconstruction result of the reconstruction model to obtain a transmission result;

[0044] The video transmission module is used to transmit the transmission result to the client through the WebRTC audio and video channel and the SCTP data channel;

[0045] The frame pairing module is used to receive the transmission results from the edge server and perform frame-level joint processing to obtain display results.

[0046] The beneficial effects of the present invention are as follows:

[0047] This invention provides a method and system for 3D immersive content interaction based on 3D Gaussian splashing. By employing a 3D Gaussian splashing-based 3D immersive content interaction technology, it significantly improves the quality of 3D scene reconstruction and the smoothness of the user interaction experience. Compared to traditional 3D modeling methods, this invention significantly reduces computational complexity while maintaining high-fidelity modeling. This allows 3D scenes constructed from conventional 2D images to possess greater spatial consistency and visual realism, providing users with a more immersive visual experience.

[0048] The high-precision camera pose matrix obtained by motion structure recovery in the present invention lays a solid foundation for subsequent 3D reconstruction, ensures the geometric accuracy of the reconstructed model, and avoids the common spatial distortion and texture dislocation problems in traditional methods; the 3D reconstruction processing based on the 3D Gaussian splash algorithm can effectively retain the high-frequency detail information of the scene, and the generated reconstructed model not only has continuous spatial representation capabilities, but also can support high-quality rendering of any perspective, providing an efficient data foundation for real-time interactive applications; the behavioral analysis mechanism of the user's historical viewport trajectory can accurately predict the user's next perspective change through deep learning of the user's viewing habits and interaction patterns. This prediction capability significantly reduces the waiting time caused by perspective switching, allowing users to experience a more natural and coherent interactive response when browsing 3D content; the collaborative working mechanism of the perspective prediction result and the reconstruction model ensures the rendering resources The reasonable allocation avoids unnecessary computational waste, while ensuring the timely generation and transmission of key perspective content. The deployment strategy of the real-time rendering engine on the client effectively reduces the delay pressure of network transmission. Through localized video frame generation and processing, users can obtain more immediate visual feedback, especially in an environment with unstable network conditions, which significantly enhances the robustness and availability of the system. In addition, by establishing multiple transmission mechanisms, the real-time transmission of data is guaranteed, which avoids the common audio and video asynchrony problems in traditional solutions, and ensures the integrity of spatial information and the smooth playback of visual content. The frame-level joint processing mechanism ensures the precise correspondence between RGB video frames and depth map data through an intelligent data pairing algorithm. The precise correspondence capability provides reliable data support for subsequent 3D rendering and interactive operations, avoiding visual errors and interaction anomalies caused by data mismatch.

[0049] Overall, the entire transmission method and system architecture of the present invention achieves a reasonable allocation of computing resources, network resources and storage resources through multi-level optimization design, which not only improves the user experience of single users, but also enhances the system's concurrent processing capabilities, laying a technical foundation for the deployment and application of large-scale three-dimensional content services, and at the same time provides efficient and reliable underlying technical support for emerging application fields such as virtual reality and augmented reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference numerals represent the same components. Obviously, the drawings described below are only some of the embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings.

[0051] Figure 1 A schematic flow chart of a three-dimensional immersive content interaction method based on 3D Gaussian splashing provided in an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the structure of a three-dimensional immersive content interaction system based on 3D Gaussian splashing provided in an embodiment of the present invention.

[0053] Reference numerals:

[0054] 100, image acquisition module; 200, 3D reconstruction module; 300, viewpoint prediction module; 400, real-time rendering module; 500, video transmission module; 600, frame matching module. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the scope of protection of the present invention.

[0056] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0057] In the description of the present invention, it should be noted that, unless otherwise expressly specified and limited, the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second" and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0058] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with certain aspects of the present invention, as detailed in the appended claims.

[0059] In order to better understand the present invention, the research background of the present invention is first explained in detail below.

[0060] With the rapid development of immersive technologies such as virtual reality (VR) and augmented reality (AR), users are placing higher demands on high-quality, highly realistic 3D content. However, traditional 3D reconstruction technologies—such as mesh, point cloud, and voxel-based modeling—are gradually showing limitations in efficiency and quality in current immersive application scenarios.

[0061] However, these methods often rely on complex geometric modeling, texture reconstruction, and lighting calculation processes, which not only have long production cycles and high development costs, but also require high-precision sensor equipment and specialized operators. Furthermore, explicit modeling methods often generate huge amounts of data, which not only consumes a large amount of terminal rendering resources but also places significant pressure on network bandwidth. In terms of content presentation, such methods often have difficulty accurately restoring details and are prone to problems such as holes, texture aliasing, and geometric artifacts, which seriously affect the user's immersive experience.

[0062] The emerging 3D Gaussian Spray (3DGS) technology has provided a new solution for immersive content generation in recent years. 3DGS overcomes the limitations of traditional geometric modeling by representing a three-dimensional scene as a set of Gaussian distributions with spatial position, orientation, color, and opacity attributes, and rendering them in a rasterized manner. Compared with existing methods, 3DGS offers the following significant advantages: it eliminates the need to construct an explicit mesh topology, offering greater flexibility and scalability; it can efficiently and cost-effectively restore complex geometric details, supporting a highly realistic free-viewpoint roaming experience; it is compatible with existing GPU rendering pipelines and offers high rendering efficiency; and its generation process is concise and can be completed through rapid training, facilitating large-scale content production and application deployment.

[0063] While 3DGS significantly lowers the barrier to entry for immersive content generation and holds broad application prospects, existing systems still face numerous challenges in ensuring real-time interaction and transmission of content. In particular, effectively reducing transmission costs and improving interactive response speed while maintaining high-fidelity rendering quality have become significant bottlenecks hindering the practical implementation of 3DGS. Therefore, a system architecture and approach that integrates content generation, real-time transmission, and user interaction is urgently needed. This approach can fully unleash the potential of new three-dimensional expression technologies like 3DGS, further lower the barrier to entry for immersive experiences, and comprehensively enhance user engagement and immersion.

[0064] Existing immersive content delivery systems are primarily categorized into two types: terminal-rendering and edge-rendering. Terminal-rendering solutions typically transmit 3D content directly to the user's terminal, where it performs real-time rendering. While this approach maximizes user interaction freedom, it places high demands on the client device's graphics processing power. Furthermore, when faced with the random access requirements of multiple objects and angles in complex scenarios (such as virtual exhibitions and e-commerce browsing), bandwidth and latency pressures increase dramatically, making it difficult to adapt to low-performance devices and weak network environments.

[0065] Edge rendering approaches attempt to perform the majority of rendering work on edge servers close to the user. Based on user perspective prediction, the rendered 2D images or video clips are then streamed to the terminal. This approach offers certain advantages in reducing terminal pressure and saving bandwidth, but it requires extremely high accuracy in predicting the user's perspective. Any deviation from the user's perspective can lead to content that doesn't meet user expectations, compromising the immersive experience. Furthermore, most existing transmission algorithms are not optimized for novel representation methods like 3DGS, failing to fully exploit their lightweight and flexible potential.

[0066] Therefore, based on the 3D Gaussian splash method, the present invention constructs an end-to-end "generation-rendering-transmission-interaction" closed-loop architecture, aiming to provide users with a low-latency, high-fidelity immersive experience. The method and system of the present invention consist of two major parts: the edge server and the client, which work together to support high-quality user interaction experience.

[0067] The embodiments of the present invention are described below with reference to the accompanying drawings.

[0068] like Figure 1 As shown, the present invention provides a three-dimensional immersive content interaction method based on 3D Gaussian splashing, comprising:

[0069] S1: Collect multi-view images and perform motion structure recovery to obtain the camera pose matrix.

[0070] In step S1 of the present invention, the target object or scene is first photographed from multiple angles using a mobile terminal or digital camera. To ensure image data consistency and subsequent modeling quality, the following parameters must be kept constant during the photographing process: ISO sensitivity, shutter speed, white balance, focal length, etc. Image acquisition should ensure sufficient perspective overlap between adjacent frames while maintaining constant lighting conditions to avoid noise introduced by motion blur and exposure variations, thereby improving the robustness and accuracy of the reconstruction phase.

[0071] After image acquisition, the present invention uses the COLMAP tool to estimate camera pose. Through feature extraction and image pairing, the Structure-from-Motion (SfM) process is performed to obtain the extrinsic parameter matrix of each image in 3D space. Global pose optimization is then used to further improve pose estimation accuracy, ensure geometric consistency between images, and provide a high-quality input foundation for 3D reconstruction.

[0072] Wherein, step S1 further includes:

[0073] S11: Shoot the target content from multiple angles to obtain multi-view images.

[0074] Furthermore, in step S11 of the present invention, the target content is first photographed from multiple angles, wherein each image contains the color information and spatial projection relationship of the target at a specific viewing angle. In addition, sufficient viewing angle overlap areas need to be maintained between adjacent images. The overlapping areas provide a data basis for subsequent feature matching and geometric constraint establishment, and the shooting trajectory needs to cover the main surface area of ​​the target to form a closed or semi-closed shooting path to ensure that complete spatial geometric information can be obtained during three-dimensional reconstruction.

[0075] S12: extracting feature points from the multi-view image using a feature extraction algorithm to obtain image feature points.

[0076] Specifically, in step S12, the input image is first subjected to multi-scale Gaussian filtering to construct a scale space pyramid. Each scale layer contains the filtering results of the original image under different Gaussian kernel standard deviations. Then, a Gaussian difference image is obtained by calculating the difference images between adjacent scale layers. Extreme points are detected in the Gaussian difference image as candidate feature points. Finally, each candidate point is compared with its neighboring points in space and scale dimensions. A point is determined as a feature point only when its gradient amplitude is greater than or less than all neighboring points.

[0077] S13: Based on the image feature points, geometric consistency verification is performed using an image pairing algorithm to obtain a valid image pair.

[0078] Furthermore, in step S13 of the present invention, first, for each feature point in the image, the distance between it and all feature point descriptors in the other image is calculated, and the two candidate matching points with the smallest distance are selected. When the ratio of the minimum distance to the second smallest distance is less than a preset threshold, the match is considered to be a reliable preliminary match; after the preliminary matching is completed, the present invention uses a random sampling consistency algorithm to perform geometric constraint verification, that is, randomly select multiple pairs of matching points to calculate the basic matrix. The basic matrix describes the geometric relationship of the epipolar lines between the two images. The basic matrix is ​​solved by the least squares method, and the coordinates of the multiple pairs of points are substituted into the constraint equation to form a linear equation group. The optimal solution of the basic matrix is ​​obtained by singular value decomposition. The calculated basic matrix is ​​then used to verify all preliminary matching point pairs, and the distance from each pair of matching points to the corresponding epipolar line is calculated. Matching points with a distance less than the pixel threshold are considered to meet the geometric consistency constraints. The obtained matching points constitute the above-mentioned feature point set of valid image pairs.

[0079] S14: Estimating an extrinsic parameter matrix based on the valid image pair to obtain the camera pose matrix.

[0080] Furthermore, the above-mentioned method for estimating the extrinsic parameter matrix adopts a progressive structure recovery algorithm. First, the image pair with the most valid matching points is selected as the initial baseline, and the relative rotation matrix and translation vector are obtained by decomposing the basic matrix. Subsequently, the three-dimensional coordinates of the matching points in the initial image pair are calculated by a triangulation method, that is, the matching point pairs corresponding to the projection matrices of the two cameras, whose three-dimensional coordinates are obtained by solving two linear equations = 0, and then new images are gradually added. For each new image, its extrinsic parameter matrix relative to the reconstructed three-dimensional points is first calculated by the perspective n-point algorithm. After each new image is added, the algorithm performs a global bundle adjustment optimization. The bundle adjustment simultaneously optimizes the extrinsic parameter matrices and three-dimensional point coordinates of all cameras, and minimizes the sum of the reprojection errors of all observation points. Finally, the objective function is optimized to obtain the extrinsic parameter matrix of the camera.

[0081] S2: According to the camera pose matrix, the image data is subjected to three-dimensional reconstruction processing by a 3D Gaussian splashing algorithm to obtain a reconstructed model.

[0082] In step S2, after obtaining the high-precision camera pose in step S1, the present invention uses a 3D Gaussian splattering method to perform 3D reconstruction of the image data. Using a sparse point cloud as the initial representation, the present invention optimizes the Gaussian distribution representation through parameter learning, combining image color, depth, and spatial position, to achieve continuous modeling of scene surface details.

[0083] 3DGS not only improves modeling efficiency but also preserves high-frequency texture information, resulting in a highly realistic and structurally accurate 3D scene. Finally, the model is stored in the system's model database, accessible to real-time rendering and interaction modules. Through this process, the present invention can efficiently construct high-quality 3D scenes from conventional 2D images, meeting the dual requirements of spatial accuracy and real-time responsiveness in immersive applications.

[0084] Wherein, step S2 further includes:

[0085] S21: Generate an initial representation of the multi-view image according to the camera pose matrix to obtain sparse point cloud data.

[0086] In step S2, the present invention first generates an initial representation of the multi-view image according to the camera pose matrix in step S21 to obtain sparse point cloud data. The sparse point cloud data is a set of discrete points in three-dimensional space obtained by multi-view geometric constraint calculation. Each three-dimensional point contains the X, Y, and Z coordinate values ​​in the world coordinate system and the corresponding RGB color information, which is realized by triangulation calculation.

[0087] S22: Continuously modeling the sparse point cloud data through Gaussian distribution parameter learning to obtain Gaussian distribution parameters.

[0088] Furthermore, step S22 is implemented by a 3D Gaussian splashing algorithm. Specifically, each three-dimensional point in the sparse point cloud is first initialized to a 3D Gaussian kernel, the initial center position is directly set to the world coordinates of the three-dimensional point, the initial covariance matrix is ​​obtained by calculating the distance variance between the point and its neighboring points, the opacity is initialized to 0.1, and the spherical harmonic function coefficients are calculated by fitting the color values ​​of the point in all observed images. The spherical harmonic function uses the first three-order coefficients to represent the relationship between color and viewing angle. Subsequently, the gradient-based optimization algorithm learns the parameter values, that is, the gradient of the loss function with respect to the Gaussian distribution parameters is calculated by backpropagation, and the Adam optimizer is used to update the parameter values, and finally the optimal Gaussian distribution parameters are learned.

[0089] S23: Reconstruct the scene of the multi-view image according to the Gaussian distribution parameters to obtain a reconstruction model and a reconstruction result.

[0090] Furthermore, in step S23, the present invention performs adaptive splitting and merging operations of the Gaussian kernel and constructs a final model, and finally reconstructs the scene of the multi-view image according to the Gaussian distribution parameters to obtain a reconstruction model and a reconstruction result. The reconstruction model is a data structure containing all optimized 3D Gaussian kernel parameters, and the reconstruction result refers to the multi-view image generated by rendering the model.

[0091] Specifically, the adaptive splitting operation is judged based on the gradient amplitude of the Gaussian kernel. When the position gradient of the Gaussian kernel is continuously greater than the threshold in multiple iterations, the Gaussian kernel is split into two new Gaussian kernels along the main direction of the maximum gradient. The merging operation is performed on the Gaussian kernel whose opacity is less than the threshold or the covariance matrix determinant is too small. Specifically, the Mahalanobis distance between adjacent Gaussian kernels is calculated. When the distance is less than the merging threshold, the Gaussian kernel is merged into a single new kernel, and the parameters of the new kernel are calculated by weighted averaging. In the scene reconstruction, a depth-based hybrid rendering method is used to generate the final image. For each pixel in the image, the algorithm performs color mixing on all Gaussian kernels that affect the pixel in depth order. The final data structure of the reconstructed model contains an optimized number of Gaussian kernels, a complete parameter set for each Gaussian kernel, and a spatial index structure for fast rendering.

[0092] S3: Perform behavioral analysis on the user's historical viewport trajectory to obtain the viewpoint prediction results.

[0093] This paper integrates 3D Gaussian splattering technology with the GStreamer video stream processing framework in its edge rendering module, creating a high-precision rendering and efficient transmission mechanism that supports immersive scenes. This module generates corresponding RGB images based on the client's perspective prediction results. Regarding the perspective prediction and prefetching mechanism, the client builds a behavior prediction model based on historical viewport trajectories and user motion characteristics, inferring future perspective change trends and generating prefetch requests based on this information, which are then sent to the edge server.

[0094] Wherein, step S3 further includes:

[0095] S31: Extract motion features from the user's historical viewport trajectory to obtain user motion features.

[0096] Specifically, in step S31, the present invention first obtains the historical operation data of the user in the three-dimensional immersive scene, including the three-dimensional rotation parameters of the user's perspective, the position coordinate change, and the scaling ratio sequence. Subsequently, the original viewport trajectory data is preprocessed, and the high-frequency noise caused by device jitter is removed by filtering. Then, the angular velocity vector, displacement velocity vector and scaling change rate between adjacent time points are calculated. Then, based on the statistical analysis of these velocity parameters, the motion pattern feature vector of the user in different time windows is calculated. The obtained feature vector includes the average angular velocity, angular acceleration change trend, displacement direction consistency index and scaling frequency distribution parameter. The obtained motion feature parameters constitute the input basis of the subsequent prediction algorithm.

[0097] S32: According to the user motion characteristics, a time series fitting of the user's historical viewport trajectory is performed using a timestamp linear fitting algorithm to obtain prediction window parameters.

[0098] Furthermore, when obtaining the prediction window parameters, a time axis coordinate system is first established, and the historical viewport trajectories are arranged in timestamp order to form a multidimensional time series containing rotation angles, position coordinates, and scaling parameters. The linear fitting process uses the least squares method to independently fit the data of each dimension, and the changing trend of the parameters of each dimension over time is determined by calculating the regression coefficient.

[0099] Specifically, for the rotation angle sequence, each component of the Euler angle is used as the dependent variable, and the timestamp is used as the independent variable for linear regression analysis to obtain the slope parameter and intercept parameter of the angle change. For the position coordinate sequence, linear fitting is performed on the three coordinate axes of X, Y, and Z respectively to calculate the direction vector and movement rate of the position change. The scaling parameter is fitted by linear regression after logarithmic transformation to adapt to the exponential characteristics of the scaling operation. After fitting is completed, the algorithm determines the time length of the prediction window based on the correlation coefficient and residual analysis of the fitting results. When the correlation coefficient is higher than the threshold, the prediction window is increased, and when the residual is large, the prediction window is reduced. The final output includes the prediction window parameters including the prediction time length, fitting parameters and confidence interval.

[0100] S33: Perform future viewing angle inference according to the prediction window parameters to obtain the viewing angle prediction result.

[0101] Furthermore, in step S33, the present invention first calculates the predicted value of the viewing angle parameter at the future moment based on the regression equation obtained by linear fitting, adds the current timestamp and the prediction window time length as the target time point, and substitutes it into the regression equation. For rotation prediction, the angle change rate obtained by fitting is multiplied by the prediction time interval, and the current angle value is added to obtain the future Euler angle prediction result, which is then converted into quaternion form to avoid the universal lock problem; position prediction is performed by multiplying the fitted displacement velocity vector by the prediction time interval and adding the current position coordinates to obtain the future position; and scaling prediction is performed by converting the linear prediction result in the logarithmic domain back to the original scaling ratio through an exponential function. The viewing angle prediction result of the present invention also takes into account the randomness of user behavior, adds an uncertainty range based on historical variance to the prediction result, and finally forms a viewing angle prediction result including a central prediction value and a confidence interval.

[0102] S4: Based on the view prediction result and the reconstruction result of the reconstruction model, a video frame is generated on the client through a real-time rendering engine to obtain a transmission result.

[0103] Wherein, step S4 further includes:

[0104] S41: Calculating rendering parameters of the reconstructed model according to the viewing angle prediction result.

[0105] Furthermore, in step S41, the quaternion rotation data, three-dimensional position coordinate data and scaling data from the view prediction result are first received, and the 3D Gaussian distribution parameter set of the corresponding scene is extracted from the reconstruction model database. The quaternion rotation parameters in the view prediction result are then converted into a camera extrinsic parameter matrix. A 3×3 rotation matrix is ​​calculated using the standard conversion formula from quaternion to rotation matrix. A 4×4 camera extrinsic parameter transformation matrix is ​​then constructed in combination with the predicted three-dimensional position coordinates. The camera intrinsic parameter matrix is ​​then constructed based on the resolution parameters and field of view parameters of the client display device. The intrinsic parameter matrix includes key parameters such as horizontal focal length, vertical focal length, and principal point coordinates. The rendering parameters also include frustum clipping parameters. By calculating the visibility cone range under the current view angle, the near clipping plane distance and the far clipping plane distance are determined to form a complete MVP transformation matrix. The calculation module also adjusts the field of view angle parameters according to the scaling ratio. When the scaling ratio increases, the field of view angle is correspondingly reduced to achieve a magnification effect. When the scaling ratio decreases, the field of view angle is expanded to achieve a reduction effect. Finally, a complete rendering parameter set including the camera parameter matrix, cropping parameters and lighting parameters is output.

[0106] S42: performing image rendering on the rendering parameters using a 3DGS rendering algorithm to obtain an original image frame.

[0107] In step S42, the present invention first reads the three-dimensional center coordinates, RGB color value, opacity value and 3×3 covariance matrix parameters of each Gaussian distribution from the reconstructed model, then transforms the three-dimensional center coordinates of all Gaussian distributions into the camera coordinate system through the camera extrinsic parameter matrix, and then uses the camera intrinsic parameter matrix to project the three-dimensional coordinates onto the two-dimensional image plane, and calculates the projection center position of each Gaussian distribution on the image plane; subsequently, the projection shape of each Gaussian distribution on the two-dimensional image plane is calculated, and the two-dimensional covariance matrix is ​​obtained by transforming the three-dimensional covariance matrix through the camera transformation matrix. The two-dimensional covariance matrix describes the elliptical shape and direction of the Gaussian distribution on the image plane; then, the overlapping Gaussian distributions are color fused, and each pixel point is processed in sequence from front to back according to the depth, and the contribution weights of all Gaussian distributions to the pixel point are calculated. The influence intensity of each Gaussian distribution on the pixel is calculated by the Gaussian function, and finally the RGB color value of the pixel is obtained to form a complete original image frame.

[0108] S43: Generating depth information through a depth extraction algorithm according to the original image frame to obtain a transmission result including an RGB video frame and depth map data.

[0109] In step S43, the present invention uses a depth extraction algorithm and, based on the depth buffer information during the 3DGS rendering process, synchronously records the depth value corresponding to each pixel when rendering the pixel. Specifically, a depth buffer data structure is first established. The depth buffer has the same resolution as the image frame, and the corresponding depth value is stored at each pixel position. Subsequently, during the 3DGS rendering process, when calculating the contribution of each Gaussian distribution to the pixel, it is necessary to simultaneously record the Z coordinate value of the center point of the Gaussian distribution in the camera coordinate system as the depth information. In addition, since a single pixel is affected by multiple Gaussian distributions, the depth value calculation requires weighted averaging. The corresponding depth values ​​are weighted summed according to the contribution weight of each Gaussian distribution to the pixel to obtain the final depth value of the pixel.

[0110] After the depth information is generated, the depth value is normalized and the original floating-point depth value is mapped to an integer range of 0-255 to form a standard 8-bit depth map data. At the same time, the original image frame is preprocessed for video encoding, and the RGB image data is converted into a standard video frame format. It is encapsulated together with the depth map data to form a transmission result data packet containing the RGB video frame and depth map data.

[0111] Wherein, step S4 further includes:

[0112] S44: Dynamically adjust the video frame rate and the video bit rate according to the network status of the client receiving the transmission result.

[0113] Wherein, step S44 further includes:

[0114] S441: Based on the frame quality and interaction delay, the user experience is quantified using the QoE evaluation model to obtain the experience quality index result. S442: Based on the experience quality index result, the parameters are adjusted using the frame-level optimization algorithm to obtain the optimal frame rate and optimal bit rate.

[0115] The present invention also introduces a dynamic frame rate and bit rate control algorithm based on user behavior and network status adaptation, aiming to maximize the user's perceived video quality (QoE) while meeting low-latency transmission requirements. The specific implementation is as follows.

[0116] First, parameter initialization and information input are performed. The input parameters of the algorithm include the initial frame rate of each frame , bit rate , the user's current viewing mode (e.g., fast browsing or slow viewing), available network bandwidth , prediction window length, prefetch delay and transmission queuing delay and other key factors, where i is the frame number index value. In the initialization phase, the default frame rate and bit rate parameters are set based on the bandwidth status and user preferences as the starting point for subsequent adaptive adjustment.

[0117] Secondly, QoE modeling and real-time control strategy are carried out. During the processing cycle of each frame i, the system calculates the impact of the current frame on the overall user experience quality (QoE), frame quality Usually it is positively correlated with the bit rate and frame rate of the frame, while the interactive delay It reflects the response delay caused by transmission, queuing, and decoding. To quantify QoE, the present invention adopts the following evaluation model:

[0118]

[0119] in, is the delay penalty weight coefficient, which is used to balance the trade-off between visual quality and interactive response. The parameter settings of the next frame are adjusted according to the dynamic change trend to continuously optimize the user experience while meeting the real-time constraints.

[0120] (3) Overall optimization objective function

[0121] To maximize QoE within the entire session window, the algorithm constructs the following frame-level joint optimization objective function:

[0122]

[0123] in, and It is the coefficient that controls the impact of bit rate and frame rate on QoE. is the number of frames in the transmission queue when the i-th frame arrives, is the total number of frames contained in the transmission queue, where is the number of in-flight frames when frame i arrives at the transmission queue. This objective function takes into account the network bandwidth constraint and playback smoothness requirements while optimizing the overall QoE, thereby achieving efficient and robust video transmission control.

[0124] In order to avoid freeze or empty frames during playback, the present invention adopts the following two protection mechanisms during the control process: Network congestion avoidance: By limiting the bit rate of each frame Does not exceed the current available network bandwidth , to prevent frame loss or delay caused by network congestion; Queue buffer control: through queue constraints, ensure that there is always sufficient frame data in the playback cache to avoid playback interruption due to insufficient prefetching.

[0125] The present invention adopts a frame-by-frame iterative optimization strategy. When processing each frame, the optimal result of the previous frame is used as the initial estimate value, and the frame rate of the current frame is dynamically adjusted in combination with real-time network feedback and user interaction status. and bitrate The iterative process continues until all frame parameters are optimized and finally output to the video encoding and transmission module to ensure the playback continuity and QoE optimization of the entire content.

[0126] Among them, also include:

[0127] S5: performing prediction error correction on the transmission result to obtain a correction result including a viewport prediction error correction result and a rotation error correction result.

[0128] The present invention also introduces a viewport prediction error correction mechanism to process translation and scaling errors and rotation errors respectively to improve the accuracy and consistency of the visual experience.

[0129] Wherein, step S5 specifically includes:

[0130] Calculate the viewport prediction error, and transform the image drawing area used to display the transmission result according to the viewport prediction error to obtain the viewport prediction error correction result; according to the depth map data, perform pixel-level remapping on the rotation error through a ray tracing remapping algorithm to obtain the rotation error correction result.

[0131] To address viewport prediction errors caused by user panning or zooming, the present invention appropriately expands the rendering canvas to improve content coverage, thereby alleviating the image edge loss caused by prediction bias. Specifically, the present invention uses the currently received video frame viewport parameters and the user's real-time viewport state as input, estimates the viewing angle offset, and adjusts the image rendering area accordingly, achieving dynamic expansion or scaling of the canvas.

[0132] set up and are the original image width and the corrected image width, and For the radius and radius error, the corrected image width can be calculated using the following trigonometric relationship:

[0133]

[0134]

[0135] Through the above formula, the expansion ratio can be obtained , the specific expression is:

[0136]

[0137] The obtained compensation ratio guides the rendering module to expand the image boundaries at the canvas level, which can effectively correct the viewport deviation in the translation or scaling direction and ensure the integrity of the visual content.

[0138] As for the prediction error caused by perspective rotation, the present invention further introduces a pixel-level remapping method based on depth map and ray tracing to accurately correct the rotation error through three-dimensional space reconstruction and re-projection.

[0139] First, the present invention calculates the corresponding light starting point and direction for each pixel in the image, assuming that the translation vector of the source camera is , the rotation matrix is , the pixel coordinates are , the principal point coordinates are , the horizontal and vertical focal lengths are and , then the pixel light direction can be expressed as:

[0140]

[0141] in, The coordinates are The pixel light direction of the coordinate point, The world coordinates of each pixel are the transposed matrix of the rotation matrix, based on the light starting point and direction, and the depth map information. It can be calculated by the following formula:

[0142]

[0143] in It's a pixel The corresponding depth value.

[0144] Next, transform the 3D point in the world coordinate system into the target camera coordinate system:

[0145]

[0146] in, is the translation vector of the target camera, is the rotation matrix of the target camera. Finally, the 3D point is projected onto the 2D image plane of the target camera through the camera's intrinsic parameter matrix:

[0147]

[0148] in, is the camera’s intrinsic parameter matrix, is a three-dimensional point in the target camera coordinate system, is the horizontal coordinate of the three-dimensional point in the target camera coordinate system, is the vertical coordinate of the three-dimensional point in the target camera coordinate system, is the depth value in the target camera coordinate system. Through the above steps, the system can accurately correct the rotation error.

[0149] like Figure 2 As shown, the present invention also provides a three-dimensional immersive content interaction system based on 3D Gaussian splashing, comprising:

[0150] An image acquisition module 100, a 3D reconstruction module 200, a real-time rendering module 400, and a video transmission module 500 provided on an edge server, and a viewpoint prediction module 300 and a frame matching module 600 provided on a client; wherein the client and the edge server establish a real-time communication connection based on the WebRTC protocol;

[0151] Image acquisition module 100, used to acquire multi-view images and perform motion structure recovery to obtain camera pose matrix;

[0152] A 3D reconstruction module 200 is configured to perform 3D reconstruction processing on the image data using a 3D Gaussian splashing algorithm according to the camera pose matrix to obtain a reconstructed model, and store the reconstructed model in a model database;

[0153] The view prediction module 300 is used to perform behavioral analysis on the user's historical viewport trajectory, obtain a view prediction result, and send the view prediction result as a pre-fetch request to the edge server through the SCTP data channel;

[0154] A real-time rendering module 400 is configured to generate video frames through a real-time rendering engine based on the view prediction result received from the client and the reconstruction result of the reconstruction model to obtain a transmission result;

[0155] The video transmission module 500 is used to transmit the transmission result to the client through the WebRTC audio and video channel and the SCTP data channel;

[0156] The frame matching module 600 is used to receive the transmission result from the edge server and perform frame-level joint processing to obtain a display result.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0158] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0159] The following describes in detail the actual deployment and operation of the edge server in the present invention in conjunction with another specific embodiment, including key processes such as receiving client requests, dynamic image rendering, video encoding and multi-channel real-time transmission.

[0160] In this embodiment, the edge server is built based on Python 3 and integrates the Socket.IO real-time communication framework, the GStreamer video processing pipeline, the NVIDIA NVENC encoder, and the WebRTC and mediasoup protocol stacks.

[0161] When the server starts:

[0162] Initialize the GStreamer multimedia framework;

[0163] Create a data buffer queue (image frame, view information, encoding parameters) for thread communication;

[0164] Start the WebSocket signaling service (port 6009) to listen for client connections and view control data;

[0165] Load the pre-trained 3DGS 3D model for subsequent perspective rendering.

[0166] When the client sends a view prediction and encoding request through the WebRTC data channel, the edge server processes the data through the following steps:

[0167] Receive the request in JSON format in the controlData() callback function decorated with @sio.event;

[0168] Extract the viewing angle parameters, encoding frame rate (Frame_rate) and bit rate (Bitrate) from each frame request;

[0169] Write the processed triplet (frame_info, frame_rate, bitrate) into view_info_queue for use by subsequent rendering threads.

[0170] The above implementation can ensure the asynchronous reception and sequential buffering of multi-frame requests from the client by the edge server, and lay the foundation for refined encoding control.

[0171] For each view request from the client, the server performs the following real-time rendering and encoding process:

[0172] 3D perspective rendering: Call the gaussian_renderer.render() function to generate the RGB image and depth map under the target perspective based on the camera pose parameters in the requested frame and the specified model.

[0173] Image encoding and queue distribution: Rendered images are fed into the GStreamer pipeline for H.264 encoding. The encoding process is based on NVIDIA hardware acceleration (NVENC), and the encoded H.264 stream is pushed into the h264_stream_queue, awaiting calls from the transport module.

[0174] Multi-channel real-time transmission: RTP channel: Sends the encoded image to the client in the form of RTP packets through the WebRTC protocol to achieve low-latency video transmission; SCTP channel: Synchronously sends the depth map, frame timestamp and metadata through the WebRTC data channel to ensure the consistent arrival of multimodal data.

[0175] The above process effectively combines 3D image modeling and real-time transmission architecture, ensuring low-latency rendering and transmission when the user switches perspectives in 6DoF scenarios.

[0176] The server integrates the ConnectMediasoup class to connect to the Mediasoup real-time media server, including:

[0177] Check if a connection exists (REST API);

[0178] Dynamically create Broadcaster instances;

[0179] Get the audio and video transmission port and SSRC identifier;

[0180] Manage audio and video synchronization and multi-client broadcasting.

[0181] The module supports multiple clients to simultaneously subscribe to edge-generated video streams, providing expansion capabilities for immersive multi-user applications.

[0182] When the server exits abnormally or an interrupt signal (such as SIGINT) is triggered, the cleanup function registered with atexit releases GPU, GStreamer, and thread resources to ensure stable reuse and deployment of the system.

[0183] Table 1 Key functions and parameters of edge servers

[0184] Function name Functional Description parameter connect(sid, environ) Process client connection events and initialize connection status sid (session ID), environ (environment information) controlData(sid, remotedata) Receive client perspective prediction data and encoding parameters, parse and write to the processing queue sid (session ID), remotedata (JSON format control data) disconnect(sid) Handle client disconnection events and release resources sid (session ID) send_frame_info(info) Push frame metadata information to the client via SocketIO info (frame description information) web_server() Start the WebSocket signaling service and listen for client connections and message communications none

[0185] Table 2 ConnectMediasoup class key functions

[0186] Function name Functional Description parameter __init__(self, SERVER_URL, ROOM_ID, BROADCASTER_ID, AUDIO_PT,VIDEO_PT, AUDIO_SSRC, VIDEO_SSRC) Constructor, initializes Mediasoup connection configuration SERVER_URL (server URL), ROOM_ID (room ID), BROADCASTER_ID (broadcaster ID), AUDIO_PT (audio PT), VIDEO_PT (video PT), AUDIO_SSRC (audio SSRC), VIDEO_SSRC (video SSRC) check_room_existence(self) Check whether the specified room exists to ensure that the media routing configuration is valid none create_broadcaster(self) Create and register a broadcast instance to start media streaming none delete_broadcaster(self) Delete the broadcast instance and stop media streaming none ready_for_rtp(self) Verify that the RTP media stream is ready and perform related configurations none

[0187] Table 3 Mediasoup connection configuration and experimental parameters

[0188] category Parameters / configuration items illustrate Connection Information SERVER_URL (server address), ROOM_ID (room ID), BROADCASTER_ID (broadcaster ID) Configure the connection to the Mediasoup server and room resource management. Media Transmission AUDIO_PT (audio PT), VIDEO_PT (video PT) Configure the Payload Type (PT) information for audio and video transmission to ensure correct routing of audio and video data. Media Logo AUDIO_SSRC (audio SSRC), VIDEO_SSRC (video SSRC) Used to identify the synchronization source of audio and video streams to ensure correct matching and synchronization of data streams. RTP Configuration ready_for_rtp(): Checks if the RTP media stream is ready This method checks and ensures that the RTP transport of the media stream is ready to be used for real-time video transmission.

[0189] The following describes the implementation of the client. This implementation describes the key functions of the client, especially the implementation of perspective prediction, frame prefetching and adaptive bit rate control, frame-to-frame information pairing mechanism, video stream management, etc. The core logic and functional modules of the client will be gradually introduced below.

[0190] When the client application starts, the system performs the following initialization operations: configuring viewport parameters and camera parameters (cameraParam) to prepare for subsequent perspective control and rendering.

[0191] The client uses the HistoryList and PredictionList classes to model and predict the user's viewport trajectory:

[0192] Historical Data Management: The HistoryList class is used to store and manage a user's historical viewing angle data, including the yaw, pitch, and radius of the view. Each time the view angle changes, new data is appended to the historical data, ensuring that the system can predict future views based on the user's historical behavior.

[0193] Viewport Prediction: The PredictionList class predicts viewport changes over multiple time steps in the future based on historical data. For example, the system uses historical viewport data (such as yaw, pitch, and radius) to predict future viewport changes and generate corresponding prefetch requests.

[0194] The client manages video encoding and prefetching frames through the FrameScheduler class:

[0195] Update encoding parameters: The updateEncodeParameters(frameRate, bitRate) method adjusts the bitrate and frame rate of the video stream based on the current network bandwidth and frame rate. The client dynamically adjusts the encoding frameRate and bitRate based on network conditions (such as bandwidth and latency) to balance video quality and smoothness.

[0196] Dynamic prefetching: The updatePrefetchFrames(predictionList) method updates the prefetched frames based on the view prediction list. The client calculates and stores the corresponding prefetched frames based on the future view index, ensuring that the images are prefetched and ready for rendering when the client needs them.

[0197] Generate prefetch request: The generateRequest() method packages the prefetched frame information and encoding parameters into a JSON format request and sends it to the server for image rendering and encoding processing.

[0198] During frame pairing and processing, the FramePairingProcessor class is used to ensure that the image frames and depth information received by the client are paired and processed in sequence:

[0199] Frame pairing: The client maintains a mapping between frame information and image frames, receives each video frame through the receiveFrame() method, and waits for all expected frames to arrive through the waitForExpectedPairs() method. When the system receives the expected frame, it updates the pairing status to ensure that each frame is correctly aligned with the depth map information.

[0200] Frame loss handling: If a frame is lost, the system will wait until all expected frames arrive or time out, and use a sliding window mechanism to compensate to ensure the continuity and smoothness of rendering.

[0201] On the client side, the viewport correction module corrects image misalignment caused by perspective prediction errors, including rotation errors and translation / scaling errors. This module is integrated into the frame processing flow and is automatically triggered when the system detects that the prediction error exceeds a threshold.

[0202] Rotation Error Correction: First, the client reads the camera intrinsics and extrinsics for the current and target frames using read_camera_parameters(file_path). These parameters are parsed and used for rotation and translation correction, respectively. Next, get_rays_np is called to calculate the ray direction for each pixel. The reconstruct_world_point function then performs a 3D reconstruction of each pixel, generating a point cloud in the world coordinate system. The 3D points are then projected to the target viewpoint, and reproject_to_image_plane is called to generate aligned image plane coordinates, completing view reprojection.

[0203] Translation and scaling error correction: When there is a significant error in translation or scaling, the system uses the compute_expansion_ratio function to calculate and set a canvas expansion ratio, and dynamically adjusts the rendering canvas size based on the expansion ratio to compensate for the visual area offset caused by translation / scaling.

[0204] The client controls the movement and zoom of the view through timers and user input (such as mouse, keyboard, scroll wheel, etc.):

[0205] View control: Event handling functions such as handleMouseDown(), handleMouseMove(), and handleMouseUp() allow users to change the view by dragging the mouse, and rotate and zoom the view using keyboard shortcuts.

[0206] Control loop: The controlloop() method periodically performs tasks such as view prediction, frame prefetching, and encoding parameter adjustment on the client to ensure synchronization with the server and maintain smooth real-time interaction.

[0207] The present invention provides a three-dimensional immersive content interaction method and system based on 3D Gaussian splash, which relates to the field of immersive multimedia content generation, real-time interaction and transmission. It proposes a three-dimensional immersive content generation, real-time interaction and high-fidelity, low-latency transmission system based on 3D Gaussian splash (3DGS) technology, aiming to solve the key problems of current immersive content in terms of high production cost, low rendering and transmission efficiency, and large interactive response delay, thereby promoting the widespread popularization and application of such content in education, culture and tourism, e-commerce and other fields.

[0208] Based on 3DGS, the present invention constructs a lightweight three-dimensional content generation process suitable for immersive scenes, as well as a joint transmission mechanism of edge rendering and view prediction for 3DGS representation. This solves the problems of traditional three-dimensional content generation processes, which are complex and costly, rely on high-precision equipment and professional modeling knowledge, and are difficult to achieve large-scale production and transmission. It significantly lowers the threshold for three-dimensional content production, simplifies the process, improves efficiency, supports the rapid generation and deployment of high-quality content, and reduces terminal load and bandwidth requirements through the collaboration of edge rendering and view prediction, thus achieving a high-fidelity, low-latency interactive experience.

[0209] The present invention also proposes a viewport prediction error correction mechanism to address the problem of users frequently switching perspectives in 6DoF scenes, which can easily lead to prediction errors, resulting in delayed loading or content errors. It supports rapid content recovery under perspective prediction errors, enhances the continuity and immersion of free-view browsing, and improves system robustness. The present invention also proposes an adaptive frame rate and bit rate transmission algorithm for real-time interactive immersive scene transmission. This solves the problem that existing adaptive bit rate strategies are difficult to cope with real-time interactive immersive scene transmission, affecting transmission stability and a good experience. It improves the real-time and adaptability of content transmission, ensures smooth playback, and reduces network transmission pressure.

[0210] Overall, the present invention achieves deep coupling and closed-loop optimization of the entire process, improves the overall operating efficiency of the system and the consistency of user experience, and promotes the large-scale implementation of immersive content.

[0211] Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents; and such modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be covered by the scope of protection of the present invention.

Claims

1. A three-dimensional immersive content interaction method based on 3D Gaussian splashing, characterized in that: include: S1: Collect multi-view images and perform motion structure recovery to obtain the camera pose matrix; S2: performing three-dimensional reconstruction processing on the image data using a 3D Gaussian splash algorithm according to the camera pose matrix to obtain a reconstructed model; S3: Perform behavioral analysis on the user's historical viewport trajectory to obtain the viewpoint prediction result; S4: generating video frames on the client through a real-time rendering engine according to the view prediction result and the reconstruction result of the reconstruction model to obtain a transmission result; Step S4 further includes: S41: calculating rendering parameters of the reconstructed model according to the view angle prediction result; S42: performing image rendering on the rendering parameters using a 3DGS rendering algorithm to obtain an original image frame; S43: generating depth information using a depth extraction algorithm based on the original image frame to obtain a transmission result including an RGB video frame and depth map data.

2. The three-dimensional immersive content interaction method based on 3D Gaussian splashing according to claim 1, characterized in that: Step S1 further comprises: S11: Shoot the target content from multiple angles to obtain multi-view images; S12: extracting feature points from the multi-view image using a feature extraction algorithm to obtain image feature points; S13: performing geometric consistency verification using an image pairing algorithm based on the image feature points to obtain a valid image pair; S14: Estimating an extrinsic parameter matrix based on the valid image pair to obtain the camera pose matrix.

3. The three-dimensional immersive content interaction method based on 3D Gaussian splashing according to claim 1, characterized in that: Step S2 further comprises: S21: Generate an initial representation of the multi-view image according to the camera pose matrix to obtain sparse point cloud data; S22: Continuously modeling the sparse point cloud data through Gaussian distribution parameter learning to obtain Gaussian distribution parameters; S23: Reconstruct the scene of the multi-view image according to the Gaussian distribution parameters to obtain a reconstruction model and a reconstruction result.

4. The three-dimensional immersive content interaction method based on 3D Gaussian splashing according to claim 1, characterized in that: Step S3 further comprises: S31: Extract motion features from the user's historical viewport trajectory to obtain user motion features; S32: performing a perspective time series fitting on the user's historical viewport trajectory using a timestamp linear fitting algorithm according to the user's motion characteristics to obtain prediction window parameters; S33: Perform future viewing angle inference according to the prediction window parameters to obtain the viewing angle prediction result.

5. The three-dimensional immersive content interaction method based on 3D Gaussian splashing according to claim 1, characterized in that: Also includes: S5: performing prediction error correction on the transmission result to obtain a correction result including a viewport prediction error correction result and a rotation error correction result.

6. The three-dimensional immersive content interaction method based on 3D Gaussian splashing according to claim 5, characterized in that: Step S5 specifically includes: Calculating a viewport prediction error, and transforming an image drawing area for displaying a transmission result according to the viewport prediction error to obtain a viewport prediction error correction result; According to the depth map data, the rotation error is remapped at the pixel level by a ray tracing remapping algorithm to obtain the rotation error correction result.

7. The three-dimensional immersive content interaction method based on 3D Gaussian splashing according to claim 1, characterized in that: Step S4 further includes: S44: Dynamically adjust the video frame rate and the video bit rate according to the network status of the client receiving the transmission result.

8. The three-dimensional immersive content interaction method based on 3D Gaussian splashing according to claim 7, characterized in that: Step S44 further includes: S441: Quantify the user experience based on the frame quality and interaction delay using a QoE evaluation model to obtain a quality of experience indicator result. S442: According to the experience quality indicator result, parameter adjustment processing is performed through a frame-level optimization algorithm to obtain an optimal frame rate and an optimal bit rate.

9. A 3D immersive content interaction system based on 3D Gaussian splashing, configured to execute a 3D immersive content interaction method based on 3D Gaussian splashing according to any one of claims 1 to 8, characterized in that: include: An image acquisition module, a 3D reconstruction module, a 3D reconstruction module, a real-time rendering module, and a video transmission module are provided on the edge server, and a view prediction module and a frame matching module are provided on the client; wherein the client and the edge server establish a real-time communication connection based on the WebRTC protocol; Image acquisition module, used to collect multi-view images and perform motion structure recovery to obtain the camera pose matrix; A three-dimensional reconstruction module is used to perform three-dimensional reconstruction processing on the image data using a 3D Gaussian splash algorithm according to the camera pose matrix to obtain a reconstructed model, and store the reconstructed model in a model database; The view prediction module is used to perform behavioral analysis on the user's historical viewport trajectory, obtain the view prediction result, and send the view prediction result as a pre-fetch request to the edge server through the SCTP data channel; A real-time rendering module, configured to generate video frames through a real-time rendering engine based on the view prediction result received from the client and the reconstruction result of the reconstruction model to obtain a transmission result; The real-time rendering module is specifically used to calculate the rendering parameters of the reconstructed model according to the perspective prediction result; perform image rendering on the rendering parameters using a 3DGS rendering algorithm to obtain an original image frame; and generate depth information based on the original image frame using a depth extraction algorithm to obtain a transmission result including an RGB video frame and depth map data; The video transmission module is used to transmit the transmission results to the client through the WebRTC audio and video channel and the SCTP data channel; the frame matching module is used to receive the transmission results from the edge server and perform frame-level joint processing to obtain the display results.

Images