Scene interaction method, device, storage medium and program product

Abstract

The application discloses a scene interaction method and device, a storage medium and a program product, relates to the technical field of virtual reality, and comprises the following steps: acquiring a target scene model, wherein the target scene model is composed of at least one three-dimensional Gaussian element; generating an agent grid sheet corresponding to the three-dimensional Gaussian element based on a geometric parameter of the three-dimensional Gaussian element; in response to an interaction operation in a virtual environment, determining a first agent grid sheet that moves from the agent grid sheet; updating the geometric parameter of the three-dimensional Gaussian element corresponding to the first agent grid sheet according to a pose parameter of the first agent grid sheet; and rendering the target scene model based on the updated geometric parameter to realize scene interaction in the virtual environment. The application realizes efficient and real-time scene interaction while maintaining the original high rendering quality of the three-dimensional Gaussian sputtering.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to scene interaction methods, scene interaction devices, storage media, and computer program products. Background Technology

[0002] With the development of 3D scene rendering technology, 3D Gaussian sputtering technology has attracted widespread attention due to its ability to render complex scenes with high fidelity and efficiency. This technology uses a large number of 3D Gaussian elements to represent the geometry and appearance of the scene. To enable such high-fidelity scenes to be used in interactive environments such as virtual reality, a physics engine needs to be introduced to achieve real-time interaction between the user and objects in the scene.

[0003] However, traditional physics engines are designed based on the calculation and collision detection of explicit surface mesh models, and cannot be directly applied to scene representations composed of discrete, non-mesh 3D Gaussian elements. When conventional techniques are used to implement interaction, either the computational load is too large to achieve real-time results, or reconstruction distortion reduces rendering quality. Therefore, traditional techniques struggle to achieve real-time interaction while maintaining high rendering quality. Summary of the Invention

[0004] The main purpose of this application is to provide a scene interaction method, scene interaction device, storage medium and computer program product, which aims to solve the technical problem that traditional technologies are difficult to be compatible with high rendering quality and real-time interaction.

[0005] To achieve the above objectives, this application proposes a scene interaction method, which includes:

[0006] Obtain a target scene model, wherein the target scene model is composed of at least one three-dimensional Gaussian element; Based on the geometric parameters of the three-dimensional Gaussian element, a proxy mesh corresponding to the three-dimensional Gaussian element is generated; In response to interactive operations in the virtual environment, the first proxy mesh piece that has moved is determined from the proxy mesh pieces; Based on the pose parameters of the first proxy mesh, update the geometric parameters of the three-dimensional Gaussian elements corresponding to the first proxy mesh; The target scene model is rendered based on the updated geometric parameters to enable scene interaction in the virtual environment.

[0007] In one embodiment, the step of obtaining the target scene model includes: Acquire multi-view 2D images of the target scene; Scene training is performed based on the multi-view two-dimensional images to obtain a two-dimensional Gaussian representation of the target scene, wherein the two-dimensional Gaussian representation contains at least one two-dimensional Gaussian element; The two-dimensional scale parameters of each of the two-dimensional Gaussian elements are extended to three-dimensional scale parameters to obtain the three-dimensional Gaussian representation after the transformation of the two-dimensional Gaussian representation; A target scene model is constructed based on at least one three-dimensional Gaussian element in the three-dimensional Gaussian representation.

[0008] In one embodiment, the geometric parameters include position parameters, rotation parameters, and scale parameters. The step of generating a proxy mesh corresponding to the three-dimensional Gaussian element based on the geometric parameters of the three-dimensional Gaussian element includes: The spatial point corresponding to the position parameters of the three-dimensional Gaussian element is taken as the first vertex; Based on the rotation parameters of the three-dimensional Gaussian element, the first direction and the second direction of the three-dimensional Gaussian element are determined, and based on the first scale value corresponding to the first direction in the scale parameters, the second vertex is obtained by extending from the first vertex along the first direction, wherein the first direction and the second direction are orthogonal to each other; Based on the second scale value corresponding to the second direction in the scale parameters, the third vertex is obtained by extending from the first vertex along the second direction; Based on the first vertex, the second vertex, and the third vertex, a proxy mesh corresponding to the three-dimensional Gaussian element is constructed.

[0009] In one embodiment, the pose parameters include vertex information, edge vectors, and mesh size; the geometric parameters include position parameters, rotation parameters, and scale parameters; and the step of updating the geometric parameters of the 3D Gaussian elements corresponding to the first proxy mesh based on the pose parameters of the first proxy mesh includes: Based on the vertex information of the first proxy mesh, determine the position parameters of the corresponding three-dimensional Gaussian elements; Based on the edge vectors of the first proxy mesh, determine the rotation parameters of the corresponding three-dimensional Gaussian elements; Based on the mesh size of the first proxy mesh, determine the scale parameters of the corresponding three-dimensional Gaussian elements.

[0010] In one embodiment, the vertex information includes an updated first vertex, an updated second vertex, and an updated third vertex, and the step of determining the rotation parameters of the corresponding 3D Gaussian elements based on the edge vectors of the first proxy mesh includes: Obtain the first edge vector and the second edge vector of the first proxy mesh piece, wherein the first edge vector points from the updated first vertex to the updated second vertex, and the second edge vector points from the updated first vertex to the updated third vertex; The first side vector and the second side vector are normalized respectively to obtain the first direction vector and the second direction vector. The first direction vector and the second direction vector are determined as the first direction component and the second direction component in the rotation parameters of the three-dimensional Gaussian element; The cross product of the first direction component and the second direction component is used as the third direction component in the rotation parameters of the three-dimensional Gaussian element. The rotation parameters of the three-dimensional Gaussian element are determined based on the first direction component, the second direction component, and the third direction component.

[0011] In one embodiment, the step of rendering the target scene model based on the updated geometric parameters includes: Based on the updated geometric parameters of each of the three-dimensional Gaussian elements, the target scene model is image synthesized to obtain the synthesized image; The synthesized image is displayed in the virtual environment.

[0012] In one embodiment, the step of determining the first proxy mesh piece that has moved from the proxy mesh piece in response to an interactive operation in the virtual environment includes: Determine the target area of ​​action of the interactive operation in the virtual environment; From the proxy meshes, the proxy mesh that is located within the target area and undergoes physical movement due to the interaction operation is selected as the first proxy mesh.

[0013] Furthermore, to achieve the above objectives, this application also proposes a scene interaction device, which includes: An acquisition module is used to acquire a target scene model, wherein the target scene model is composed of at least one three-dimensional Gaussian element; The generation module is used to generate a proxy mesh corresponding to the three-dimensional Gaussian element based on the geometric parameters of the three-dimensional Gaussian element. A determination module is used to determine, in response to interactive operations in the virtual environment, the first proxy mesh piece that has moved from the proxy mesh pieces; The update module is used to update the geometric parameters of the three-dimensional Gaussian elements corresponding to the first proxy mesh based on the pose parameters of the first proxy mesh. The rendering module is used to render the target scene model based on the updated geometric parameters in order to enable scene interaction in the virtual environment.

[0014] Furthermore, to achieve the above objectives, this application also proposes a scene interaction device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the scene interaction method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the scene interaction method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the scene interaction method described above.

[0017] The one or more technical solutions proposed in this application have at least the following technical effects: By generating a corresponding proxy mesh for each 3D Gaussian element based on its geometric parameters in the target scene model, a binding relationship is established between the proxy mesh and the 3D Gaussian element. This proxy mesh serves as a medium for physical interaction between the physics engine and the target scene, enabling the traditional physics engine to perform calculations based on the proxy mesh, thereby indirectly applying physical interaction to the 3D Gaussian scene. When an interactive operation occurs in the virtual environment, only the first proxy mesh that has moved needs to be determined, and the geometric parameters of the 3D Gaussian element bound to it are updated synchronously according to the updated pose parameters of the first proxy mesh. Thus, without global surface mesh reconstruction of the target scene model or changing the spatial distribution of Gaussian elements, synchronous updates of physical simulation and visual rendering are achieved. The technical solution of this application avoids the huge overhead of direct physical calculations on a massive number of 3D Gaussian elements, and also avoids the loss of Gaussian information and rendering distortion caused by global mesh reconstruction and redistribution, thereby achieving efficient and real-time scene interaction while maintaining the original high rendering quality of 3D Gaussian sputtering. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an embodiment of the interaction method in this application. Figure 1 ; Figure 2 A brief illustration of an embodiment of the scenario interaction method of this application. Figure 2 ; Figure 3 A brief illustration of an embodiment of the scenario interaction method of this application. Figure 3 ; Figure 4 A brief illustration of an embodiment of the scenario interaction method of this application. Figure 4 ; Figure 5 This is a schematic diagram of the module structure of the interactive device in this application. Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the scenario interaction method of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] In conventional techniques, achieving scene interaction based on 3D Gaussian sputtering (3DGS) primarily relies on two methods. One method involves directly performing physical calculations on each of the millions of 3D Gaussian elements in the Gaussian scene. However, due to the sheer number of Gaussian elements in a Gaussian scene, the computational overhead is enormous, making it impossible to meet the real-time interactive frame rates required by applications such as virtual reality. The other method reconstructs the entire Gaussian scene into a unified mesh surface and constrains the Gaussian elements to this reconstructed mesh surface for physical simulation. However, the mesh surface reconstruction process inevitably loses Gaussian element information from non-surface areas of objects (such as interiors and back faces), and the reconstructed mesh surface struggles to accurately replicate the original complex spatial distribution of Gaussian elements, resulting in a significant decrease in visual fidelity and rendering quality. Therefore, traditional techniques struggle to achieve real-time interaction while maintaining high rendering quality.

[0025] In this embodiment, by assigning geometric parameters to each 3D Gaussian element in the target scene model, a corresponding proxy mesh is generated for each 3D Gaussian element to establish a binding relationship between the proxy mesh and the 3D Gaussian element. This proxy mesh serves as a medium for physical interaction between the physics engine and the target scene, enabling the traditional physics engine to perform calculations based on the proxy mesh, thereby indirectly applying physical interaction to the 3D Gaussian scene. When an interaction occurs in the virtual environment, only the first proxy mesh that has moved needs to be determined, and the geometric parameters of the bound 3D Gaussian element are synchronously updated according to the updated pose parameters of the first proxy mesh. Thus, without global surface mesh reconstruction of the target scene model or alteration of the spatial distribution of Gaussian elements, synchronous updates of physical simulation and visual rendering are achieved. The technical solution of this application avoids the enormous overhead of direct physical calculations on a massive number of 3D Gaussian elements, and also avoids the loss of Gaussian information and rendering distortion caused by global mesh reconstruction and redistribution, thereby achieving efficient and real-time scene interaction while maintaining the original high rendering quality of 3D Gaussian sputtering.

[0026] It should be noted that the executing entity of the scene interaction method can be a scene interaction device, such as a VR (Virtual Reality) device, or the main processor or graphics processing unit in the VR device; the executing entity of the scene interaction method can also be other local devices, such as game consoles, graphics workstations, AR (Augmented Reality) devices, etc.; the executing entity of the scene interaction method can also be a virtual device, and this application embodiment does not limit this. For ease of description, the execution entity is omitted in the following description of each embodiment.

[0027] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the scene interaction method of this application. In this embodiment, the scene interaction method includes the following steps: Step S1: Obtain the target scene model, wherein the target scene model is composed of at least one three-dimensional Gaussian element; In one feasible embodiment, the scene interaction device acquires the target scene model.

[0028] Optionally, the target scene model consists of at least one (i.e., one or more) three-dimensional Gaussian elements, where a three-dimensional Gaussian element refers to a basic unit obtained by three-dimensional Gaussian sputtering technology to represent a target scene (such as a Gaussian scene), and each three-dimensional Gaussian element is an ellipsoidal representation with position, rotation and scale properties in three-dimensional space.

[0029] Optionally, a three-dimensional Gaussian element can be called a 3D (Three-Dimensional) Gaussian sphere, and at least one such 3D Gaussian sphere can be combined to form a complete target scene model that can achieve high-fidelity and fast rendering.

[0030] Optionally, the target scene model can be obtained by reconstructing the three-dimensional image from multiple perspectives using techniques such as three-dimensional Gaussian sputtering, or it can be directly loaded from an existing three-dimensional Gaussian model library.

[0031] Step S2: Generate a proxy mesh corresponding to the 3D Gaussian element based on the geometric parameters of the 3D Gaussian element. In one feasible embodiment, the scene interaction device generates a proxy mesh corresponding to each three-dimensional Gaussian element based on the geometric parameters of that element.

[0032] Optionally, the geometric parameters include the position parameters, rotation parameters, and scale parameters of the three-dimensional Gaussian elements in three-dimensional space.

[0033] Optionally, the position parameter can be the center position coordinates of the three-dimensional Gaussian element in three-dimensional space; the rotation parameter can be a three-dimensional rotation matrix describing the orientation of the three-dimensional Gaussian element; and the scale parameter can be the scale value describing the three-dimensional Gaussian element in the three principal axis directions (e.g., x, y, z).

[0034] Optionally, the process of generating a proxy mesh patch can be as follows: using the center coordinates of the three-dimensional Gaussian element as one vertex of the proxy mesh patch, and then extending the corresponding scale value along the two main scale directions (e.g., x and y) determined by the three-dimensional rotation matrix, thereby determining the other two vertices of the proxy mesh patch. Connecting these three vertices forms a triangular planar patch, i.e., the proxy mesh patch. For example, the proxy mesh patch can also be a quadrilateral.

[0035] Optionally, the surrogate mesh is a simplified geometric representation used for physical simulation. of The shape and direction are derived from the bound 3D Gaussian elements, but the proxy mesh itself does not have rendering properties and only serves as a proxy carrier for physical interaction.

[0036] Step S3: In response to interactive operations in the virtual environment, determine the first proxy mesh piece that has moved from the proxy mesh pieces; In one feasible embodiment, when an interactive operation occurs in the virtual environment, such as when a user grabs something with a controller or when objects collide, the scene interaction device responds to this interactive operation by using an integrated physics engine to calculate which proxy meshes have changed state due to the force applied. These proxy meshes that have changed pose parameters due to physics calculations are collectively referred to as the first proxy meshes.

[0037] Optionally, the pose parameters of the moving proxy mesh, such as vertex coordinates and edge vector directions, may change numerically due to the impact of collision reaction forces calculated by the physics engine, continuously applied virtual forces, or constraint solutions. Therefore, these parameters need to be identified and processed for subsequent parameter synchronization.

[0038] Optionally, the interactive forces in the virtual environment are converted into forces or impulses that the physics engine can handle, and applied to the proxy mesh pieces of components with physical properties (such as being given rigid body) and mesh colliders, driving the proxy mesh pieces to perform translation, rotation around an axis or uniform scaling in three-dimensional space, resulting in positional movement, rotation or scale changes of the proxy mesh pieces.

[0039] Optionally, a physics engine is a software component integrated into a virtual reality development platform (such as PhysX in the Unity engine, PhysX in NVIDIA, or Bullet, etc.), which is responsible for simulating the physical laws of the real world (such as gravity, friction, collision detection and response). Proxy meshes can be directly recognized, calculated and have physical effects applied by the physics engine because they have a standard triangular mesh structure.

[0040] Step S4: Update the geometric parameters of the 3D Gaussian elements corresponding to the first proxy mesh according to the pose parameters of the first proxy mesh. In one feasible embodiment, after determining which proxy meshes have moved, the scene interaction device reads the updated pose parameters of the first proxy mesh and uses the updated pose parameters to synchronously update the geometric parameters of the three-dimensional Gaussian elements bound to the first proxy mesh, so as to ensure the consistency between the visual representation and the physical state.

[0041] It is understandable that the pose parameters of the first proxy mesh at this time have changed compared to the pose parameters of the proxy mesh constructed based on the geometric parameters of the three-dimensional Gaussian elements. That is, the pose parameters of the first proxy mesh can be considered as the updated pose parameters.

[0042] Optionally, the pose parameters include the new position of the surrogate mesh in 3D space (such as new vertices), new rotation state (such as new edge vector directions), and new scaling information (such as the mesh size reflected by the changed edge length).

[0043] Optionally, the update process is the inverse operation of the process of generating the proxy mesh. Specifically, from the current coordinates of the three vertices and the direction and length of the two edge vectors of the proxy mesh, the new center position (position parameter), the new spatial orientation (rotation parameter), and the new size (scale parameter) of the three-dimensional Gaussian element bound to it can be deduced in reverse and uniquely.

[0044] Understandably, since the generation of proxy meshes follows a deterministic mathematical mapping from the geometric parameters of their bound 3D Gaussian elements to the coordinates of the mesh vertices, this reverse update process is a mathematically rigorous and lossless data synchronization mechanism, thereby ensuring that the visual representation of the 3D Gaussian elements in subsequent rendering can accurately and without distortion follow the movement of their proxy meshes in the physical simulation.

[0045] Step S5: Render the target scene model based on the updated geometric parameters to enable scene interaction in the virtual environment.

[0046] In one feasible embodiment, the scene interaction device re-renders the target scene model based on the updated geometric parameters of all three-dimensional Gaussian elements, thereby presenting a visual effect in the virtual environment where the scene changes in real time with the interaction, and realizing natural scene interaction for users in the virtual reality environment.

[0047] Optionally, the rendering process uses the updated geometric parameters to calculate the contribution of each 3D Gaussian element to the final image based on the rendering pipeline of 3D Gaussian sputtering.

[0048] Optionally, the target scene model can be rendered using a shader program running on a graphics processor, which efficiently synthesizes high-quality images using the properties of 3D Gaussian elements. Through this process, the scene interaction device enables 3D Gaussian scenes, which originally lacked physical entities and were difficult to interact with directly, to be seamlessly integrated into a physically-simulated virtual reality environment using proxy meshes as an intermediary. This allows users to manipulate and change objects in the scene in a natural way, while maintaining the high rendering fidelity of the original 3D Gaussian scene throughout the process.

[0049] For example, please refer to Figure 2 , Figure 2 A simplified schematic diagram of a scene interaction method is provided. Specifically, the target scene is meshed, that is, the target scene is meshed into a set of meshes composed of basic mesh units; then, the mesh surface is extracted and reconstructed from the mesh set to form a single, continuous mesh surface, and the three-dimensional Gaussian elements (not directly drawn in the figure) that were originally freely distributed in the three-dimensional space are constrained on the reconstructed mesh surface, and scene interaction is then carried out based on this.

[0050] Understandably, not all 3D Gaussian elements in the target scene are distributed on the surface. Directly meshing the target scene would result in the loss of a significant amount of Gaussian information beyond the surface. Since 3D Gaussian elements are ellipsoidal, the meshing algorithm cannot completely recreate the target scene surface. Furthermore, meshes typically contain numerous holes and defects, resulting in uneven surfaces. The spatial positions of the newly generated 3D Gaussian elements are confined to the mesh surface, leading to a final scene rendering quality far inferior to the original target scene.

[0051] For example, please refer to Figure 3 , Figure 3 A simplified schematic diagram of a scene interaction method is provided. Specifically, after the target scene is trained using a two-dimensional Gaussian sputtering technique, it is represented by a two-dimensional Gaussian representation ( Figure 2 The ellipse in the image is then transformed and expanded into a three-dimensional Gaussian representation that includes three-dimensional information. Figure 2 (The ellipsoid in the image). Finally, based on the geometric parameters represented by a three-dimensional Gaussian, a carrier for physical interaction, namely a proxy mesh, is generated.

[0052] It is understandable that the two-dimensional Gaussian representation is converted into a three-dimensional Gaussian representation. Then, the corresponding proxy mesh is calculated based on the geometric parameters of each three-dimensional Gaussian element in the three-dimensional Gaussian representation. The three-dimensional Gaussian element is then bound to the proxy mesh generated by itself. Scene interaction is based on this. This process does not change the parameters of the three-dimensional Gaussian element itself, and the target scene can be perfectly restored during scene interaction.

[0053] In this embodiment, by assigning geometric parameters to each 3D Gaussian element in the target scene model, a corresponding proxy mesh is generated for each 3D Gaussian element to establish a binding relationship between the proxy mesh and the 3D Gaussian element. This proxy mesh serves as a medium for physical interaction between the physics engine and the target scene, enabling the traditional physics engine to perform calculations based on the proxy mesh, thereby indirectly applying physical interaction to the 3D Gaussian scene. When an interaction occurs in the virtual environment, only the first proxy mesh that has moved needs to be determined, and the geometric parameters of the bound 3D Gaussian element are synchronously updated according to the updated pose parameters of the first proxy mesh. Thus, without global surface mesh reconstruction of the target scene model or alteration of the spatial distribution of Gaussian elements, synchronous updates of physical simulation and visual rendering are achieved. The technical solution of this application avoids the enormous overhead of direct physical calculations on a massive number of 3D Gaussian elements, and also avoids the loss of Gaussian information and rendering distortion caused by global mesh reconstruction and redistribution, thereby achieving efficient and real-time scene interaction while maintaining the original high rendering quality of 3D Gaussian sputtering.

[0054] Based on the first embodiment described above, a second embodiment of the scene interaction method of this application is proposed. In this embodiment, step S1, the step of obtaining the target scene model, includes: Step S11: Obtain multi-view two-dimensional images of the target scene; In one feasible embodiment, the scene interaction device acquires multi-view two-dimensional images of the target scene, and the acquisition operation involves image acquisition of a selected target object or environment.

[0055] Optionally, the target scene can be a physical scene that exists in the real world or a computer-generated virtual object.

[0056] Optionally, in order to obtain visual data that can comprehensively describe the appearance of the target scene, it is necessary to take pictures or render from at least one different three-dimensional spatial position and viewing angle surrounding the target scene, so as to obtain a set of two-dimensional digital images that complement each other in terms of perspective, i.e., multi-view two-dimensional images.

[0057] Alternatively, the acquisition of multi-view 2D images can be achieved by setting up a calibrated and time-synchronized camera array in a real scene for real-world shooting, or by using 3D computer graphics software to render an existing digital 3D model using at least one virtual camera for synthesis.

[0058] Step S12: Perform scene training based on multi-view two-dimensional images to obtain a two-dimensional Gaussian representation of the target scene, wherein the two-dimensional Gaussian representation contains at least one two-dimensional Gaussian element. In one feasible embodiment, the scene interaction device performs scene training based on the acquired multi-view two-dimensional images. The purpose is to obtain a digital model that can synthesize a new view consistent with the input image from any viewpoint. The output of this training process is called the two-dimensional Gaussian representation of the target scene, which consists of a massive set of discrete two-dimensional Gaussian elements.

[0059] Optionally, the two-dimensional Gaussian representation of the target scene obtained in this step can be referred to as a two-dimensional scene model.

[0060] Optionally, scene training refers to using reconstruction algorithms, such as optimization frameworks based on 2D Gaussian sputtering technology, to analyze the color correspondence and geometric constraints of pixels between multiple input 2D images, and to continuously optimize and adjust the attribute parameters of each 2D Gaussian element in the 2D scene model in an iterative manner using differentiable rendering technology and gradient descent. The attribute parameters include geometric parameters (such as position, rotation, and scale) and appearance parameters (such as color and opacity).

[0061] Optionally, each two-dimensional Gaussian element obtained from training is a mathematical entity existing in three-dimensional space. The two-dimensional Gaussian element has three-dimensional position coordinates, but its geometry is defined by a two-dimensional scale parameter, and its projection shape in three-dimensional space can be regarded as an ellipse. The two-dimensional Gaussian element has an orientation in three-dimensional space defined by a rotation parameter. It has non-zero scale values ​​in two principal axis directions (corresponding to the major and minor axes of the ellipse, such as the x-axis and y-axis), while the scale value in the third principal axis direction (i.e., the normal direction of the ellipse, such as the z-axis) is zero or infinitely close to zero. This makes each two-dimensional Gaussian element actually appear as a flat (i.e., elliptical) sheet with no thickness in three-dimensional space.

[0062] Optionally, the optimization objective of the entire training process is to minimize the difference between the virtual image generated by rendering with two-dimensional Gaussian elements and the input real multi-view two-dimensional image, until the loss function of the two-dimensional scene model converges to a preset threshold.

[0063] Step S13: Expand the two-dimensional scale parameters of each two-dimensional Gaussian element into three-dimensional scale parameters to obtain the three-dimensional Gaussian representation after the two-dimensional Gaussian representation is transformed. In one feasible embodiment, the scene interaction device extends the two-dimensional scale parameter of each two-dimensional Gaussian element in the two-dimensional Gaussian representation to a three-dimensional scale parameter. Specifically, for each two-dimensional Gaussian element that originally had a scale of zero or undefined in the third dimension, a defined, non-zero third-dimensional component is added to its two-dimensional scale parameter.

[0064] Optionally, extending the two-dimensional scale parameter to a three-dimensional scale parameter means adding a very small scale value corresponding to the third principal axis direction to the two-dimensional scale parameter while keeping the original position parameter, rotation parameter, and scale values ​​of the two principal directions (such as the major axis and minor axis directions of the ellipse) of each two-dimensional Gaussian element unchanged.

[0065] Understandably, after the expansion operation, each two-dimensional Gaussian element has a three-dimensional position, a three-dimensional rotation, and a scale value in three dimensions. It can be considered that the mathematical form of the two-dimensional Gaussian element is consistent with that of the three-dimensional Gaussian element. All the sets that have completed the transformation from two-dimensional Gaussian elements to three-dimensional Gaussian elements constitute a new mathematical model of the target scene, namely, the three-dimensional Gaussian representation.

[0066] Optionally, the added three-dimensional scale parameter is a preset minimal normal number (e.g., 1e-8). The selection criteria are: to ensure that the value is greater than zero so that it can be recognized by the physics engine as a mathematical entity with a non-zero volume in subsequent physical simulation calculations, thus enabling it to participate in collision detection and mechanical simulation; and to ensure that the value is small enough so that the newly added thickness dimension will not cause changes in the appearance of the object that are perceptible to the human eye during the visual rendering stage, thereby maintaining a high degree of consistency in the visual performance of the scene before and after the conversion and achieving visual lossless conversion.

[0067] Step S14: Construct the target scene model based on at least one three-dimensional Gaussian element in the three-dimensional Gaussian representation.

[0068] In one feasible embodiment, the scene interaction device uses a three-dimensional Gaussian representation as a basis and constructs the final target scene model by organizing and integrating the three-dimensional Gaussian elements contained in the three-dimensional Gaussian representation.

[0069] Optionally, the process of constructing a target scene model based on each three-dimensional Gaussian element in the three-dimensional Gaussian representation refers to encoding, encapsulating and storing the three-dimensional Gaussian elements with three-dimensional geometric attributes (position, rotation, scale) and appearance attributes (such as color, transparency) according to their spatial relationships and logical connections, as a unified, structured data object or standardized file.

[0070] Optionally, the target scene model defines the shape, appearance, and spatial distribution of the target scene in three-dimensional space.

[0071] Optionally, the constructed target scene model can be a 3D scene representation based on 3D Gaussian elements, enabling real-time, high-quality scene rendering in the future.

[0072] In this embodiment, a tiny third-dimensional scale parameter is introduced into the high-quality rendered 2D Gaussian representation, transforming it into a 3D Gaussian representation composed of extremely flattened ellipsoids, and this is used to construct the target scene model. This transformation process is visually lossless because the added third-dimensional scale value is extremely small, producing almost no perceptible thickness change during rendering. This allows the target scene model to fully inherit the dual advantages of 2D Gaussian sputtering technology in terms of rendering speed and image quality, enabling high-fidelity reproduction of the scene's complex appearance and fine details from any viewpoint. More importantly, the computational cost of this transformation is extremely low; the operation only involves adding a preset constant to each Gaussian element, far less complex than the scene mesh reconstruction or physical property recalculation processes in traditional methods.

[0073] Based on any of the above embodiments, a third embodiment of the scene interaction method of this application is proposed. In this embodiment, the geometric parameters include position parameters, rotation parameters, and scale parameters. Step S2, the step of generating a proxy mesh corresponding to the three-dimensional Gaussian element based on the geometric parameters of the three-dimensional Gaussian element, includes: Step S21: Take the spatial point corresponding to the position parameter of the three-dimensional Gaussian element as the first vertex; In one feasible embodiment, the scene interaction device generates a corresponding proxy mesh based on the geometric parameters of the three-dimensional Gaussian elements.

[0074] Optionally, the geometric parameters include position parameters describing the precise coordinates of the three-dimensional Gaussian element in three-dimensional space, rotation parameters defining the orientation of the three-dimensional Gaussian element in three-dimensional space, and scale parameters characterizing the size of the three-dimensional Gaussian element in three mutually perpendicular principal axis directions.

[0075] Optionally, the starting point for generating the proxy mesh is that the scene interaction device takes the three-dimensional spatial point corresponding to the position parameter of the three-dimensional Gaussian element as the first vertex of the proxy mesh to be constructed, which is called the first vertex.

[0076] Optionally, the first vertex mathematically coincides with the center point of the three-dimensional Gaussian ellipsoid, thus determining the reference position of the proxy mesh patch in three-dimensional space.

[0077] Step S22: Based on the rotation parameters of the three-dimensional Gaussian element, determine the first direction and the second direction of the three-dimensional Gaussian element, and based on the first scale value corresponding to the first direction in the scale parameters, extend from the first vertex along the first direction to obtain the second vertex, wherein the first direction and the second direction are orthogonal to each other; In one feasible embodiment, the scene interaction device determines the main extension direction of the three-dimensional Gaussian element in space based on the rotation parameters of the three-dimensional Gaussian element, specifically a first direction and a second direction, wherein the first direction and the second direction are orthogonal to each other.

[0078] Optionally, the rotation parameters exist in the form of a three-dimensional rotation matrix.

[0079] Optionally, based on the specific value in the scale parameter corresponding to the first direction, i.e. the first scale value, the scene interaction device starts from the first vertex and extends along the first direction in three-dimensional space for a distance that is exactly equal to the first scale value, thereby obtaining the second spatial point of the proxy mesh patch, i.e. the second vertex.

[0080] Alternatively, this extension process can be accomplished by vector addition, which adds the coordinates of the first vertex to the product of the unit vector in the first direction and the first scale value.

[0081] Optionally, the relationship between the first direction and the second direction in three-dimensional space can be that they are perpendicular to each other, and together they define the spatial plane (such as the xy plane) represented by the three-dimensional Gaussian element.

[0082] Step S23: Based on the second scale value corresponding to the second direction in the scale parameters, extend from the first vertex along the second direction to obtain the third vertex; In one feasible embodiment, the scene interaction device, based on another specific value in the scale parameter corresponding to the second direction, namely the second scale value, starts from the first vertex and extends along the second direction in three-dimensional space for a distance exactly equal to the second scale value, thereby obtaining the third spatial point of the proxy mesh patch, namely the third vertex.

[0083] Optionally, the second and third vertices obtained in this way represent the spatial extension boundaries of the three-dimensional Gaussian element in two directions, respectively.

[0084] Step S24: Based on the first vertex, the second vertex, and the third vertex, construct a proxy mesh corresponding to the three-dimensional Gaussian elements.

[0085] In one feasible embodiment, the scene interaction device constructs the final proxy mesh using the determined first vertex, second vertex, and third vertex.

[0086] Optionally, the specific construction method can be: connecting the three vertices in sequence to form a triangular plane in space, which is a proxy mesh that is bound one-to-one with the three-dimensional Gaussian element.

[0087] Optionally, the geometry, size, and spatial orientation of the proxy mesh are derived from the geometric parameters of the bound 3D Gaussian element, thus ensuring a strict spatial correlation between the two.

[0088] Understandably, the proxy mesh will serve as the direct target for subsequent physics engine simulation operations such as collision detection and force calculation.

[0089] In this embodiment, by mapping the geometric parameters of 3D Gaussian elements to the vertices and edges of proxy meshes, a simple and geometrically corresponding proxy mesh is generated for each 3D Gaussian element, achieving high computational efficiency and geometric fidelity. Specifically, since the process of generating proxy meshes involves only basic vector operations, the computational overhead is extremely low, meeting the stringent real-time requirements of virtual reality interaction. Furthermore, because the shape, size, and spatial orientation of the proxy mesh are entirely determined by the 3D Gaussian element it is bound to, a strict and one-to-one spatial relationship is established between them. This not only ensures that the physics engine can directly apply to the standard, lightweight proxy mesh for efficient collision detection and dynamic simulation, but also avoids the geometric information loss or distortion problems caused by global coarse meshing of complex scenes in traditional methods, ensuring lossless synchronization between the physical state of the proxy mesh and the visual state of the 3D Gaussian element. Therefore, the method described in this embodiment achieves a high-fidelity Gaussian scene in virtual reality that combines natural physical interaction with real-time rendering performance.

[0090] Based on any of the above embodiments, a fourth embodiment of the scene interaction method of this application is proposed. In this embodiment, the pose parameters include vertex information, edge vectors, and mesh size, and the geometric parameters include position parameters, rotation parameters, and scale parameters. Step S4, the step of updating the geometric parameters of the three-dimensional Gaussian elements corresponding to the first proxy mesh according to the pose parameters of the first proxy mesh, includes: Step D1: Based on the vertex information of the first proxy mesh patch, determine the position parameters of the corresponding 3D Gaussian elements; In one feasible embodiment, after the scene interaction device detects the first proxy mesh piece that has moved due to the interaction operation in the virtual environment, it synchronously updates the geometric parameters of the three-dimensional Gaussian elements bound to it according to the updated pose parameters of the first proxy mesh piece.

[0091] Optionally, the pose parameters specifically include information describing the shape and position of the proxy mesh, namely, the vertex information formed by the coordinates of its three vertices in three-dimensional space, the edge vector represented by the direction and length of the edges defined by the connection of the vertices in three-dimensional space, and the mesh size that directly reflects the overall size of the proxy mesh.

[0092] Optionally, the scene interaction device determines the new position parameters of the corresponding 3D Gaussian element based on the vertex coordinate information of the first proxy mesh. Specifically, the first proxy mesh is a triangular face previously generated from the 3D Gaussian element, and the coordinates of its first vertex (i.e., the first vertex) are the center point of the 3D Gaussian element during generation. When the proxy mesh moves under the action of the physics engine, the spatial coordinates of its vertices change. The scene interaction device reads the new 3D coordinates of the current first vertex and uses them as the center point of the bound 3D Gaussian element to update the position parameters of the 3D Gaussian element.

[0093] Optionally, vertex information is a set of data containing three three-dimensional coordinates that reflects the precise position of the proxy mesh in space. Position updates ensure that the motion of the three-dimensional Gaussian element in visual space is consistent with the motion of its physical proxy mesh.

[0094] Step D2: Determine the rotation parameters of the corresponding 3D Gaussian elements based on the edge vectors of the first proxy mesh. In one feasible embodiment, the scene interaction device determines the new rotation parameters of the corresponding three-dimensional Gaussian elements based on the edge vectors of the first proxy mesh.

[0095] Optionally, an edge vector refers to a directed line segment formed by connecting the vertices of the proxy mesh, including a vector pointing from the first vertex to the second vertex and a vector pointing from the first vertex to the third vertex. The vector defines the two principal axis directions of the plane in which the proxy mesh is located.

[0096] Optionally, the scene interaction device extracts the edge vectors from the geometric data of the proxy mesh to update the rotation parameters of the three-dimensional Gaussian elements.

[0097] Optionally, the rotation parameter defines the orientation of the 3D Gaussian element in space. Updating the rotation parameter based on the edge vector ensures that the visual orientation of the 3D Gaussian element changes synchronously with the planar orientation of its physical proxy mesh.

[0098] Step D3: Determine the scale parameters of the corresponding 3D Gaussian elements based on the mesh size of the first proxy mesh patch.

[0099] In one feasible embodiment, the scene interaction device determines the new scale parameters of the corresponding three-dimensional Gaussian elements based on the mesh size of the first proxy mesh patch.

[0100] Optionally, the mesh size refers to the length of the proxy mesh, that is, the distance from the first vertex to the second vertex and the distance from the first vertex to the third vertex. These two distances reflect the size of the proxy mesh in two dimensions, respectively.

[0101] Optionally, the scene interaction device can obtain the distance value by calculating the magnitude of the current edge vector, and use these two values ​​as the scale values ​​of the first two dimensions in the three-dimensional Gaussian element scale parameters, that is, the new dimensions in the X-axis and Y-axis directions of its coordinate system.

[0102] For the third dimension scale value of a three-dimensional Gaussian element, since the proxy mesh is a two-dimensional planar piece, it cannot directly provide the third dimension size information. The scene interaction device can use a preset, extremely small fixed constant as its third dimension scale value, or derive the small change amount by the scaling ratio implicit in its area or normal vector direction when generating the proxy mesh.

[0103] In this way, the scene interaction device maps the mesh size to the new scale values ​​of the three-dimensional Gaussian elements on the three principal axes, thereby completing the update of the scale parameters of the three-dimensional Gaussian elements.

[0104] Optionally, the scale parameter determines the size of the 3D Gaussian element during rendering. Updating the scale parameter according to the mesh size ensures that the visual size change of the object matches the deformation or scaling of its physical proxy, so that all geometric parameters of the 3D Gaussian element are accurately updated in reverse according to the motion state of its bound proxy mesh.

[0105] In one feasible implementation, the vertex information includes the updated first vertex, the updated second vertex, and the updated third vertex. Step D2, the step of determining the rotation parameters of the corresponding 3D Gaussian elements based on the edge vectors of the first proxy mesh patch, includes: Step D21: Obtain the first edge vector and the second edge vector of the first proxy mesh piece, wherein the first edge vector points from the updated first vertex to the updated second vertex, and the second edge vector points from the updated first vertex to the updated third vertex. In one feasible embodiment, when the scene interaction device updates the rotation parameters of the three-dimensional Gaussian elements based on the first proxy mesh piece that has moved, it needs to update the first vertex, the second vertex, and the third vertex contained in the vertex information.

[0106] Optionally, the updated first vertex, the updated second vertex, and the updated third vertex are because the pose of the proxy mesh has changed, thus the coordinates of the vertices have changed; that is, the vertices at this time are the updated vertices.

[0107] Optionally, to obtain the rotation parameters used to define the new orientation of the 3D Gaussian elements, the scene interaction device obtains the edge vectors from the new vertex coordinates. Specifically, the scene interaction device calculates the difference between the 3D coordinates of the new second vertex and the 3D coordinates of the new first vertex to obtain the directed line segment from the new first vertex to the new second vertex, i.e., the first edge vector. Simultaneously, by calculating the difference between the 3D coordinates of the new third vertex and the 3D coordinates of the new first vertex, the device obtains the directed line segment from the new first vertex to the new third vertex, i.e., the second edge vector.

[0108] Optionally, the edge vector is a three-dimensional vector with direction and length, representing the spatial orientation and geometric dimensions of two edges of the proxy mesh.

[0109] Step D22: Normalize the first side vector and the second side vector respectively to obtain the first direction vector and the second direction vector; In one feasible embodiment, the scene interaction device normalizes the acquired first side vector and second side vector respectively, in order to retain the direction information of the vector and eliminate the influence of its length.

[0110] Optionally, normalization means dividing each component of a non-zero vector by the magnitude of that vector to obtain a unit vector with a magnitude of 1.

[0111] Understandably, after normalization, the scene interaction device obtains two unit vectors, namely the first direction vector and the second direction vector.

[0112] Optionally, normalization ensures that the rotation matrices constructed subsequently are orthogonal.

[0113] Step D23: Determine the first direction vector and the second direction vector as the first direction component and the second direction component in the rotation parameters of the three-dimensional Gaussian element; In one feasible embodiment, the scene interaction device uses the normalized first direction vector as the first direction component in the rotation parameters of the three-dimensional Gaussian element to be updated, and uses the second direction vector as the second direction component in the rotation parameters.

[0114] Optionally, the rotation parameter can be a three-dimensional rotation matrix, whose first three columns (or first three rows) consist of three mutually orthogonal unit direction vectors. The first and second direction components correspond to the new orientations of the two principal axes (e.g., the X-axis and Y-axis) in the three-dimensional Gaussian element coordinate system in three-dimensional space.

[0115] Optionally, the direction of the edge vector is used as a component of the rotation parameter because the edges of the proxy mesh are generated along the principal axis of the 3D Gaussian element. This inverse mapping relationship ensures the consistency of parameter updates.

[0116] Step D24: The cross product of the first direction component and the second direction component is used as the third direction component in the rotation parameters of the three-dimensional Gaussian element. In one feasible embodiment, since three-dimensional rotation requires three mutually orthogonal direction components for definition, the scene interaction device needs to calculate the third direction component. To this end, the scene interaction device performs a vector cross product operation on the already determined first and second direction components.

[0117] Optionally, the cross product is a vector operation in three-dimensional space, the result of which is a new vector that is perpendicular to the other two vectors. The scene interaction device uses the calculated new vector as the third direction component in the three-dimensional Gaussian element rotation parameters, which corresponds to the direction of the third principal axis (e.g., the Z-axis) in its coordinate system.

[0118] Optionally, the cross product operation ensures that the third directional component is perpendicular to the first and second directional components, thus forming an orthogonal basis describing the rotation in three-dimensional space.

[0119] Step D25: Determine the rotation parameters of the three-dimensional Gaussian element based on the first direction component, the second direction component, and the third direction component.

[0120] In one feasible embodiment, the scene interaction device determines the rotation parameters of the final three-dimensional Gaussian element based on the first direction component, the second direction component, and the third direction component obtained through cross product operation.

[0121] Specifically, the scene interaction device combines three mutually orthogonal unit direction vectors in sequence to form a third-order orthogonal matrix, which is the new rotation parameter of the three-dimensional Gaussian elements in space.

[0122] Alternatively, the rotation parameters can also be converted from this orthogonal matrix to other equivalent representations, such as quaternions, for use in different graphics engines or computation modules.

[0123] Understandably, the rotation parameters of the 3D Gaussian element are precisely derived and updated based on the geometry of its bound, moved proxy mesh.

[0124] In this embodiment, a physics-driven vision parameter synchronization scheme is achieved by inversely and precisely mapping the pose parameters of the first proxy mesh back to the geometric parameters of its bound 3D Gaussian element. This parameter synchronization process is reversible and lossless. Specifically, when the proxy mesh is translated, rotated, or scaled under the action of the physics engine, its updated vertex coordinates, edge vector directions, and lengths, etc., pose information, can be uniquely and accurately restored to the new position, new rotation, and new scale parameters of the corresponding 3D Gaussian element through a defined mathematical inverse operation (e.g., directly mapping the first vertex coordinates to the center position, normalizing and cross-productting the edge vectors to reconstruct the rotation matrix, and inferring the scale value from the edge length). This one-to-one inverse derivation ensures that the state of the visual model and the state of the physical simulation remain equivalent, thereby realizing the information transfer from physical interaction to visual rendering.

[0125] Therefore, the method in this embodiment enables a high-fidelity Gaussian scene to become an interactive entity that responds to real physical laws without sacrificing visual detail. User-applied interactive forces such as grabbing, throwing, or colliding are calculated and processed by the physics engine via proxy meshes, and their effects are accurately reflected in the real-time visual representation of the Gaussian scene through this solution. This solves the problem of the disconnect between physical simulation and high-quality rendering in traditional methods, providing a feasible way to achieve immersive, highly realistic virtual reality scene interaction.

[0126] Based on any of the above embodiments, a fifth embodiment of the scene interaction method of this application is proposed. In this embodiment, step S5, the step of rendering the target scene model based on the updated geometric parameters, includes: Step S51: Based on the updated geometric parameters of each 3D Gaussian element, perform image synthesis on the target scene model to obtain the synthesized image; In one feasible embodiment, after updating the geometric parameters of all interactive 3D Gaussian elements in the target scene model, the scene interaction device performs a step of rendering the target scene model based on the updated geometric parameters.

[0127] Optionally, the rendering process requires image compositing. That is, the scene interaction device uses the graphics rendering pipeline to project and fuse the set of Gaussian elements in three-dimensional space onto a two-dimensional image plane based on the current virtual camera's viewpoint, projection matrix, and environmental parameters such as lighting, combined with the updated geometric parameters (including position parameters, rotation parameters, and scale parameters) of each three-dimensional Gaussian element and its stored appearance attributes (such as color and opacity). This is done using a three-dimensional Gaussian sputtering algorithm, thereby generating a digital image that reflects the current state of the target scene model, i.e., the composite image.

[0128] Optionally, the 3D Gaussian sputtering algorithm is a real-time rendering technique based on point cloud or ellipsoid rasterization. It achieves high-quality and high-efficiency image synthesis by calculating the coverage and contribution of each 3D Gaussian element in the screen space.

[0129] Step S52: Display the synthesized image in a virtual environment.

[0130] In one feasible embodiment, the scene interaction device outputs the synthesized image to the display system of the virtual reality environment for display.

[0131] Specifically, the scene interaction device sends the synthesized image data to the display hardware connected to the scene interaction device, such as the screen of a virtual reality head-mounted display, and refreshes the display content according to the set refresh rate (such as 90Hz or 120Hz), so that the user can see the visual changes of the target scene model under the influence of interactive operations in real time.

[0132] Optionally, the virtual environment is a computer-generated, interactive three-dimensional digital space whose display system can include two independent display channels to support stereoscopic vision and ensure that users have an immersive visual experience.

[0133] Understandably, through this continuous rendering and display cycle, users can see the visual effects of scene objects moving and deforming caused by their interactive operations such as grabbing and colliding with the controller in the virtual environment, thus completing a high-quality and real-time virtual reality interaction process.

[0134] In one feasible implementation, step S3, the step of determining the first proxy mesh piece that has moved from the proxy mesh piece in response to an interactive operation in the virtual environment, includes: Step S31: Determine the target area of ​​action of the interactive operation in the virtual environment; In one feasible embodiment, when the scene interaction device performs the step of responding to interactive operations in the virtual environment to determine the agent mesh piece that is moving, it needs to locate the range affected by the interactive operation in the virtual environment, that is, determine the target area of ​​the interactive operation in the virtual environment.

[0135] Optionally, the target area of ​​action refers to the spatial range in the virtual environment determined by the specific type and parameters of this interaction operation. For example, when a user grabs with a controller, the target area of ​​action may be a spherical range centered on the contact point between the virtual hand model and the object; when two virtual objects collide, the target area of ​​action is the collision contact surface and its adjacent surrounding space.

[0136] Optionally, the scene interaction device calculates the boundary of the target's area of ​​action in the global three-dimensional coordinate system by parsing the data stream of the interaction operation, such as the pose data of the handle and the intersection information of the collision bodies returned by the collision detection algorithm.

[0137] Optionally, the interactive operation can be direct handle selection and dragging, or collision and bounce between objects automatically triggered by the physics engine.

[0138] Step S32: From the proxy meshes, select the proxy mesh that is located within the target area and undergoes physical movement due to the interaction operation as the first proxy mesh.

[0139] In one feasible embodiment, after the target area of ​​action is determined, the scene interaction device identifies the proxy mesh patch within the target area of ​​action that has actually undergone a physical state change due to the current interaction operation.

[0140] Specifically, the scene interaction device queries the simulation results of the physics engine, selects proxy meshes that fall within the target area and have undergone physical motion (such as position movement, rotation or scaling changes) based on the calculations of the physics engine (such as being subjected to collision force, continuous thrust or torque).

[0141] Optionally, the selected proxy mesh that has undergone physical movement is the first proxy mesh that needs to be synchronized with parameters.

[0142] Optionally, the physics engine continuously calculates the motion state of the proxy mesh. The scene interaction device compares the pose parameters (such as changes in vertex coordinates) of the proxy mesh before and after the physics simulation to determine whether the proxy mesh has moved and the degree of movement, thereby identifying the first proxy mesh.

[0143] In this embodiment, by determining the target area of ​​the interactive operation and selecting proxy meshes within that area that move due to physical actions, the 3D Gaussian elements affected by the current interactive operation can be located efficiently and accurately. This avoids the global computational overhead of traversing all 3D Gaussian elements in the Gaussian scene in traditional methods, significantly improving response speed and resource utilization efficiency. Subsequently, the pose changes of the proxy meshes driven by the physics engine are synchronized back to their bound 3D Gaussian elements through a reversible and lossless parameter mapping relationship, thereby achieving physical and visual consistency. Finally, a high-quality image is synthesized in real time based on all 3D Gaussian elements that have completed state updates and output to the display device, providing smooth and coherent visual feedback.

[0144] Through the above process, this embodiment achieves natural, real-time, and distortion-free scene interaction of a high-fidelity 3D Gaussian scene in a virtual reality environment. This method effectively meets the stringent requirements of virtual reality applications for high real-time performance and low latency while ensuring an immersive visual experience without distortion, thus providing an interactive experience that combines realistic physical feedback and visual presentation.

[0145] For example, please refer to Figure 4 , Figure 4 A simplified schematic diagram of a scene interaction method is provided. Specifically, a Gaussian scene model with a two-dimensional Gaussian representation obtained from scene training is obtained. A minimum value is added to each two-dimensional Gaussian element in the two-dimensional Gaussian representation as a third-dimensional scale, which is then converted into a target scene model with a three-dimensional Gaussian representation to form a scene representation that can be used for interaction.

[0146] For a 3D Gaussian element represented by a 3D Gaussian, a corresponding surrogate mesh is generated based on the geometric parameters of the 3D Gaussian element. Specifically, taking the center point of the 3D Gaussian element as the first vertex, the corresponding scale distance is extended along the two scale directions of the original 2D Gaussian element to obtain the second and third vertices, constructing a surrogate mesh, and binding the surrogate mesh to the corresponding 3D Gaussian element.

[0147] In a virtual environment, when a user interacts with the scene, the physics engine acts on proxy meshes, causing them to shift, rotate, or deform. At this time, the geometric parameters of the bound 3D Gaussian elements are updated in real time based on the pose parameters of the moving proxy mesh (i.e., the first proxy mesh), ensuring that the state of the 3D Gaussian elements is synchronized with their bound proxy meshes.

[0148] Based on the updated geometric parameters of the 3D Gaussian elements, a new 3D Gaussian representation is obtained. Using the new 3D Gaussian representation, the target scene is re-rendered, enabling users to interact with Gaussian objects composed of multiple 3D Gaussian elements in a virtual environment in real time without distortion, while maintaining the high rendering quality of the original target scene.

[0149] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the interaction method of this application. Any simple changes based on this technical concept, such as the interaction and combination of various embodiments, are all within the protection scope of this application.

[0150] This application also provides a scene interaction device, please refer to... Figure 5 The scene interaction device includes: The acquisition module 10 is used to acquire a target scene model, wherein the target scene model is composed of at least one three-dimensional Gaussian element; The generation module 20 is used to generate a proxy mesh patch corresponding to the three-dimensional Gaussian element based on the geometric parameters of the three-dimensional Gaussian element. The determination module 30 is used to determine the first proxy mesh piece that has moved from the proxy mesh pieces in response to interactive operations in the virtual environment; The update module 40 is used to update the geometric parameters of the three-dimensional Gaussian elements corresponding to the first proxy mesh based on the pose parameters of the first proxy mesh. The rendering module 50 is used to render the target scene model based on the updated geometric parameters in order to enable scene interaction in the virtual environment.

[0151] The scene interaction device provided in this application, employing the scene interaction method in the above embodiments, can solve the technical problem that traditional technologies struggle to achieve compatibility with high rendering quality and real-time interaction. Compared with the prior art, the beneficial effects of the scene interaction device provided in this application are the same as those of the scene interaction method provided in the above embodiments, and other technical features in the scene interaction device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0152] This application provides a scene interaction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the scene interaction method in the first embodiment described above.

[0153] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the scene interaction device in the embodiments of this application. The scene interaction device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The scene interaction device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0154] like Figure 6As shown, the scene interaction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the scene interaction device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the scene interaction device to communicate wirelessly or wiredly with other devices to exchange data. Although scene interaction devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0155] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0156] The scene interaction device provided in this application, employing the scene interaction method in the above embodiments, can solve the technical problem that traditional technologies struggle to achieve compatibility with high rendering quality and real-time interaction. Compared with the prior art, the beneficial effects of the scene interaction device provided in this application are the same as those of the scene interaction method provided in the above embodiments, and other technical features of this scene interaction device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0157] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0158] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0159] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the scene interaction method in the above embodiments.

[0160] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0161] The aforementioned computer-readable storage medium may be included in the scene interaction device; or it may exist independently and not be assembled into the scene interaction device.

[0162] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0164] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0165] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described scene interaction method, thereby solving the technical problem that traditional technologies struggle to achieve compatibility with high rendering quality and real-time interaction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the scene interaction method provided in the above embodiments, and will not be elaborated upon here.

[0166] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the scene interaction method described above.

[0167] The computer program product provided in this application can solve the technical problem that traditional technologies are difficult to integrate with high rendering quality and real-time interaction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the scene interaction method provided in the above embodiments, and will not be repeated here.

[0168] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A scene interaction method, characterized in that, The scene interaction method includes: Obtain a target scene model, wherein the target scene model is composed of at least one three-dimensional Gaussian element; Based on the geometric parameters of the three-dimensional Gaussian element, a proxy mesh corresponding to the three-dimensional Gaussian element is generated; In response to interactive operations in the virtual environment, the first proxy mesh piece that has moved is determined from the proxy mesh pieces; Based on the pose parameters of the first proxy mesh, update the geometric parameters of the three-dimensional Gaussian elements corresponding to the first proxy mesh; The target scene model is rendered based on the updated geometric parameters to enable scene interaction in the virtual environment.

2. The method as described in claim 1, characterized in that, The steps for obtaining the target scene model include: Acquire multi-view 2D images of the target scene; Scene training is performed based on the multi-view two-dimensional images to obtain a two-dimensional Gaussian representation of the target scene, wherein the two-dimensional Gaussian representation contains at least one two-dimensional Gaussian element; The two-dimensional scale parameter of the two-dimensional Gaussian element is extended to a three-dimensional scale parameter to obtain the three-dimensional Gaussian representation after the two-dimensional Gaussian representation is transformed. The target scene model is constructed based on at least one three-dimensional Gaussian element in the three-dimensional Gaussian representation.

3. The method as described in claim 1, characterized in that, The geometric parameters include position parameters, rotation parameters, and scale parameters. The step of generating a proxy mesh corresponding to the three-dimensional Gaussian element based on the geometric parameters of the three-dimensional Gaussian element includes: The spatial point corresponding to the position parameters of the three-dimensional Gaussian element is taken as the first vertex; Based on the rotation parameters of the three-dimensional Gaussian element, the first direction and the second direction of the three-dimensional Gaussian element are determined, and based on the first scale value corresponding to the first direction in the scale parameters, the second vertex is obtained by extending from the first vertex along the first direction, wherein the first direction and the second direction are orthogonal to each other; Based on the second scale value corresponding to the second direction in the scale parameters, the third vertex is obtained by extending from the first vertex along the second direction; Based on the first vertex, the second vertex, and the third vertex, a proxy mesh corresponding to the three-dimensional Gaussian element is constructed.

4. The method as described in claim 1, characterized in that, The pose parameters include vertex information, edge vectors, and mesh size; the geometric parameters include position parameters, rotation parameters, and scale parameters; and the step of updating the geometric parameters of the 3D Gaussian elements corresponding to the first proxy mesh based on the pose parameters of the first proxy mesh includes: Based on the vertex information of the first proxy mesh, determine the position parameters of the corresponding three-dimensional Gaussian elements; Based on the edge vectors of the first proxy mesh, determine the rotation parameters of the corresponding three-dimensional Gaussian elements; Based on the mesh size of the first proxy mesh, determine the scale parameters of the corresponding three-dimensional Gaussian elements.

5. The method as described in claim 4, characterized in that, The vertex information includes the updated first vertex, the updated second vertex, and the updated third vertex. The step of determining the rotation parameters of the corresponding 3D Gaussian elements based on the edge vectors of the first proxy mesh includes: Obtain the first edge vector and the second edge vector of the first proxy mesh piece, wherein the first edge vector points from the updated first vertex to the updated second vertex, and the second edge vector points from the updated first vertex to the updated third vertex; The first side vector and the second side vector are normalized respectively to obtain the first direction vector and the second direction vector. The first direction vector and the second direction vector are determined as the first direction component and the second direction component in the rotation parameters of the three-dimensional Gaussian element; The cross product of the first direction component and the second direction component is used as the third direction component in the rotation parameters of the three-dimensional Gaussian element. The rotation parameters of the three-dimensional Gaussian element are determined based on the first direction component, the second direction component, and the third direction component.

6. The method as described in claim 1, characterized in that, The step of rendering the target scene model based on the updated geometric parameters includes: Based on the updated geometric parameters of each of the three-dimensional Gaussian elements, the target scene model is image synthesized to obtain the synthesized image; The synthesized image is displayed in the virtual environment.

7. The method as described in claim 1, characterized in that, The step of determining the first proxy mesh piece that has moved from the proxy mesh piece in response to an interactive operation in the virtual environment includes: Determine the target area of ​​action of the interactive operation in the virtual environment; From the proxy meshes, the proxy mesh that is located within the target area and undergoes physical movement due to the interaction operation is selected as the first proxy mesh.

8. A scene interaction device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the scene interaction method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the scene interaction method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the scene interaction method as described in any one of claims 1 to 7.

Images