A particle-based multi-media interactive simulation method for real scenes

Abstract

The present invention discloses a particle-based real-scene multi-media interactive simulation method, which belongs to the field of computer graphics and physical simulation technology and includes the following steps: S1, collecting RGB images of a target scene; S2, obtaining a three-dimensional scene represented by particles; S3, inputting the collected RGB images into a video tracking model DEVA to obtain a multi-view mask with IDs; S4, rendering the original three-dimensional scene represented by particles into a surface depth map using an extended Gaussian renderer; S5, back-projecting the 2D IDs onto particles using the 2D ID information and depth information obtained in S3 and S4, and assigning an ID to each particle; S6, assigning physical materials to all objects in the scene, and customizing specific material properties according to the properties of the objects; S7, performing multi-media physical simulation of the real scene. The present invention can meet the needs of physical simulation in most scenarios and has high versatility and strong ease of use.

Description

Technical Field

[0001] The present invention relates to the technical field of computer graphics and physical simulation, and in particular to a particle-based real-scene multi-media interactive simulation method. Background Art

[0002] In recent years, with the development of industries such as virtual reality (VR), augmented reality (AR), and film and television special effects, the demand for high-precision, realistic simulations of physical phenomena has increased. Traditional fluid simulation methods primarily rely on gridded numerical solutions, such as the finite difference method and the finite element method, which simulate fluid behavior by solving the Navier-Stokes equations. However, these methods have significant limitations when dealing with complex boundary conditions and interactions between different substances. In particular, they are inefficient in real-time applications and cannot meet high-performance requirements.

[0003] Currently, existing methods for simulating multi-media interactions between particles in real-world scenarios still have the following technical problems:

[0004] (1) The speed and accuracy of three-dimensional reconstruction are insufficient, and the requirements for original data are high, which limits the practical value of existing methods.

[0005] (2) When performing object-level segmentation, existing methods often introduce large models to obtain features and then retrain the scene to inject semantic information into the scene. This method will bring about a huge additional time cost.

[0006] (3) Existing segmentation methods often fail to distinguish the entire object from the scene, resulting in noisy segmented objects or incomplete segmentation, which hinders the visual effects of subsequent physical simulations.

[0007] (4) Existing dynamic scene reconstruction methods often lack physical priors and cannot realistically simulate scenes where objects of different materials interact with each other. Summary of the Invention

[0008] The purpose of the present invention is to provide a particle-based real-scene multi-media interaction simulation method that supports fast real-time reconstruction and rich multi-media interaction, thereby covering the needs of physical simulation in most scenarios and having high versatility and strong ease of use.

[0009] To achieve the above object, the present invention provides a particle-based real-scene multi-media interactive simulation method, comprising the following steps:

[0010] S1. Collect the RGB image of the target scene and obtain the internal and external parameter information of the RGB image;

[0011] S2, using 3D Gaussian splashing to quickly reconstruct the scene in three dimensions to obtain a three-dimensional scene represented by particles;

[0012] S3. Input the collected RGB image into the video tracking model DEVA to obtain a multi-view mask with an ID, so that the pixels of the same object have the same ID under different viewing angles;

[0013] S4, rendering the original particle-represented 3D scene into a surface depth map through an extended Gaussian renderer;

[0014] S5, through the 2D ID information and depth information obtained in S3 and S4, the 2D ID is back-projected onto the particles, an ID is assigned to each particle, and the object-level segmentation task of the 3D scene is completed to obtain a Gaussian scene with an ID;

[0015] S6. Assign physical materials to all objects in the scene, and customize the specific material properties according to the nature of the objects;

[0016] S7. Conduct multi-media physical simulation of real scenes.

[0017] Preferably, in S1, the collected RGB image is subjected to colmap processing.

[0018] Preferably, in S4, the method of extending the Gaussian renderer is: given a mask, for any pixel p on it, convert it to homogeneous coordinates P, and obtain the ray in the camera coordinate system through the camera intrinsic parameter K and the camera extrinsic parameter E:

[0019]

[0020] Where C represents the position of the camera in the world coordinate system, and d represents the depth;

[0021] As the depth d increases, the transmittance T decreases sharply when the light hits the surface of the object. When it falls below the set transmittance threshold, it means that the object surface corresponding to the pixel has been found and the depth is recorded.

[0022] Preferably, in S6, the physical material is one of soft body, rigid body, and fluid.

[0023] Preferably, in S7, the scene is physically simulated using a particle-based physical simulation method mpm.

[0024] Therefore, the beneficial effects of the present invention using the above-mentioned particle-based real-scene multi-media interactive simulation method are as follows:

[0025] (1) The method of the present invention has low hardware requirements. During the data acquisition phase, only RGB image data collected by the mobile phone is required. During the semantic segmentation phase, only simple back-projection is required without retraining. In addition, the present invention supports fast real-time reconstruction and rich multi-media interaction, thus covering the needs of physical simulation in most scenarios, and has high versatility and strong ease of use.

[0026] (2) Existing methods for 3D reconstruction + semantic segmentation usually perform semantic segmentation on 2D images, so they can only generate various image training sets and cannot edit or even physically simulate scenes. The present invention expands the application scope of segmentation and elevates it to 3D space, thereby obtaining the ability to select particles and assign physical properties for physical simulation.

[0027] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of a particle-based real-scene multi-media interactive simulation method of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0030] Unless otherwise defined, the technical or scientific terms used in the present invention shall have the usual meanings understood by persons of ordinary skill in the field to which the present invention belongs. The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0031] Example 1

[0032] The present invention provides a particle-based method for simulating real-world multi-media interactions. To reduce data requirements and improve the accuracy and speed of three-dimensional reconstruction, the present invention introduces 3D Gaussian splattering, replacing the implicit representation previously represented by NeRF with an explicit representation in the form of particles. To accurately select scene objects, the present invention uses a video tracking model to obtain the classification of objects in the scene, and uses back-projection to upgrade the 2D classification to 3D, obtaining the classification of the 3D scene, thereby completing the segmentation task. Finally, we assign physical properties to different objects and introduce MPM as a particle-based physical simulation method to complete the multi-media interactive physical simulation.

[0033] like Figure 1 As shown, the specific steps include:

[0034] S1. Use a mobile phone or camera to collect RGB images of the target scene, perform colmap processing on the collected RGB images, and obtain the internal and external parameter information of the RGB images.

[0035] S2. Use 3D Gaussian splashing to quickly reconstruct the scene in three dimensions to obtain a three-dimensional scene represented by particles.

[0036] S3. Input the collected RGB image into the video tracking model DEVA to obtain a multi-view mask with an ID. That is, each RGB image is converted into a grayscale image, and each pixel on it represents an ID, that is, the ID assigned to the object at the pixel position. Under different viewing angles, the pixels where the same object is located have the same ID.

[0037] S4. Through the extended Gaussian renderer, the original particle representation of the three-dimensional scene is rendered into a surface depth map, that is, the depth of the first object encountered by the camera light under the current camera perspective. This will help the subsequent back projection and supplement the missing depth information during the 2D to 3D conversion.

[0038] The method of extending the Gaussian renderer is: given a mask, for any pixel p on it, convert it to homogeneous coordinates P, and obtain the ray in the camera coordinate system through the camera intrinsic parameter K and the camera extrinsic parameter E:

[0039]

[0040] Where C represents the position of the camera in the world coordinate system, and d represents the depth;

[0041] As the depth d increases, the transmittance T decreases sharply when the light hits the surface of the object. When it falls below the set transmittance threshold, it means that the object surface corresponding to the pixel has been found and the depth is recorded.

[0042] S5. Through the 2D ID information and depth information obtained in S3 and S4, the 2D ID is back-projected onto the particles, and an ID is assigned to each particle, thus completing the object-level segmentation task of the three-dimensional scene and obtaining a Gaussian scene with an ID.

[0043] S6. Assign physical materials to all objects in the scene. The physical material can be one of soft body, rigid body, or fluid (sand, water, snow, etc.). At the same time, customize the specific material properties according to the nature of the object.

[0044] S7. Perform physical simulation of the scene using the particle-based physical simulation method mpm to complete the particle-based real-scene multi-media physical simulation.

[0045] In this embodiment, rapid, high-precision 3D reconstruction is achieved using only multi-view images captured by a mobile phone. 3D object-level semantic segmentation is then performed on the reconstructed scene, and finally, different materials are assigned to different objects, completing a realistic multi-media physics simulation. This process avoids the use of expensive time and hardware methods. For example, industrial cameras, infrared cameras, and remote sensing data are not used in the 3D reconstruction phase, and the semantic segmentation phase does not require retraining of the scene. Instead, geometric back-projection is used to infuse semantic information into the scene.

[0046] This embodiment uses 3D Gaussian splattering to represent the scene in the form of particles. After completing data collection, high-precision three-dimensional reconstruction can be completed in a relatively short time.

[0047] The 3D reconstruction method used in this embodiment is data-driven and does not require manual modeling. In addition, reconstruction can be completed by simply taking RGB images at different viewing angles. Even ordinary mobile phone cameras can be used to collect data without the need for complex equipment such as stereo cameras and infrared cameras, which provides a basic guarantee for the high versatility of the present invention. Unlike the previous 3D reconstruction method based on NeRF, which often takes dozens of hours to complete, the 3D reconstruction method used in the present invention can converge within tens of minutes, reflecting the potential for real-time reconstruction and real-time simulation.

[0048] After completing the 3D reconstruction, the particles in the scene are classified by objects to facilitate the subsequent allocation of physical materials and physical simulation. All objects in the scene are represented by Gaussian particles, which only contain appearance attributes such as color, position, transparency, etc., and are not classified. This embodiment adds an ID attribute to each particle to ensure that particles belonging to the same object have the same ID, thereby facilitating the assignment of the same physical properties to particles of the same type of object. After completing the object-level segmentation, different physical materials can be assigned to the physical particles in the scene, including rigid bodies, soft bodies, and fluids, to complete a multi-media interactive simulation with realistic visual effects.

[0049] Therefore, the present invention adopts the above-mentioned particle-based real-scene multi-media interaction simulation method, which supports fast real-time reconstruction and rich multi-media interaction, and can therefore cover the needs of physical simulation in most scenarios, with high versatility and strong ease of use.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A particle-based real-scene multi-media interactive simulation method, characterized by: The following steps are involved: S1. Collect the RGB image of the target scene and obtain the internal and external parameter information of the RGB image; S2, using 3D Gaussian splashing to quickly reconstruct the scene in three dimensions to obtain a three-dimensional scene represented by particles; S3. Input the collected RGB image into the video tracking model DEVA to obtain a multi-view mask with an ID, so that the pixels of the same object have the same ID under different viewing angles; S4, rendering the original particle-represented 3D scene into a surface depth map through an extended Gaussian renderer; S5, through the 2Did information and depth information obtained in S3 and S4, the 2Did is back-projected onto the particles, and an id is assigned to each particle, thus completing the object-level segmentation task of the three-dimensional scene and obtaining a Gaussian scene with an id; S6. Assign physical materials to all objects in the scene, and customize the specific material properties according to the nature of the objects; S7. Conduct multi-media physical simulation of real scenes.

2. The particle-based real-scene multi-media interactive simulation method according to claim 1, characterized in that: In S1, the collected RGB image is subjected to colmap processing.

3. The particle-based real-scene multi-media interactive simulation method according to claim 1, characterized in that: In S4, the method of extending the Gaussian renderer is: given a mask, for any pixel p on it, convert it to homogeneous coordinates P, and obtain the ray in the camera coordinate system through the camera intrinsic parameter K and the camera extrinsic parameter E: Where C represents the position of the camera in the world coordinate system, and d represents the depth; As the depth d increases, the transmittance T decreases sharply when the light hits the surface of the object. When it falls below the set transmittance threshold, it means that the object surface corresponding to the pixel has been found and the depth is recorded.

4. The particle-based real-scene multi-media interactive simulation method according to claim 1, characterized in that: In S6, the physical material is one of soft body, rigid body and fluid.

5. The particle-based real-scene multi-media interactive simulation method according to claim 1, characterized in that: In S7, the scene is physically simulated using the particle-based physics simulation method mpm.

Images