Information processing system and information processing method

Abstract

To improve the accuracy of reproducing the behavior of thin films in 3D Gaussian Splatting. [Solution] The information processing system includes a processing circuit that simulates the behavior of a thin film using a group of Gaussian particles that form a thin film representation based on 3D Gaussian splatting, specifically a first particle which constitutes the thin film, a second particle which is placed at each vertex of a mesh representing the surface of the thin film, and a third particle which is placed within each polygon of the mesh. This technology can be applied, for example, to a system that reproduces a 3D space.

Description

[Technical Field]

[0001] The present technology relates to an information processing system and an information processing method, and particularly relates to an information processing system and an information processing method suitable for use when reproducing the behavior of thin films in 3D Gaussian Splatting (3DGS). [Background Art]

[0002] 3DGS can reproduce a static 3D space through learning using a plurality of images from different viewpoints. Further, technologies have been proposed that apply the Material Point Method (MPM) to 3DGS to reproduce various physical behaviors (see, for example, Non-Patent Document 1). [Prior Art Documents] [Non-Patent Documents]

[0003] [Non-Patent Document 1] Tianyi Xie, six others, "PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics", 2024, IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, although the invention described in Non-Patent Document 1 supports reproducing the behavior of elastic bodies, sand and the like, it does not support reproducing the behavior of thin films such as cloth.

[0005] The present technology has been made in view of such circumstances, and is intended to improve the accuracy of reproducing the behavior of thin films in 3DGS. [Means for Solving the Problem]

[0006] An information processing system according to one aspect of the present technology includes a processing circuit that simulates the behavior of a thin film using first particles that are particles constituting the thin film among a Gaussian particle group forming a thin film representation based on 3D Gaussian Splatting, second particles that are particles disposed at each vertex of a mesh representing a surface of the thin film, and third particles that are particles disposed within each polygon of the mesh.

[0007] In an information processing method according to one aspect of the present technology, an information processing system simulates the behavior of a thin film using first particles that are particles constituting the thin film among a Gaussian particle group forming a thin film representation based on 3D Gaussian Splatting, second particles that are particles disposed at each vertex of a mesh representing a surface of the thin film, and third particles that are particles disposed within each polygon of the mesh.

[0008] A program according to one aspect of the present technology causes a computer to execute a process of simulating the behavior of a thin film using first particles that are particles constituting the thin film among a Gaussian particle group forming a thin film representation based on 3D Gaussian Splatting, second particles that are particles disposed at each vertex of a mesh representing a surface of the thin film, and third particles that are particles disposed within each polygon of the mesh.

[0009] In one aspect of the present technology, the behavior of the thin film is simulated using first particles that are particles constituting the thin film among a Gaussian particle group forming a thin film representation based on 3D Gaussian Splatting, second particles that are particles disposed at each vertex of a mesh representing a surface of the thin film, and third particles that are particles disposed within each polygon of the mesh. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] [Figure 1] It is a figure for explaining 3DGS. [Figure 2] It is a block diagram showing a configuration example of an information processing system to which the present technology is applied. [Figure 3] This is a flowchart to explain the thin-film behavior simulation process. [Figure 4] This diagram illustrates the extraction of thin film segments, the extraction of thin film particles, and the generation of a surface mesh. [Figure 5] This figure shows examples of Type 1 to Type 3 particles. [Figure 6] This figure shows examples of Type 1 to Type 3 particles and computational grids. [Figure 7] This is a flowchart illustrating the details of the thin-film behavior calculation process. [Figure 8] This is a diagram illustrating the thin film behavior calculation process. [Figure 9] This is a diagram illustrating the projection method of physical quantities between particles and lattice points. [Figure 10] This is a diagram illustrating how to update the Gaussian kernel. [Figure 11] This is a diagram illustrating the deformation method of Type 1 particles. [Figure 12] This figure shows an example of a GUI. [Figure 13] This is a diagram illustrating variations of the computational grid. [Figure 14] This is a diagram showing an example of a computer configuration. [Modes for carrying out the invention]

[0011] The following describes the configurations for implementing this technology. The explanation will proceed in the following order. 0. Overview of this technology 1. Embodiment 2. Variations 3. Others

[0012] <<0. Overview of this technology>> First, we will explain the overview of this technology with reference to Figure 1.

[0013] 3DGS enables high-precision and high-speed rendering by representing three-dimensional space with Gaussian particles, which are particles with a Gaussian distribution. Each Gaussian particle is an elliptical particle and is represented by parameters including position (coordinates), rotation, scale, color, and opacity.

[0014] The color of a Gaussian particle is represented by spherical harmonics. The opacity of a Gaussian particle follows a Gaussian distribution, meaning that the particle becomes more transparent as it moves away from the center. Therefore, the color of a Gaussian particle becomes lighter as it moves away from the center.

[0015] When rendering a three-dimensional space represented by 3DGS (hereinafter referred to as 3DGS space) into a two-dimensional image, the color of each pixel in the image is represented by the overlapping of Gaussian particles.

[0016] For example, as shown in Figure 1, the color of each pixel is set to the color of virtual light that passes through Gaussian particles 11-1 to 11-3 and is projected onto a virtual two-dimensional screen 13 by a virtual lens 12. In this case, the further a Gaussian particle is from the screen 13, the lower its contribution to the color of each pixel.

[0017] Hereafter, Gaussian particles based on 3DGS will also be referred to as 3DGS particles.

[0018] 3DGS is limited to the static reconstruction of three-dimensional space based on multiple images from multiple viewpoints.

[0019] In response to this, 3DGS is required to support a wider range of physical behaviors. As mentioned above, Non-Patent Document 1 supports the reproduction of the behavior of elastic materials and sand, but does not support the reproduction of the behavior of thin films such as cloth.

[0020] For example, among the 3DGS particle group that makes up a thin film, the 3DGS particles that constitute the thin film are scattered near the surface and lack regularity. Therefore, it is not possible to separate the normal contact response and in-plane deformation, which are necessary for reproducing the behavior of the thin film.

[0021] Surface meshes are commonly used to separate in-plane deformations. However, it is difficult to construct a thin film surface mesh and a group of 3DGS particles placed on that surface mesh without additional training.

[0022] In contrast, this technology improves the accuracy of reproducing the behavior of thin films in 3DGS.

[0023] <<1. Embodiment>> Next, embodiments of this technology will be described with reference to Figures 2 to 12.

[0024] <Example configuration of information processing system 101> Figure 2 shows an example configuration of an information processing system 101 to which this technology is applied.

[0025] The information processing system 101 is a system that reproduces a 3D space using 3DGS. The information processing system 101 comprises an input unit 111, an information processing unit 112, an output unit 113, and a storage unit 114.

[0026] The input unit 111 includes an input device for the user to operate the information processing unit 112 or to input various types of data into the information processing unit 112.

[0027] The information processing unit 112 performs various information processing tasks. The information processing unit 112 includes an image processing unit 121, an output control unit 122, and a file generation unit 123.

[0028] The image processing unit 121 performs various image processing operations. The image processing unit 121 includes a 3DGS particle group generation unit 131, a thin film particle extraction unit 132, a mesh generation unit 133, a mesh particle generation unit 134, an analysis condition setting unit 135, a thin film behavior calculation unit 136, a 3DGS particle group update unit 137, and a rendering unit 138.

[0029] The 3DGS particle generation unit 131 generates a 3DGS particle group, which is a group of Gaussian particles representing a three-dimensional space, using 3DGS based on multi-view image data input via the input unit 111, for example. The 3DGS particle generation unit 131 supplies the 3DGS particle group data representing the generated 3DGS particle group to the output control unit 122, the file generation unit 123, and the thin-film particle extraction unit 132.

[0030] The thin-film particle extraction unit 132 extracts 3DGS particles that constitute thin films such as cloth from the 3DGS particle group as Type 1 particles. The thin-film particle extraction unit 132 supplies the 3DGS particle group data, to which information about the extracted Type 1 particles has been added, to the mesh generation unit 133.

[0031] The mesh generation unit 133 generates a surface mesh in the 3DGS particle group, which is a mesh representing the surface of the thin film. The mesh generation unit 133 supplies the 3DGS particle group data, to which information about the generated surface mesh has been added, to the mesh particle generation unit 134.

[0032] The mesh particle generation unit 134 generates particles that constitute the surface mesh. For example, the mesh particle generation unit 134 generates type 2 particles at the positions of each vertex of the surface mesh and type 3 particles at the positions of the centers of each polygon of the surface mesh. The mesh particle generation unit 134 supplies 3DGS particle group data, to which information about the generated type 2 and type 3 particles has been added, to the analysis condition setting unit 135.

[0033] The analysis condition setting unit 135 sets the analysis conditions for simulating the behavior of the thin film. The analysis condition setting unit 135 supplies 3DGS particle group data and analysis condition information indicating the analysis conditions to the thin film behavior calculation unit 136.

[0034] The thin film behavior calculation unit 136 calculates the behavior of the thin film using MPM or the like, based on the 3DGS particle group data and analysis condition information. The thin film behavior calculation unit 136 supplies the 3DGS particle group data and information indicating the calculation results of the thin film behavior to the 3DGS particle group update unit 137. The thin film behavior calculation unit 136 supplies information indicating the calculation results of the thin film behavior to the output control unit 122.

[0035] The 3DGS particle group update unit 137 updates the 3DGS particle group based on the calculation results of the thin film behavior. The 3DGS particle group update unit 137 supplies the updated 3DGS particle group data to the output control unit 122, the file generation unit 123, the thin film behavior calculation unit 136, and the rendering unit 138.

[0036] The rendering unit 138 renders an image based on the 3DGS particle swarm data. The rendering unit 138 supplies image data representing the rendered image to the output control unit 122 and the file generation unit 123.

[0037] The output control unit 122 controls the output of various types of information, such as images and audio, from the output unit 113. For example, the output control unit 122 controls the display of the calculation results of the thin film behavior by the output unit 113 based on information indicating the calculation results of the thin film behavior. The output control unit 122 controls the display of the 3DGS particle group by the output unit 113 based on the 3DGS particle group data. The output control unit 122 controls the display of images by the output unit 113 based on image data.

[0038] The file generation unit 123 generates, for example, a file containing 3DGS particle group data and a file containing image data, according to a predetermined format. The file generation unit 123 stores the generated files in the storage unit 114.

[0039] The output unit 113 includes output devices that output various types of information. For example, the output unit 113 includes a display device that displays images such as a display, and an audio output device that outputs sound such as a speaker, headphones, or earphones.

[0040] The storage unit 114 stores various files generated by the file generation unit 123 and various data used in the processing of the information processing unit 112.

[0041] <Thin film behavior simulation processing> Next, with reference to the flowchart in Figure 5, the thin-film behavior simulation process performed by the information processing system 101 will be explained.

[0042] In step S1, the 3DGS particle group generation unit 131 generates a 3DGS particle group. For example, the 3DGS particle group generation unit 131 generates a 3DGS particle group based on multiple multi-viewpoint (different viewpoint) image data input via the input unit 111. The 3DGS particle group generation unit 131 supplies the 3DGS particle group data representing the generated 3DGS particle group to the thin-film particle extraction unit 132.

[0043] In step S2, the thin film particle extraction unit 132 extracts thin film particles. For example, as schematically shown in Figure 4A, the thin film particle extraction unit 132 extracts segments containing thin films such as cloth (hereinafter referred to as thin film segments) from the 3DGS particle group.

[0044] The method for extracting thin-film segments is not particularly limited. For example, the user may specify the source image from which the 3DGS particle group was generated, or a thin-film segment within the 3DGS particle group, via the input unit 111. For example, the thin-film particle extraction unit 132 may detect thin-film segments in the source image using a method such as semantic segmentation, and then extract thin-film segments from the 3DGS particle group based on the results.

[0045] Next, for example, the thin-film particle extraction unit 132 extracts 3DGS particles (thin-film particles) that constitute the thin film within the thin-film segment as type 1 particles, as shown in Figure 4B.

[0046] Type 1 particles are treated as weakly elastic materials in the MPM and follow the behavior of the thin film. The elastic behavior of the thin film is then simulated using the type 1 particles.

[0047] The thin-film particle extraction unit 132 supplies 3DGS particle group data, to which information about the extracted Type 1 particles has been added, to the mesh generation unit 133.

[0048] In step S3, the mesh generation unit 133 generates a surface mesh. For example, as schematically shown in Figure 4C, the mesh generation unit 133 generates a surface mesh in the 3DGS particle group, which is a mesh representing the surface of the thin film.

[0049] The surface mesh is composed of, for example, triangular polygons. The surface mesh does not require very high precision, and discontinuous regions (perforated regions) are acceptable.

[0050] The mesh generation unit 133 supplies 3DGS particle group data, to which information about the generated surface mesh has been added, to the mesh particle generation unit 134.

[0051] In step S4, the mesh particle generation unit 134 generates mesh particles. Specifically, the mesh particle generation unit 134 generates type 2 particles at the positions of each vertex of the surface mesh. The type 2 particles are used to track the in-plane deformation of the thin film (material).

[0052] The mesh particle generation unit 134 generates type 3 particles at the center of each polygon in the surface mesh. The type 3 particles are used as calculation points for calculating the forces arising from the normal deformation of the thin film (material) and the in-plane deformation obtained from the type 2 particles.

[0053] Then, the behavior of the thin film is simulated using type 2 and type 3 particles.

[0054] Here, the mass of the type 1 particle is set to be sufficiently small compared to the type 2 and type 3 particles. Conversely, the masses of the type 2 and type 3 particles are set to be sufficiently large compared to the type 1 particle.

[0055] This allows the momentum of the surface mesh to become dominant, enabling the creation of 3DGS animations that more accurately reproduce the behavior of thin films.

[0056] Figure 5 schematically shows examples of Type 1 to Type 3 particles. In Figure 5, the round dots represent Type 1 particles, the square dots represent Type 2 particles, and the triangular dots represent Type 3 particles. The triangular frames represent polygons of the surface mesh.

[0057] Type 1 particles construct a continuum that encompasses the surface mesh of the thin film.

[0058] The mesh particle generation unit 134 supplies 3DGS particle group data, to which information regarding the generated type 2 and type 3 particles has been added, to the analysis condition setting unit 135.

[0059] In step S5, the analysis condition setting unit 135 sets the analysis conditions based on information input by the user, for example, via the input unit 111. The analysis conditions include, for example, boundary conditions (e.g., displacement conditions at the fixed end of the thin film), gravitational acceleration, properties of the thin film, information about the fluid (e.g., wind or water), time step width, and resolution of the computational grid. Information about the fluid includes, for example, the fluid's drag coefficient, velocity, and direction. Properties of the thin film include, for example, density, Young's modulus, and Poisson's ratio. The time step width indicates the time interval for simulating the behavior of the thin film.

[0060] The analysis condition setting unit 135 supplies 3DGS particle group data and analysis condition information indicating the analysis conditions to the thin film behavior calculation unit 136.

[0061] In step S6, the thin film behavior calculation unit 136 generates a computational grid. Specifically, the analysis condition setting unit 135 generates a computational grid that is a three-dimensional grid in the Eulerian coordinate system (Eulerian grid).

[0062] FIG. 6 schematically shows the state of each particle and the computational grid. An elliptical frame indicates type 1 particles (3DGS particles), square points indicate type 2 particles, and triangular points indicate type 3 particles. A rectangular frame indicates the computational grid. The computational grid is generated independently of each particle.

[0063] Hereinafter, each cubic region of the computational grid is referred to as a cell.

[0064] In step S7, the thin film behavior calculation unit 136 executes a thin film behavior calculation process.

[0065] Here, details of the thin film behavior calculation process will be described with reference to the flowchart of FIG. 7.

[0066] In step S51, the thin film behavior calculation unit 136 calculates the inertial resistance force of fluid for each polygon of the surface mesh. For example, the thin film behavior calculation unit 136 calculates the inertial resistance force f received by each polygon of the surface mesh according to the following formula (1) fruid .

[0067] f fruid =-1 / 2ρ|v v | 2 c d av n ···(1)

[0068] v n represents a unit normal vector of the polygon, ρ represents the density of the thin film, c d represents the inertial resistance coefficient of the fluid. v v represents a relative velocity vector indicating the relative speed and direction of the polygon with respect to the fluid. The relative velocity vector v v is calculated by the following formula (2).

[0069] v v=v surface -v fruid ...(2)

[0070] v surface v is the velocity vector of the polygon. fruid This indicates the fluid velocity vector.

[0071] 'a' represents the area component of the polygon affected by the fluid, and is calculated by the following equation (3).

[0072] a=a0(v v ·v n / |n|) ···(3)

[0073] a0 represents the area of ​​the polygon.

[0074] In step S52, the thin film behavior calculation unit 136 projects the particle physical quantities of each particle and the fluid's inertial resistance force onto the computational grid. That is, the thin film behavior calculation unit 136 calculates the influence of the particle physical quantities of each particle and the fluid's inertial resistance force acting on each particle at each grid point of the computational grid.

[0075] Particle physical quantities include, for example, particle internal forces (the internal forces of each particle), particle mass (the mass of each particle), and particle momentum (the momentum of each particle).

[0076] For example, as schematically shown in Figure 8A, the thin film behavior calculation unit 136 distributes the internal force, particle mass, and particle momentum of each particle (type 1 to type 3 particles) to the grid points of the cells near each particle, using weighting coefficients.

[0077] In Figure 8, the black circles represent individual particles, and the white squares represent lattice points.

[0078] For example, as shown in Figure 9A, the mass m of particle p (where p is an index for identifying the particle) p When the particles are distributed to neighboring lattice points, the mass m distributed from each particle p to the lattice point i (where i is an index for identifying the particle) i This is calculated by the following equation (4).

[0079]

number

[0080] Figure 9B shows the shape function S for calculating the weight coefficients in equation (4). ip The graph shows the position (coordinates) of the particle. The horizontal axis represents the position (coordinates) of the particle, x i indicates the position (coordinate) of lattice point i to which the mass is distributed. h indicates the distance between lattice points.

[0081] Therefore, the closer a particle p is to lattice point i, the greater the mass distributed from particle p to lattice point i, and the further a particle p is from lattice point i, the smaller the mass distributed from particle p to lattice point i. And the mass m of each particle p in the vicinity of lattice point i p However, by adding them using a weighting coefficient based on the distance from the grid point i, the mass m at the grid point i is calculated. i This is calculated

[0082] The internal force and momentum of each particle are similarly represented by the shape function S. ip It is distributed to each grid point using this method.

[0083] Furthermore, the thin-film behavior calculation unit 136 distributes the fluid's inertial resistance force acting on each particle (type 2 particles to type 3 particles) to the grid points of the cells in the vicinity of each particle.

[0084] Here, an inertial resistance force f is applied equally to the type 2 and type 3 particles placed in each polygon. fruid Assuming that this is acting, the inertial resistance force f of the fluid is distributed from each particle p to the lattice point i. i,fruid This is calculated by the following equation (5).

[0085]

number

[0086] x² in equation (5) pThis shows the position vector (coordinates) of a type 2 particle, x3 p ∫ represents the position vector (coordinates) of a type 3 particle. The first term on the right-hand side of equation (5) represents the sum of the inertial drag forces distributed from type 2 particles in the vicinity of lattice point i, and the second term represents the sum of the inertial drag forces distributed from type 3 particles in the vicinity of lattice point i.

[0087] In equation (5), as in equation (4), the shape function S ip This is used. Therefore, the closer a particle p is to lattice point i, the greater the inertial resistance force distributed from particle p to lattice point i, and the further a particle p is from lattice point i, the smaller the inertial resistance force distributed from particle p to lattice point i.

[0088] In step S53, the thin film behavior calculation unit 136 updates the grid point physical quantities. That is, the thin film behavior calculation unit 136 calculates the grid point physical quantities at each grid point of the computational grid based on the internal force, mass, momentum, and fluid inertial resistance force projected from each particle to each grid point.

[0089] Specifically, the thin film behavior calculation unit 136 calculates the grid point internal force, which is the internal force at each grid point, by adding the internal force projected onto each grid point and the fluid's inertial resistance force. The thin film behavior calculation unit 136 calculates the grid point mass, which is the mass at each grid point, by adding the mass projected onto each grid point. The thin film behavior calculation unit 136 calculates the grid point momentum, which is the momentum at each grid point, by adding the momentum projected onto each grid point.

[0090] The thin film behavior calculation unit 136 calculates the grid point velocity, which is the velocity of each grid point, based on the grid point internal force, grid point mass, and grid point momentum at each grid point, as well as the grid point external force, which is the external force acting on each grid point, using the following equation (6).

[0091] Grid point velocity = Δt × (grid point external force + grid point internal force) / grid point mass + Lattice point momentum / Lattice point mass ···(6)

[0092] Δt represents the time step width.

[0093] In step S54, the thin film behavior calculation unit 136 applies boundary conditions to the grid points. Specifically, the thin film behavior calculation unit 136 corrects the grid point velocity of each grid point based on the boundary conditions of each grid point. Based on the corrected grid point velocity, the thin film behavior calculation unit 136 calculates and updates the position of each grid point.

[0094] As a result, as shown in Figure 8B, each grid point in the computational grid moves and the cells deform based on the particle physical quantities of neighboring particles and the inertial resistance force of the fluid.

[0095] In step S55, the thin film behavior calculation unit 136 projects the lattice physical quantities of the computational grid onto each particle. That is, the thin film behavior calculation unit 136 calculates the influence of the lattice physical quantities of each lattice point on the position of each particle.

[0096] For example, the thin film behavior calculation unit 136 calculates the velocity of each lattice point using a weighting coefficient (e.g., shape function S) based on the distance to each particle. ip Using this, the particles (Type 1 particles to Type 3 particles) in the vicinity of each lattice point are distributed accordingly.

[0097] This allows us to calculate the velocity projected from each lattice point to the position of each particle.

[0098] In steps S56-1 to S56-3, the thin film behavior calculation unit 136 updates the particle physical quantities of each particle based on the lattice point physical quantities projected from each lattice point to the position of each particle.

[0099] Specifically, in step S56-1, the thin film behavior calculation unit 136 updates the velocity, position, affine matrix, and deformation gradient of the type 1 particles.

[0100] Specifically, the thin film behavior calculation unit 136 calculates (updates) the particle velocity of each type 1 particle by adding the velocities (velocity vectors) projected onto the position of each type 1 particle. Based on the updated particle velocity of each type 1 particle, the thin film behavior calculation unit 136 calculates (updates) the position of each type 1 particle. Based on the updated position and particle velocity of each type 1 particle, the thin film behavior calculation unit 136 calculates (updates) an affine matrix that shows the spatial velocity gradient at the position of each type 1 particle. Based on the updated position, particle velocity, and affine matrix of each type 1 particle, the thin film behavior calculation unit 136 calculates (updates) a deformation gradient that shows the deformation rate of each type 1 particle.

[0101] The thin film behavior calculation unit 136 supplies the 3DGS particle group data and information indicating the particle physical quantities of each type 1 particle after the update to the 3DGS particle group update unit 137.

[0102] In step S56-2, the thin film behavior calculation unit 136 updates the velocity, position, and affine matrix of the type 2 particles.

[0103] Specifically, the thin film behavior calculation unit 136 calculates (updates) the particle velocity of each type 2 particle by adding the velocities (velocity vectors) projected onto the position of each type 2 particle. Based on the updated particle velocity of each type 2 particle, the thin film behavior calculation unit 136 calculates (updates) the position of each type 2 particle. Based on the updated position and particle velocity of each type 2 particle, the thin film behavior calculation unit 136 calculates (updates) an affine matrix that shows the spatial velocity gradient at the position of each type 2 particle.

[0104] In step S56-3, the thin film behavior calculation unit 136 updates the affine matrix and normal vector of the type 3 particles.

[0105] Specifically, the thin film behavior calculation unit 136 calculates the particle velocity of each type 3 particle by adding the velocities (velocity vectors) projected onto the positions of each type 3 particle. Based on the calculated particle velocities of each type 3 particle, the thin film behavior calculation unit 136 calculates the position of each type 3 particle. Based on the calculated position and grid point velocity of each type 3 particle, the thin film behavior calculation unit 136 calculates an affine matrix that shows the spatial velocity gradient at the position of each type 3 particle. Based on the affine matrix of each type 3 particle, the thin film behavior calculation unit 136 calculates a normal vector that shows the normal direction of the polygon containing each type 3 particle.

[0106] In step S57, the thin film behavior calculation unit 136 updates the velocity and position of the type 3 particles based on the type 2 particles. Specifically, the thin film behavior calculation unit 136 calculates the shape, position, and orientation of each polygon composed of each type 2 particle based on the updated position of each type 2 particle. Based on the shape, position, and orientation of each polygon, the thin film behavior calculation unit 136 calculates the velocity and position of the type 3 particle located at the center of each polygon.

[0107] Through the processing in steps S56-1 to S57, each particle moves based on the lattice point physical quantities of neighboring lattice points, for example, as shown in Figure 8C.

[0108] In step S58, the thin film behavior calculation unit 136 resets the calculation grid. That is, as shown in Figure 8D, each cell of the calculation grid is restored to its pre-deformation shape, and each grid point returns to its original position.

[0109] After that, the thin film behavior processing is completed.

[0110] Returning to Figure 3, in step S8, the 3DGS particle group update unit 137 updates the Gaussian kernel based on the analysis results of type 1 particles. That is, the 3DGS particle group update unit 137 updates the Gaussian kernel, which is a kernel function representing the Gaussian distribution of type 1 particles, which are Gaussian particles.

[0111] Specifically, in MPM, each particle records a deformation gradient. The deformation gradient shows how a small region around the particle has deformed.

[0112] In response, the thin film behavior calculation unit 136 applies the deformation gradient of the type 1 particles calculated in step S56-1 to the Gaussian kernel of the type 1 particles to reproduce the physical deformation of the thin film in the 3DGS particle group.

[0113] Type 1 particle Gaussian kernel G p (x) can be expressed by equation (7) below, using a Gaussian kernel of a standard Gaussian distribution.

[0114]

number

[0115] p represents an index for identifying type 1 particles, x represents a coordinate vector on a Gaussian distribution, and μ p This shows the mean vector of the Gaussian distribution, C p This shows the covariance matrix of a Gaussian distribution.

[0116] Here, the Gaussian kernel Gp(x',t) of a type 1 particle at time t is the deformation map φ at time t. p It can be expressed by the following equation (8) using (x,t).

[0117]

number

[0118] The transformation map φ(x,t) is a function that maps the coordinate vector x in the original Gaussian distribution to the coordinate vector x' in the transformed Gaussian distribution.

[0119] In this case, if we treat the local deformation of the Gaussian distribution as a linear transformation, the deformation gradient F of the type 1 particle is p This can be applied to the deformation map φ(x,t), and the deformation map φ(x,t) is the deformation map φ' of equation (9) below.p It is approximated by (x,t).

[0120]

number

[0121] Transformation mapping φ' p (x,t) is the mean vector μ, as schematically shown in Figure 10. p ,covariance matrix C p The Gaussian distribution before deformation, represented by the mean vector μ' p , covariance matrix C' p This is a function that maps to the modified Gaussian distribution represented by .

[0122] The Gaussian kernel Gp(x',t) in equation (8) can be expressed by equation (10) using equation (9).

[0123]

number

[0124] Then, by using equation (10), the Gaussian kernel of the deformed type 1 particle is updated using the deformation gradient of the type 1 particle.

[0125] As a result, the type 1 particle, which is a Gaussian particle, is deformed, as schematically shown in Figure 11.

[0126] Specifically, Figure 11A shows the type 1 particle 201 and the computational grid cell 202 before deformation. Then, as shown in Figure 11B, the cell 202 is deformed by the particle physical quantities of the particles in its vicinity and the inertial resistance force of the fluid, and this deformation of the cell 202 is reflected in the type 1 particle 202. As a result, the type 1 particle moves and deforms. Subsequently, as shown in Figure 11C, the position and shape of the cell 202 are reset.

[0127] In this way, the 3DGS particle group is updated in accordance with the behavior of the thin film.

[0128] In step S9, the 3DGS particle group update unit 137 determines whether or not to save the update result. For example, if the user inputs a command to save the update result via the input unit 111, the file generation unit 123 determines to save the update result, and the process proceeds to step S10.

[0129] In step S10, the 3DGS particle group update unit 137 saves the update results. Specifically, the 3DGS particle group update unit 137 decomposes the covariance matrix of each Gaussian kernel to which the deformation of type 1 particles has been applied into a rotation matrix and a scaling matrix.

[0130] Specifically, the covariance matrix C of the Gaussian kernel of the deformed type 1 particle. * It is defined by the following equation (11).

[0131] C * =F·C·F T ...(11)

[0132] F represents the deformation gradient, and C represents the covariance matrix of the Gaussian kernel of the type 1 particle before deformation.

[0133] Here, the covariance matrix A * When we perform singular value decomposition on this, we get equation (12) below.

[0134] A * =U·Σ·V T ...(12)

[0135] U represents a unitary matrix with left singular vectors as columns, Σ represents a singular value matrix with singular values ​​as diagonal elements, and V represents a unitary matrix with right singular vectors as columns.

[0136] At this time, the covariance matrix A * Since it is a diagonal matrix, equation (12) can be expressed as equation (13).

[0137] A * =U·Σ·U T ...(13)

[0138] Furthermore, the covariance matrix A * This is the scaling matrix S * and rotation matrix R * Using this, it can be expressed by the following equation (14).

[0139] A * =R * ·S * ·S *T ·R T ...(14)

[0140] Therefore, the rotation matrix R * and scaling matrix S * This is expressed by the following equations (15) and (16).

[0141] R * =U ···(15) S * =sqrt(Σ) ···(16)

[0142] Rotation matrix R * This indicates the amount and direction of rotation of the deformed type 1 particle, and the scaling matrix S * This indicates the scaling amount of the deformed type 1 particle.

[0143] This allows the deformation state of 3DGS particles to be redefined based on the calculated deformation gradient of Type 1 particles, and the static state to be saved. For example, the axis length and rotation of 3DGS particles can be redefined, so the deformed 3DGS particles can be baked in conventional formats such as PLY (Polygon File Format).

[0144] The process then proceeds to step S11.

[0145] On the other hand, if it is determined in step S9 that the update result should not be saved, the process in step S10 is skipped, and the process proceeds to step S11.

[0146] In step S11, the rendering unit 138 determines whether or not to output a rendering. For example, if the user inputs a command to output a rendering via the input unit 111, the file generation unit 123 determines to output a rendering, and the process proceeds to step S12.

[0147] In step S12, the rendering unit 138 outputs a rendering. Specifically, the rendering unit 138 obtains 3DGS particle group data showing the updated 3DGS particle group from the 3DGS particle group update unit 137. The rendering unit 138 renders an image based on the 3DGS particle group data. The rendering unit 138 supplies image data showing the rendered image to the output control unit 122.

[0148] The output control unit 122 displays the rendered image on the output unit 113 based on the image data.

[0149] The process then proceeds to step S13.

[0150] On the other hand, if it is determined in step S11 that rendering should not be output, the process in step S12 is skipped, and the process proceeds to step S13.

[0151] In step S13, the thin film behavior calculation unit 136 determines whether or not to terminate the analysis. If it is determined not to terminate the analysis, the process returns to step S7.

[0152] Subsequently, in step S13, the processes from steps S7 to S13 are repeatedly executed until it is determined that the analysis is complete.

[0153] On the other hand, in step S13, for example, if the user inputs a command to terminate the analysis via the input unit 111, the thin film behavior calculation unit 136 determines that the analysis should be terminated, and the thin film behavior simulation process ends.

[0154] As described above, by treating the 3DGS particles (type 1 particles) constituting the thin film as weakly elastic bodies and handling thin film behavior using a surface mesh with a certain degree of accuracy (allowing holes) obtained from a group of 3DGS particles, a continuum containing the surface mesh of the thin film is constructed.

[0155] This improves the accuracy of reproducing thin film behavior in 3DGS without using additional training or complex techniques. Furthermore, thin film behavior can be applied to 3DGS particle groups, enabling rendering results of animation that are closer to reality to be obtained.

[0156] Furthermore, by applying the inertial resistance force of fluid, for example, realistic behavior of cloth affected by wind can be reproduced with high accuracy. For example, since the force applied to the thin film changes depending on the angle relative to the wind, complex movements of the thin film can be reproduced.

[0157] <Example of GUI (Graphical User Interface)> FIG. 12 shows an example of a GUI displayed on the output unit 113 of the information processing system 101.

[0158] The GUI in FIG. 12 includes a menu bar 301, a tool bar 302, a 3D viewport 303, a control panel 304, and a status bar 305.

[0159] The menu bar 301 and the tool bar 302 are horizontally extending bars, and are arranged vertically side by side at the top of the GUI. The status bar 305 is a horizontally extending bar arranged at the bottom end of the GUI. The 3D viewport 303 and the control panel 304 are arranged horizontally side by side between the tool bar 302 and the status bar 305.

[0160] On the menu bar 301, for example, operation menus such as Display and Help are displayed.

[0161] Toolbar 302 displays menus for tools used to perform 3DGS operations, such as data loading, segmentation execution, and mesh generation.

[0162] The 3D viewport 303 is an area that displays 3DGS particle clusters (point clouds), surface meshes, 3D images based on 3DGS, etc. For example, it is possible to perform operations such as zooming, rotating, and panning on the image displayed in the 3D viewport 303.

[0163] The control panel 304 is an area for making various settings. For example, the control panel 304 includes settings for "Segmentation," "Mesh Generation," and "Simulation."

[0164] "Segmentation" includes settings related to segmentation for extracting thin film regions from the original image or 3DGS particle group. For example, "Segmentation" includes settings for selection tools for manual segmentation, automatic segmentation, etc.

[0165] "Mesh generation" includes settings related to the generation of surface meshes. For example, "Mesh generation" includes settings such as surface mesh density and quality adjustment.

[0166] The "Simulation" section includes settings related to the simulation of thin film behavior. For example, the "Simulation" section includes settings for the physical properties of the thin film, environmental conditions such as wind and gravitational acceleration, and simulation operation (start, stop, pause).

[0167] The status bar 305 displays information about the status and operation of the image displayed in the 3D viewport 303. For example, it displays information about the frame displayed in the 3D viewport 303, operation buttons (e.g., play, stop, etc.), and messages.

[0168] <<2. Variant>> The following describes some modifications of the embodiments of the present technology described above.

[0169] The above explanation illustrates an example where the deformation of each cell in the computational grid is projected onto each particle, and after each particle moves, the computational grid is reset to its original shape and position, as shown in Figure 13A.

[0170] In contrast, as shown in Figure 13B, for example, the position or resolution of the computational grid may be adjusted based on the distribution of each particle when the computational grid is reset. For example, the computational grid may be moved to match the position of each particle. This allows simulations of thin films moving over a wide area to be performed with fewer cells, thereby reducing the computational load and the amount of memory used. For example, the resolution (cell size) of the computational grid may be dynamically changed to match the distribution of each particle. This reduces the computational load, for example, when the movement of the thin film is small.

[0171] For example, the shape of the polygons in the surface mesh is not limited to triangles; polygons with four or more sides may also be used.

[0172] For example, type 3 particles may be placed at locations other than the center of the polygon in the surface mesh (e.g., at the centroid of the polygon).

[0173] This technology is applicable to systems, devices, programs, etc., that reproduce 3D space using 3DGS. For example, this technology can be applied to information processing devices that reproduce 3D space, such as servers, and display devices that reproduce 3D space, such as HMDs.

[0174] <<3.B>> <Description of a computer using this technology> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0175] Figure 14 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0176] In a computer, the processing circuit 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0177] An input / output interface 1005 is further connected to the bus 1004. An input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010.

[0178] The input unit 1006 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 1006 may include sensors for inputting various physical quantities into the computer. For example, the input unit 1006 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 1006 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 1007 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 1008 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 1009 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 1010 drives removable media 1011 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory.

[0179] The processing circuit 1001 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 1001 (its processor) performs the series of processes described above by loading the program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it. The processing circuit 1001 can output the processing results of the series of processes from the output unit 1007, for example, via the bus 1004 and the input / output interface 1005, as needed. The processing circuit 1001 can also store the processing results in the memory unit 1008 or transmit them from the communication unit 1009.

[0180] The program executed by the computer (processing circuit 1001) can be provided by recording it on a removable medium 1011, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0181] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media 1011 into the drive 1010. Alternatively, a program can be received by the communication unit 1009 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008.

[0182] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0183] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0184] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0185] When the above-described series of processes are performed by a computer executing a program, for example, the input unit 1006 functions as the input unit 111 in Figure 2. For example, the processing circuit 1001 (processor) functions as the information processing unit 112 in Figure 2 by executing a program. For example, the output unit 1007 functions as the output unit 113 in Figure 4.

[0186] In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of a processor and memory (for example, only the processing circuit 1001, or a combination of processing circuit 1001 to bus 1004, etc.), constitute a system. Regarding a collection of multiple components, it is not necessary whether all components reside in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, an entire computer, or a combination of a computer and other devices such as a server (not shown), also constitute a system.

[0187] The components (blocks) of the apparatus illustrated in this specification are functional conceptual blocks, and the actual apparatus does not need to have the illustrated configuration. That is, the apparatus can have any configuration in which the functions of the illustrated components are divided and / or integrated into any unit, for example, a configuration having one block in which the functions of all components are integrated.

[0188] The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0189] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0190] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0191] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0192] <Examples of configuration combinations> This technology can also be configured as follows:

[0193] (1) The behavior of the thin film is simulated using a group of Gaussian particles that form a thin film representation based on 3D Gaussian splatting, specifically: first particles which constitute the thin film; second particles which are placed at each vertex of the mesh representing the surface of the thin film; and third particles which are placed within each polygon of the mesh. Equipped with a processing circuit Information processing system. (2) The processing circuit projects the physical quantity of each particle onto each grid point of the computational grid, which is an Euler grid, updates the physical quantity of each grid point, projects the updated physical quantity of each grid point onto each particle, and updates the physical quantity of each particle. The information processing system described in (1) above. (3) The processing circuit further projects the fluid's inertial resistance force acting on each particle onto each grid point and updates the physical quantities at each grid point. The information processing system described in (2) above. (4) The processing circuit calculates the inertial resistance force for each polygon of the mesh and projects the inertial resistance forces experienced by the second and third particles placed in each polygon onto each grid point. The information processing system described in (3) above. (5) The processing circuit simulates the behavior of the thin film based on boundary conditions, gravitational acceleration, the physical properties of the thin film, and analysis conditions including the fluid's drag coefficient, velocity, and direction. The information processing system described in (3) or (4) above. (6) The processing circuit projects physical quantities, including the internal force, mass, and momentum of each particle, onto each of the grid points. An information processing system as described in any of (2) to (5) above. (7) The processing circuit projects physical quantities, including the velocity of each lattice point, onto each particle. An information processing system as described in any of (2) to (6) above. (8) The processing circuit updates the physical quantities including the velocity, position, affine matrix, and deformation gradient of each first particle, the velocity, position, and affine matrix of each second particle, and the velocity, position, affine matrix, and normal vector of each third particle. An information processing system as described in any of (2) to (7) above. (9) The processing circuit updates a physical quantity including the deformation gradient of each first particle, and deforms each first particle based on the deformation gradient of each first particle. An information processing system as described in any of (2) to (7) above. (10) The processing circuit updates the Gaussian kernel of each first particle based on the deformation gradient of each first particle. The information processing system described in (9) above. (11) The processing circuit calculates the scaling amount, rotation amount, and rotation direction of the deformed first particle based on the deformation gradient of the first particle. The information processing system described in (9) or (10) above. (12) The processing circuit applies boundary conditions to the physical quantities of each updated grid point. An information processing system as described in any of (2) to (11) above. (13) The processing circuit updates the physical quantities of each particle and then returns the computational grid to its original state. An information processing system as described in any of (2) to (12) above. (14) The processing circuit adjusts the position or resolution of the computational grid based on the distribution of each particle. An information processing system as described in any of (2) to (12) above. (15) The processing circuit simulates the elastic behavior of the thin film using the first particle, and simulates the behavior of the thin film using the second and third particles. An information processing system as described in any of (1) to (14) above. (16) The processing circuit updates the 3DGS particle group in accordance with the behavior of the thin film. An information processing system as described in any of (1) to (15) above. (17) The processing circuit renders an image based on the particle group of the 3DGS. An information processing system as described in any of (1) to (16) above. (18) A display device that displays the aforementioned image The information processing system described in (17) further comprises the above. (19) Information processing system, The behavior of the thin film is simulated using a group of Gaussian particles that form a thin film representation based on 3D Gaussian splatting, specifically: first particles which constitute the thin film; second particles which are placed at each vertex of the mesh representing the surface of the thin film; and third particles which are placed within each polygon of the mesh. Information processing methods. (20) On the computer, The behavior of the thin film is simulated using a group of Gaussian particles that form a thin film representation based on 3D Gaussian splatting, specifically: first particles which constitute the thin film; second particles which are placed at each vertex of the mesh representing the surface of the thin film; and third particles which are placed within each polygon of the mesh. A program to execute a process.

[0194] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur. [Explanation of symbols]

[0195] 101 Information Processing System, 111 Input Unit, 112 Information Processing Unit, 113 Output Unit, 121 Image Processing Unit, 122 Output Control Unit, 123 File Generation Unit, 131 3DGS Particle Group Generation Unit, 132 Thin Film Particle Extraction Unit, 133 Mesh Generation Unit, 134 Mesh Particle Generation Unit, 135 Analysis Condition Setting Unit, 136 Thin Film Behavior Calculation Unit, 137 3DGS Particle Group Update Unit, 138 Rendering Unit

Claims

1. The behavior of the thin film is simulated using a group of Gaussian particles that form a thin film representation based on 3D Gaussian splatting, specifically: first particles which constitute the thin film; second particles which are placed at each vertex of the mesh representing the surface of the thin film; and third particles which are placed within each polygon of the mesh. Equipped with a processing circuit Information processing system.

2. The processing circuit projects the physical quantity of each particle onto each grid point of the computational grid, which is an Euler grid, updates the physical quantity of each grid point, projects the updated physical quantity of each grid point onto each particle, and updates the physical quantity of each particle. The information processing system according to claim 1.

3. The processing circuit further projects the fluid's inertial resistance force acting on each particle onto each grid point and updates the physical quantities at each grid point. The information processing system according to claim 2.

4. The processing circuit calculates the inertial resistance force for each polygon of the mesh and projects the inertial resistance forces experienced by the second and third particles placed in each polygon onto each grid point. The information processing system according to claim 3.

5. The processing circuit simulates the behavior of the thin film based on boundary conditions, external forces, the physical properties of the thin film, and analysis conditions including the fluid's inertial resistance coefficient, velocity, and direction. The information processing system according to claim 3.

6. The processing circuit projects physical quantities, including the internal force, mass, and momentum of each particle, onto each of the grid points. The information processing system according to claim 2.

7. The processing circuit projects physical quantities, including the velocity of each lattice point, onto each particle. The information processing system according to claim 2.

8. The processing circuit updates the physical quantities including the velocity, position, affine matrix, and deformation gradient of each first particle, the velocity, position, and affine matrix of each second particle, and the velocity, position, affine matrix, and normal vector of each third particle. The information processing system according to claim 2.

9. The processing circuit updates a physical quantity including the deformation gradient of each first particle, and deforms each first particle based on the deformation gradient of each first particle. The information processing system according to claim 2.

10. The processing circuit updates the Gaussian kernel of each first particle based on the deformation gradient of each first particle. The information processing system according to claim 9.

11. The processing circuit calculates the scaling amount, rotation amount, and rotation direction of the deformed first particle based on the deformation gradient of the first particle. The information processing system according to claim 9.

12. The processing circuit applies boundary conditions to the physical quantities of each updated grid point. The information processing system according to claim 2.

13. The processing circuit updates the physical quantities of each particle and then returns the computational grid to its original state. The information processing system according to claim 2.

14. The processing circuit adjusts the position or resolution of the computational grid based on the distribution of each particle. The information processing system according to claim 2.

15. The processing circuit simulates the elastic behavior of the thin film using the first particle, and simulates the behavior of the thin film using the second and third particles. The information processing system according to claim 1.

16. The processing circuit updates the particle group of the 3DGS in accordance with the behavior of the thin film. The information processing system according to claim 1.

17. The processing circuit renders an image based on the particle group of the 3DGS. The information processing system according to claim 1.

18. A display device that displays the aforementioned image The information processing system according to claim 17, further comprising:

19. Information processing system, The behavior of the thin film is simulated using a group of Gaussian particles that form a thin film representation based on 3D Gaussian splatting, specifically: first particles which constitute the thin film; second particles which are placed at each vertex of the mesh representing the surface of the thin film; and third particles which are placed within each polygon of the mesh. Information processing methods.

20. On the computer, The behavior of the thin film is simulated using a group of Gaussian particles that form a thin film representation based on 3D Gaussian splatting, specifically: first particles which constitute the thin film; second particles which are placed at each vertex of the mesh representing the surface of the thin film; and third particles which are placed within each polygon of the mesh. A program to execute a process.

Images