System identification device, system identification method, and program

Abstract

This system identification device converts a lattice point field representing a geometric structure of a target object at one point in time into a particle field representing the geometric structure of the target object at that point in time, subjects the particle field to a physical simulation, calculates particle fields at a plurality of points in time after the one point in time, converts the particle fields at the one point in time and the plurality of points in time into a lattice point field, subjects the converted lattice point field to volume rendering, estimates measurement data that is a result of capturing images of the target object at each of the one point in time and the plurality of points in time, calculates a loss between the measurement data of the target object estimated at each of the one point in time and the plurality of points in time and measurement data obtained as a result of actually capturing images of the target object, and optimizes, on the basis of the loss, a field indicating the geometric structure of the target object.

Description

System identification device, system identification method and program

[0001] The present invention relates to a system identification device, a system identification method, and a program.

[0002] There is a technology called system identification that estimates the geometric structure and physical properties of an object from measurement data and constructs a dynamic model. Measurement data can include, for example, video captured from one or more viewpoints. The geometric structure represents the external factors of an object, such as its appearance and shape. The physical properties represent the internal factors of an object, such as the object's fluid viscosity, bulk modulus, shear modulus, yield stress, plastic viscosity, Young's modulus, Poisson's ratio, and friction angle. System identification is a technology that can be applied to a wide range of technologies, including environmental understanding, simulation, and content generation, and is actively researched in fields such as physics and computer vision.

[0003] One system identification technique is called "Geometric-Agnostic System Identification," which estimates the geometric structure and physical properties of an object solely from measurement data such as video, without making any assumptions about the object's geometric structure. The advantage of this technique is that, because it does not require any assumptions about the object's geometric structure, there are fewer constraints on the object, making it applicable to a wide range of objects.

[0004] However, this technique has a weakness in that it does not make assumptions about the geometric structure of the object, resulting in a high degree of freedom in the solution and making it difficult to solve the problem. To overcome this weakness, for example, there is a method called Physics-Augmented Continuum Neural Radiance Fields (PAC-NeRF) (see Non-Patent Document 1). PAC-NeRF uses Neural Radiance Fields (NeRF) to represent the geometric structure of an object on lattice points in Euler coordinates (world coordinates), and uses Material Point Method (MPM) to simulate the physical motion of the object based on particles in Lagrangian coordinates (object coordinates). By using grid-to-particle and particle-to-grid transformations, the geometric structure of the object represented on Euler coordinates is associated with the physical motion represented on Lagrangian coordinates.

[0005] Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. PAC-NeRF: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In ICLR, 2023.

[0006] However, in PAC-NeRF, optimization is performed in Euler coordinates (world coordinates), in which it is difficult to explicitly impose physical constraints on the geometric structure of an object, and optimization is performed using only the initial frames of measurement data over a time series. Therefore, when it is difficult to estimate the geometric structure of an object, the accuracy of system identification may deteriorate.

[0007] In view of the above circumstances, an object of the present invention is to provide a system identification device, a system identification method, and a program that are capable of performing system identification with higher accuracy.

[0008] One aspect of the present invention is a system identification device that converts a lattice point field representing a geometric structure of a target object at a single time point into a particle field representing the geometric structure of the target object at the single time point, performs a physical simulation on the particle field, calculates particle fields at multiple time points after the single time point, converts the particle fields at the single time point and the multiple time points into lattice point fields, performs volume rendering on the converted lattice point field, estimates measurement data resulting from photographing the target object at the single time point and each of the multiple time points, calculates a loss between the estimated measurement data of the target object at the single time point and each of the multiple time points and measurement data obtained as a result of actually photographing the target object, and optimizes a field representing the geometric structure of the target object based on the loss.

[0009] One aspect of the present invention is a system identification method that converts a lattice point field representing a geometric structure of a target object at a single time point into a particle field representing the geometric structure of the target object at the single time point, performs a physical simulation on the particle field, calculates particle fields at multiple time points after the single time point, converts the particle fields at the single time point and the multiple time points into lattice point fields, performs volume rendering on the converted lattice point field, estimates measurement data resulting from photographing the target object at the single time point and each of the multiple time points, calculates a loss between the estimated measurement data of the target object at the single time point and each of the multiple time points and measurement data obtained as a result of actually photographing the target object, and optimizes the field representing the geometric structure of the target object based on the loss.

[0010] The present invention allows for more accurate system identification.

[0011] FIG. 1 is a diagram showing a system identification device 1 according to a first embodiment. FIG. 2 is a diagram showing an optimization method according to the first embodiment. FIG. 3 is a flowchart showing an optimization method according to the first embodiment. FIG. 4 is a diagram showing an optimization method according to a second embodiment. FIG. 5 is a flowchart showing an optimization method according to the second embodiment. FIG. 6 is a diagram showing Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) for each method. FIG. 7 is a diagram showing the accuracy of physical properties estimated in a comparison method and a third method. FIG. 8 is a diagram showing an example of the hardware configuration of a system identification device 1 according to an embodiment.

[0012] (First Embodiment) Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Fig. 1 is a diagram showing a system identification device 1 according to a first embodiment. The system identification device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, which are connected via a bus.

[0013] The control unit 11 executes, for example, a measurement data acquisition process, a system identification process, and an output process.

[0014] (Measurement Data Acquisition Process) The measurement data acquisition process is a process for acquiring measurement data of an object A that is the subject of system identification (hereinafter referred to as "target object"). The measurement data is, for example, video data captured from multiple viewpoints by multiple cameras 3. In FIG. 1, the target object A is captured from four viewpoints by four cameras 3-1 to 3-4, and video data is acquired. In the following descriptions except for mathematical formulas, symbols with a hat are represented by writing a ^ next to the symbol. For example, a symbol with a hat on α is written as α^.

[0015] Video data can be expressed as a set of pixels C^(r,t) corresponding to time t (t is a positive real number) and camera rays r(s) (r(s) is a three-dimensional vector with real components). More specifically, the set of camera rays r(s) is R^, and the set of times t is T^={t ０ , ..., t Ｎ－１} (N is the number of frames of the video), the measurement data is a set of pixels corresponding to a data space consisting of R^ and T^.

[0016] The camera's ray r(s) is expressed as r(s) = o + sd, where o is the camera's position (viewpoint position) in three-dimensional space (o is the three-dimensional coordinate), d is the camera's direction (viewpoint direction) (d is the coordinate on the two-dimensional sphere), and s is the distance from the camera. s is the minimum value s ｎ and the maximum value s ｆ It is defined as a value between

[0017] (System Identification Processing) The system identification processing is processing for estimating the geometric structure and physical properties of the target object A based on measurement data of the target object A.

[0018] Geometry describes the external characteristics of an object, such as its appearance and shape, while physical properties describe the internal characteristics of an object, such as fluid viscosity, bulk modulus, shear modulus, yield stress, plastic viscosity, Young's modulus, Poisson's ratio, and friction angle.

[0019] In the system identification process, the geometric structure and physical properties of the target object A are estimated based on a set of measurement data C^(r,t), for example. The estimation results from the system identification process can be expressed using a pixel C(r,t) corresponding to the ray r of an arbitrary camera and time t. C(r,t) is the result of estimating the corresponding measurement data C^(r,t). By obtaining C(r,t) as the estimation result through the system identification process, an image obtained when the target object A is observed by an arbitrary camera at an arbitrary time is generated. Here, "any camera" includes a camera with a viewpoint not included in the measurement data. In other words, the system identification process can generate a video of the target object A from more viewpoints than the measured viewpoints, using measurement data from a limited number of viewpoints.

[0020] The output process is a process for outputting the generated result, for example, by displaying the estimation result C(r, t) on a display device.

[0021] (Method of Representing C(r,t)) C(r,t) is expressed using, for example, Continuum Neural Radiance Fields (Continuum NeRF). Continuum NeRF is an extension of NeRF to handle continuums. Specifically, NeRF, which is designed for still images, is extended to Dynamic NeRF, which is designed for video. Dynamic NeRF uses a point x in three-dimensional space, a camera direction d, and time t to represent a volume density field (σ(x,t)) and a color field (c(x,d,t)). The volume density field represents the density per unit volume of a point in three-dimensional space; smaller values ​​indicate lower density, and larger values ​​indicate higher density. Density has an inverse relationship with transmittance; the higher the density, the lower the transmittance, making objects behind a point less visible. Conversely, the lower the density, the higher the transmittance, making objects behind a point more visible. The color field represents the color of a point in three-dimensional space. In Dynamic NeRF, a pixel C(r, t) corresponding to a camera ray r and time t is calculated by volume rendering using equations (1) and (2).

[0022] In formula (1), c ｂｇ represents the background color. In the calculation of equations (1) and (2), C(r, t) and T(s, t) may be calculated by discretizing the integral.

[0023] Hereinafter, a method for improving the accuracy of the estimation result C(r, t) by performing optimization in the system identification process will be described. First, the first measurement time t ０ F shows the geometric structure in Ｇ’ (t0) is generated. F Ｇ’ (t0) is the grid point field represented by grid points in Euler coordinates. F Ｇ’(t0) is optimized by the final calculated loss.

[0024] Geometric structure F Ｇ’ The initial value of (t0) is set randomly by sampling from a predetermined distribution (such as a Gaussian distribution). Ｇ’ The initial value of (t0) is set to a predetermined value, for example. Ｇ’ The method for setting the initial value of (t0) is not particularly limited.

[0025] F Ｇ’ (t0) is F Ｐ (t0). F Ｐ (t0) is time t ０ The transformation from the representation of a geometric structure using a grid in Eulerian coordinates to a geometric structure using particles in Lagrangian coordinates is expressed by equation (3).

[0026] In equation (3), F represents the field, the superscripts P and G represent the particle-based field and the lattice-based field, respectively, and the subscripts p and i represent the particle index and the lattice index, respectively. ｐ Ｐ denotes a particle-based field denoted by p, and F ｉ Ｇ denotes the field based on the lattice indicated by i. ｉｐ represents the weight of the grid i when interpolating the particle indicated by p on the grid. ｉｐ The interpolation method is not particularly limited, and examples include trilinear interpolation and spline interpolation. Hereinafter, the conversion from a grid in Eulerian coordinates to particles in Lagrangian coordinates will also be referred to as G2P (Grid-to-Particle).

[0027] F Ｐ A physical simulation is performed based on (t0), and time t ｉ = t １ , ..., t Ｎ－１ F when Ｐ (t ｉ) (i∈{1, ..., N-1}) is calculated. For example, the Material Point Method (MPM) is used for the physical simulation. In the physical simulation, the volume density field σ(x, t) is constrained by equation (4), and the color field c(x, d, t) is constrained by equation (5).

[0028] That is, the simulation is performed so that the Lagrangian derivative of the volume density field σ(x, t) and the color field c(x, d, t) always becomes 0. In addition, an arbitrary time-dependent field φ(x, t) is expressed by Equation (6).

[0029] In equation (6), v represents the velocity field, which follows the law of conservation of momentum for a continuum expressed by equation (7).

[0030] In equation (7), ρ is the physical density field, T is the Cauchy stress tensor, and g is the gravitational acceleration. A physical simulation is performed on the particles in Lagrangian coordinates, taking into account the physical constraints of equations (4) to (7), and the particle field F Ｐ (t ｉ ) (i∈{1,...,N-1}) Ｐ (x, t ｉ ) and color field c Ｐ (x, d, t ｉ ) is calculated, and F Ｐ Based on (t0) F Ｐ (t ｉ ) (iε{1, . . . , N−1}) is calculated.

[0031] F Ｐ (t ｉ ) (i∈{0, 1, ..., N-1}) each is F Ｇ (t ｉ ) is converted into F Ｇ (t ｉ ) is a geometric structure represented by lattice points in Eulerian coordinates. The transformation from a geometric structure representation using particles in Lagrangian coordinates to a geometric structure representation using a grid in Eulerian coordinates is expressed by equation (8).

[0032] In formula (8), as in formula (3), F ｐ Ｐ denotes a particle-based field denoted by p, and F ｉ Ｇ denotes the field based on the lattice indicated by i, and w ｉｐ represents the weight of grid i when interpolating a particle indicated by p on the grid. Hereinafter, the conversion from a particle in Lagrangian coordinates to a grid in Eulerian coordinates will also be referred to as P2G (Particle-to-Grid).

[0033] F Ｇ (t ｉ ) are volume rendered, and C(r, t ｉ ) is calculated. The volume rendering method will be explained in detail below. The volume density field σ(x, t) in equation (1) is calculated by the lattice point field F Ｇ The volume density field σ at (t) Ｇ The color field c(x, d, t) in equation (1) is calculated using the lattice point field F Ｇ Color field c in (t) Ｇ It is calculated by using equation (10).

[0034] In equation (9), softplus is a softplus function, which is expressed by equation (11).

[0035] In Equation (9) and Equation (10), "interp" represents an interpolation process. "MLP" represents a process of applying a multi-layer perceptron. The interpolation method represented by "interp" is not particularly limited, and examples include trilinear interpolation and spline interpolation. Note that in Equation (9), any function, including an activation function such as a normalized linear function, may be applied instead of a softplus function.

[0036] The volume density field σ(x, t ｉ ) and the color field c(x, d, t ｉ ) and apply equations (1) and (2) to obtain C(r, t ｉ) is calculated.

[0037] The estimation result C(r, t ｉ ) and the measurement data C^(r,t ｉ ) is calculated. pixel is calculated, for example, by equation (12).

[0038] The loss is not limited to that calculated based on the mean square error shown in equation (12), but may be a loss based on any distance measure, such as an L1 error, an Lp error, or a hinge error.

[0039] Loss L pixel The geometric structure F Ｇ (t ０ ) is adjusted. As a result, the geometric structure F Ｇ (t ０ ) is optimized.

[0040] FIG. 2 is a diagram illustrating an optimization method according to the first embodiment. Ｇ’ (t0) is F Ｐ (t0) (G2P). Then, F Ｐ By physical simulation based on (t0), F Ｐ (t ｉ ) (i∈{1, . . . , N−1}) is calculated. Ｐ (t ｉ ) (i∈{0, 1, ..., N-1}) is F Ｇ (t ｉ ) (P2G). Then, F Ｇ (t ｉ ) is volume rendered to C(r,t ｉ ) is calculated. ｉ ) and the measurement data C^(r,t ｉ ) and the loss is calculated based on F Ｇ’ (t0) is optimized.

[0041] 3 is a flowchart showing an optimization method according to the first embodiment. The system identification device 1 first acquires measurement data through a measurement data acquisition process (step S11). The system identification device 1 then determines F Ｇ’ An initial value for (t0) is set (step S12).

[0042] The system identification device 1 performs system identification processing to Ｇ’ (t0) is F Ｐ (t0) (step S13). After that, the system identification device 1 performs the system identification process to convert F Ｐ By physical simulation based on (t0), F Ｐ (t ｉ ) (i∈{1, . . . , N−1}) (step S14). After that, the system identification device 1 calculates F Ｐ (t ｉ ) to F Ｇ (t ｉ ) (step S15). The system identification device 1 converts F Ｇ (t ｉ ) is subjected to volume rendering to obtain the estimated result C(r,t ｉ ) is calculated (step S16).

[0043] Thereafter, the system identification device 1 determines whether or not optimization has been performed a predetermined number of times through the system identification process (step S17). If optimization has not been performed a predetermined number of times (step S17: NO), the system identification device 1 performs the system identification process to optimize the measurement data C^(r,t ｉ ) and the estimated result C(r, t ｉ ) and based on F Ｇ’ (t0) is optimized (step S18). After step S18, the process returns to step S13. If optimization has been performed a predetermined number of times (step S17: YES), the operation ends.

[0044] Note that the operation may be terminated when the loss calculated from the measurement data and the estimation result becomes equal to or less than a predetermined amount. The criterion for terminating the operation is not limited to when the loss becomes equal to or less than a predetermined amount, and the operation may be terminated based on any judgment criterion using the measurement data and the estimation result. For example, the operation may be terminated when the fluctuation in loss (the difference between the loss in one time and the loss in the next time) becomes equal to or less than a predetermined amount, or when an evaluation index other than the loss (e.g., any evaluation index such as PSNR, SSIM, or LPIPS) becomes equal to or less than a predetermined amount.

[0045] The system identification process according to the first embodiment differs from the system identification method disclosed in Non-Patent Document 1 in that it calculates losses from measurement data and estimation results at different times, and then calculates a grid point field F Ｇ’ (t0) is optimized. This allows the grid point field F Ｇ’ (t0) can be optimized based on all data over time, not just the initial frame of measurement data. As a result, it is possible to compensate for the lack or deterioration of measurement data in the initial frame, thereby improving the accuracy of system identification.

[0046] Second Embodiment Fig. 4 is a diagram showing an optimization method according to a second embodiment. The system identification process according to the first embodiment is performed by optimizing the grid point field F Ｇ’ (t0), whereas the system identification process according to the second embodiment optimizes the particle field F Ｐ In the second embodiment, the loss L pixel The particle field F Ｐ (t0) is adjusted.

[0047] The system identification process is Ｇ’ (t0) is F Ｐ (t0) (G2P). Then, the system identification process is Ｐ By physical simulation based on (t0), F Ｐ (t ｉ ) (i∈{1, . . . , N−1}). Then, the system identification process calculates F Ｐ (tｉ ) (i∈{0, 1, ..., N-1}) into F Ｇ (t ｉ ) (P2G). Then, the system identification process is Ｇ (t ｉ ) by performing volume rendering on C(r, t ｉ ) is calculated. The system identification process is ｉ ) and the measurement data C^(r,t ｉ ) and calculate the loss based on the particle field F Ｐ Optimize (t0).

[0048] 5 is a flowchart showing an optimization method according to the second embodiment. The operations from step S21 to step S27 are the same as the operations from step S11 to step S17 in the first embodiment.

[0049] If optimization has not been performed a predetermined number of times (step S27: NO), the system identification process ｉ ) and the estimated result C(r, t ｉ ) and based on F Ｐ (t0) is optimized (step S28). After step S28, the process returns to step S24, and the system identification process is Ｐ Based on (t0) F Ｐ (t ｉ ) is calculated. When optimization has been performed a predetermined number of times (step S27: YES), the system identification process ends its operation. Note that, similar to the end determination in the first embodiment, the operation may be ended based on any determination criterion using the measurement data and the estimation result.

[0050] In the first embodiment, the system identification process optimizes the volume density and color of the grid points while keeping the grid point positions fixed. On the other hand, in the second embodiment, the system identification process optimizes not only the volume density and color of the particles but also the particle positions. This allows the second embodiment to directly optimize the shape of the object.

[0051] In the first embodiment, the lattice point field F Ｇ’(t0) is optimized, and in the second embodiment, the particle field F Ｐ Therefore, the first and second embodiments have in common the point that the field indicating the geometric structure of the target object A is optimized.

[0052] (Optimization Combination) In the first embodiment, the system identification process is performed by estimating the grid point field F based on the estimation results and measurement data over time. Ｇ’ In the second embodiment, the system identification process optimizes the particle field F based on the estimation results and measurement data over time. Ｐ In the system identification method described in Non-Patent Document 1, the particle field F Ｐ (t0) to F Ｐ (t ｉ ) (i∈{1, ..., N-1}) are calculated based on the estimation results and measurement data over time. These optimization methods may be combined as appropriate.

[0053] For example, the system identification process is performed by using the grid point field F Ｇ’ (t0) and physical characteristics may be optimized. In this case, the system identification process may be performed by, for example, calculating the loss and then calculating the grid point field F Ｇ’ The optimization of (t0) and the optimization of the physical properties are performed simultaneously. The system identification process is performed by using the grid point field F Ｇ’ The grid point field optimization phase for optimizing (t0) and the physical property optimization phase for optimizing the physical properties may be performed alternately. By repeating the grid point field optimization phase and the physical property optimization phase, the grid point field F Ｇ’ (t0) and the physical properties are optimized.

[0054] In the grid field optimization phase, the grid field F Ｇ’ (t0) may be optimized once or twice or more times. In the physical property optimization phase, the physical property may be optimized once or twice or more times. Therefore, in the grid point field optimization phase, the grid point field F Ｇ’After optimizing (t), the loss is calculated. Then, the phase shifts to the physical property optimization phase, and the physical property may be optimized based on the calculated loss. In the grid point field optimization phase, the grid point field F Ｇ’ (t0) may be optimized. In addition, after the physical properties are optimized in the physical property optimization phase, the loss is calculated, and then the phase shifts to the grid point field optimization phase, in which the grid point field F Ｇ’ (t0) may be optimized, and the physical properties may be optimized again in the physical property optimization phase.

[0055] Geometric structures and physical properties interact with each other. The more accurately the geometric structure can be estimated, the more accurate the estimation of physical properties will be. Conversely, the more accurately the physical properties can be estimated, the more accurate the estimation of geometric structures will be. Therefore, by repeatedly estimating the geometric structure and estimating the physical properties based on the estimated geometric structure, the estimation accuracy of the geometric structure and the physical properties will be improved in parallel.

[0056] For example, the system identification process may be performed using the particle field F Ｐ (t0) and physical properties may be optimized. In this case, the system identification process may be performed by, for example, calculating the loss and then estimating the particle field F Ｐ The optimization of (t0) and the optimization of physical properties are performed simultaneously. The system identification process is performed by Ｐ The particle field optimization phase for optimizing (t0) and the physical property optimization phase for optimizing the physical properties may be alternately performed. By repeating the particle field optimization phase and the physical property optimization phase, the particle field F Ｐ (t0) and optimize the physical properties.

[0057] In the particle field optimization phase, the particle field F Ｐ (t0) may be optimized once or twice or more times. In the physical property optimization phase, the physical property may be optimized once or twice or more times. Therefore, in the particle field optimization phase, the particle field F ＰAfter optimizing (t0), the loss is calculated. Then, the phase shifts to the physical property optimization phase, and the physical property is optimized based on the calculated loss. Then, in the particle field optimization phase, the particle field F Ｐ (t0) may be optimized. In addition, after the physical properties are optimized in the physical property optimization phase, the loss is calculated, and then the process moves to the particle field optimization phase, in which the particle field F is optimized based on the calculated loss. Ｐ Alternatively, (t0) may be optimized, or the physical properties may be optimized again in the physical property optimization phase. This method also improves the accuracy of estimating the geometric structure and the physical properties in parallel by repeatedly estimating the geometric structure and estimating the physical properties based on the estimated geometric structure.

[0058] For example, the system identification process is performed by using the grid point field F Ｇ’ Optimization of (t0) and the particle field F in the second embodiment Ｐ (t0) may be optimized. In this case, the system identification process is performed by optimizing the grid point field F Ｇ’ The grid point field optimization phase, which optimizes (t0), and the particle field F Ｐ This may be followed by an alternating particle field optimization phase that optimizes (t0).

[0059] In the grid field optimization phase, the grid field F Ｇ’ (t0) may be optimized once or more than once. In the particle field optimization phase, the particle field F Ｐ (t0) may be optimized once or twice or more. Therefore, in the grid field optimization phase, the grid field F Ｇ’ After optimizing (t0), the loss is calculated, and then the particle field optimization phase is started. Based on the calculated loss, the particle field F Ｐ (t0), and again in the grid field optimization phase, the grid field F Ｇ’ (t0) may be optimized. Also, in the particle field optimization phase, the particle field F Ｐ After optimizing (t0), the loss is calculated, and then the phase shifts to the grid point field optimization phase, where the grid point field F Ｇ’(t0), and again in the particle field optimization phase, the particle field F Ｐ (t0) may be optimized.

[0060] In addition, the particle field optimization phase is shifted to the grid point field optimization phase, and the particle field F Ｐ After optimizing (t0), the grid point field F Ｇ’ When optimizing (t0), the grid point field F Ｇ’ F calculated from (t0) Ｇ (t ｉ ) and the particle field F Ｐ F calculated from (t0) Ｇ (t ｉ ) and calculate the difference between the grid point field F Ｇ’ The grid point field F is calculated so as to minimize the loss calculated from (t0) and the difference. Ｇ’ (t0). Note that the grid point field F Ｇ’ When optimizing (t0), the grid point field F Ｇ’ C(r, t) calculated from (t0) ｉ ) and the optimized particle field F Ｐ C(r, t) calculated from (t0) ｉ ) so as to minimize the difference between the grid point field F Ｇ’ (t0) may be calculated. Ｇ’ When optimizing (t0), the grid point field F Ｇ’ F calculated from (t0) Ｐ (t0) and the optimized particle field F Ｐ (t0) to minimize the difference between the grid point field F Ｇ’ (t0) may be calculated. Note that the object of minimization is not limited to the above, and the lattice point field F Ｇ’ Any data calculated from (t0) and F Ｇ’ (t0) belongs to the same data space as the data calculated from the particle field F Ｐ The grid point field F is calculated so as to minimize the difference between the data calculated from (t0) and Ｇ’ (t0) may be calculated.

[0061] For example, the system identification process calculates the grid point field F Ｇ’ Optimization of (t0) and particle field F ＰThe optimization of (t0) may be performed simultaneously. The system identification process is performed by Ｇ’ (t0) and particle field F Ｐ (t0) simultaneously, the grid point field F Ｇ’ F calculated from (t0) Ｇ (t ｉ ) and particle field F Ｐ F calculated from (t0) Ｇ (t ｉ ) and calculate the difference between the grid point field F Ｇ’ (t0) or particle field F Ｐ C(r, t) calculated from (t0) ｉ ) and C^(r,t ｉ ) and the difference between the grid point field F Ｇ’ (t0) and particle field F Ｐ (t0) are simultaneously optimized. Ｇ’ (t0) and particle field F Ｐ (t0) simultaneously, the grid point field F Ｇ’ C(r, t) calculated from (t0) ｉ ) and particle field F Ｐ C(r, t) calculated from (t0) ｉ ) so as to minimize the difference between the grid point field F Ｇ’ (t0) and particle field F Ｐ (t0) may be calculated simultaneously. Ｇ’ (t0) and particle field F Ｐ (t0) simultaneously, the grid point field F Ｇ’ F calculated from (t0) Ｐ (t0) and particle field F Ｐ (t0) to minimize the difference between the grid point field F Ｇ’ (t0) and particle field F Ｐ (t0) may be calculated at the same time. The object of minimization is not limited to the above, and the lattice point field F Ｇ’ Any data calculated from (t0) and F Ｇ’ (t0) belongs to the same data space as the data calculated from the particle field F Ｐ The grid point field F is calculated so as to minimize the difference between the data calculated from (t0) and Ｇ’ (t0) and particle field F Ｐ(t0) may be calculated at the same time.

[0062] For example, in addition to the minimization, the system identification process may include auxiliary system identification processes, such as eliminating particles with a volume density below a certain level, constraining color learning to be independent of the viewing direction, or adjusting the learning time to prevent overlearning to specific measurement data.

[0063] For example, the system identification process may be performed in three phases: a grid field optimization phase, a particle field optimization phase, and a physical property optimization phase. Two or more of the three phases may be performed simultaneously or in a predetermined order.

[0064] When combining optimization methods, the particle field F Ｐ After optimizing (t0), the particle field F Ｐ (t0) to the lattice point field F Ｇ’ (t0) may be calculated. Ｇ’ By applying G2P to (t0), the particle field F Ｐ (t0) may be resampled. When optimizing only the particle field, the high degree of freedom in the point positions can lead to overfitting due to the measurement data, which can result in a decrease in generalization performance. By applying G2P to the particle field and then applying P2G for resampling, overfitting can be mitigated by the effects of discretization and resampling.

[0065] As described above, the estimation results C(r, t ｉ ) can be obtained. When one or more optimization methods are performed iteratively, the measurement data C^(r,t ｉ ) is the estimation result C(r, t ｉ ) may be added to the measurement data, and the system identification process and optimization may be performed again. This allows optimization to be performed based on measurement data from more perspectives.

[0066] In the above explanation, the calculated loss is ｉC(r, t ｉ ) and C^(r,t ｉ ), but the loss was based on C(r,t ｉ ) and C^(r,t ｉ ) and may be calculated and optimized based on the

[0067] In the above description, a case has been described in which time-series two-dimensional pixel data is used as measurement data, but time-series 2.5-dimensional data such as depth data collected using a depth sensor or the like, or time-series three-dimensional data such as point clouds collected using a lidar or the like may also be used. Furthermore, a combination of these data may also be used.

[0068] In the above description, it is assumed that rendering is based on a grid point field, and the particle field F Ｐ (t ｉ ) is calculated, and then P2G is used to calculate F Ｇ (t ｉ ) where, when rendering is performed based on a particle field, the particle field F Ｐ (t ｉ ) is calculated, the particle field F Ｐ (t ｉ ) is rendered to obtain the estimation result C(r, t ｉ ) may be calculated.

[0069] In the above explanation, the geometric structure is defined as the lattice point field F Ｇ’ (t0), and the lattice point field F Ｇ’ By performing G2P at (t0), the particle field F Ｐ (t0). Here, the geometric structure is transformed into the particle field F Ｐ (t0). In this case, the lattice point field F Ｇ’ (t0) and so on. Ｇ’ If no processing is performed for (t0), the grid point field F Ｇ’ (t0) does not need to be calculated. When rendering is performed based on the particle field, the lattice point field F Ｇ’ (t0) and F Ｇ (t0) is not calculated, and the particle field F Ｐ(t ｉ ) only needs to be calculated.

[0070] (Experimental Results) The experiments conducted will be described below. The target object A was composed of two Newtonian fluids (physical properties: fluid viscosity μ, bulk modulus κ), and two non-Newtonian fluids (physical properties: shear modulus μ, bulk modulus κ, yield stress τ Ｙ , plastic viscosity η), two compressible solids (physical properties: Young's modulus E, Poisson's ratio ν), two plasticine (physical properties: Young's modulus E, Poisson's ratio ν, yield stress τ Ｙ ) two, granular medium (physical properties: friction angle θ fric A total of nine objects were used, one for each object, and one for each object. Each object was measured as it fell freely under gravity and impacted the ground. Object A was rendered under various ambient lighting conditions and with various textures on the ground, and this rendered data was used as measurement data.

[0071] Target object A was photographed from 11 viewpoints. To verify the effectiveness of the inventive method in cases where it is difficult to estimate the geometric structure using only the initial frames of measurement data, optimization was performed using only videos photographed from three viewpoints. Videos photographed from the eight viewpoints not used in optimization were used to evaluate the accuracy of the optimization.

[0072] Four optimization methods were performed: a comparison method, a first method, a second method, and a third method. The comparison method is the method described in Non-Patent Document 1. In other words, the comparison method calculates the grid point field F Ｇ’ (t0) and optimizes the physical properties based on the measurement data over time. The first method is a method that adds the method according to the first embodiment to the comparison method. That is, the first method optimizes the grid point field F Ｇ’ (t0) is optimized, and the physical properties are optimized based on the measurement data over time. Then, the grid point field F Ｇ’ This is a method for optimizing (t0).

[0073] The second method is a method that adds the method according to the second embodiment to the comparison method. That is, the second method calculates the grid point field FＧ’ (t0) is optimized, and the physical properties are optimized based on the measurement data over time. Then, the particle field F Ｐ Optimize (t0).

[0074] The third method is a method of repeating the second method. In this experiment, the second method was repeated four times in the third method.

[0075] FIG. 6 shows the peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS) for each method. The larger the PSNR and SSIM values, the higher the degree of agreement between the estimation results and the measurement data. The smaller the LPIPS value, the higher the degree of agreement between the estimation results and the measurement data. FIG. 6 shows the average scores for nine different target objects A. As shown in FIG. 6, the first method outperformed the comparison method in all evaluation indices. Also, as shown in FIG. 6, the second method outperformed both the comparison method and the first method in all evaluation indices. From this, it can be seen that the grid point field F Ｇ’ (t0) and the particle field F Ｐ It can be seen that optimizing (t0) is effective in improving the accuracy of system identification.

[0076] Furthermore, the third method outperformed the comparison method, the first method, and the second method in all evaluation indices. This shows that the method of iteratively optimizing the geometric structure and physical properties is effective in improving the accuracy of system identification.

[0077] FIG. 7 is a diagram showing the accuracy of physical properties estimated by the comparative method and the third method. The difference between the actual physical properties of target object A and the estimated physical properties was calculated. The smaller the difference, the higher the accuracy of the physical property estimation. A Newtonian fluid was selected as target object A. The index used was the difference Δlog 10(μ) and the difference between the common logarithm of the bulk modulus κ, Δlog 10 (κ) was used.

[0078] For all evaluation indices, the physical properties estimated by the third method were more accurate than those estimated by the comparative method. This shows that the method of iteratively optimizing the geometric structure and physical properties is effective in improving the accuracy of physical property estimation.

[0079] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the present invention that do not deviate from the gist of the present invention.

[0080] 8 is a diagram showing an example of the hardware configuration of the system identification device 1 according to the embodiment. The system identification device 1 includes a control unit 11 having a processor 91 and a memory 92 connected by a bus, and executes a program. By executing the program, the system identification device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0081] More specifically, the processor 91 reads out a program stored in the storage unit 13 and stores the read out program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the system identification device 1 functions as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0082] The control unit 11 controls the operation of each functional unit included in the calculation device 1. The control unit 11, for example, acquires information stored in the memory unit 13. The process of acquiring information stored in the memory unit 13 is specifically reading. The control unit 11 may, for example, output various types of information to the memory unit 13. The memory unit 13 records the information output to the memory unit 13. The control unit 11, for example, acquires information acquired by the interface unit 12. The control unit 11, for example, controls the interface unit 12 to transmit the information to be transmitted to the destination.

[0083] The control unit 11 executes, for example, a measurement data acquisition process, a system identification process, and an output process.

[0084] The interface unit 12 includes a communication interface for connecting the system identification device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits network information. The interface unit 12 acquires network information by communicating with the device that transmits the network information.

[0085] The interface unit 12 may be configured to include input devices such as a mouse, a keyboard, a touch panel, etc. The interface unit 12 may be configured as an interface that connects these input devices to the computing device 1. In this way, the input devices of the interface unit 12 accept input of various information to the computing device 1 via wired or wireless connections. Note that various information such as network information does not necessarily need to be input to the communication interface of the interface unit 12, and may also be input to the input devices of the interface unit 12.

[0086] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL (Electro-Luminescence) display, or a three-dimensional display, and a speaker. The interface unit 12 may be configured as an interface that connects these display devices or speakers to the computing device 1. Therefore, the display device and speaker included in the interface unit 12 output information input to an input device of the interface unit 12, for example, as an image, a three-dimensional video, or sound.

[0087] The control unit 11 may, for example, control the operation of a display device or speaker that constitutes the interface unit 12 to output the estimation results and the like.

[0088] The storage unit 13 is configured using a non-transitory computer-readable recording medium such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the calculation device 1. The storage unit 13 stores, for example, measurement data and estimation results. The storage unit 13 may exist on, for example, a cloud.

[0089] 1 System identification device, 3 Camera, 11 Control unit, 12 Interface unit, 13 Storage unit, 91 Processor, 92 Memory

Claims

1. A system identification device that converts a lattice point field representing the geometric structure of a target object at a single time point into a particle field representing the geometric structure of the target object at said time point, performs a physical simulation on the particle field to calculate particle fields at multiple time points after said single time point, converts the particle fields at said single time point and at said multiple time points into lattice point fields, performs volume rendering on the converted lattice point field, and estimates measurement data resulting from photographing the target object at said single time point and each of said multiple time points, calculates a loss between the estimated measurement data of the target object at said single time point and each of said multiple time points and the measurement data obtained as a result of actually photographing the target object, and optimizes the field representing the geometric structure of the target object based on the loss.

2. The system identification device according to claim 1, wherein the physical properties of the target object in the physical simulation are optimized based on the loss.

3. A system identification method comprising: converting a lattice point field representing the geometric structure of a target object at a single time point into a particle field representing the geometric structure of the target object at said time point; performing a physical simulation on the particle field to calculate particle fields at multiple time points after said single time point; converting the particle fields at said single time point and at said multiple time points into lattice point fields; performing volume rendering on the converted lattice point field to estimate measurement data resulting from photographing the target object at said single time point and at each of said multiple time points; calculating a loss between the estimated measurement data of the target object at said single time point and at each of said multiple time points and the measurement data obtained as a result of actually photographing the target object; and optimizing the field representing the geometric structure of the target object based on the loss.

4. A program for causing a computer to execute the system identification method according to claim 3.

Images