Soft asset reconstruction method and device, electronic equipment, storage medium and product

Abstract

This disclosure provides a method, apparatus, electronic device, storage medium, and product for soft body asset reconstruction. The method includes: acquiring multi-view videos of a target soft body; extracting features from the first frame of each video using an aggregator in a feedforward neural network to obtain first image features of the first frame; predicting point cloud data of the target soft body using a first prediction network in the feedforward neural network based on the first image features of each first frame; obtaining a digital model of the target soft body using a second prediction network in the feedforward neural network based on the first image features and point cloud data of each first frame; extracting features from each video using the aggregator to obtain motion state features corresponding to each video; and predicting the physical parameters of the target soft body using a third prediction network in the feedforward neural network based on the motion state features corresponding to each video. This improves the efficiency and accuracy of soft body asset reconstruction.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence technology, and in particular to a method, apparatus, electronic device, storage medium, and product for soft body asset reconstruction. Background Technology

[0002] In recent years, embodied AI technology has developed rapidly. Experimenting with soft body assets in simulation environments is one of the main research methods in embodied AI technology. Therefore, a large number of soft body assets need to be reconstructed in this field. Soft body assets typically refer to digital models of soft bodies, which are deformable physical entities; soft body assets usually include parameters of the physical properties of the soft body during its deformation process.

[0003] In related technologies, a scene-by-scene optimization approach is required for soft body reconstruction. For example, in algorithms like Spring-Gauss and PhysTwin, the algorithm needs to be optimized individually for each type of soft body. When a new type of soft body is introduced, the algorithm must be optimized using a large amount of data from the new soft body before a digital model of the new soft body can be reconstructed. Furthermore, the physical property parameters of the digital model must be manually labeled to ultimately obtain soft body data for simulation experiments in embodied intelligence technology.

[0004] However, the above process requires individual optimization for each soft body, which is too time-consuming, and the accuracy and precision of manually annotated parameters are also poor, resulting in low efficiency and accuracy of soft body asset reconstruction. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure is proposed. This disclosure provides a method, apparatus, electronic device, storage medium, and product for soft-body asset reconstruction.

[0006] On one hand, embodiments of this disclosure provide a method for reconstructing soft-body assets, including:

[0007] Acquire multi-view videos of the target soft body, the multi-view videos including multiple videos captured by cameras distributed from multiple perspectives of the target soft body;

[0008] The first image features of the first frame of each video are obtained by extracting features from the first frame of the multi-view video through the aggregator in the feedforward neural network.

[0009] Based on the first image features of each first frame image, the first prediction network in the feedforward neural network is used to predict the point cloud data of the target soft body. The point cloud data represents the position information of the pixel points corresponding to the target soft body in each first frame image.

[0010] Based on the first image features of each first frame image and the point cloud data, the digital model of the target soft body is obtained by using the second prediction network in the feedforward neural network.

[0011] The aggregator extracts features from each video to obtain the motion state features corresponding to each video.

[0012] Based on the motion state features corresponding to each video, the physical parameters of the target soft body are predicted by the third prediction network in the feedforward neural network, so as to obtain soft body assets based on the digital model and physical parameters.

[0013] In one possible implementation, the third prediction network includes a first multilayer perceptron and at least one physical parameter prediction head;

[0014] The physical parameters of the target soft body are predicted using the third prediction network in the feedforward neural network based on the motion state features corresponding to each video, including:

[0015] The motion state features of multiple videos are fused along the viewpoint dimension to obtain multi-viewpoint fused features;

[0016] Based on the multi-view fusion features, compression processing is performed to obtain global context features;

[0017] The global context features are extracted by the first multilayer perceptron to obtain high-dimensional features, and the high-dimensional features are predicted by at least one physical parameter prediction head to obtain at least one first physical parameter corresponding to each mass point of the target soft body.

[0018] In one possible implementation, the third prediction network further includes a second multilayer perceptron, and the method further includes:

[0019] The second multilayer perceptron performs point-by-point dynamic attribute prediction on the multi-view fusion features to obtain the second physical parameter corresponding to each mass point of the target soft body. The second physical parameter represents the initial motion state of the mass point.

[0020] In one possible implementation, the at least one first physical parameter includes at least one of the following: the mass parameter of the mass point, the spring stiffness coefficient of the spring corresponding to the two mass points, or the frictional force parameter of the mass point.

[0021] The second physical parameter includes the initial three-dimensional velocity vector field corresponding to the particle, which includes the initial velocity corresponding to the particle.

[0022] In one possible implementation, the second prediction network includes a three-dimensional Gaussian sputtering 3DGS prediction head and an adapter module, wherein the digital model is a Gaussian model;

[0023] The process of obtaining a digital model of the target soft body using the first image features of each first frame image and the point cloud data, employing a second prediction network in a feedforward neural network, includes:

[0024] Based on the first image features of each first frame image and the point cloud data, the 3DGS prediction head is used to make a prediction to obtain the original Gaussian representation of the target soft body;

[0025] Based on the original Gaussian representation, the point cloud data, and the depth map of each first frame image, the adapter module is used for parsing to obtain the Gaussian model of the target soft body.

[0026] In one possible implementation, the first prediction network includes a camera prediction head and a depth prediction head;

[0027] The point cloud data of the target soft body is predicted by the first prediction network in the feedforward neural network based on the first image features of each first frame image, including:

[0028] The camera prediction head performs parameter prediction on the first image features of each first frame image to obtain the camera parameters of the camera corresponding to each first frame image; and the depth prediction head performs depth prediction on the first image features of each first frame image to obtain the depth map of each first frame image, wherein the depth map represents the distance from a pixel in the first frame image to the camera at the corresponding viewpoint.

[0029] For each viewpoint, based on the depth map and camera parameters of the first frame image corresponding to each viewpoint, the pixels of the target soft body in the corresponding first frame image are back-projected to obtain the local point cloud data of the target soft body corresponding to each viewpoint.

[0030] The point cloud data of the target soft body is obtained based on the local point cloud data corresponding to each viewpoint.

[0031] In another aspect, embodiments of this disclosure provide a soft-body asset reconstruction apparatus, comprising:

[0032] The video acquisition module is used to acquire multi-view videos of the target soft body, the multi-view videos including multiple videos captured by cameras distributed on the target soft body from multiple perspectives;

[0033] The first feature extraction module is used to extract features from the first frame image of the multi-view video through the aggregator in the feedforward neural network, so as to obtain the first image features of the first frame image of each video.

[0034] The first prediction module is used to predict the point cloud data of the target soft body based on the first image features of each first frame image and using the first prediction network in the feedforward neural network. The point cloud data represents the position information of the pixel points corresponding to the target soft body in each first frame image.

[0035] The second prediction module is used to obtain the digital model of the target soft body based on the first image features of each first frame image and the point cloud data, using the second prediction network in the feedforward neural network.

[0036] The second feature extraction module is used to extract features from each video through the aggregator to obtain the motion state features corresponding to each video.

[0037] The third prediction module is used to predict the physical parameters of the target soft body based on the motion state features corresponding to each video, using the third prediction network in the feedforward neural network, so as to obtain soft body assets based on the digital model and physical parameters.

[0038] In one possible implementation, the third prediction network includes a first multilayer perceptron and at least one physical parameter prediction head;

[0039] The third prediction module is used for:

[0040] The motion state features of multiple videos are fused along the viewpoint dimension to obtain multi-viewpoint fused features;

[0041] Based on the multi-view fusion features, compression processing is performed to obtain global context features;

[0042] The global context features are extracted by the first multilayer perceptron to obtain high-dimensional features, and the high-dimensional features are predicted by at least one physical parameter prediction head to obtain at least one first physical parameter corresponding to each mass point of the target soft body.

[0043] In one possible implementation, the third prediction network further includes a second multilayer perceptron, and the third prediction module is further configured to:

[0044] The second multilayer perceptron performs point-by-point dynamic attribute prediction on the multi-view fusion features to obtain the second physical parameter corresponding to each mass point of the target soft body. The second physical parameter represents the initial motion state of the mass point.

[0045] In one possible implementation, the at least one first physical parameter includes at least one of the following: the mass parameter of the mass point, the spring stiffness coefficient of the spring corresponding to the two mass points, or the frictional force parameter of the mass point.

[0046] The second physical parameter includes the initial three-dimensional velocity vector field corresponding to the particle, and the initial three-dimensional velocity vector field includes the initial velocity corresponding to the particle.

[0047] In one possible implementation, the second prediction network includes a three-dimensional Gaussian sputtering (3DGS) prediction head and an adapter module, wherein the digital model is a Gaussian model; the third prediction module is used for:

[0048] Based on the first image features of each first frame image and the point cloud data, the 3DGS prediction head is used to make a prediction to obtain the original Gaussian representation of the target soft body;

[0049] Based on the original Gaussian representation, the point cloud data, and the depth map of each first frame image, the adapter module is used for parsing to obtain the Gaussian model of the target soft body.

[0050] In one possible implementation, the first prediction network includes a camera prediction head and a depth prediction head; the first prediction module is configured to:

[0051] The camera prediction head performs parameter prediction on the first image features of each first frame image to obtain the camera parameters of the camera corresponding to each first frame image; and the depth prediction head performs depth prediction on the first image features of each first frame image to obtain the depth map of each first frame image, wherein the depth map represents the distance from a pixel in the first frame image to the camera at the corresponding viewpoint.

[0052] For each viewpoint, based on the depth map and camera parameters of the first frame image corresponding to each viewpoint, the pixels of the target soft body in the corresponding first frame image are back-projected to obtain the local point cloud data of the target soft body corresponding to each viewpoint.

[0053] The point cloud data of the target soft body is obtained based on the local point cloud data corresponding to each viewpoint.

[0054] In another aspect, embodiments of this disclosure provide an electronic device, including: a memory for storing a computer program product; and a processor for executing the computer program product stored in the memory, wherein when the computer program product is executed, the above-described soft-body asset reconstruction method is implemented.

[0055] In another aspect, embodiments of this disclosure provide a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the aforementioned soft-body asset reconstruction method.

[0056] In another aspect, embodiments of this disclosure provide a computer program product, including computer program instructions that, when executed by a processor, implement the aforementioned soft-body asset reconstruction method.

[0057] The soft body asset reconstruction method provided in this disclosure acquires multi-view videos of the target soft body and extracts features from the first frame of each video using an aggregator in a feedforward neural network to obtain first image features of the first frame. Based on the first image features of each first frame, a first prediction network in the feedforward neural network predicts the point cloud data of the target soft body. Based on the first image features of each first frame and the point cloud data, a second prediction network in the feedforward neural network obtains the digital model of the target soft body. Since the model is directly constructed using the feedforward neural network, it remains applicable to new soft body types without requiring scene-by-scene optimization algorithms. This disclosure can quickly obtain the digital model of the target soft body, greatly improving the efficiency of digital model construction. Furthermore, the aggregator extracts features from each video to obtain motion state features corresponding to each video. Based on the motion state features corresponding to each video, a third prediction network in the feedforward neural network predicts the physical parameters of the target soft body. Based on this, a soft body asset including a digital model and physical parameters can be obtained. Since no manual parameter annotation is required, and parameter prediction is performed through a feedforward neural network, the accuracy and precision of the parameters can be greatly improved, thereby improving the efficiency and accuracy of soft body asset reconstruction.

[0058] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0059] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0060] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0061] Figure 1 A flowchart of a soft-body asset reconstruction method provided in this disclosure embodiment;

[0062] Figure 2 A flowchart of another flexible asset reconstruction method provided in this disclosure embodiment;

[0063] Figure 3 A flowchart of another flexible asset reconstruction method provided in this disclosure embodiment;

[0064] Figure 4 A flowchart of another flexible asset reconstruction method provided in this disclosure embodiment;

[0065] Figure 5 A flowchart of another flexible asset reconstruction method provided in this disclosure embodiment;

[0066] Figure 6 A block diagram of a soft-body asset reconstruction device provided in an embodiment of this disclosure;

[0067] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0068] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0069] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0070] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0071] It should also be understood that in the embodiments of this disclosure, "a plurality of" can refer to two or more, and "at least one" can refer to one, two or more.

[0072] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0073] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0074] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0075] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0076] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0077] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0078] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0079] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0080] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0081] Figure 1 This is a schematic flowchart of a soft-body asset reconstruction method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, it includes the following steps:

[0082] Step 101: Obtain multi-view video of the target soft body.

[0083] The multi-view video includes multiple videos captured by cameras distributed across the target soft body from multiple perspectives.

[0084] For example, the target soft body is the soft body to be reconstructed. In this embodiment of the disclosure, the soft body asset corresponding to the target soft body is obtained by reconstructing the target soft body. The soft body asset is a digital representation of the target soft body and may include the digital model and physical parameters of the target soft body.

[0085] It should be noted that soft bodies can be deformable objects. For example, a soft body refers to a physical entity with flexible characteristics, whose materials and structure can deform, such as the flexible limbs of a soft robot or the flexible sensors in wearable devices. The physical properties of soft bodies enable them to exhibit high flexibility and adaptability when interacting with the environment, allowing them to better adapt to complex and uncertain scenarios. In the field of embodied intelligence, soft bodies can be studied through simulation experiments to achieve bio-like movement capabilities through biomimetic flexible structures or flexible actuation technologies. Therefore, in this field, it is necessary to reconstruct a large number of soft body assets for simulation experiments, and the simulation experimental data can be used as training data for robots in the field of embodied intelligence.

[0086] For example, the target soft body can be an animal such as a cat or dog, or an object such as a fluttering flag or leaves. The multiple viewpoints can include, but are not limited to, viewpoints from all directions surrounding the target soft body. For instance, a camera can be positioned in front of, behind, to the left, and to the right of the target soft body to capture images, thus obtaining videos corresponding to the front, rear, left, and right viewpoints of the target soft body. This application does not specifically limit the target soft body or the multiple viewpoints.

[0087] Step 102: Extract features from the first frame of the multi-view video using the aggregator in the feedforward neural network to obtain the first image features of the first frame of each video.

[0088] In this embodiment of the disclosure, a feedforward neural network is used for soft body asset reconstruction. For example... Figure 2 As shown, this feedforward neural network includes an Aggregator, which can input the first frame images of multi-view videos for feature extraction. For example, the first frame images of four videos corresponding to four different viewpoints can be input into the Aggregator to obtain the first image features corresponding to the four first frame images respectively.

[0089] The video includes multiple frames, and the first frame can be the first frame among the multiple frames included in the video.

[0090] Step 103: Based on the first image features of each first frame image, the first prediction network in the feedforward neural network is used to predict the point cloud data of the target soft body.

[0091] The point cloud data represents the positional information of the pixels corresponding to the target soft body in each first frame image.

[0092] In this step, the first image features of each first frame image can be input into the first prediction network. Based on the first image features of each first frame image, the first prediction network predicts the camera parameters for each viewpoint and the depth map of the first frame image for each viewpoint. Back projection is performed based on the camera parameters and depth map of each viewpoint to map the pixels of the target soft body in each first frame image to obtain point cloud data.

[0093] In one possible implementation, the first prediction network includes a camera prediction head and a depth prediction head; correspondingly, such as Figure 3 As shown, step 103 includes steps 1031 to 1033:

[0094] Step 1031: Perform parameter prediction on the first image features of each first frame image using the camera prediction head to obtain the camera parameters of the camera corresponding to each first frame image; and perform depth prediction on the first image features of each first frame image using the depth prediction head to obtain the depth map of each first frame image.

[0095] The depth map represents the distance from a pixel in the first frame image to the camera at the corresponding viewpoint. For example, the depth map may include the depth value of each pixel.

[0096] like Figure 2 As shown, the first image features of the first frame image of each viewpoint can be input into the camera prediction head (CameraHead) to predict camera parameters and obtain the intrinsic and extrinsic parameters (Extr, Intr) of the camera for that viewpoint; and the first image features of the first frame image of each viewpoint can be input into the depth prediction head (Depth Head) to predict depth and obtain the depth map corresponding to the first frame image of each viewpoint.

[0097] Step 1032: For each viewpoint, based on the depth map and camera parameters of the first frame image corresponding to each viewpoint, perform back projection processing on the pixels of the target soft body in the corresponding first frame image to obtain the local point cloud data of the target soft body corresponding to each viewpoint.

[0098] In this step, pixels belonging to the foreground portion of the first frame image can be selected based on the depth map; these are the pixels corresponding to the target soft body. Backprojection is then performed on the depth values ​​and pixel coordinates of these pixels to convert them into three-dimensional point cloud coordinates.

[0099] For example, the back projection process may include the following steps A1 to A3:

[0100] Step A1: The camera plane coordinates can be obtained by normalizing the pixel coordinates of the pixels using the following formula:

[0101]

[0102] Where, x n y n Represents the camera plane coordinates, where u and v represent pixel coordinates. K represents the camera parameters; for example, K can be a camera parameter matrix as follows:

[0103]

[0104] Among them, f x f y Indicates camera focal length; c x c y This represents the principal point, also known as the optical center.

[0105] Step A2: Based on the camera plane coordinates, obtain the three-dimensional coordinates in the camera coordinate system using the following formula:

[0106]

[0107] Among them, P c This represents the 3D coordinates of the pixel in the camera coordinate system. d(u,v) represents the depth value of the pixel (u,v). Here, the point corresponding to the pixel in the 3D camera coordinate system can be referred to as a 3D point.

[0108] Step A3: Based on camera extrinsic parameters, the 3D coordinates of a 3D point in the camera coordinate system can be converted to 3D coordinates in the world coordinate system.

[0109] Based on this, local point cloud data containing the spatial coordinates of multiple points can be obtained, including the coordinate positions of three-dimensional points in the world coordinate system.

[0110] Step 1033: Based on the local point cloud data of the target soft body corresponding to each viewpoint, obtain the point cloud data of the target soft body.

[0111] In this step, local point cloud data from multiple perspectives can be integrated to obtain point cloud data of the target soft body from multiple perspectives. For example, this can be achieved through feature stitching.

[0112] Step 104: Based on the first image features of each first frame image and the point cloud data, the second prediction network in the feedforward neural network is used to obtain the digital model of the target soft body.

[0113] The digital model is a digital representation of the target soft body, which can characterize the color, shape, position, etc. of the target soft body from multiple perspectives.

[0114] For example, the second prediction network may include a 3D Gaussian Splatting (3DGS) prediction head, and the digital model may be a 3D Gaussian model. Figure 2 As shown, the 3D Gaussian model corresponding to the target soft body can be generated through this 3DGS prediction head.

[0115] In one possible implementation, the second prediction network includes a 3D Gaussian sputtering 3DGS prediction head and an adapter module, wherein the digital model can be a Gaussian model; correspondingly, such as Figure 4 As shown, step 104 includes steps 1041 to 1042:

[0116] Step 1041: Based on the first image features of each first frame image and the point cloud data, the 3DGS prediction head is used to make a prediction to obtain the original Gaussian representation of the target soft body.

[0117] Step 1042: Based on the original Gaussian representation, the point cloud data, and the depth map of each first frame image, the adapter module is used for parsing to obtain the Gaussian model of the target soft body.

[0118] The original Gaussian representation may include scales, rotations, and spherical harmonic coefficients (SH coefficients). The scales, rotations, and SH coefficients are represented as vectors, quaternions, and matrices, respectively. It should be noted that the original Gaussian representation is a digitized form of intermediate parameters in the 3DGS workflow.

[0119] The depth map of each first frame image can be the depth map predicted by the depth prediction head for the first image features of each first frame image in step 103. That is, in step 1042, the original Gaussian representation, the point cloud data, and the depth map obtained in step 103 can be input into the adapter module.

[0120] In this step, the original Gaussian representation is parsed by the adapter module before it can be used for rendering. For example, this parsing process may include further decomposition and normalization by the adapter module. Specifically, for the scale parameter, it can be activated and scaled to a small range using the softplus (soft normal distribution, a smooth activation function) function to ensure the Gaussian is sufficiently compact. For the rotation parameter, the quaternion-based rotation is normalized to ensure its unit length, thus forming a valid rotation representation. For the spherical harmonic coefficients, they can be rearranged into an expansion of the basis functions in three dimensions for illumination-invariant color rendering. Further, based on the decomposition and normalization process, the adapter generates a Gaussian distribution model that can be directly rendered; that is, it obtains Gaussian model parameters including position data, covariance matrix, and color data. The position data can include the position information of each point in the point cloud data. The covariance matrix includes the scale and rotation parameters processed by the adapter module. The color data includes the spherical harmonic coefficients processed by the adapter module.

[0121] It should be noted that existing technologies typically only use the 3DGS algorithm for Gaussian model construction. When a new soft body type is introduced, the parameters of the 3DGS algorithm need to be optimized using a large amount of data from the new soft body to adapt them to the new soft body type. In other words, a scene-by-scene optimization approach is required for soft body reconstruction. Therefore, the current method of optimizing each soft body individually is too time-consuming and inefficient.

[0122] In this embodiment, the feedforward neural network includes an aggregator, a first prediction network, a 3DGS prediction head, and a second prediction network. The aggregator extracts features from the first frame of the multi-view video to obtain the first image features of the first frame of each video. The first prediction network predicts the point cloud data of the target soft body, and the second prediction network uses the 3DGS prediction head and adapter module to obtain the complete Gaussian model data of the target soft body. This feedforward neural network is pre-trained, so it remains applicable even to new soft body types. Therefore, this embodiment can quickly obtain the digital model of the target soft body without scene-by-scene optimization, greatly improving the efficiency of digital model construction.

[0123] Step 105: Extract features from each video using the aggregator to obtain the motion state features corresponding to each video.

[0124] In this step, videos corresponding to multiple perspectives can be input into the aggregator. The aggregator extracts the time series information of each video. That is, the aggregator extracts the time series information of multiple consecutive frames in the video to obtain features including the motion state information of the target soft body.

[0125] Step 106: Based on the motion state features corresponding to each video, the third prediction network in the feedforward neural network is used to predict the physical parameters of the target soft body, so as to obtain the soft body asset based on the digital model and physical parameters.

[0126] The physical parameters of the target flexible body are physical parameters that characterize its motion properties. For example, physical parameters may include, but are not limited to: the mass parameters of the corresponding particles, the spring stiffness coefficients of the springs between the particles, and the frictional force parameters of the particles. In this step, parameter prediction can be performed using the multilayer perceptron and physical parameter prediction head in the third prediction network. The frictional force parameter can be the coefficient of friction.

[0127] like Figure 2 As shown, the third prediction network can be based on a spring-mass system. For example, the third prediction network may include a spring-mass system prediction head (SMS Head), which is a prediction head for each physical parameter. The SMS Head predicts the various physical parameters of the target flexible body corresponding to the spring-mass system. Based on this, in the simulation experiment of embodied intelligence technology, the movement of the mass drives the movement of the 3DGS Gaussian model, thereby visually satisfying the physical credibility.

[0128] In one possible implementation, the third prediction network includes a first multilayer perceptron and at least one physical parameter prediction head; correspondingly, such as Figure 5 As shown, step 106 includes steps 1061 to 1063:

[0129] Step 1061: Perform feature fusion on the motion state features of multiple videos in the viewpoint dimension to obtain multi-viewpoint fused features.

[0130] In this step, the motion state features of multiple videos can be averaged and pooled along the viewpoint dimension to obtain a cross-viewpoint fused feature representation, which is a multi-viewpoint fused feature.

[0131] For example, the motion state features output by the Aggregator can be in the form of a list of corresponding tokens (units). This token list includes multiple dimensions of information, such as [batch_size, num_views, n_tokens, dim_in]. Here, dim_in represents the visual features of the corresponding region; num_views represents the viewpoint dimension, with different values ​​for num_views depending on the viewpoint; batch_size represents the batch size. n_tokens represents the spatial dimension and can characterize the number of blocks in the image segmentation; for example, the image can be segmented into multiple patches, each patch corresponding to its own token list. The token list information for each patch represents a local feature unit of the image.

[0132] In this step, cross-view information fusion can be achieved through average pooling in the num_views dimension.

[0133] Step 1062: Based on the multi-view fusion feature, perform compression processing to obtain global context features.

[0134] In this step, to obtain the global physical parameters that control the overall dynamic behavior of the target soft body, the feature vectors of the multi-view fusion features can be averaged in the spatial dimension to further compress them into a global context feature vector. For example, the token list information of all patches can be averaged in the n_tokens spatial dimension to achieve global feature compression.

[0135] Based on this, by fusing the multi-dimensional information in the feature vectors of multi-view fusion features, the information represented by the feature vectors becomes denser, which helps to improve the computational accuracy of the subsequent prediction process.

[0136] Step 1063: Extract features from the global context features using the first multilayer perceptron to obtain high-dimensional features, and predict the high-dimensional features using at least one physical parameter prediction head to obtain at least one first physical parameter corresponding to each mass point of the target soft body.

[0137] In this step, the global context features can be input into an MLP (Multilayer Perceptron). The MLP extracts motion characteristic information from the global context features to transform the original visual features into a high-dimensional representation related to dynamics, that is, high-dimensional features.

[0138] The first physical parameter may include, but is not limited to, at least one of the following: the mass parameter of a particle, the stiffness coefficient of the spring corresponding to the two particles, or the frictional force parameter of the particles. Correspondingly, at least one physical parameter prediction head may correspond to, but is not limited to, at least one of the following: a mass prediction head, a spring stiffness prediction head, and a frictional force prediction head. Taking a first physical parameter comprising three parameters as an example, in this step, the mass prediction head, the spring stiffness prediction head, and the frictional force prediction head can be used respectively for parameter prediction. That is, the high-dimensional features are input into the mass prediction head, the spring stiffness prediction head, and the frictional force prediction head respectively to obtain the mass parameter of each particle, the spring stiffness coefficient of the corresponding spring, and the frictional force parameter of the particle.

[0139] In one possible implementation, the third prediction network further includes a second multilayer perceptron, and step 106 further includes the following step 1064:

[0140] Step 1064: The second multilayer perceptron performs point-by-point dynamic attribute prediction on the multi-view fusion feature to obtain the second physical parameter corresponding to each mass point of the target soft body. The second physical parameter represents the initial motion state of the mass point.

[0141] In this step, the multi-view fusion features can be input into an MLP, and the MLP can be used to extract features and predict the motion state of each patch of the image point by point to obtain the second physical parameters.

[0142] In one possible approach, the second physical parameter includes an initial three-dimensional velocity vector field corresponding to the particle, which includes the initial velocity of the particle. For example, it includes the initial velocity of each particle in the x, y, and z axes of the world coordinate system.

[0143] In the third prediction network, a dual-path parallel prediction approach can be used. The dual paths include a global physical parameter regression path and a point-by-point dynamic attribute prediction path. The global physical parameter regression path uses the first multilayer perceptron and the physical parameter prediction head to predict the mass parameters of the particle, the corresponding spring stiffness coefficient, and the friction parameters of the particle; this corresponds to steps 1062 to 1063 above. The point-by-point dynamic attribute prediction path uses the second multilayer perceptron to predict the initial three-dimensional velocity vector field of the particle; this corresponds to step 1064 above.

[0144] It should be noted that the two paths are executed in parallel. Specifically, in step 106, steps 1062 to 1063 and step 1064 can be executed in parallel. That is, step 1061 is executed first. Then, steps 1062 to 1063 can be executed using the multi-view fusion features of step 1061; at the same time, step 1064 can be executed in parallel using the multi-view fusion features of step 1061.

[0145] It should be noted that the point mass and physical parameters together constitute the complete spring-mass system corresponding to the target flexible body. For example, two points mass are connected by a spring to form a dynamic system. The initial velocity of the point mass determines its trajectory, and the mass parameters of the point mass determine the inertia of the target flexible body. By obtaining the spring-mass system corresponding to the target flexible body, the motion process of the target flexible body in the real world can be simulated in simulation experiments.

[0146] In this system, each image patch corresponds to a potential mass point. A second multilayer perceptron (MLP) maps the patch features to an initial velocity vector. The required number of mass points can be represented by `n_points`. When `n_points` is less than the number of patches, multiple patches in the image can be uniformly sampled to obtain the required mass points. When `n_points` is greater than the number of patches, the second MLP can first predict the velocity of all patches. Then, the physical parameter values ​​of the mass points can be zero-padded, or a linear interpolation algorithm can be used to calculate the physical parameters of the additional mass points beyond the number of patches.

[0147] In this embodiment, the feedforward neural network can be a pre-trained neural network using a large number of samples. This feedforward neural network can learn knowledge about different types of soft bodies and still maintains good performance for new soft body types. Therefore, in the online application phase, it can quickly handle the digital model construction process of new soft bodies and can rapidly and accurately predict the physical parameters of new soft bodies.

[0148] For example, the feedforward neural network can adopt a Transformer structure, such as an aggregator, camera prediction head, depth prediction head, 3DGS prediction head, or physical parameter prediction head.

[0149] The reconstruction of soft body assets in this embodiment includes the construction of a digital model of the target soft body and the prediction of the physical parameters of the target soft body.

[0150] like Figure 2 As shown, after predicting the SMS physical parameters through the Spring-Mass System Prediction Head (SMS head), a soft body asset for simulation in the field of embodied intelligence can be constructed based on the Gaussian model of the target soft body and the physical parameters.

[0151] It should be noted that in existing technologies, physical parameters need to be manually calibrated, but the accuracy and precision of manual calibration are often very low. In this embodiment of the present disclosure, the physical parameters of the target soft body are predicted by a third prediction network in a feedforward neural network. That is, the neural network replaces manual calibration, which can greatly improve the accuracy and precision of the physical parameters.

[0152] The soft body asset reconstruction method provided in this disclosure acquires multi-view videos of the target soft body and extracts features from the first frame of each video using an aggregator in a feedforward neural network to obtain first image features of the first frame. Based on the first image features of each first frame, a first prediction network in the feedforward neural network predicts the point cloud data of the target soft body. Based on the first image features of each first frame and the point cloud data, a second prediction network in the feedforward neural network obtains the digital model of the target soft body. Since the model is built directly through the feedforward neural network, it remains applicable to new soft body types without requiring scene-by-scene optimization, thus improving the efficiency of digital model construction. Furthermore, the aggregator extracts features from each video to obtain motion state features corresponding to each video. Based on the motion state features corresponding to each video, a third prediction network in the feedforward neural network predicts the physical parameters of the target soft body. Based on this, a soft body asset including a digital model and physical parameters can be obtained. Since no manual parameter annotation is required, and parameter prediction is performed through a feedforward neural network, the accuracy and precision of the parameters can be greatly improved, thereby improving the efficiency and accuracy of soft body asset reconstruction.

[0153] Figure 6 A block diagram of a soft-body asset reconstruction apparatus provided in an embodiment of this disclosure is shown. Figure 6 As shown, the device includes:

[0154] The video acquisition module 601 is used to acquire multi-view videos of the target soft body, which include multiple videos captured by cameras distributed at multiple viewpoints of the target soft body.

[0155] The first feature extraction module 602 is used to extract features from the first frame image of the multi-view video through the aggregator in the feedforward neural network to obtain the first image features of the first frame image of each video.

[0156] The first prediction module 603 is used to predict the point cloud data of the target soft body based on the first image features of each first frame image and using the first prediction network in the feedforward neural network. The point cloud data represents the position information of the pixel points corresponding to the target soft body in each first frame image.

[0157] The second prediction module 604 is used to obtain the digital model of the target soft body based on the first image features of each first frame image and the point cloud data, using the second prediction network in the feedforward neural network.

[0158] The second feature extraction module 605 is used to extract features from each video through the aggregator to obtain the motion state features corresponding to each video.

[0159] The third prediction module 606 is used to predict the physical parameters of the target soft body based on the motion state features corresponding to each video using the third prediction network in the feedforward neural network, so as to obtain soft body assets based on the digital model and physical parameters.

[0160] In one possible implementation, the third prediction network includes a first multilayer perceptron and at least one physical parameter prediction head;

[0161] This third prediction module is used for:

[0162] The motion state features of multiple videos are fused along the viewpoint dimension to obtain multi-viewpoint fused features;

[0163] Based on this multi-view fusion feature, compression processing is performed to obtain global context features;

[0164] The first multilayer perceptron extracts features from the global context features to obtain high-dimensional features, and then uses at least one physical parameter prediction head to predict the high-dimensional features to obtain at least one first physical parameter corresponding to each mass point of the target soft body.

[0165] In one possible implementation, the third prediction network further includes a second multilayer perceptron, and the third prediction module is further configured to:

[0166] The second multilayer perceptron performs point-by-point dynamic attribute prediction on the multi-view fusion features to obtain the second physical parameter corresponding to each mass point of the target soft body. The second physical parameter represents the initial motion state of the mass point.

[0167] In one possible implementation, the at least one first physical parameter includes at least one of the following: the mass parameter of the mass, the spring stiffness coefficient of the spring corresponding to the two mass points, or the frictional force parameter of the mass.

[0168] The second physical parameter includes the initial three-dimensional velocity vector field corresponding to the particle, which includes the initial velocity corresponding to the particle.

[0169] In one possible implementation, the second prediction network includes a 3D Gaussian sputtering 3DGS prediction head and an adapter module; the third prediction module is used for:

[0170] Based on the first image features of each first frame image and the point cloud data, the 3DGS prediction head is used to make a prediction to obtain the original Gaussian representation of the target soft body.

[0171] Based on the original Gaussian representation, the point cloud data, and the depth map of each first frame image, the adapter module is used for parsing to obtain the Gaussian model of the target soft body.

[0172] In one possible implementation, the first prediction network includes a camera prediction head and a depth prediction head; the first prediction module is used for:

[0173] The camera prediction head performs parameter prediction on the first image features of each first frame image to obtain the camera parameters of the camera corresponding to each first frame image; and the depth prediction head performs depth prediction on the first image features of each first frame image to obtain the depth map of each first frame image, which represents the distance from the pixel in the first frame image to the camera at the corresponding viewpoint.

[0174] For each viewpoint, based on the depth map and camera parameters of the first frame image corresponding to each viewpoint, the pixels of the target soft body in the corresponding first frame image are back-projected to obtain the local point cloud data of the target soft body corresponding to each viewpoint.

[0175] Based on the local point cloud data corresponding to each viewpoint of the target soft body, the point cloud data of the target soft body is obtained.

[0176] The soft body asset reconstruction apparatus provided in this disclosure acquires multi-view videos of a target soft body and extracts features from the first frame of each video using an aggregator in a feedforward neural network to obtain first image features of the first frame. Based on the first image features of each first frame, a first prediction network in the feedforward neural network predicts point cloud data of the target soft body. Based on the first image features of each first frame and the point cloud data, a second prediction network in the feedforward neural network obtains a digital model of the target soft body. Since the model is built directly through the feedforward neural network, it remains applicable to new soft body types without requiring scene-by-scene algorithm optimization, thus improving the efficiency of digital model construction. Furthermore, the aggregator extracts features from each video to obtain motion state features corresponding to each video. Based on the motion state features corresponding to each video, a third prediction network in the feedforward neural network predicts the physical parameters of the target soft body. Based on this, a soft body asset including a digital model and physical parameters can be obtained. Since no manual parameter annotation is required, and parameter prediction is performed through a feedforward neural network, the accuracy and precision of the parameters can be greatly improved, thereby improving the efficiency and accuracy of soft body asset reconstruction.

[0177] Any of the soft body asset reconstruction methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the soft body asset reconstruction methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the soft body asset reconstruction methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0178] Below, for reference Figure 7 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0179] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0180] like Figure 7 As shown, the electronic device includes one or more processors and memory.

[0181] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0182] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the sound source localization methods of the various embodiments of this disclosure described above and / or other desired functions.

[0183] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0184] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0185] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0186] Of course, for the sake of simplicity, Figure 7 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0187] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the soft body asset reconstruction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0188] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0189] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the soft-body asset reconstruction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0190] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0191] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0192] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0193] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0194] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0195] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0196] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0197] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for reconstructing soft-body assets, characterized in that, include: Acquire multi-view videos of the target soft body, the multi-view videos including multiple videos captured by cameras distributed from multiple perspectives of the target soft body; The first image features of the first frame of each video are obtained by extracting features from the first frame of the multi-view video through the aggregator in the feedforward neural network. Based on the first image features of each first frame image, the first prediction network in the feedforward neural network is used to predict the point cloud data of the target soft body. The point cloud data represents the position information of the pixel points corresponding to the target soft body in each first frame image. Based on the first image features of each first frame image and the point cloud data, the digital model of the target soft body is obtained by using the second prediction network in the feedforward neural network. The aggregator extracts features from each video to obtain the motion state features corresponding to each video. Based on the motion state features corresponding to each video, the third prediction network in the feedforward neural network is used to predict the physical parameters of the target soft body, so as to obtain soft body assets based on digital models and physical parameters; The first prediction network includes a camera prediction head and a depth prediction head. The point cloud data of the target soft body is predicted using the first prediction network in a feedforward neural network based on the first image features of each first frame image, including: The camera prediction head performs parameter prediction on the first image features of each first frame image to obtain the camera parameters of the camera corresponding to each first frame image; and the depth prediction head performs depth prediction on the first image features of each first frame image to obtain the depth map of each first frame image, wherein the depth map represents the distance from a pixel in the first frame image to the camera at the corresponding viewpoint. For each viewpoint, based on the depth map and camera parameters of the first frame image corresponding to each viewpoint, the pixels of the target soft body in the corresponding first frame image are back-projected to obtain the local point cloud data of the target soft body corresponding to each viewpoint. Based on the local point cloud data of the target soft body corresponding to each viewpoint, the point cloud data of the target soft body is obtained; The third prediction network includes a first multilayer perceptron and at least one physical parameter prediction head. The step of predicting the physical parameters of the target soft body using the third prediction network in the feedforward neural network based on the motion state features corresponding to each video includes: The motion state features of multiple videos are fused along the viewpoint dimension to obtain multi-viewpoint fused features; Based on the multi-view fusion features, compression processing is performed to obtain global context features; The global context features are extracted by the first multilayer perceptron to obtain high-dimensional features, and the high-dimensional features are predicted by at least one physical parameter prediction head to obtain at least one first physical parameter corresponding to each mass point of the target soft body. The at least one first physical parameter includes at least one of the following: the mass parameter of the mass point, the spring stiffness coefficient of the spring corresponding to the two mass points, or the friction parameter of the mass point.

2. The method according to claim 1, characterized in that, The third prediction network further includes a second multilayer perceptron, and the method further includes: The second multilayer perceptron performs point-by-point dynamic attribute prediction on the multi-view fusion features to obtain the second physical parameter corresponding to each mass point of the target soft body. The second physical parameter represents the initial motion state of the mass point.

3. The method according to claim 2, characterized in that, The at least one first physical parameter includes at least one of the following: the mass parameter of the particle, the spring stiffness coefficient of the spring corresponding to the two particles, or the frictional force parameter of the particle. The second physical parameter includes the initial three-dimensional velocity vector field corresponding to the particle, and the initial three-dimensional velocity vector field includes the initial velocity corresponding to the particle.

4. The method according to claim 1, characterized in that, The second prediction network includes a 3D Gaussian sputtering 3DGS prediction head and an adapter module, wherein the digital model is a Gaussian model; The process of obtaining a digital model of the target soft body using the first image features of each first frame image and the point cloud data, employing a second prediction network in a feedforward neural network, includes: Based on the first image features of each first frame image and the point cloud data, the 3DGS prediction head is used to make a prediction to obtain the original Gaussian representation of the target soft body; Based on the original Gaussian representation, the point cloud data, and the depth map of each first frame image, the adapter module is used for parsing to obtain the Gaussian model of the target soft body.

5. A flexible asset reconstruction device, characterized in that, include: The video acquisition module is used to acquire multi-view videos of the target soft body, the multi-view videos including multiple videos captured by cameras distributed on the target soft body from multiple perspectives; The first feature extraction module is used to extract features from the first frame image of the multi-view video through the aggregator in the feedforward neural network, so as to obtain the first image features of the first frame image of each video. The first prediction module is used to predict the point cloud data of the target soft body based on the first image features of each first frame image and using the first prediction network in the feedforward neural network. The point cloud data represents the position information of the pixel points corresponding to the target soft body in each first frame image. The second prediction module is used to obtain the digital model of the target soft body based on the first image features of each first frame image and the point cloud data, using the second prediction network in the feedforward neural network. The second feature extraction module is used to extract features from each video through the aggregator to obtain the motion state features corresponding to each video. The third prediction module is used to predict the physical parameters of the target soft body based on the motion state features corresponding to each video, using the third prediction network in the feedforward neural network, so as to obtain soft body assets based on the digital model and physical parameters. The first prediction network includes a camera prediction head and a depth prediction head; the first prediction module is used for: The camera prediction head performs parameter prediction on the first image features of each first frame image to obtain the camera parameters of the camera corresponding to each first frame image; and the depth prediction head performs depth prediction on the first image features of each first frame image to obtain the depth map of each first frame image, wherein the depth map represents the distance from a pixel in the first frame image to the camera at the corresponding viewpoint. For each viewpoint, based on the depth map and camera parameters of the first frame image corresponding to each viewpoint, the pixels of the target soft body in the corresponding first frame image are back-projected to obtain the local point cloud data of the target soft body corresponding to each viewpoint. Based on the local point cloud data of the target soft body corresponding to each viewpoint, the point cloud data of the target soft body is obtained; The third prediction network includes a first multilayer perceptron and at least one physical parameter prediction head; the third prediction module is used for: The motion state features of multiple videos are fused along the viewpoint dimension to obtain multi-viewpoint fused features; Based on the multi-view fusion features, compression processing is performed to obtain global context features; The global context features are extracted by the first multilayer perceptron to obtain high-dimensional features, and the high-dimensional features are predicted by at least one physical parameter prediction head to obtain at least one first physical parameter corresponding to each mass point of the target soft body. The at least one first physical parameter includes at least one of the following: the mass parameter of the mass point, the spring stiffness coefficient of the spring corresponding to the two mass points, or the friction parameter of the mass point.

6. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-4.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-4.

8. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-4.

Images