Based on 3D parametric reconstruction and prediction methods and electronic devices
By using unified topology modeling and dimensionality reduction techniques and deep learning with a small number of multi-view images to extract shape basis vectors, the problem of device and data dependence in existing technologies is solved, and efficient and interpretable 3D shape modeling and physical parameter prediction are achieved, which is suitable for real-time interaction in AR/VR and robots.
Patent Information
- Application Number
- CN202510381805.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-03-28
AI Technical Summary
Existing 3D model reconstruction methods rely on expensive equipment, large amounts of data, or complex algorithms, making it difficult to achieve efficient and interpretable 3D shape modeling and physical parameter prediction under limited data conditions.
By using unified topology modeling and dimensionality reduction techniques, deep learning is performed on a small number of multi-view images to extract shape basis vectors, reconstruct 3D models and predict physical parameters. Dimensionality reduction methods such as PCA, LLE or Isomap are used to map high-dimensional meshes to low-dimensional shape parameter spaces.
It enables efficient and interpretable 3D shape modeling and physical parameter prediction under limited data conditions, reducing the dependence on expensive equipment and massive amounts of data, and is suitable for AR/VR, online inspection and real-time robot interaction scenarios.
Smart Images

Figure CN120318455B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of 3D model reconstruction technology, specifically relating to a 3D parametric reconstruction and prediction technology. Background Technology
[0002] 3D model reconstruction has wide applications in computer vision, graphics, industrial design, robotics, and other fields. In existing technologies, 3D model reconstruction and physical parameter prediction mainly involve capturing high-quality images, constructing 3D models from these images, and then predicting the physical parameters (such as length, width, and height) of objects in the images based on the 3D models. The key to this type of method is the construction of a 3D model from a 2D image.
[0003] Existing 3D model reconstruction methods mainly include 3D scanning technology, multi-view stereo vision technology, and Neural Radiation Field (NeRF) methods, but they still have many limitations. 3D scanning technology can typically generate point cloud data directly, but it relies on expensive and complex laser or structured light equipment to acquire the 3D coordinates of the object's surface. Furthermore, the scanning process requires collecting a large number of 3D coordinates from the object's surface to form a dense point cloud, making large-scale, low-cost deployment difficult. Multi-view stereo vision technology reconstructs the same scene from multiple perspectives using multiple cameras, reconstructing the 3D structure of the scene represented by dense or sparse point clouds based on image feature matching. This technology has high requirements for image quality, viewpoint coverage, and matching algorithms, and is computationally intensive and time-consuming, making it difficult to meet the demands for real-time performance and high efficiency. While NeRF methods can generate high-quality 3D reconstructions and synthesize new perspectives, they are highly dependent on training data and computational resources, making them difficult to implement effectively in scenarios with limited data. Furthermore, existing deep learning-based 3D model reconstruction methods rely on a large amount of data for deep learning, and many deep learning models have opaque internal mechanisms, making it difficult to clearly explain shape changes, which is not conducive to understanding and application. Summary of the Invention
[0004] In view of this, the present invention proposes a 3D parametric reconstruction and prediction method. By performing unified topological modeling on objects and using a small number of multi-view images for deep learning, it can achieve 3D shape modeling and physical parameter prediction on various objects (including animals, plants, and inanimate objects) under limited data conditions. It features high modeling efficiency, low-dimensional representation of 3D shapes, interpretability of shape changes, and low cost. Furthermore, the present invention also provides an electronic device and a computer-readable storage medium for implementing the above method.
[0005] The technical solution of the present invention is as follows:
[0006] The first aspect of this invention discloses a 3D parametric reconstruction and prediction method, comprising the following steps:
[0007] S1: Acquire 2D images of the object under test from one or more perspectives;
[0008] S2: Input the 2D image into the target parameter prediction model to obtain the shape parameter vector of the object to be tested;
[0009] S3: Based on the shape parameter vector and the pre-extracted shape basis vector of the object to which the object belongs, reconstruct the 3D model of the object to be tested;
[0010] S4: Perform geometric analysis on the 3D model of the object under test to obtain the key physical parameters of the object under test;
[0011] The shape basis vectors are extracted using dimensionality reduction techniques and are used to characterize the main deformation patterns of the object to which the test object belongs.
[0012] As an optional approach, the shape basis vectors have k elements, each representing the main deformation pattern of an object belonging to a certain category; the k shape basis vectors constitute a basis vector space, which can interpret the deformation data of objects belonging to a certain category of the object to be tested at a preset ratio or higher; the shape parameter vectors have k projection coefficients, which are obtained by projection based on the basis vector space.
[0013] As an alternative, the dimensionality reduction techniques include linear principal component analysis (PCA), nonlinear local linear embedding (LLE), and nonlinear isomap.
[0014] As an optional approach, step S2, the process of constructing the target parameter prediction model, includes the following steps:
[0015] S21: Provide multiple target objects belonging to the same category as the analyte;
[0016] S22: Acquire 2D images of each target object from multiple perspectives, wherein the 2D images can reflect the size and shape characteristics of the target object;
[0017] S23: Measure each target object separately to obtain physical parameters that reflect the size and shape characteristics of the target object;
[0018] S24: Create training data based on the 2D images and physical parameters, wherein the training data includes 2D images and shape parameter vectors corresponding to each target object;
[0019] S25: Construct an initial parameter prediction model that can fuse features from multiple perspectives;
[0020] S26: Train the initial parameter prediction model using the training dataset to obtain the target parameter prediction model.
[0021] As an optional approach, step S24, the process of creating training data, includes:
[0022] A1: Select one of the target objects and establish a baseline 3D model based on the 2D image and physical parameters of the selected target object;
[0023] A2: Based on the 2D images and physical parameters of other target objects besides the selected target object, the base 3D model is deformed to obtain a corresponding derived 3D model. The derived 3D model has the same mesh topology as the base 3D model.
[0024] A3: Based on the coordinate differences of vertices at corresponding positions between each derived 3D model and the baseline 3D model, k shape basis vectors of the target object are extracted using dimensionality reduction techniques, and the k shape basis vectors constitute the basis vector space;
[0025] A4: Based on the derived 3D model and the basis vector space, the shape parameter vector of the derived 3D model is obtained by projection.
[0026] A5. Use the 2D images and shape parameter vectors of each target object as training data.
[0027] As an optional approach, the dimensionality reduction technique is principal component analysis; step A3 includes:
[0028] A31: Calculate the coordinate differences of vertices at corresponding positions in each derived 3D model and the base 3D model to obtain multiple sets of high-dimensional difference vectors;
[0029] A32: Pull multiple sets of high-dimensional difference vectors into one dimension by row or column respectively, forming multiple sets of 3N-dimensional one-dimensional difference vectors;
[0030] A33: Perform PCA statistical analysis on the obtained difference vectors to extract the shape basis vector b of the target object. j A basis vector space; the basis vector space contains k basis vectors of shape b. j
[0031] The projection method in step A4 is as follows:
[0032] The one-dimensional difference vectors of each 3D model are projected onto the basis vector space, and the shape parameter vector β of each 3D model is calculated respectively; the 3D models include derived 3D models and newly created 3D models; the shape parameter vector β contains k projection coefficients β. j .
[0033] As an optional approach, step A2 involves deforming the reference 3D model, specifically by editing the reference 3D model using modeling software while ensuring that the total number of triangular facets constituting the model and the total number of vertices of the triangular facets remain unchanged.
[0034] As an optional approach, in step S2, the initial parameter prediction model includes an input layer, a feature extraction unit, a feature fusion unit, a shape parameter regression unit, and an output layer. The input layer receives 2D images from multiple perspectives. The feature extraction unit extracts features from the 2D images. The feature fusion unit merges the extracted image features to obtain overall shape features. The shape parameter regression unit maps the overall features to a shape parameter vector. The output layer outputs the shape parameter vector. During model training, the model weight parameters are continuously optimized until the model loss function converges or the number of iterations reaches a preset value, at which point the target model is obtained. The model loss function is the mean squared error between the predicted shape parameter vector and the actual shape parameter vector in the training set.
[0035] As an optional approach, in step S3, the 3D model of the object under test is linearly reconstructed based on the following formula:
[0036]
[0037] In the formula, M(β) is the 3D model of the test object; b. A 3D baseline model of the object belonging to the category to which the object under test belongs; j Let β be the j-th shape basis vector of the target object; j is the projection coefficient of the object under test on the j-th shape basis vector.
[0038] As an optional approach, step S4 specifically includes: importing the 3D model of the object under test into a preset library function to calculate the key physical parameters of the object under test; the preset library function includes at least one of Meshlab, Blender, and Open3D.
[0039] A second aspect of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect of the present invention or any alternative embodiment thereof.
[0040] A third aspect of the present invention discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect of the present invention or any alternative embodiment thereof.
[0041] The present invention has the following beneficial effects:
[0042] (1) This invention establishes a 3D model with a unified topological structure (referred to as "unified mesh") and combines it with the PCA statistical method to map the 3D mesh deformation of the target object to a low-dimensional shape parameter space, thereby providing a compact description of the object's shape and achieving 3D parametric modeling. This allows for the prediction of the object's physical parameters (volume, weight, geometric dimensions, etc.) even with limited data. Compared with traditional large-scale voxel or point cloud methods, this invention can reconstruct a complete 3D model by regressing a small number of parameters. This low-dimensional parametric approach significantly improves computational efficiency and enhances the interpretability of shape changes.
[0043] (2) The present invention pre-establishes a 3D model (unified mesh) with a unified topology, ensuring that all object deformations are performed on the same number of triangles and triangle vertices (mesh vertices), without the need for frequent topology rearrangement or reliance on non-rigid registration.
[0044] (3) This invention extracts the main deformation patterns (i.e., shape parameter vectors) by performing principal component analysis on mesh deformation in the training dataset, reducing the high-dimensional 3D mesh (tens of thousands of vertices) to a low-dimensional shape parameter space (tens or hundreds of dimensions). This achieves shape space dimensionality reduction, and the shape of the new object can be regarded as a linear combination of these patterns, greatly simplifying the representation and processing of high-dimensional 3D data. It has the advantage of strong real-time performance and is very suitable for AR / VR, online detection, or real-time robot interaction scenarios. Furthermore, the dimensionality-reduced shape parameter space is more robust to noise and can also prevent the network from overfitting to singular shapes.
[0045] (4) This invention has the advantages of high computational efficiency, strong interpretability and adaptability to various object shapes. It overcomes the dependence of existing technologies on expensive equipment, large amounts of data or complex algorithms, and provides an efficient and universal solution for various 3D modeling and parameter prediction tasks (such as animal and plant morphology evaluation, industrial parts inspection, robot interaction, etc.). It is very practical.
[0046] (5) The 3D parametric reconstruction based on unified topology modeling in this invention not only meets the purpose of this invention for physical parameter prediction, but also has derivative value. It can be directly applied to subsequent tasks such as rendering, measurement, animation, collision detection, and physical simulation, and can be used to interpolate or regress other physical quantities in the same space. Compared with point clouds or implicit representations, which require additional mesh reconstruction, this greatly reduces the workload. This also makes the invention more scalable and industrially valuable in practical 3D scene applications. Attached Figure Description
[0047] Figure 1 This is a flowchart of the 3D parametric reconstruction and prediction method in Example 1;
[0048] Figure 2 This is a flowchart illustrating the method for constructing the target parameter prediction model in Example 2.
[0049] Figure 3 This is a schematic diagram of the process for creating model training data in Example 3;
[0050] Figure 4 This is a schematic diagram of the process for shape basis vector extraction and shape parameter vector calculation in Example 4;
[0051] Figure 5 A schematic diagram of the mesh for the baseline mushroom model;
[0052] Figure 6 A schematic diagram of a partially derived mushroom model with a unified topological structure;
[0053] Figure 7 This is a schematic diagram of the neural network model structure;
[0054] Figure 8 This is a schematic diagram of 3D shape reconstruction and parameter extraction. Detailed Implementation
[0055] This invention is based on the idea of "parametric model + unified mesh + PCA dimensionality reduction". By combining the modeler's unified topological modeling of the object and the deep learning of a small number of multi-view images through neural networks, it can achieve high-precision three-dimensional (3D) shape modeling and physical parameter prediction of the object in the case of limited data, and realize interpretable low-dimensional representation of shape changes.
[0056] The technical solution of the present invention will now be described in more detail with reference to specific embodiments and accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.
[0057] For ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, embodiments and features described herein can be combined with each other.
[0058] like Figure 1 As shown, Embodiment 1 of the present invention discloses a 3D parametric reconstruction and prediction method, which mainly includes the following steps:
[0059] Step 1: Acquire 2D images of the target object from one or more viewpoints;
[0060] Step 2: Input the 2D image into the target parameter prediction model to obtain the shape parameter vector of the target object to be tested;
[0061] Step 3: Reconstruct the 3D model of the target object based on the shape parameter vector and the pre-extracted shape basis vector of the object belonging to the category of the target object;
[0062] Step 4: Perform geometric analysis on the 3D model of the target object to obtain the key physical parameters of the target object.
[0063] The shape basis vectors are extracted using dimensionality reduction techniques and are used to reflect the deformation patterns of the object belonging to the category of the target object.
[0064] Understandably, the shape basis vector b extracted through dimensionality reduction techniques... j There are k basis vectors, where k is a natural number greater than or equal to 1. Each extracted shape basis vector b... j The representative deformation pattern is also the main deformation pattern among the many deformation patterns of the object to which the test object belongs. In other words, k deformation patterns can explain more than a preset proportion of the deformation data of the object to which the test object belongs, usually reaching 90% or even higher. k shape basis vectors b j The basis vector space is formed. The shape parameter vector has k projection coefficients β. j Based on the basis vector space, it can be obtained through projection.
[0065] It is worth noting that the dimensionality reduction techniques used in this invention may include linear principal component analysis (PCA), or nonlinear local linear embedding (LLE) or nonlinear isomap, and this invention does not limit these techniques.
[0066] Furthermore, such as Figure 2 As shown in Embodiment 2 of the present invention, a method for constructing a target parameter prediction model is disclosed, which mainly includes the following steps:
[0067] Step 21. Multi-view image data acquisition.
[0068] Provide multiple target objects belonging to the same category, and take multiple photos of each target object from multiple angles to obtain multiple two-dimensional (2D) images from different perspectives. The obtained 2D images should cover the overall size and shape features of the target object as much as possible.
[0069] In one specific embodiment, the target object is a single mushroom (referred to as "mushroom"). Because the target object has a relatively simple structure, 30 mushrooms can be selected for photography. The camera is used to photograph the 30 mushrooms from various angles, including the front, left, right, top, and back, to obtain sufficient image information. These multi-view 2D mushroom images can reflect the overall shape characteristics of the mushroom.
[0070] It is worth noting that the number of target objects selected in this invention is usually only tens to hundreds, compared to tens of thousands or even hundreds of thousands required for deep learning, which greatly reduces the requirement for the amount of prior data.
[0071] Step 22. Measurement of physical parameters.
[0072] The physical parameters of each target object are measured separately. The physical parameters to be measured can be selected based on the user's focus on the target object and their needs, typically reflecting the object's size and shape characteristics. These include size parameters such as length, width, height, volume, and surface area, and shape parameters such as curvature and maximum diameter. These physical parameters usually include key parameters that reflect the target object's main deformation modes.
[0073] Continuing with the previous specific embodiment, after the filming is completed, the dimensional parameters such as cap diameter, cap thickness, stem length, and overall height of these 30 mushrooms can be measured on-site. At the same time, shape parameters such as surface area, average stem diameter, and maximum stem diameter can be calculated based on these dimensional parameters, and these measured physical parameters can be recorded.
[0074] It is understood that this invention does not limit the selection of the required physical parameters for measurement, and can be flexibly adjusted according to the characteristics of the target object, the user's focus on the target object, and the user's needs. Furthermore, the measurement referred to in this step includes not only direct measurement but also simple calculations based on the measurement parameters.
[0075] Step 23. Training data creation.
[0076] A mapping relationship is established between the 2D images of each target object and the shape parameter vectors of its 3D model, ensuring that each 3D model can find corresponding coordinates in the parameter space. This is equivalent to establishing a mapping relationship between "2D image - shape parameter vector - 3D model - physical parameters". The 2D images and the shape parameter vectors of their corresponding 3D models for each target object are combined to form a training dataset. The 2D images serve as input to the subsequent parameter prediction model, while the shape parameter vectors of the corresponding 3D models are used to calculate the loss function, providing supervision information for the training of the parameter prediction model.
[0077] Continuing from the previous specific embodiment, a mapping relationship is established between 2D mushroom images of 30 mushrooms and shape parameter vectors of 3D mushroom models, and these are used as the training dataset for subsequent model training.
[0078] Therefore, it is evident that because this invention confines the deformation range to the principal component subspace prior to deep learning or regression, it only needs to regress the shape parameter β. The dimension of the shape parameter vector β, i.e., the number of principal components, is much smaller than the tens of thousands of dimensions of the original 3D mesh, thereby reducing the data scale requirement. Thus, only a small amount of training data is needed during deep learning. Compared to conventional 3D point cloud / voxel methods, this invention significantly reduces its dependence on hardware and massive amounts of data, making it easier to achieve accurate 3D shape reconstruction in situations with limited data.
[0079] Step 24: Initial parameter prediction model construction.
[0080] Once the training dataset is obtained, an initial parameter prediction model can be built, or it can be done simultaneously. This invention does not have specific requirements for the selection of the initial parameter prediction model; generally, it only needs to satisfy the requirements of multi-view feature fusion. For example, a convolutional neural network (CNN) or other models that fuse multi-view features can be used.
[0081] Combination Figure 7 As shown, the initial model selected in this invention mainly consists of an input layer, a feature extraction unit, a feature fusion unit, a shape parameter regression unit, and an output layer. The input layer receives one or more normalized 2D images of the target object from different perspectives; the feature extraction unit extracts features from the 2D images; the feature fusion unit fuses the extracted features from multiple perspectives to form a fused feature for overall shape representation; and the shape parameter regression unit maps the fused feature to a shape parameter vector and outputs it through the output layer. Specifically, network structures such as ConvNext, EfficientNet, RegNet, and Swin Transformer can be used.
[0082] In specific embodiments, for example, a combination of ResNet and MLP can be used. First, a ResNet network is used to extract depth features from images at different viewpoints. Then, an MLP (Multilayer Perceptron) is used to fuse (weight or average) the depth features of the multi-view images to obtain a fused comprehensive feature. Finally, the output layer of the MLP maps the comprehensive feature to a shape parameter vector for 3D reconstruction or physical parameter measurement. Another example is using a Swing Transformer network. First, images at different viewpoints are divided into blocks and encoded using a transformer to output a local-global combined image representation. Then, the Transformer encoded representation is fused using mechanisms such as attention to form an overall feature. Finally, a small regression network (such as a fully connected layer) outputs a shape parameter vector. Yet another example is using an energy-efficient and highly accurate feature extractor like EfficientNet to extract features from multi-view images. Then, the multi-view features are simply averaged or weighted and merged to obtain a fused global shape feature, which also yields good results. Finally, one or more fully connected layers are used to transform the global shape feature into a shape parameter vector.
[0083] Step 25. Model training.
[0084] Understandably, before extracting features from a 2D image of a target object, the image usually needs to be normalized to a similar size or resolution so that the network has a consistent scale reference when inputting. If the shooting distance or camera parameters are different, a calibration object can be placed in the image or other calibration methods can be used for size calibration. This invention does not limit this.
[0085] This invention trains a parameter prediction model using training data and uses a loss function to characterize the model's prediction performance on shape parameter vectors. During training, the loss function value is continuously reduced to ensure geometric consistency with the target object as much as possible. This invention can use the mean squared error (MSE) of the shape parameter vectors as the loss function value to measure the difference between the predicted shape parameter vector values and the actual shape parameter vector values in the training set. During the training process, this invention can use optimization algorithms such as Adam and SGD to iteratively update the network weights of the parameter prediction model until the loss function value tends to converge or the number of iterations reaches a threshold, at which point training stops, resulting in the target parameter prediction model.
[0086] Based on a trained target parameter prediction model, this invention can predict the shape parameter vector in the 3D model of the object under test, thereby realizing the reconstruction of the 3D model of the object under test and the prediction of key physical parameters.
[0087] Continuing from the previous specific embodiment, the 3D model reconstruction and key physical parameter prediction of all similar mushrooms can be realized based on the trained mushroom parameter prediction model. Specifically, any mushroom can be photographed to obtain one or more 2D mushroom images; the obtained 2D mushroom images are input into the mushroom parameter prediction model to obtain the shape parameter vector of the mushroom; the obtained shape parameter vector is used as known parameters, and the corresponding 3D model is calculated by formula (2); then, by calling library functions, the key physical parameters of the mushroom are calculated, such as the surface area, surface length, surface width, stem length, average stem diameter, maximum stem diameter, and overall height. Figure 8 The image shows the reconstructed 3D shape of the mushroom. The key physical parameters obtained by calling library functions are as follows: {'Fungal surface area': 1.0526140914811715, 'Fungal surface length': 1.1448038816452026, 'Fungal surface width': 1.168502688407898, 'Stem length': 0.9566632509231567, 'Average stem diameter': 0.2134493738412857, 'Maximum stem diameter': 0.3072476089000702, 'Overall height': 1.7578308582305908}.
[0088] It is worth noting that the 3D parametric reconstruction and prediction method provided in this invention is not only applicable to jointless objects such as mushrooms, but can also be extended to other similar objects, and even to targets with skeletons but no pose modeling required (such as certain plants and animals, industrial parts, etc.). When modeling various objects, the key is to ensure that all training samples have a consistent mesh topology and to extract principal components using statistical methods. Relying on prior parametric shape space and a small number of multi-view images, this invention can accurately predict the 3D model and key physical dimensions of objects, greatly reducing reliance on expensive 3D scanning equipment or massive amounts of data, and has significant advantages in scenarios with limited data.
[0089] Furthermore, such as Figure 3 As shown, Embodiment 3 of the present invention discloses a method for generating training data for a target parameter prediction model, which is also one of the core aspects of the present invention. Specifically, it includes the following sub-steps:
[0090] Step A1: Manually construct the baseline 3D model.
[0091] Select one of the target objects, and a professional 3D modeler can manually create a high-precision 3D model of the target object using modeling software such as Maya and Blender based on the corresponding 2D image and physical parameters of the target object. The established high-precision 3D model (also known as "3D mesh") is then used as the reference 3D model (also known as "reference 3D mesh").
[0092] Understandably, 3D modeling software typically models in a standard 3D mesh interface. The resulting 3D model consists of multiple interconnected triangular facets, each with three vertices. Adjacent triangular facets share the same vertex. Therefore, the 3D model of an object is also called the 3D mesh of the object.
[0093] Step A2: Obtain multiple derived 3D models through shape editing.
[0094] Modelers can use a baseline 3D model as a foundation, based on 2D images and physical parameters of other target objects, to perform linear deformation modeling, resulting in multiple derived 3D models with different shapes corresponding to each target object. Specifically, within the 3D modeling software, the baseline 3D model can be edited—for example, through dragging and stretching—to deform its shape while maintaining the 3D mesh topology, thus obtaining multiple derived 3D models. This avoids frequent non-rigid registration or topological rearrangement in statistical learning (such as PCA) and subsequent parameter prediction model training. Furthermore, the mesh structure of these 3D models is coherent and unified, allowing for easy measurement of object dimensions, surface area, volume, etc., and intuitive visualization, comparison, or statistical analysis.
[0095] It is understood that in this invention, "invariant topology" and related expressions such as "having a unified topology," "having a unified mesh," and "consistent topology" all mean the same thing. Invariant topology refers to the consistency of the number of triangles and vertices in each 3D model. By having 3D modelers manually create and edit 3D models, it is possible to ensure that the training data is consistent in mesh topology. Subsequently, the low-dimensional representation of PCA can be used to improve the accuracy and generalization ability of 3D shape reconstruction and key physical parameter prediction.
[0096] It's worth noting that, while maintaining the same number of triangles and vertices, modelers can generate more deformation samples by making diverse edits to the baseline 3D model. All deformations occur within a fixed mesh structure, eliminating the problem of facet or vertex rearrangement. This results in a 3D mesh with a consistent topology, generating a complete and renderable mesh, thus enriching the training dataset and improving the model's generalization ability to new shapes. In contrast, existing technologies lack mesh topology in point clouds, requiring additional algorithms (e.g., Poisson Surface Reconstruction) to fill in the faces, and the quality is not always stable. NeRF, on the other hand, uses implicit volume rendering, requiring voxelization or isosurface extraction during mesh extraction, a complex process that cannot guarantee an ideal topology.
[0097] Step A3: Extract the shape basis vectors.
[0098] This section uses Principal Components Analysis (PCA) as an example, aiming to utilize the concept of dimensionality reduction to replace a large number of variables with fewer variables, while still reflecting most of the information from the original multiple variables. In this step, the coordinate differences of vertices at corresponding positions in each derived 3D model and the baseline 3D model are first calculated. Then, based on these vertex coordinate differences, the PCA statistical method is used to extract k shape basis vectors b of the target object. j .
[0099] Understandably, other dimensionality reduction techniques, including linear or nonlinear ones, can also be used in other embodiments. For example, nonlinear methods such as LLE (Locally Linear Embedding) and Isomap (Iso-mapping) can also perform dimensionality reduction analysis on high-dimensional mesh deformation vectors. LLE, by preserving the linear structure within the local neighborhood, can capture more complex bending manifold changes in shape space; Isomap, based on approximate geodesic distance, maintains the overall geometry and can handle strongly nonlinear deformations. Both can also replace PCA as the dimensionality reduction method in this invention.
[0100] Step A4: Calculate the shape parameter vector.
[0101] Based on the obtained shape basis vectors, the shape parameter vectors of each target object are calculated through projection. Specifically, the shape parameter vector β corresponding to each model, including the baseline 3D model and the derived 3D model, is calculated, and β has k projection coefficients β'. j .
[0102] It is worth noting that the present invention mainly considers statistical analysis of a large number of high-dimensional difference vectors (such as grid coordinate difference vectors) in PCA dimensionality reduction, to find the orthogonal basis vectors that can explain the largest data variance, thereby extracting the most significant deformation patterns (i.e., the main deformation patterns) and reducing the dimensionality. Specifically, the following principles can be considered: 1) PCA seeks to project onto the direction with the largest variance in the dataset, corresponding to the eigenvector with the largest eigenvalue, in order to retain the most shape change information; hence, it is also called the maximum variance principle. 2) Each principal component is pairwise orthogonal to avoid redundancy and correlation, ensuring independence between deformation patterns (e.g., "becoming taller" and "becoming fatter" can be represented separately). 3) Sorting by eigenvalue from largest to smallest, generally selecting the first k principal components, whose cumulative variance accounts for 90% to 95%, avoids high-dimensional noise and overfitting. This way, it can retain the main geometric changes while significantly compressing the data dimensionality. 4) From a practical perspective, the grid coordinate difference is centered to eliminate the influence of overall translation. For deformation patterns, it is required to learn the most critical directions even with a small amount of data. PCA's linear statistical method is simple, efficient, and interpretable. Based on the above principles, the principal component vector extracted by PCA, namely the "shape basis vector," can summarize the main changes and significantly reduce the dimensionality that subsequent models need to process.
[0103] Understandably, PCA generates multiple principal components, each corresponding to an eigenvalue (representing the amount of data variance that the principal component can explain). These eigenvalues are ordered from largest to smallest, and typically the first few principal components can explain the vast majority of shape variations. For example, if the first k principal components can explain 95% of the data variation, then only these k principal components can be retained. The explanatory power of the remaining principal components is relatively small, and retaining them would introduce noise or lead to excessive dimensionality. Therefore, it is common practice to select the eigenvalues (i.e., principal components) that collectively explain a predetermined proportion (e.g., 95%) of the total data variance to avoid excessive complexity or noise accumulation caused by selecting too many principal components later. In other words, the final number of principal components retained is determined by a predetermined threshold for the "cumulative variance contribution rate".
[0104] Understandably, if k principal components are extracted using the PCA statistical method, each principal component represents a major deformation pattern, such as length increasing or decreasing, overall height increasing, edge thickening, or a certain part bulging. in, Let represent the low-dimensional (k-dimensional) space in which the shape parameter vector resides. The shape parameter vector β is a k-dimensional vector, therefore denoted as . Wherein, each component β jEach of these corresponds to a projection coefficient along a principal component direction, representing the degree of change of the object under that deformation mode. β is also the projection coefficient of each sample or new input on these principal deformation modes, which means that the shape parameter vector β includes at least one component (usually 5 to 50), thus being able to express the diverse geometric changes of the target object.
[0105] It is worth noting that this invention utilizes dimensionality reduction techniques such as PCA to map high-dimensional 3D meshes with tens of thousands of vertices to a low-dimensional shape parameter space (tens or at most hundreds of dimensions). Only a small number of parameters need to be regressed to reconstruct the complete 3D mesh. Unlike existing technologies such as NeRF or point cloud processing, which require extensive sampling or matching in a 3D field, this invention's parameterized model only regresses or optimizes low-dimensional parameters, making training and inference more efficient. Furthermore, in real-world applications, the deformation of target objects (such as mushrooms or fish) is often limited and regular. Incorporating them into a low-dimensional space reduces dependence on high-dimensional noise. Based on this, this invention, through dimensionality reduction techniques, restricts all legal deformations to a low-dimensional subspace. This means that the network or algorithm can only search in these main deformation directions, making it less prone to strange shapes or outrageous deformations. Therefore, it has strong deformation priors. The model restricts all legal deformations to a low-dimensional space obtained through statistical learning, enhancing robustness to noise and missing data. Even with missing viewpoints, a reasonable 3D shape can still be obtained. Furthermore, because the shape parameter space has prior knowledge, the network is less likely to fit extreme or strange shapes, reducing blind overfitting of abnormal data and making it less likely to overfit to abnormal geometric shapes.
[0106] Continuing with the previous specific example, from the 30 mushrooms, one is selected as the longest and most standard or average-sized mushroom. Based on multiple 2D images of this mushroom and measured physical parameters, the modeler uses Blender modeling software to create a 3D model (3D mesh) of this mushroom within a mesh. The resulting model can be called the baseline mushroom model. The baseline mushroom model is a 3D mesh composed of 5000 triangular faces and 10000 vertices of triangular faces, such as... Figure 5 As shown.
[0107] Next, the modelers continued to refer to the 2D images and physical parameters of 29 other mushrooms, and based on the baseline mushroom model, performed drag-and-drop operations in the modeling software to obtain 29 derived mushroom models. Figure 6As shown. It is important to note that during dragging, stretching, and other operations, it is crucial to ensure that the topological structure of the derived mushroom model and the baseline mushroom model is consistent, meaning they both have 5000 triangular faces and 10000 vertices, ensuring that each model shares the same mesh topology. Next, by comparing the coordinate differences of each vertex in each derived mushroom model and the baseline mushroom model, k principal components, i.e., the mushroom's shape basis vector b, are extracted using the PCA statistical method based on these vertex coordinate differences. j These shape basis vectors are used to form a shape blending matrix B. S And calculate the shape parameter vector β for all mushroom models, each shape parameter vector β containing k projection coefficients β. j For each derived mushroom β i ={β i,1 ,β i,2 ,β i,3 ,β i,4 ,β i,5}, which correspond to the projections of these principal components.
[0108] In the data accuracy phase, several mushroom grids with identical topological structures were acquired, containing deformation samples with different stipe lengths, cap sizes, and cap thicknesses. Each grid was unfolded into a 3N-dimensional vector, and the difference was calculated between this vector and the baseline mushroom to obtain the difference value {d}. i By using PCA to statistically extract the shape basis vectors of mushrooms, assuming that 5 principal components are obtained (k=5), these 5 principal components can explain more than 95% of the shape changes of mushrooms. Therefore, these principal component vectors can be retained to describe most of the deformation patterns of mushrooms, without having to process the subsequent small noise effects in high-dimensional coordinates, so as to achieve both a full description of shape changes and model simplification.
[0109] In this embodiment, the five principal components of the main deformation modes of the reaction mushroom include: a) stipe length, for example, the stipe becomes longer or shorter; b) cap diameter, for example, the cap diameter increases or decreases; c) cap thickness, for example, the top bulges or collapses; d) stipe thickness, for example, the radius of the central stipe changes; and e) cap edge curvature, for example, the mushroom edge curls downward or outward. Each principal component provides a key deformation mode of the mushroom.
[0110] A5: Finally, the 2D images of each target object and the calculated shape parameter vectors are used as training data to train the constructed initial parameter prediction model, so as to obtain the target parameter prediction model.
[0111] In other embodiments, when the target object is an animal, this invention can remove unnecessary dynamic factors such as bones / joints, retaining only the parts that play a major role in shape changes, and focusing on describing the geometric deformation of the target object. This can also be achieved using PCA statistics. If PCA statistics yield several principal components, each principal component represents a major deformation mode, and the shape parameters are the projection coefficients of the model along these principal component directions. This invention extracts the main feature components of the data, i.e., PCA shape basis vectors, through the PCA statistical method, achieving dimensionality reduction of high-dimensional data.
[0112] Furthermore, such as Figure 4 As shown, Embodiment 4 of the present invention, taking the PCA method as an example, discloses a method for extracting shape basis vectors and calculating shape parameter vectors, which mainly includes the following steps:
[0113] Step A31: Calculate the coordinate differences of vertices at corresponding positions in each derived 3D model and the base 3D model to obtain multiple sets of high-dimensional difference vectors.
[0114] The formula for calculating coordinate differences is as follows:
[0115]
[0116] In the formula, V represents the coordinate difference between the vertex at position t in the i-th derived 3D model and the vertex at position t in the base 3D model; it This represents the vertex coordinates at the t-th position in the derived 3D model; This represents the vertex coordinates at the t-th position in the baseline 3D model.
[0117] Assume the 3D mesh of the target object has N vertices, t = 1, 2, ..., N, and each vertex contains three coordinates (x, y, z). The coordinate difference of vertex 1 at position 1 is... The coordinates are (Δx1, Δy1, Δz1), and the coordinate difference of the second vertex at the second position is... The coordinates are (Δx2, Δy2, Δz2), ..., the difference in coordinates between the Nth and Nth vertices. The coordinates are (Δx) N ,Δy N ,Δz N These are arranged sequentially to form a set of 3N high-dimensional difference vectors, i.e., an N×3 difference vector matrix with N rows and 3 columns. This process is repeated for all derived 3D models to obtain multiple sets of N×3 difference vector matrices (high-dimensional difference vectors).
[0118] Step A32: Pull the multiple sets of high-dimensional difference vectors obtained in step A31 into one dimension by row or column respectively, forming multiple sets of 3N-dimensional one-dimensional difference vectors.
[0119] Taking the i-th derived 3D model as an example, expanding the N×3 difference vector matrix of the corresponding vertices of the i-th derived 3D model and the base 3D model by rows or columns yields a 3N-dimensional one-dimensional difference vector d. i , that is {Δx1,Δy1,Δz1,Δx2,Δy2,Δz2,K,Δx N ,Δy N ,Δz N}, Repeat this process for all derived 3D models to obtain multiple sets of 3N-dimensional one-dimensional difference vectors. Based on this, a set of difference vector datasets can be constructed for subsequent high-dimensional statistical analysis of PCA.
[0120] Understandably, the 3D mesh here can be a known 3D mesh or a new 3D mesh that appears later. A new mesh typically refers to a 3D mesh that appears later and is not in the original derived dataset, or it can be a 3D mesh that needs to be recalculated from the original derived mesh, but all of them must have a unified topological structure.
[0121] Step A33: Perform PCA statistical analysis on the obtained difference vectors to extract the shape basis vector b of the target object. j This forms a basis vector space.
[0122] In this step, PCA statistics are performed on the 3N-dimensional one-dimensional difference vector obtained in step A32 to obtain several principal components, which are the shape basis vectors of the target object, and finally a shape basis vector matrix containing the principal component vectors is formed.
[0123] Specifically, firstly, the difference vector Transform into a centered vector ,in, The mean vector (for) (Take the average); then calculate the covariance of the centered vector set to obtain the covariance matrix; finally select... The eigenvectors (orthogonal basis vectors) corresponding to the largest eigenvalues (covariance values) are used as shape basis vectors and combined to form a shape mixing matrix. That is, the basis vector space. ,in, That is, the number of principal components. For the first Shape basis vectors, , This represents a major deformation mode. The formula for calculating the covariance is:
[0124] (2);
[0125] The number of 3D meshes.
[0126] Step A34: Obtain the shape parameter vector of the derived 3D model by projection.
[0127] Project the one-dimensional difference vectors of each 3D model onto the basis vector space, and calculate the shape parameter vector β of each 3D model. For a known derived 3D model, the shape parameter vector β can be obtained by projection; for a new 3D mesh, the difference between the vertices of the new 3D mesh and the corresponding vertices of the base 3D model can be calculated first, and then projected onto the basis vector space to obtain the shape parameter vector β.
[0128] Specifically, the shape parameter vector of the i-th derived 3D model The projection coefficient of the i-th derived 3D model onto the j-th shape basis vector is β i,j Therefore, the shape parameter vector β of the i-th derived 3D model i It contains k projection coefficients, meaning that the shape parameter vector β of each 3D model contains k projection coefficients β. j .
[0129] It is worth noting that the shape basis vector b j Principal components extracted by PCA statistics describe the main deformation patterns that an object may undergo; their number k is much smaller than the original dimension 3N. The shape parameter vector β characterizes the projection coefficients of each derived 3D model or new mesh onto these shape basis vectors, i.e., how much deformation each principal component contributes. In other words, the shape basis vector b... j Just like a base, the shape parameter vector β is like a coordinate. By linearly superimposing the two, the three-dimensional shape of the object can be reconstructed.
[0130] It should also be noted that the shape mixing matrix B S It is a matrix that stores several shape basis vectors (also called "principal component vectors", "basic shape transformation vectors", or "basis vectors"), typically with dimensions of 1. Where N is the number of grid vertices (each vertex has three-dimensional coordinates, totaling 3N dimensions), and k is the number of principal components selected. This is understandable, since there are N vertices, each containing three coordinates (x, y, z), therefore each shape basis vector b... j Both are 3N-dimensional vectors used to describe the coordinate offset from the reference grid. The shape parameter vector β is the projection coefficient (e.g., βa). i,j ) In these shape basis vectors b j A linear combination of these components, typically 5 to 50 dimensions, represents the components of the current object in each principal component direction and is a low-dimensional vector.
[0131] It is worth noting that, compared to point clouds or implicit representations which require additional mesh reconstruction, the parameter prediction model constructed in this invention only needs to regress a small number of shape parameters, and the network inference speed is fast, making it very suitable for AR / VR, online detection, or real-time robot interaction scenarios.
[0132] Furthermore, in another embodiment, for step 3, based on the shape parameter vector and the pre-extracted shape basis vector of the object to which the target object belongs, a complete 3D mesh can be reconstructed to restore the 3D shape of each target object.
[0133] In this embodiment, the coordinates of the mesh vertices can be recovered using a linear combination formula, as follows:
[0134]
[0135] In the formula, M i (β) represents the 3D model of the i-th target object; The baseline 3D model of the target object (representing the average shape of the target object, i.e., the baseline 3D model built by the modeler); b j Let β be the j-th shape basis vector of the target object; i,j Let be the projection coefficient of the i-th target object onto the j-th shape basis vector.
[0136] As can be seen from the above, the difference between each derived 3D model and the baseline 3D model is calculated on a "one-to-one" basis for each vertex, without requiring a specific vertex to correspond to all vertices. This ensures that under a unified topology, PCA statistical analysis has a good correspondence, and the three-dimensional shape (i.e., 3D model) of the target object can be efficiently reconstructed based on formula (3).
[0137] Furthermore, in another embodiment, for step 4, after obtaining the 3D model of the object to be tested, the 3D model of the object to be tested is imported into a pre-defined library function. By performing geometric analysis, the key physical parameters of the object to be tested can be obtained. The pre-defined library function can be a commonly used tool such as Meshlab, Blender, or Open3D. By selecting the relevant commands or APIs, the required key physical parameters, such as length, width, height, volume, and surface area, can be returned.
[0138] In summary, this invention combines three core technologies: parametric model, unified mesh, and PCA dimensionality reduction. By having 3D modelers manually create and edit 3D models, the consistency of training data in mesh topology is ensured. Dimensionality reduction techniques are used to improve the accuracy and generalization ability of 3D shape reconstruction and key physical parameter prediction. This enables high-precision, high-efficiency, and interpretable 3D model reconstruction and physical parameter prediction for various objects in situations with limited data. It provides an efficient, interpretable, and low-cost technical path for numerous application fields such as industrial inspection, agricultural product evaluation, plant and animal morphology analysis, and multi-view interaction.
[0139] Furthermore, another embodiment of the present invention discloses an electronic device, which mainly includes a processor and a memory. The memory stores program code and transmits the program code to the processor; the processor executes the 3D parametric reconstruction and prediction method provided by the present invention according to the instructions in the program code.
[0140] Electronic devices may include processing units (such as central processing units, graphics processing units, etc.) that can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or loaded from storage devices into random access memory (RAM). RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0141] Furthermore, another embodiment of the present invention provides a computer-readable storage medium for storing program code for executing the 3D parametric reconstruction and prediction method provided by the present invention.
[0142] In some embodiments of the present invention, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, (but not limited to) an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable medium may be included in the electronic device or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, enable the electronic device to run the 3D parametric reconstruction and prediction method provided by the present invention.
[0143] For ease of description and brevity, the specific working process of the electronic device, the program code in the computer-readable storage medium, and the computer program product described above can be clearly understood by those skilled in the art by referring to the content of the foregoing method embodiments, combined with common knowledge and conventional technology in the field. Therefore, it will not be repeated here.
[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method based on 3D parametric reconstruction and prediction, characterized in that, Includes the following steps: S1: Acquire 2D images of the object under test from multiple perspectives; S2: Input the 2D image into the target parameter prediction model to obtain the shape parameter vector of the object to be tested; S3: Based on the shape parameter vector and the pre-extracted shape basis vector of the object to which the object belongs, reconstruct the 3D model of the object to be tested; S4: Perform geometric analysis on the 3D model of the object under test to obtain the key physical parameters of the object under test; The shape basis vector is extracted from the 3D model of the object to which the test object belongs using dimensionality reduction technology, and is used to characterize the main deformation mode of the object to which the test object belongs; the shape parameter vector is used to characterize the degree of change of the test object under the main deformation mode. The training data for the target parameter prediction model is prepared as follows: Multiple target objects belonging to the same category as the object to be tested are provided; 2D images of each target object from multiple perspectives are acquired, and physical parameters reflecting the size and shape characteristics of the target object are measured; a baseline 3D model is constructed based on the 2D image and physical parameters of one of the target objects; the baseline 3D model is deformed to obtain a derived 3D model with a consistent mesh topology; based on the coordinate differences of vertices at corresponding positions between each derived 3D model and the baseline 3D model, the shape basis vectors are extracted using the dimensionality reduction technique; and the shape parameter vectors corresponding to each target object are determined based on the shape basis vectors. The 2D images of each target object and their corresponding shape parameter vectors are used as training data. The key physical parameters are physical parameters that reflect the main deformation modes of the object under test; The target parameter prediction model is trained based on multi-view 2D images of objects belonging to the category to which the test object belongs, and has the ability to extract features and fuse features from multiple perspectives.
2. The 3D parametric reconstruction and prediction method as described in claim 1, characterized in that, The shape basis vectors have Each shape basis vector represents a main deformation mode of the objects belonging to the category of the object to be measured; The basis vector space composed of the shape basis vectors can explain the deformation data of the objects belonging to the category of the object to be measured by more than a predetermined ratio. The shape parameter vector has The projection coefficients are obtained by projection based on the basis vector space.
3. The 3D parametric reconstruction and prediction method as described in claim 2, characterized in that, In step S3, the 3D model of the object under test is reconstructed based on the following formula: ; In the formula, A 3D model of the object to be tested; A pre-built baseline 3D model of the object to which the test object belongs; The first target object Shape basis vectors; For the test substance at the 1st Projection coefficients on shape basis vectors.
4. The 3D parametric reconstruction and prediction method as described in any one of claims 1 to 3, characterized in that, The dimensionality reduction techniques include principal component analysis, local linear embedding, and nonlinear isometric mapping.
5. The 3D parametric reconstruction and prediction method as described in any one of claims 1 to 3, characterized in that, In step S2, the process of constructing the target parameter prediction model includes the following steps: S21: Provide multiple target objects belonging to the same category as the analyte; S22: Acquire 2D images of each target object from multiple perspectives, wherein the 2D images reflect the size and shape characteristics of the target object; S23: Measure each target object separately to obtain physical parameters that reflect the size and shape characteristics of the target object; S24: Create a training dataset based on the 2D images and physical parameters. The training data in the training dataset includes 2D images and shape parameter vectors corresponding to each target object. S25: Train the pre-built initial parameter prediction model using the training dataset to obtain the target parameter prediction model; the initial parameter prediction model has the ability to fuse multi-view features.
6. The 3D parametric reconstruction and prediction method as described in claim 5, characterized in that, The initial parameter prediction model includes an input layer, a feature extraction unit, a feature fusion unit, a shape parameter regression unit, and an output layer. The input layer receives 2D images from multiple viewpoints. The feature extraction unit extracts features from the 2D images. The feature fusion unit fuses the extracted image features to obtain overall shape features. The shape parameter regression unit maps the overall features to a shape parameter vector. The output layer outputs the shape parameter vector. During model training, the model weight parameters are continuously optimized. When the model loss function value tends to converge or the number of iterations reaches a preset value, the target parameter prediction model is obtained. The model loss function value is the mean square error between the predicted value of the shape parameter vector and the true value of the shape parameter vector in the corresponding training data.
7. The 3D parametric reconstruction and prediction method as described in any one of claims 1 to 3, characterized in that, Step S4 specifically includes: importing the 3D model of the object under test into a preset library function for geometric analysis, and calculating the key physical parameters of the object under test; the preset library function includes at least one of Meshlab, Blender, and Open3D.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the 3D parametric reconstruction and prediction method according to any one of claims 1 to 7.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the 3D parametric reconstruction and prediction method according to any one of claims 1 to 7.
Citation Information
Patent Citations
3D heart shape reconstruction method based on deep neural network and model prior
CN115601295A
Method and system for generating three-dimensional (3D) model(s) from a two-dimensional (2D) image of an object
US20240054722A1
