Three-dimensional reconstruction and model optimization method and device, storage medium and program product
Through the differentiable grid reconstruction and microroutable rendering technology, the problem of irregular geometric structure and topological structure of the three-dimensional reconstruction model is solved, and higher quality three-dimensional model generation is achieved, meeting the accuracy requirements of application scenarios.
Patent Information
- Application Number
- CN202411375726.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-29
AI Technical Summary
The model generated by the existing three-dimensional reconstruction method has poor geometric structure and irregular topological structure, which leads to increased post-processing costs and is difficult to meet the accuracy requirements of application scenarios.
Using microgrid reconstruction and microrendering techniques, by obtaining the initial mesh model, microgrid reconstruction is performed to obtain the target reconstruction model with differentiable properties, and optimize it using the loss function between the original image and the rendered image. , improve the geometric structure accuracy and topological regularity of the model.
It significantly improves the geometric accuracy and topological regularity of the three-dimensional model, improves the overall quality of the model, and reduces the cost of post-processing.
Smart Images

Figure CN118941742A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of model processing technology, and in particular to a three-dimensional reconstruction and model optimization method, device, storage medium and program product. Background Art
[0002] 3D reconstruction refers to the use of computer vision and graphics technology to digitally restore real 3D objects on computers to obtain digital models suitable for computer representation and processing. It has a wide range of applications. For example, in the field of e-commerce, by creating a 3D digital model of a product, consumers' online shopping experience can be greatly enriched. For example, in the field of medicine, by 3D reconstruction of the human bronchi, disease research and teaching analysis can be conducted more accurately.
[0003] With the development of 3D reconstruction technology, more and more 3D reconstruction schemes have emerged, such as 3D reconstruction based on structured light, 3D reconstruction based on laser scanning, 3D reconstruction based on stereo vision, and 3D reconstruction based on deep learning methods. These 3D reconstruction methods can reconstruct the 3D model of the object, but the reconstructed 3D model may have problems of unsatisfactory quality, such as poor geometric structure accuracy or irregular topological structure, which cannot meet the requirements of model accuracy in application scenarios. Summary of the invention
[0004] Multiple aspects of the present application provide a three-dimensional reconstruction and model optimization method, device, storage medium and program product to improve the accuracy of the geometric structure and the regularity of the topological structure of the three-dimensional model, thereby improving the overall model quality.
[0005] An embodiment of the present application provides a model optimization method, comprising: obtaining an initial mesh model of a target object, wherein the initial mesh model is obtained by performing three-dimensional reconstruction on an original image containing the target object; performing differentiable mesh reconstruction on the initial mesh model to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; and optimizing the target reconstruction model using a loss function between the original image and a rendered image obtained by performing differentiable rendering on the target reconstruction model as an optimization target to obtain a target mesh model.
[0006] An embodiment of the present application also provides a three-dimensional reconstruction method, comprising: acquiring an original image containing a target object, and performing three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model; performing differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; optimizing the reconstructed model using a loss function between the original image and a rendered image obtained by performing differentiable rendering of the target reconstructed model as an optimization target to obtain a target mesh model.
[0007] An embodiment of the present application also provides a model optimization method, including: receiving an initial mesh model of a target object sent by a terminal device, wherein the initial mesh model is obtained by performing three-dimensional reconstruction of an original image containing the target object; performing differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; optimizing the reconstructed model with a loss function between the original image and a rendered image obtained by performing differentiable rendering of the target reconstructed model as an optimization target to obtain a target mesh model; and sending the target mesh model to the terminal device so that the terminal device can use the target mesh model.
[0008] An embodiment of the present application also provides a three-dimensional reconstruction method, including: receiving an original image containing a target object sent by a terminal device, and performing three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model; performing differentiable mesh reconstruction on the initial mesh model to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; optimizing the target reconstruction model with a loss function between the original image and a rendered image obtained by differentiable rendering of the target reconstruction model as an optimization target to obtain a target mesh model; and sending the target mesh model to the terminal device so that the terminal device can use the target mesh model.
[0009] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the model optimization method or the three-dimensional reconstruction method.
[0010] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the model optimization method or the three-dimensional reconstruction method.
[0011] An embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the steps in the model optimization method or the three-dimensional reconstruction method are executed.
[0012] In this embodiment, either an initial mesh model of the target object is obtained, or a three-dimensional reconstruction is performed based on the original image of the target object to obtain an initial mesh model; a differentiable mesh reconstruction technology is introduced on the basis of the initial mesh model, and a differentiable mesh reconstruction is performed on the initial mesh model to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; on this basis, a differentiable rendering technology is introduced, and the loss function between the original image and the rendered image obtained by differentiable rendering of the target reconstruction model is used as the optimization target, and the target reconstruction model is optimized to obtain a target mesh model. In this way, the model optimization process based on differentiable mesh reconstruction and differentiable rendering technology can make the geometric structure of the three-dimensional model more accurate and the topological structure more regular, thereby improving the overall quality of the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flow chart of a model optimization method provided for an exemplary embodiment of the present application; Figure 2 A flowchart of a model optimization method provided by an exemplary embodiment of the present application in an actual scenario; Figure 3 A rendering of topological structure optimization provided by an exemplary embodiment of the present application; Figure 4 A schematic diagram of a flow chart of a three-dimensional reconstruction method provided by an exemplary embodiment of the present application; Figure 5 A flow chart of a model optimization method provided for an exemplary embodiment of the present application; Figure 6 A schematic diagram of a flow chart of a three-dimensional reconstruction method provided by an exemplary embodiment of the present application; Figure 7 A schematic diagram of the structure of a model optimization device provided by an exemplary embodiment of the present application; Figure 8 A schematic structural diagram of a three-dimensional reconstruction device provided by an exemplary embodiment of the present application; Fig. 9 A schematic diagram of the structure of a model optimization device provided by an exemplary embodiment of the present application; Fig.10 A schematic structural diagram of a three-dimensional reconstruction device provided by an exemplary embodiment of the present application; Fig.11 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0014] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0015] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.
[0016] In the application scenarios related to 3D reconstruction, although some 3D reconstruction methods can be used to reconstruct the 3D model of an object, the reconstructed 3D model may have problems with poor quality, such as poor geometric structure accuracy, the topological structure of the generated 3D model is often uncontrollable and very messy, and the subsequent texture mapping of the 3D model is also prone to blurring, and the UV (2D texture coordinates corresponding to the vertex information of the geometric figure) map expansion complexity is also correspondingly higher. These problems can usually only be solved by manual processing in the later stage, which greatly increases the cost of 3D reconstruction.
[0017] In view of the above technical problems, in an embodiment of the present application, an initial mesh model of the target object can be obtained, or an initial mesh model can be obtained by three-dimensional reconstruction based on the original image of the target object; a differentiable mesh reconstruction technology is introduced on the basis of the initial mesh model, and the initial mesh model is reconstructed in a differentiable mesh to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; on this basis, a differentiable rendering technology is introduced, and the loss function between the original image and the rendered image obtained by differentiable rendering of the target reconstruction model is used as the optimization target, and the target reconstruction model is optimized to obtain a target mesh model. In this way, based on the differentiable mesh reconstruction and the optimization based on the differentiable rendering technology, the geometric structure of the three-dimensional model is more accurate and the topological structure is more regular, thereby improving the overall quality of the three-dimensional model.
[0018] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0019] Figure 1This is a flow chart of a model optimization method provided by an exemplary embodiment of the present application. The model optimization method can be executed by a model optimization platform. In terms of implementation form, the model optimization platform can be implemented as an electronic device. The electronic device can be a terminal device such as a mobile phone, a computer, and a tablet computer, or a server such as a cloud server or an ordinary server. This embodiment does not limit the type of electronic device. Figure 1 As shown, the method comprises the following steps: Step 11: Obtain an initial mesh model of the target object. The initial mesh model is obtained by three-dimensionally reconstructing the original image containing the target object.
[0020] Step 12: Perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties.
[0021] Step 13: Taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstructed model as the optimization target, the target reconstructed model is optimized to obtain a target mesh model.
[0022] This embodiment does not limit the specific type of the target object, which may be an object, a commodity or a person, or an animal or a plant, etc. In the embodiment of the present application, an initial mesh model of the target object may be obtained, and the initial mesh model is obtained by three-dimensionally reconstructing the original image containing the target object. This embodiment does not limit the specific implementation method of three-dimensional reconstruction. In some exemplary embodiments, three-dimensional reconstruction can be obtained by a model optimization platform, or by a three-dimensional reconstruction platform provided by a third party, which is not limited in this embodiment.
[0023] Among them, the mesh model is composed of a series of interconnected triangular planes (faces), and each triangular plane is a mesh. Each triangular plane can be composed of a set of vertices (vertices) and edges (edges), and these triangular planes together constitute a complete mesh model surface. Correspondingly, the initial mesh model refers to a model obtained by three-dimensional reconstruction of the original image and expressed based on the mesh / triangular plane. Optionally, the initial mesh model does not have differentiable properties. Among them, the differentiable property refers to the model property that can calculate a continuous and differentiable rate of change in the surface or parameter space of the three-dimensional model. Based on the differentiable properties of the three-dimensional model, the gradient information can be used to optimize the model parameters. The initial mesh model does not have differentiable properties means that the model parameters such as the surface normal, surface curvature, vertex position, and topological structure of the initial mesh model are not continuously differentiable, and the initial mesh model does not support the optimization of model parameters using optimization algorithms such as gradient descent.
[0024] After obtaining the initial mesh model, the initial mesh model can be further reconstructed into a microgrid to obtain a target reconstructed model that can express the initial mesh model and has microgrid properties. The target reconstructed model refers to a reconstructed microgrid model. Microgrid reconstruction refers to the process of reconstructing the initial mesh model into a microgrid model. In other words, it is to re-express the initial mesh model expressed in a mesh in another way. This embodiment does not limit the specific implementation of microgrid reconstruction. For example, FlexiCubes (a technology for implementing microgrid rendering) can be used to reconstruct the initial mesh model into a microgrid, or other technologies such as DMTet (Deep Marching Tetrahedra) or DMesh (Differentiable Mesh) can be used to reconstruct the initial mesh model into a microgrid. This implementation does not limit it. FlexiCubes is an algorithm for efficiently representing and manipulating three-dimensional meshes. A series of deformable cube units (cubes) can be used to approximate the surface of a three-dimensional model. Each unit can be stretched or compressed according to the details and needs of the model to form a highly adaptable mesh representation. Among them, DMTet uses a series of stacked tetrahedral units to express the three-dimensional model in space, and adapts to different geometric structures by changing the deformation, distance and other properties of each unit. DMesh is an algorithm based on tetrahedral unit representation and uses Delaunay triangulation to adaptively express differentiable three-dimensional grids.
[0025] In this embodiment, a differentiable mesh reconstruction technology is introduced, and a target reconstruction model is obtained by performing differentiable mesh reconstruction on the initial mesh model. The target reconstruction model can be used to express the initial mesh model, and the target reconstruction model has differentiable properties. That is to say, the model parameters of the target reconstruction model, such as surface normals, surface curvatures, vertex positions, topological structures, etc., are continuously differentiable, and these model parameters are referred to as differentiable. For differentiable model parameters, these model parameters can be adjusted using optimization algorithms such as gradient descent to achieve specific optimization goals. Based on this, in this embodiment, a differentiable rendering technology is further introduced, and the target reconstruction model can be differentiably rendered to obtain a rendered image of the target object. Among them, differentiable rendering is a technology used to optimize three-dimensional models in computer graphics. It is a rendering process that can be differentiated and derived. It can be divided into forward and reverse processes. In the forward process, the model and related model parameters can be input to obtain a rendered image. The reverse process refers to the process of calculating the gradient of the rendered image relative to the model parameters. In the reverse process, the partial derivatives of the loss function with respect to each model parameter can be calculated through back propagation, so as to determine how each model parameter affects the final rendering result based on the calculated partial derivatives. After determining the gradient of the rendered image with respect to the scene parameters through the reverse process, the model parameters can be further adjusted using optimization algorithms such as gradient descent to update the model, making the model parameters more accurate and the rendering results closer to the target image or the desired effect.
[0026] In this embodiment, the implementation method of the rendered image is not limited. For example, the rendered image may include a reconstructed image of the original image, may include a silhouette image corresponding to the reconstructed image, or may include both the reconstructed image and the silhouette image. The reconstructed image is the reconstruction result of the original image, is a complete image, and may be an RGB (Red-Green-Blue, a color mode) image or an RGBD image. The RGB image refers to an image expressed based on the three colors of red, green, and blue, and the RGBD image is an image expressed based on the three colors of red, green, and blue and depth information. The silhouette image refers to an image used to describe the outline of the target object, which can be obtained by binarizing the RGB image. The binarization process refers to marking the foreground object (i.e., the target object) of the RGB image as white (usually 1), and the rest of the background is marked as black (usually 0).
[0027] After obtaining the rendered image of the target object, the target reconstruction model can be optimized by taking the loss function between the original image and the rendered image as the optimization target to obtain the target mesh model. The target reconstruction model can be optimized by taking the loss function between the original image and the rendered image convergence to a preset range as the optimization target to obtain the target network model. The preset range can be set to any range according to actual design requirements, and this embodiment does not impose any limitation.
[0028] In the embodiment of the present application, the implementation method of optimizing the target reconstruction model is not limited. For example, the optimization target may be that the loss function between the original image and the rendered image converges to a preset range, and the target reconstruction model is optimized in a single stage; or, the optimization target may be that the loss function between the original image and the rendered image converges to a preset range, and the target reconstruction model is optimized in two stages from coarse to fine (Coarse-to-fine) to obtain a target mesh model; of course, according to the accuracy requirements of the target reconstruction model, the target reconstruction model may also be optimized in more stages (for example, three stages or more). Among them, no matter how many stages of optimization are used, the optimization target is that the loss function between the original image and the rendered image converges to a preset range, and the optimization object is different according to the optimization stage. For example, in a single-stage optimization process, the optimization object may include one or more of the texture information, camera pose and geometric structure of the target reconstruction model. Preferably, the texture information, camera pose and geometric structure of the target reconstruction model may be optimized at the same time. For another example, in a two-stage optimization process, the optimization object in the first stage may include the texture information and camera pose of the target reconstruction model, and the optimization object in the second stage may include the geometric structure of the target reconstruction model; further optionally, the optimization object in the first stage may also include the geometric structure of the target reconstruction model. For another example, in a three-stage optimization process, the optimization object in the first stage may include the texture information of the target reconstruction model, the optimization object in the second stage may include the camera pose of the target reconstruction model, and the optimization object in the third stage may include the geometric structure of the target reconstruction model, that is, different objects can be optimized in the three stages respectively. Which optimization objects are optimized in each of the above stages is only an example and is not limited thereto.
[0029] In the embodiment, an initial mesh model of the target object is obtained, and a microgrid reconstruction technology is introduced on the basis of the initial mesh model, and the initial mesh model is microgrid reconstructed to obtain a target reconstruction model that can express the initial mesh model and has microgrid properties; on this basis, a microgrid rendering technology is introduced, and the loss function between the original image and the rendered image obtained by microgrid rendering the target reconstruction model is used as the optimization target, and the target reconstruction model is optimized to obtain a target mesh model. In this way, based on microgrid reconstruction and optimization based on microgrid rendering technology, the geometric structure of the three-dimensional model is more accurate and the topological structure is more regular, thereby improving the overall quality of the three-dimensional model.
[0030] The embodiment of the present application does not limit the specific implementation method of the aforementioned step 12 "reconstructing the initial mesh model into a differentiable mesh to obtain a target reconstructed model that can represent the initial mesh model and has differentiable properties". In an exemplary embodiment, the initial mesh model can be reconstructed into a differentiable mesh using a stereo unit, and the reconstruction process can be implemented based on the following steps 121-123: Step 121: Determine initial differentiable parameters and initial solid units required to represent differentiable meshes. The initial solid units have a default shape.
[0031] Among them, differentiable mesh is an advanced representation method used in the field of computer graphics and machine learning, which can differentiate 3D geometric shapes during optimization and training. Differentiable meshes are usually composed of a set of vertices and faces connecting these vertices, and the vertex positions and other geometric properties can be continuous differentiable functions, so that optimization algorithms such as gradient descent can be used based on differentiable meshes to subsequently optimize the target reconstruction model.
[0032] Among them, the initial three-dimensional unit refers to the smallest unit used to fit the initial grid model. The fitting of the initial grid model can be completed by adjusting the relative positions of the initial three-dimensional units one by one. The initial three-dimensional unit has a default shape. This embodiment does not limit the specific implementation method of the default shape. It can be a rectangular parallelepiped, a cube, or other tetrahedral shapes. Preferably, a rectangular parallelepiped can be used as the initial three-dimensional unit. The rectangular parallelepiped has a simple geometric shape, is easy to understand and process, and is easy to segment and combine. Therefore, using a rectangular parallelepiped as the initial three-dimensional unit can improve the efficiency of fitting the initial grid model.
[0033] The initial differentiable parameters required for the differentiable mesh are used to represent the geometric shapes of the initial stereo unit and the initial mesh model, that is, the initial differentiable parameters include both the parameters of each initial stereo unit and the parameters of the initial reconstruction model composed of the initial stereo units as a whole. These parameters work together to represent complex geometric shapes in a flexible and detailed manner. The parameters of each initial stereo unit may include at least one of the following: position parameters, deformation parameters, and detail level parameters. The position parameters are used to describe the position of each initial stereo unit in its space, usually represented by three-dimensional coordinates. The deformation parameters are used to control the ability of each initial stereo unit to deform from the original default shape so that each initial stereo unit can adapt to the local details of the initial mesh model. The deformation methods may include translation, rotation, scaling, and more complex nonlinear deformations. The detail level parameter is used to control the resolution of each initial stereo unit, that is, to represent the fineness of the details of the initial reconstruction model. The parameters of the entire initial reconstruction model composed of the initial stereo units may include at least one of the following: camera pose parameters, size ratio parameters, connectivity parameters, texture parameters, optimization parameters, and constraint parameters. The size ratio parameters are used to define the size and ratio of the entire initial reconstruction model. The size ratio parameters can affect the relative size and position of each initial stereo unit. The connectivity parameters are used to control the connection between the initial stereo units to ensure the continuity and integrity of the initial reconstruction model. The camera pose parameters are used to describe the position and pose of the camera when taking the original image. The texture parameters are used to define the visual appearance of the initial reconstruction model, including color, reflection characteristics, transparency, etc. The optimization parameters refer to the parameters related to the fitting process of the initial mesh model, which may include at least one of the following: number of iterations, convergence threshold and regularization term. The optimization parameters affect how the initial stereo units approximate the initial mesh model. The constraint parameters are used to constrain the fitting process of the initial mesh model. For example, the constraint parameters can keep certain parts of the initial mesh model from being deformed, or limit the type and degree of deformation of certain parts of the initial mesh model to meet specific engineering or aesthetic requirements.
[0034] Step 122: adjust the shape of the initial three-dimensional unit to obtain a target three-dimensional unit that matches the shape of the target object. The shape refers to the proportional relationship between the components of the initial three-dimensional unit. If the initial three-dimensional unit is a cuboid, the corresponding shape is the proportional relationship between the length, width and height.
[0035] In this embodiment, the specific shape adjustment method is not limited. In some exemplary embodiments, the initial stereo unit can be proportionally enlarged or proportionally reduced. In other exemplary embodiments, the size information of the initial grid model can be obtained. The size information is used to describe the length, width and height of the initial grid model. According to the size information, the initial stereo unit is adjusted in shape to obtain a target stereo unit that matches the shape of the target object. Among them, according to the size information, when the initial stereo unit is adjusted in shape, the first ratio between the length, width and height of the initial grid model can be calculated, and the initial stereo unit is adjusted in shape using the first ratio, so that the second ratio between the length, width and height of the initial stereo unit is the same or similar to the first ratio. Similar in this case means that the first ratio and the second ratio are within a certain error range. By adjusting the shape of the initial stereo unit, the adaptability of the shape of the initial stereo unit and the target object can be improved, so as to finally obtain the target stereo unit; compared with directly using the initial stereo unit to fit the initial grid model, using the target stereo unit that matches the shape of the target object can fit the initial grid model more accurately and efficiently.
[0036] Step 123, fitting the initial mesh model according to the initial differentiable parameters and the target stereo unit to obtain the target reconstruction model. The target reconstruction model obtained by the fitting process has the target differentiable parameters. The process of fitting the initial mesh model is the process of continuously adjusting the initial differentiable parameters of the initial mesh model to finally obtain the target differentiable parameters. The target differentiable parameters refer to the differentiable parameters that meet the fitting termination conditions after adjusting the initial differentiable parameters.
[0037] This embodiment does not limit the specific implementation of step 123. In an exemplary embodiment, the initial mesh model can be covered with the target stereo unit according to the initial differentiable parameters to obtain the initial reconstruction model. The number, size and position of the target stereo unit can be adjusted according to the initial differentiable parameters, and the target stereo units can be combined to cover the initial mesh model. Afterwards, the similarity error between the initial mesh model and the initial reconstruction model can be used as the optimization target, and the gradient descent algorithm can be used to iteratively optimize the differentiable parameters of the initial reconstruction model to obtain the target reconstruction model.
[0038] In the embodiments of the present application, the implementation method for expressing the similarity between the initial mesh model and the initial reconstruction model is not limited, for example, it can be adopted but not limited to: the chamfer distance between the initial mesh model and the initial reconstruction model, the Hausdorff distance, the average point-to-point distance, or the similarity between geometric features, etc. The following examples are given: Optionally, the chamfer distance between the initial reconstruction model and the initial mesh model can be calculated as the similarity error between the two. The average value of the distance from each position point of the initial reconstruction model to the nearest position point of the initial mesh model can be calculated as the chamfer distance. When the chamfer distance does not meet the set first distance condition, the differentiable parameters of the initial reconstruction model are iteratively optimized according to the gradient vector of the chamfer distance until the chamfer distance meets the first set distance condition, and the initial reconstruction model when the chamfer distance meets the first set distance condition is used as the target reconstruction model. The first set distance condition can be set to any condition, which is not limited in this embodiment. The iterative optimization of the differentiable parameters of the initial reconstruction model includes but is not limited to: optimizing the position parameters, deformation parameters and detail level parameters of each stereo unit in the initial reconstruction model, as well as optimizing the parameters of the entire model level, such as size ratio parameters, connectivity parameters, optimization parameters and constraint parameters.
[0039] Optionally, the Hausdorff distance between the initial reconstruction model and the initial grid model can be calculated as the similarity error between the two. The Hausdorff distance is a measure of the similarity between two sets of point sets. The distance between the first point set of the initial reconstruction model and the farthest point pair in the second point set of the initial grid model can be calculated as the Hausdorff distance. The first point set is a point set composed of various position points of the initial reconstruction model, and the second point set is a point set composed of various position points of the initial grid model. When the Hausdorff distance does not meet the second distance condition, the differentiable parameters of the initial reconstruction model are iteratively optimized according to the gradient vector of the Hausdorff distance until the Hausdorff distance meets the second distance condition, and the initial reconstruction model when the Hausdorff distance meets the second distance condition is used as the target reconstruction model. The second set distance condition can be set to any condition, which is not limited in this embodiment.
[0040] Optionally, the average point-to-point distance (AveragePoint-to-Point Distance) between the initial reconstruction model and the initial mesh model can be calculated as the similarity error between the two. The average point-to-point distance can be calculated by calculating the average Euclidean distance between the first point set of the initial reconstruction model and the second point set of the initial mesh model as the average point-to-point distance. When the average point-to-point distance does not meet the third set distance condition, the differentiable parameters of the initial reconstruction model are iteratively optimized according to the gradient vector of the average point-to-point distance until the chamfer distance meets the third set distance condition, and the initial reconstruction model when the average point-to-point distance meets the third set distance condition is used as the target reconstruction model. The third set distance condition can be set to any condition, and this embodiment does not impose any limitation.
[0041] Optionally, the similarity error between the geometric features of the initial reconstruction model and the initial mesh model can be calculated, and the geometric features can be curvature or normal vectors, etc., which are not limited in this embodiment. In the case where the geometric features do not meet the set feature conditions, the differentiable parameters of the initial reconstruction model are iteratively optimized according to the gradient vector of the geometric features until the geometric features meet the set feature conditions, and the initial reconstruction model when the geometric features meet the set feature conditions is used as the target reconstruction model. Among them, the set feature conditions can be set to any conditions, which are not limited in this embodiment.
[0042] Based on the above optional implementations, when iteratively optimizing the differentiable parameters of the initial reconstruction model, the differentiable parameters can be regularized in each round of iteration to more effectively prevent overfitting, thereby further improving the accuracy of the target reconstruction model.
[0043] In this embodiment, based on the above steps 121 to 123, the initial mesh model can be fitted according to the target stereo unit adapted to the shape of the target object and the initial differentiable parameters to obtain a target reconstruction model more accurately.
[0044] In the embodiment of the present application, the specific implementation method of the aforementioned step 13 "taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as the optimization target, optimizing the target reconstruction model to obtain the target mesh model" is not limited. In an exemplary embodiment, it can be implemented based on the following steps 131-135: Step 131: Input the target reconstruction model and its corresponding texture information and camera pose into a differentiable renderer for differentiable rendering to obtain a first rendered image. The texture information of the target reconstruction model is used to describe the details / physical properties on the surface of the target reconstruction model, and may include at least one of the following: color, brightness, roughness, and smoothness. The camera pose is used to describe the position and posture of the camera when shooting the original image.
[0045] Optionally, before the target reconstruction model and its corresponding texture information and camera pose are input into the differentiable renderer for differentiable rendering, the camera pose can be obtained in the following way: the camera pose of the original image can be estimated to obtain the camera pose corresponding to the target reconstruction model. The camera pose estimation can be implemented based on the following steps: First, the key points in the original image can be identified. Since these key points can correspond to the feature points on the target reconstruction model, the detected key points can be matched with the feature points on the target reconstruction model. The corresponding points between the original image and the target reconstruction model are determined based on the matching results, and the camera pose corresponding to the target reconstruction model is determined based on the corresponding points between the original image and the target reconstruction model. The accuracy of the camera pose determined here may be low, so further camera pose optimization is required. The relevant content has been recorded in detail later and will not be repeated here.
[0046] Optionally, before the target reconstruction model and its corresponding texture information and camera pose are input into the differentiable renderer for differentiable rendering, the texture information can be obtained in the following manner: based on the texture prediction network corresponding to the target object, the texture information of the target reconstruction model is predicted to obtain the texture information corresponding to the target reconstruction model, wherein the texture prediction network can have the ability to predict the texture information of the three-dimensional model based on the pre-training process. Specifically, the coordinates of any position point on the target reconstruction model can be input into the texture prediction network, and the texture prediction network can output the texture information corresponding to the arbitrary position point based on the mapping relationship between the model position point and the texture information learned in advance and the coordinates of the arbitrary position point on the target reconstruction model, thereby finally obtaining the texture information corresponding to the target reconstruction model.
[0047] Among them, images of the target object under multiple perspectives can be collected as sample images with texture information; according to the sample image and the target reconstruction model, the initial network is trained to output texture information based on the position coordinates on the target reconstruction model to obtain a texture prediction network. Specifically, the sample image and the target reconstruction model can be input into the initial network, and under the supervision of the sample image, the initial network is iteratively trained with the loss function of the initial network converging to a specified range as the goal to obtain a texture prediction network. Among them, in any iteration round, the coordinates of any position point of the target reconstruction model can be input into the initial network to obtain the texture information corresponding to the coordinates. If the error between the texture information corresponding to the coordinates and the texture information on the sample image does not converge to the specified range, the parameters of the initial network are updated and the next iteration round is entered; if the error converges to the specified range, the texture prediction network is output. Based on the above training process, the texture prediction network can output the corresponding texture information based on the position points on the target reconstruction model more accurately.
[0048] Step 132: According to the first loss function between the first rendered image and the original image, at least the topological structure of the target reconstruction model is optimized to obtain a first optimized model and its corresponding texture information and camera pose. The topological structure of the target reconstruction model refers to the relationship between vertices, edges and faces in the target reconstruction model. This embodiment does not limit the specific implementation of the first loss function, which can be any function for calculating the error between the first rendered image and the original image, including but not limited to: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE) or Mean Squared Error (MSE).
[0049] Specifically, when the first loss function does not meet the first optimization termination condition, the topological structure of the target reconstruction model and at least one of the geometric structure, camera pose and texture prediction network of the target reconstruction model can be jointly optimized until the first loss function meets the first optimization termination condition to obtain the first optimized model and its corresponding texture information and camera pose. Among them, the first optimization termination condition can be set to any condition according to the actual design requirements, and this embodiment does not limit it. Among them, different from the topological structure, the geometric structure refers to the spatial layout and shape characteristics of the model, which can be represented by the position coordinates in the three-dimensional grid of the model. In other words, the position coordinates in the three-dimensional grid of the model change, and the geometric structure of the model will also change accordingly. Therefore, the accuracy of the position coordinates of the vertices in the three-dimensional grid directly affects the quality of the model shape appearance. The first optimized model refers to the model obtained after the first stage of optimization. In this embodiment, when optimizing the topological structure of the target reconstruction model, the parameters related to the topological structure in the target differentiable parameters of the target reconstruction model can be adjusted, such as connectivity parameters, etc. By optimizing the topological structure of the target reconstruction model, the complexity of the topological structure of the target reconstruction model can be reduced, and the topological structure of the first optimized model obtained is more regular. By optimizing the geometric structure of the target reconstruction model, the shape of the target reconstruction model can be made closer to the actual shape of the target object, and the shape of the first optimized model can be made more accurate. By optimizing the camera pose, the error between the actual camera pose when the original image was taken can be reduced, and the accuracy of the camera pose can be improved. By optimizing the texture prediction network, the texture information predicted by the texture prediction network can be made closer to the actual surface material of the target object. In this way, at least the topological structure and camera pose can be optimized more accurately, which effectively solves the problems of irregular model topological structure and inaccurate camera pose.
[0050] Step 133: Input the first optimization model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a second rendered image. This embodiment does not limit the type of differentiable renderer and the specific implementation method of differentiable rendering. In an exemplary embodiment, the differentiable renderer can perform vertex processing, rasterization and texture interpolation in sequence according to the first optimization model and its corresponding texture information and camera pose, so as to complete differentiable rendering. Among them, vertex processing refers to the process of transforming and operating the vertices of the first optimization model, which usually includes steps such as vertex position transformation, lighting calculation, and texture coordinate calculation. Rasterization refers to the process of converting the first optimization model into a two-dimensional image. Specifically, it can determine which pixels are covered by which geometric elements, and calculate the colors of these pixels according to the colors and lighting information of these geometric elements, so as to complete rasterization. Texture interpolation refers to interpolating the color values on the texture map during the rendering process to obtain the color of the pixel. In this way, the micro-renderer can perform differentiable rendering more efficiently based on the first optimization model and its corresponding texture information and camera pose.
[0051] Step 134: According to the second loss function between the second rendered image and the original image, at least the geometric structure of the first optimization model is optimized to obtain a second optimization model and its corresponding texture information and camera pose. The second optimization model refers to the model obtained after the second stage optimization.
[0052] It should be noted here that the above steps 131-132 describe the model optimization process of the first stage, and the above steps 133-134 describe the model optimization process of the second stage. Based on the model optimization process of the first stage, at least the topological structure and camera pose of the model can be optimized more accurately, and based on the model optimization process of the second stage, at least the geometric structure of the model can be optimized more accurately. In step 134, the topological structure and camera pose of the first optimized model can be fixed, and the topological structure and camera pose are no longer adjusted, that is, the topological structure and camera pose of the first optimized model are not adjustable. In this case, the geometric structure of the first optimized model can be optimized when the second loss function does not meet the second optimization end condition. Further, optionally, the geometric structure and texture information of the first optimized model can be optimized at the same time in the second stage to obtain a second optimized model with higher quality texture information and geometric structure. This embodiment does not limit the specific implementation method of the second loss function, which can be any function used to calculate the error between the second rendered image and the original image, including but not limited to: mean absolute error, root mean square error or mean square error. Among them, the second optimization end condition can be set to any condition according to actual needs, and this embodiment does not impose any restrictions. Among them, the number, size and position of the target stereo unit can be adjusted to optimize the geometric structure of the first optimization model. When adjusting the position of the target stereo unit, the vertex position of the target stereo unit can be offset within a first preset range near the normal of the target stereo unit, thereby fine-tuning the geometric structure. Among them, the texture information of the first optimization model can be offset within a second preset range, thereby fine-tuning the texture information. By optimizing the geometric structure and texture information of the first optimization model, the adjusted second optimization model and its texture information can be made to match the actual situation of the target object in the original image as much as possible, thereby improving the quality of the second optimization model.
[0053] Step 135: Generate a target mesh model based on the second optimized model and its corresponding texture information and camera pose, wherein texture may be added to the second optimized model based on the texture information corresponding to the second optimized model, and a target mesh model under the camera's perspective may be generated based on the camera pose.
[0054] Based on the above steps 131 to 135, when the topological structure, geometric structure, camera pose and texture prediction network of the target reconstructed model are all adjustable, at least one of them can be jointly optimized in the first stage based on the first loss function between the first rendered image and the original image; then, the topological structure and camera pose can be fixed, and the geometric structure and texture information of the first optimized model can be optimized in the second stage in a targeted manner. Through the above two-stage optimization process from coarse to fine, the geometric structure of the three-dimensional model can be made more accurate and the topological structure more regular, thereby improving the overall quality of the three-dimensional model.
[0055] The following will be combined Figure 2 and Figure 3 The above process of this scheme is further explained.
[0056] like Figure 2 As shown in the actual scene, the initial mesh model can be covered with the target stereo unit, that is, the random initialization process, to obtain the initial reconstruction model, that is, Figure 2 The random initialization model in is used; the similarity error between the initial mesh model and the initial reconstruction model is taken as the optimization target, and the gradient descent algorithm is used to iteratively optimize the differentiable parameters of the initial reconstruction model to obtain the target reconstruction model.
[0057] Based on the texture prediction network corresponding to the target object, texture information of the target reconstruction model is predicted to obtain texture information corresponding to the target reconstruction model; the target reconstruction model and its corresponding texture information and camera pose are input into a differentiable renderer for differentiable rendering to obtain a first rendered image; the first rendered image may include an RGB image and a silhouette image.
[0058] Afterwards, the first loss function between the first rendered image and the original image can be used, that is, Figure 2 The loss function between the RGB image and the original image and the loss function between the silhouette image and the original image shown in , the topological structure of the target reconstruction model and the geometric structure of the target reconstruction model, the camera pose and the texture prediction network are jointly optimized until the first loss function meets the first optimization end condition to obtain the first optimized model and its corresponding texture information and camera pose. The first optimized model and its corresponding texture information and camera pose are input into the differentiable renderer, which can perform vertex processing, rasterization and texture interpolation in sequence to obtain the second rendered image. The above is the "rough optimization" process of the first stage.
[0059] Next, the second stage of "fine optimization" process can be continued: fix the topological structure and camera pose of the first optimization model, and optimize the geometric structure and texture information of the first optimization model when the second loss function does not meet the second optimization end condition to obtain the second optimization model and its texture information and camera pose. Generate the target mesh model based on the second optimization model and its corresponding texture information and camera pose.
[0060] In this way, based on the reconstruction of differentiable meshes and the two-stage optimization process, the geometric structure of the 3D model can be made more accurate and the topological structure more regular, thereby improving the overall quality of the 3D model. Figure 3The optimization effect of the topological structure shown is that the left side is a product that has not been optimized according to the optimization method proposed in the embodiment of the present application, and the topological structure of its surface is messy. The right side is a product that has been optimized according to the optimization method proposed in the embodiment of the present application, and the topological structure of its surface is more regular and closer to the original image.
[0061] The following will be combined Figure 4 , Figure 5 as well as Figure 6 , the above process of this solution is further explained with other scenarios.
[0062] like Figure 4 As shown, the embodiment of the present application also provides a 3D reconstruction method, which can be executed by a 3D reconstruction platform. In terms of implementation form, the 3D reconstruction platform can be implemented as an electronic device, or as a 3D reconstruction tool or component running on an electronic device. The electronic device can be a terminal device such as a mobile phone, a computer, and a tablet computer, or a server such as a cloud server or an ordinary server. This embodiment does not limit the type of electronic device. The method may include: Step 41: Acquire an original image containing a target object, and perform three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model.
[0063] Step 42: Perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties.
[0064] Step 43: Taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstructed model as the optimization target, the target reconstructed model is optimized to obtain a target mesh model.
[0065] In this embodiment, the specific method of obtaining the original image containing the target object is not limited. Users or merchants can upload the original image to the 3D reconstruction platform through the upload interface provided by the terminal device they hold, or directly upload the original image to the 3D reconstruction platform through the web (World Wide Web, global local area network) page, or draw the original image on the terminal device they hold and send it to the 3D reconstruction platform. This embodiment does not impose any restrictions.
[0066] In this embodiment, an original image containing a target object can be obtained, and the target object can be three-dimensionally reconstructed based on the original image to obtain an initial mesh model; in this embodiment, the detailed implementation method of three-dimensional reconstruction based on the original image is not limited. Based on the initial mesh model, a differentiable mesh reconstruction technology is introduced, and the initial mesh model is reconstructed in a differentiable mesh to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; on this basis, a differentiable rendering technology is introduced, and the loss function between the original image and the rendered image obtained by differentiable rendering of the target reconstruction model is used as the optimization target, and the target reconstruction model is optimized to obtain a target mesh model. In this way, the model optimization process based on differentiable mesh reconstruction and differentiable rendering technology can make the geometric structure of the three-dimensional model more accurate and the topological structure more regular, thereby improving the overall quality of the reconstructed three-dimensional model.
[0067] Optionally, the initial mesh model is reconstructed into a differentiable mesh to obtain a target reconstruction model that can represent the initial mesh model and has differentiable properties, including: determining initial differentiable parameters and initial stereo units required to represent the differentiable mesh, the initial stereo units having a default shape; adjusting the shape of the initial stereo units to obtain target stereo units that match the shape of the target object; fitting the initial mesh model according to the initial differentiable parameters and the target stereo units to obtain a target reconstruction model, the target reconstruction model having target differentiable parameters.
[0068] Optionally, the initial mesh model is fitted according to the initial differentiable parameters and the target stereo units to obtain the target reconstruction model, including: covering the initial mesh model with the target stereo units according to the initial differentiable parameters to obtain the initial reconstruction model; taking the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, and iteratively optimizing the differentiable parameters of the initial reconstruction model using a gradient descent algorithm to obtain the target reconstruction model.
[0069] Optionally, taking the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, the gradient descent algorithm is used to iteratively optimize the differentiable parameters of the initial reconstruction model to obtain the target reconstruction model, including: calculating the chamfer distance between the initial reconstruction model and the initial mesh model as the similarity error between the two; when the chamfer distance does not meet the set distance condition, iteratively optimizing the differentiable parameters of the initial reconstruction model according to the gradient vector of the chamfer distance; until the chamfer distance meets the set distance condition, the initial reconstruction model when the chamfer distance meets the set distance condition is used as the target reconstruction model.
[0070] Optionally, the target reconstruction model is optimized with the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as the optimization target, so as to obtain the target mesh model, including: inputting the target reconstruction model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a first rendered image; according to the first loss function between the first rendered image and the original image, at least optimizing the topological structure of the target reconstruction model to obtain the first optimized model and its corresponding texture information and camera pose; inputting the first optimized model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a second rendered image; according to the second loss function between the second rendered image and the original image, at least optimizing the geometric structure of the first optimized model to obtain the second optimized model and its corresponding texture information and camera pose; generating the target mesh model according to the second optimized model and its corresponding texture information and camera pose. Among them, the model optimization process based on differentiable rendering technology can make the geometric structure of the three-dimensional model more accurate and the topological structure more regular, thereby improving the overall quality of the reconstructed three-dimensional model.
[0071] Optionally, before inputting the target reconstruction model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering, the method further includes: predicting the texture information of the target reconstruction model based on the texture prediction network corresponding to the target object, and obtaining the texture information corresponding to the target reconstruction model; estimating the camera pose of the original image, and obtaining the camera pose corresponding to the target reconstruction model.
[0072] Optionally, according to a first loss function between the first rendered image and the original image, at least the topological structure of the target reconstruction model is optimized to obtain a first optimized model and its corresponding texture information and camera pose, including: when the first loss function does not meet the first optimization end condition, the topological structure of the target reconstruction model and at least one of the geometric structure, camera pose and texture prediction network of the target reconstruction model are jointly optimized until the first loss function meets the first optimization end condition to obtain the first optimized model and its corresponding texture information and camera pose.
[0073] Optionally, it also includes: collecting images of the target object at multiple perspectives as sample images with texture information; and training the initial network to output texture information based on position coordinates on the target reconstruction model according to the sample images and the target reconstruction model to obtain a texture prediction network.
[0074] Optionally, if the topological structure and camera pose of the first optimization model are not adjustable, at least the geometric structure of the first optimization model is optimized according to the second loss function between the second rendered image and the original image to obtain the second optimization model and its corresponding texture information and camera pose, including: when the second loss function does not meet the second optimization end condition, optimizing the geometric structure and texture information of the first optimization model to obtain the second optimization model and its texture information and camera pose.
[0075] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.
[0076] like Figure 5 As shown, the embodiment of the present application also provides a model optimization method, which can be executed by an electronic device, which can be a terminal device such as a mobile phone, a computer, and a tablet computer, or a server such as a cloud server or a common server. This embodiment does not limit the type of electronic device. The method includes the following steps: Step 51: receiving an initial mesh model of a target object sent by a terminal device, where the initial mesh model is obtained by three-dimensionally reconstructing an original image containing the target object; Step 52: reconstructing the initial mesh model into a differentiable mesh to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; Step 53: Taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstructed model as the optimization target, the reconstructed model is optimized to obtain the target mesh model; Step 54: Send the target grid model to the terminal device so that the terminal device can use the target grid model.
[0077] In this embodiment, the terminal device may be a merchant terminal or a user terminal. In the case where the terminal device is a merchant terminal, the merchant terminal may send an initial grid model of the target object to the electronic device. The initial grid model may be generated locally by the merchant terminal according to the original image, or may be manually made by the merchant's service personnel using the merchant terminal, or may be received by the merchant terminal from other terminal devices, which is not limited in this embodiment. After receiving the initial grid model, the electronic device may execute the above steps 52-53 to obtain the target grid model. Afterwards, the electronic device may send the target grid model to the merchant terminal for use by the merchant terminal. Optionally, the merchant terminal may publish the target grid model to the product details page for the user terminal to view, or may use the target grid model and the corresponding three-dimensional scene information to construct the three-dimensional scene of the product, and obtain the grid model under the target scene for the user terminal to view. Among them, the three-dimensional scene information is used to describe the scene in which the target grid model is located. For example, if the target grid model is a cup of milk, then the three-dimensional scene information corresponding to the target grid model may be relevant information of the grassland scene. The target grid model and the corresponding three-dimensional scene information are used to construct the three-dimensional scene of the product, and the milk under the grassland scene can be obtained. Optionally, the merchant terminal also publishes the target grid model to the product details page for the user terminal to view. In the case where the terminal device is a user terminal, the user terminal may send an initial mesh model of the target object to the electronic device. The initial mesh model may be generated locally by the user terminal based on the original image, or may be manually made by the user using the user terminal, or may be received by the user terminal from other terminal devices, and this embodiment does not limit this. After receiving the initial mesh model, the electronic device may execute the above steps 52-53 to obtain the target mesh model. Afterwards, the electronic device may send the target mesh model to the user terminal for use by the user terminal. Optionally, in a makeup trial scenario, the target object may be the user himself or the user's friends, family, etc., then the target mesh model is a three-dimensional model of the user, the user's friends, family, etc., and the user terminal may display the target mesh model to the user, so that the user can add makeup effects to the target mesh model to simulate makeup trials. Optionally, in a metaverse scenario, the user terminal may display the target mesh model to the user, so that the user can control the target mesh model to move and interact in the virtual world.
[0078] Optionally, the initial mesh model is reconstructed into a differentiable mesh to obtain a target reconstruction model that can represent the initial mesh model and has differentiable properties, including: determining initial differentiable parameters and initial stereo units required to represent the differentiable mesh, the initial stereo units having a default shape; adjusting the shape of the initial stereo units to obtain target stereo units that match the shape of the target object; fitting the initial mesh model according to the initial differentiable parameters and the target stereo units to obtain a target reconstruction model, the target reconstruction model having target differentiable parameters.
[0079] Optionally, the initial mesh model is fitted according to the initial differentiable parameters and the target stereo units to obtain the target reconstruction model, including: covering the initial mesh model with the target stereo units according to the initial differentiable parameters to obtain the initial reconstruction model; taking the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, and iteratively optimizing the differentiable parameters of the initial reconstruction model using a gradient descent algorithm to obtain the target reconstruction model.
[0080] Optionally, taking the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, the gradient descent algorithm is used to iteratively optimize the differentiable parameters of the initial reconstruction model to obtain the target reconstruction model, including: calculating the chamfer distance between the initial reconstruction model and the initial mesh model as the similarity error between the two; when the chamfer distance does not meet the set distance condition, iteratively optimizing the differentiable parameters of the initial reconstruction model according to the gradient vector of the chamfer distance; until the chamfer distance meets the set distance condition, the initial reconstruction model when the chamfer distance meets the set distance condition is used as the target reconstruction model.
[0081] Optionally, the target reconstruction model is optimized with the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as the optimization target, so as to obtain the target mesh model, including: inputting the target reconstruction model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a first rendered image; according to the first loss function between the first rendered image and the original image, at least optimizing the topological structure of the target reconstruction model to obtain the first optimized model and its corresponding texture information and camera pose; inputting the first optimized model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a second rendered image; according to the second loss function between the second rendered image and the original image, at least optimizing the geometric structure of the first optimized model to obtain the second optimized model and its corresponding texture information and camera pose; generating the target mesh model according to the second optimized model and its corresponding texture information and camera pose. Among them, the model optimization process based on differentiable rendering technology can make the geometric structure of the three-dimensional model more accurate and the topological structure more regular, thereby improving the overall quality of the reconstructed three-dimensional model.
[0082] Optionally, before inputting the target reconstruction model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering, the method further includes: predicting the texture information of the target reconstruction model based on the texture prediction network corresponding to the target object, and obtaining the texture information corresponding to the target reconstruction model; estimating the camera pose of the original image, and obtaining the camera pose corresponding to the target reconstruction model.
[0083] Optionally, according to a first loss function between the first rendered image and the original image, at least the topological structure of the target reconstruction model is optimized to obtain a first optimized model and its corresponding texture information and camera pose, including: when the first loss function does not meet the first optimization end condition, the topological structure of the target reconstruction model and at least one of the geometric structure, camera pose and texture prediction network of the target reconstruction model are jointly optimized until the first loss function meets the first optimization end condition to obtain the first optimized model and its corresponding texture information and camera pose.
[0084] Optionally, it also includes: collecting images of the target object at multiple perspectives as sample images with texture information; and training the initial network to output texture information based on position coordinates on the target reconstruction model according to the sample images and the target reconstruction model to obtain a texture prediction network.
[0085] Optionally, if the topological structure and camera pose of the first optimization model are not adjustable, at least the geometric structure of the first optimization model is optimized according to the second loss function between the second rendered image and the original image to obtain the second optimization model and its corresponding texture information and camera pose, including: when the second loss function does not meet the second optimization end condition, optimizing the geometric structure and texture information of the first optimization model to obtain the second optimization model and its texture information and camera pose.
[0086] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.
[0087] In this embodiment, an initial mesh model of a target object sent by a terminal device can be received, and the initial mesh model is obtained by three-dimensional reconstruction of an original image containing the target object; a microgrid reconstruction technology is introduced on the basis of the initial mesh model, and a microgrid reconstruction is performed on the initial mesh model to obtain a target reconstruction model that can express the initial mesh model and has microgrid properties; on this basis, a microgrid rendering technology is introduced, and the loss function between the original image and the rendered image obtained by microgrid rendering of the target reconstruction model is used as the optimization target, and the target reconstruction model is optimized to obtain a target mesh model; the target mesh model is sent to the terminal device so that the terminal device can use the target mesh model. In this way, the model optimization process based on microgrid reconstruction and microgrid rendering technology can make the geometric structure of the three-dimensional model more accurate and the topological structure more regular, thereby improving the overall quality of the three-dimensional model.
[0088] like Figure 6As shown, the embodiment of the present application also provides a 3D reconstruction method, which can be executed by a 3D reconstruction platform. In terms of implementation form, the 3D reconstruction platform can be implemented as an electronic device, or as a 3D reconstruction tool or component running on an electronic device. The electronic device can be a terminal device such as a mobile phone, a computer, and a tablet computer, or a server such as a cloud server or an ordinary server. This embodiment does not limit the type of electronic device. The method includes the following steps: Step 61: receiving an original image containing a target object sent by a terminal device, and performing three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model; Step 62: Perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; Step 63: Taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstructed model as the optimization target, the reconstructed model is optimized to obtain the target mesh model; Step 64: Send the target grid model to the terminal device so that the terminal device can use the target grid model.
[0089] In this embodiment, the terminal device may be a user terminal or a merchant terminal, and the 3D reconstruction platform may provide the terminal device with a 3D reconstruction service and an original image upload portal, and the original image upload portal may be implemented as a web page or an email. For example, the 3D reconstruction platform may provide a web page for the 3D reconstruction service, in which an original image upload control is provided, and the user or merchant may upload the original image based on the original image upload control. In addition, the user or merchant may also send the original image to the 3D reconstruction platform via an email.
[0090] After receiving the initial mesh model, the 3D reconstruction platform can execute the above steps 62-63 to obtain the target mesh model. Afterwards, the 3D reconstruction platform can send the target mesh model to the user terminal for use by the user terminal. Optionally, in a makeup trial scenario, the target object can be the user himself or the user's friends, family, etc., then the target mesh model is a 3D model of the user, the user's friends, family, etc., and the user terminal can display the target mesh model to the user so that the user can add makeup effects to the target mesh model to simulate makeup trials. Optionally, in a metaverse scenario, the user terminal can display the target mesh model to the user so that the user can control the target mesh model to move and interact in the virtual world.
[0091] Optionally, the initial mesh model is reconstructed into a differentiable mesh to obtain a target reconstruction model that can represent the initial mesh model and has differentiable properties, including: determining initial differentiable parameters and initial stereo units required to represent the differentiable mesh, the initial stereo units having a default shape; adjusting the shape of the initial stereo units to obtain target stereo units that match the shape of the target object; fitting the initial mesh model according to the initial differentiable parameters and the target stereo units to obtain a target reconstruction model, the target reconstruction model having target differentiable parameters.
[0092] Optionally, the initial mesh model is fitted according to the initial differentiable parameters and the target stereo units to obtain the target reconstruction model, including: covering the initial mesh model with the target stereo units according to the initial differentiable parameters to obtain the initial reconstruction model; taking the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, and iteratively optimizing the differentiable parameters of the initial reconstruction model using a gradient descent algorithm to obtain the target reconstruction model.
[0093] Optionally, taking the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, the gradient descent algorithm is used to iteratively optimize the differentiable parameters of the initial reconstruction model to obtain the target reconstruction model, including: calculating the chamfer distance between the initial reconstruction model and the initial mesh model as the similarity error between the two; when the chamfer distance does not meet the set distance condition, iteratively optimizing the differentiable parameters of the initial reconstruction model according to the gradient vector of the chamfer distance; until the chamfer distance meets the set distance condition, the initial reconstruction model when the chamfer distance meets the set distance condition is used as the target reconstruction model.
[0094] Optionally, the target reconstruction model is optimized with the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as the optimization target, so as to obtain the target mesh model, including: inputting the target reconstruction model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a first rendered image; according to the first loss function between the first rendered image and the original image, at least optimizing the topological structure of the target reconstruction model to obtain the first optimized model and its corresponding texture information and camera pose; inputting the first optimized model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a second rendered image; according to the second loss function between the second rendered image and the original image, at least optimizing the geometric structure of the first optimized model to obtain the second optimized model and its corresponding texture information and camera pose; generating the target mesh model according to the second optimized model and its corresponding texture information and camera pose. Among them, the model optimization process based on differentiable rendering technology can make the geometric structure of the three-dimensional model more accurate and the topological structure more regular, thereby improving the overall quality of the reconstructed three-dimensional model.
[0095] Optionally, before inputting the target reconstruction model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering, the method further includes: predicting the texture information of the target reconstruction model based on the texture prediction network corresponding to the target object, and obtaining the texture information corresponding to the target reconstruction model; estimating the camera pose of the original image, and obtaining the camera pose corresponding to the target reconstruction model.
[0096] Optionally, according to a first loss function between the first rendered image and the original image, at least the topological structure of the target reconstruction model is optimized to obtain a first optimized model and its corresponding texture information and camera pose, including: when the first loss function does not meet the first optimization end condition, the topological structure of the target reconstruction model and at least one of the geometric structure, camera pose and texture prediction network of the target reconstruction model are jointly optimized until the first loss function meets the first optimization end condition to obtain the first optimized model and its corresponding texture information and camera pose.
[0097] Optionally, it also includes: collecting images of the target object at multiple perspectives as sample images with texture information; and training the initial network to output texture information based on position coordinates on the target reconstruction model according to the sample images and the target reconstruction model to obtain a texture prediction network.
[0098] Optionally, if the topological structure and camera pose of the first optimization model are not adjustable, at least the geometric structure of the first optimization model is optimized according to the second loss function between the second rendered image and the original image to obtain the second optimization model and its corresponding texture information and camera pose, including: when the second loss function does not meet the second optimization end condition, optimizing the geometric structure and texture information of the first optimization model to obtain the second optimization model and its texture information and camera pose.
[0099] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.
[0100] In this embodiment, an original image containing a target object sent by a terminal device can be received, and the target object can be reconstructed in three dimensions based on the original image to obtain an initial mesh model; a microgrid reconstruction technology is introduced on the basis of the initial mesh model, and the initial mesh model is reconstructed in a microgrid to obtain a target reconstruction model that can express the initial mesh model and has microgrid properties; on this basis, a microgrid rendering technology is introduced, and the target reconstruction model is optimized with the loss function between the original image and the rendered image obtained by microgrid rendering the target reconstruction model as the optimization target to obtain a target mesh model; the target mesh model is sent to the terminal device so that the terminal device can use the target mesh model. In this way, the model optimization process based on microgrid reconstruction and microgrid rendering technology can make the geometric structure of the three-dimensional model more accurate and the topological structure more regular, thereby improving the overall quality of the reconstructed three-dimensional model.
[0101] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 11 to 13 can be device A; for another example, the execution subject of steps 11 and 12 can be device A, and the execution subject of step 13 can be device B; and so on.
[0102] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 11, 12, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.
[0103] Figure 7 A schematic diagram of a model optimization device provided by another exemplary embodiment of the present application. Figure 7 As shown, the device comprises: The acquisition module 701 is used to: acquire an initial mesh model of the target object, wherein the initial mesh model is obtained by performing three-dimensional reconstruction on an original image containing the target object; the reconstruction module 702 is used to: perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; the optimization module 703 is used to: optimize the target reconstruction model using the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as the optimization target to obtain the target mesh model.
[0104] Optionally, when the reconstruction module 702 performs a differentiable mesh reconstruction on the initial mesh model to obtain a target reconstruction model that can represent the initial mesh model and has differentiable properties, it is specifically used to: determine initial differentiable parameters and initial three-dimensional units required to represent the differentiable mesh, and the initial three-dimensional units have a default shape; adjust the shape of the initial three-dimensional units to obtain target three-dimensional units that match the shape of the target object; fit the initial mesh model according to the initial differentiable parameters and the target three-dimensional units to obtain a target reconstruction model, and the target reconstruction model has target differentiable parameters.
[0105] Optionally, when the reconstruction module 702 fits the initial mesh model according to the initial differentiable parameters and the target stereo units to obtain the target reconstructed model, it is specifically used to: cover the initial mesh model with the target stereo units according to the initial differentiable parameters to obtain the initial reconstructed model; and iteratively optimize the differentiable parameters of the initial reconstructed model using a gradient descent algorithm with a similarity error between the initial mesh model and the initial reconstructed model as an optimization target to obtain the target reconstructed model.
[0106] Optionally, the reconstruction module 702 takes the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, and adopts the gradient descent algorithm to iteratively optimize the differentiable parameters of the initial reconstruction model to obtain the target reconstruction model, and is specifically used to: calculate the chamfer distance between the initial reconstruction model and the initial mesh model as the similarity error between the two; when the chamfer distance does not meet the set distance condition, iteratively optimize the differentiable parameters of the initial reconstruction model according to the gradient vector of the chamfer distance; until the chamfer distance meets the set distance condition, the initial reconstruction model when the chamfer distance meets the set distance condition is used as the target reconstruction model.
[0107] Optionally, when the optimization module 703 optimizes the target reconstruction model with a loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as an optimization target to obtain a target mesh model, it is specifically used to: input the target reconstruction model and its corresponding texture information and camera pose into a differentiable renderer for differentiable rendering to obtain a first rendered image; according to a first loss function between the first rendered image and the original image, at least optimize the topological structure of the target reconstruction model to obtain a first optimized model and its corresponding texture information and camera pose; input the first optimized model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a second rendered image; according to a second loss function between the second rendered image and the original image, at least optimize the geometric structure of the first optimized model to obtain a second optimized model and its corresponding texture information and camera pose; generate the target mesh model according to the second optimized model and its corresponding texture information and camera pose.
[0108] Optionally, before inputting the target reconstruction model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering, the optimization module 703 is also used to: predict the texture information of the target reconstruction model based on the texture prediction network corresponding to the target object, and obtain the texture information corresponding to the target reconstruction model; estimate the camera pose of the original image, and obtain the camera pose corresponding to the target reconstruction model.
[0109] Optionally, the optimization module 703 optimizes at least the topological structure of the target reconstruction model according to the first loss function between the first rendered image and the original image to obtain a first optimized model and its corresponding texture information and camera pose, and is specifically used to: when the first loss function does not meet the first optimization end condition, jointly optimize the topological structure of the target reconstruction model and at least one of the geometric structure, camera pose and texture prediction network of the target reconstruction model until the first loss function meets the first optimization end condition to obtain the first optimized model and its corresponding texture information and camera pose.
[0110] Optionally, the optimization module 703 is also used to: collect images of the target object from multiple perspectives as sample images with texture information; and train the initial network to output texture information based on the position coordinates on the target reconstruction model according to the sample images and the target reconstruction model to obtain the texture prediction network.
[0111] Optionally, if the topological structure and camera pose of the first optimization model are not adjustable, the optimization module 703 optimizes at least the geometric structure of the first optimization model according to the second loss function between the second rendered image and the original image to obtain the second optimization model and its corresponding texture information and camera pose. Specifically, when the second loss function does not meet the second optimization end condition, the geometric structure and texture information of the first optimization model are optimized to obtain the second optimization model and its texture information and camera pose.
[0112] Figure 8 FIG. 1 is a schematic diagram of a three-dimensional reconstruction device provided by another exemplary embodiment of the present application. Figure 8 As shown, the device comprises: The acquisition module 801 is used to: acquire an original image containing a target object, and perform three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model; the reconstruction module 802 is used to: perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; the optimization module 803 is used to: optimize the target reconstructed model using the loss function between the original image and the rendered image obtained by performing differentiable rendering of the target reconstructed model as the optimization target to obtain a target mesh model.
[0113] The detailed implementation and beneficial effects of each module in the device of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.
[0114] Fig. 9 A schematic diagram of a model optimization device provided by another exemplary embodiment of the present application. Fig. 9 As shown, the device comprises: A receiving module 901 is used to receive an initial mesh model of a target object sent by a terminal device, wherein the initial mesh model is obtained by performing three-dimensional reconstruction on an original image containing the target object; a reconstruction module 902 is used to perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; an optimization module 903 is used to optimize the target reconstruction model by taking a loss function between the original image and a rendered image obtained by performing differentiable rendering on the target reconstruction model as an optimization target to obtain a target mesh model; a sending module 904 is used to send the target mesh model to the terminal device so that the terminal device can use the target mesh model.
[0115] The detailed implementation and beneficial effects of each module in the device of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.
[0116] Fig.10 FIG. 1 is a schematic diagram of a three-dimensional reconstruction device provided by another exemplary embodiment of the present application. Fig.10 As shown, the device comprises: The receiving module 1001 is used to: receive an original image containing a target object sent by a terminal device, and perform three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model; the reconstruction module 1002 is used to: perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; the optimization module 1003 is used to: optimize the target reconstruction model with a loss function between the original image and a rendered image obtained by differentiable rendering of the target reconstruction model as an optimization target to obtain a target mesh model; the sending module 1004 is used to: send the target mesh model to the terminal device so that the terminal device can use the target mesh model.
[0117] The detailed implementation and beneficial effects of each module in the device of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.
[0118] The above describes the internal functions and structures of the model optimization device and the 3D reconstruction device. Fig.11 As shown, in practice, each of the above devices can be implemented as an electronic device, including: a memory 1101 , a processor 1102 and a communication component 1103 .
[0119] The memory 1101 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.
[0120] The memory 1101 may be implemented by any type of volatile or nonvolatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0121] In some optional embodiments, the processor 1102 is coupled to the memory 1101, and is used to execute a computer program in the memory 1101, so as to: obtain an initial mesh model of the target object, wherein the initial mesh model is obtained by performing three-dimensional reconstruction on an original image containing the target object; perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; and optimize the target reconstruction model using a loss function between the original image and a rendered image obtained by performing differentiable rendering on the target reconstructed model as an optimization target to obtain a target mesh model.
[0122] In other optional embodiments, the processor 1102 is coupled to the memory 1101, and is used to execute a computer program in the memory 1101, so as to: obtain an original image containing a target object, and perform three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model; perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; optimize the target reconstruction model using a loss function between the original image and a rendered image obtained by performing differentiable rendering of the target reconstructed model as an optimization target to obtain a target mesh model.
[0123] In some other optional embodiments, the processor 1102 is coupled to the memory 1101, and is used to execute a computer program in the memory 1101, so as to: receive an initial mesh model of a target object sent by a terminal device, wherein the initial mesh model is obtained by performing three-dimensional reconstruction on an original image containing the target object; perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; optimize the reconstructed model with a loss function between the original image and a rendered image obtained by performing differentiable rendering on the target reconstructed model as an optimization target to obtain a target mesh model; and send the target mesh model to the terminal device so that the terminal device can use the target mesh model.
[0124] In some other optional embodiments, the processor 1102 is coupled to the memory 1101, and is used to execute the computer program in the memory 1101, so as to: receive an original image containing a target object sent by a terminal device, and perform three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model; perform differentiable mesh reconstruction on the initial mesh model to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; optimize the target reconstruction model with a loss function between the original image and a rendered image obtained by differentiable rendering of the target reconstruction model as an optimization target to obtain a target mesh model; and send the target mesh model to the terminal device so that the terminal device can use the target mesh model.
[0125] Optionally, in each of the above embodiments, when the processor 1102 performs a differentiable mesh reconstruction on the initial mesh model to obtain a target reconstruction model that can represent the initial mesh model and has differentiable properties, it is specifically used to: determine initial differentiable parameters and initial three-dimensional units required to represent the differentiable mesh, and the initial three-dimensional units have a default shape; adjust the shape of the initial three-dimensional units to obtain a target three-dimensional unit that matches the shape of the target object; fit the initial mesh model according to the initial differentiable parameters and the target three-dimensional units to obtain a target reconstruction model, and the target reconstruction model has target differentiable parameters.
[0126] Optionally, in each of the above embodiments, when the processor 1102 fits the initial mesh model according to the initial differentiable parameters and the target stereo unit to obtain the target reconstruction model, it is specifically used to: cover the initial mesh model with the target stereo unit according to the initial differentiable parameters to obtain the initial reconstruction model; and iteratively optimize the differentiable parameters of the initial reconstruction model using a gradient descent algorithm with a similarity error between the initial mesh model and the initial reconstruction model as an optimization target to obtain the target reconstruction model.
[0127] Optionally, in each of the above embodiments, the processor 1102 takes the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, and adopts the gradient descent algorithm to iteratively optimize the differentiable parameters of the initial reconstruction model to obtain the target reconstruction model, and is specifically used to: calculate the chamfer distance between the initial reconstruction model and the initial mesh model as the similarity error between the two; when the chamfer distance does not meet the set distance condition, iteratively optimize the differentiable parameters of the initial reconstruction model according to the gradient vector of the chamfer distance; until the chamfer distance meets the set distance condition, the initial reconstruction model when the chamfer distance meets the set distance condition is used as the target reconstruction model.
[0128] Optionally, in each of the above embodiments, when the processor 1102 optimizes the target reconstruction model with a loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as an optimization target to obtain a target mesh model, it is specifically used to: input the target reconstruction model and its corresponding texture information and camera pose into a differentiable renderer for differentiable rendering to obtain a first rendered image; according to a first loss function between the first rendered image and the original image, at least optimize the topological structure of the target reconstruction model to obtain a first optimized model and its corresponding texture information and camera pose; input the first optimized model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a second rendered image; according to a second loss function between the second rendered image and the original image, at least optimize the geometric structure of the first optimized model to obtain a second optimized model and its corresponding texture information and camera pose; generate the target mesh model according to the second optimized model and its corresponding texture information and camera pose. Among them, the model optimization process based on differentiable rendering technology can make the geometric structure of the three-dimensional model more accurate and the topological structure more regular, thereby improving the overall quality of the reconstructed three-dimensional model.
[0129] Optionally, in each of the above embodiments, before inputting the target reconstruction model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering, the processor 1102 is also used to: predict texture information of the target reconstruction model based on a texture prediction network corresponding to the target object to obtain texture information corresponding to the target reconstruction model; and estimate the camera pose of the original image to obtain a camera pose corresponding to the target reconstruction model.
[0130] Optionally, in each of the above embodiments, the processor 1102 optimizes at least the topological structure of the target reconstruction model according to a first loss function between the first rendered image and the original image to obtain a first optimized model and its corresponding texture information and camera pose, and is specifically used to: when the first loss function does not meet the first optimization end condition, jointly optimize the topological structure of the target reconstruction model and at least one of the geometric structure, camera pose and texture prediction network of the target reconstruction model until the first loss function meets the first optimization end condition to obtain the first optimized model and its corresponding texture information and camera pose.
[0131] Optionally, in each of the above embodiments, the processor 1102 is also used to: collect images of the target object from multiple perspectives as sample images with texture information; and train the initial network to output texture information based on the position coordinates on the target reconstruction model according to the sample images and the target reconstruction model to obtain the texture prediction network.
[0132] Optionally, if the topological structure and camera pose of the first optimization model are not adjustable, the processor 1102 optimizes at least the geometric structure of the first optimization model according to a second loss function between the second rendered image and the original image to obtain a second optimization model and its corresponding texture information and camera pose. Specifically, when the second loss function does not satisfy the second optimization end condition, the geometric structure and texture information of the first optimization model are optimized to obtain the second optimization model and its texture information and camera pose.
[0133] Further, if Fig.11 As shown, the electronic device also includes: a display 1104, a power component 1105, an audio component 1106 and other components. Fig.11 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.11 In addition, Fig.11 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the working node. The working node of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server-side device such as a conventional server, a cloud server, or a server array. If the working node of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Fig.11 If the working node of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Fig.11 Components within the dashed box.
[0134] In this embodiment, either an initial mesh model of the target object is obtained, or a three-dimensional reconstruction is performed based on the original image of the target object to obtain an initial mesh model; a differentiable mesh reconstruction technology is introduced on the basis of the initial mesh model, and a differentiable mesh reconstruction is performed on the initial mesh model to obtain a target reconstruction model that can express the initial mesh model and has differentiable properties; on this basis, a differentiable rendering technology is introduced, and the loss function between the original image and the rendered image obtained by differentiable rendering of the target reconstruction model is used as the optimization target, and the target reconstruction model is optimized to obtain a target mesh model. In this way, the model optimization process based on differentiable mesh reconstruction and differentiable rendering technology can make the geometric structure of the three-dimensional model more accurate and the topological structure more regular, thereby improving the overall quality of the reconstructed three-dimensional model.
[0135] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be performed by an electronic device in the above method embodiment.
[0136] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0137] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.
[0138] The display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
[0139] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.
[0140] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.
[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program codes.
[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0145] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.
[0146] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0149] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A model optimization method, characterized in that: include: Acquire an initial mesh model of the target object, wherein the initial mesh model is obtained by three-dimensionally reconstructing an original image containing the target object; Performing differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; The target reconstruction model is optimized by taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as an optimization target to obtain a target mesh model.
2. The method according to claim 1, characterized in that Performing a differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can represent the initial mesh model and has differentiable properties, comprising: Determining initial differentiable parameters and initial solid units required to represent a differentiable grid, wherein the initial solid units have a default shape; Adjusting the shape of the initial three-dimensional unit to obtain a target three-dimensional unit that matches the shape of the target object; The initial mesh model is fitted according to the initial differentiable parameters and the target stereo unit to obtain a target reconstruction model, wherein the target reconstruction model has target differentiable parameters.
3. The method according to claim 2, characterized in that Fitting the initial mesh model according to the initial differentiable parameters and the target stereo unit to obtain the target reconstruction model includes: According to the initial differentiable parameters, the initial mesh model is covered with the target stereo unit to obtain an initial reconstructed model; Taking the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, a gradient descent algorithm is used to iteratively optimize the differentiable parameters of the initial reconstruction model to obtain the target reconstruction model.
4. The method according to claim 3, characterized in that Taking the similarity error between the initial mesh model and the initial reconstruction model as the optimization target, adopting the gradient descent algorithm to iteratively optimize the differentiable parameters of the initial reconstruction model to obtain the target reconstruction model, including: Calculating the chamfer distance between the initial reconstruction model and the initial mesh model as a similarity error between the two; When the chamfer distance does not meet the set distance condition, iteratively optimizing the differentiable parameters of the initial reconstruction model according to the gradient vector of the chamfer distance; Until the chamfer distance meets the set distance condition, the initial reconstruction model when the chamfer distance meets the set distance condition is used as the target reconstruction model.
5. The method according to any one of claims 1 to 4, characterized in that: Taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as the optimization target, optimizing the target reconstruction model to obtain a target mesh model, including: Inputting the target reconstruction model and its corresponding texture information and camera pose into a differentiable renderer for differentiable rendering to obtain a first rendered image; According to a first loss function between the first rendered image and the original image, at least the topological structure of the target reconstruction model is optimized to obtain a first optimized model and its corresponding texture information and camera pose; Inputting the first optimized model and its corresponding texture information and camera pose into the differentiable renderer for differentiable rendering to obtain a second rendered image; According to a second loss function between the second rendered image and the original image, at least the geometric structure of the first optimization model is optimized to obtain a second optimization model and its corresponding texture information and camera pose; The target mesh model is generated according to the second optimized model and its corresponding texture information and camera pose.
6. The method according to claim 5, characterized in that Before inputting the target reconstruction model and its corresponding texture information and camera pose into a differentiable renderer for differentiable rendering, the method further includes: Based on the texture prediction network corresponding to the target object, predicting the texture information of the target reconstruction model, to obtain the texture information corresponding to the target reconstruction model; The camera pose of the original image is estimated to obtain the camera pose corresponding to the target reconstruction model.
7. The method according to claim 6, characterized in that According to a first loss function between the first rendered image and the original image, at least the topological structure of the target reconstruction model is optimized to obtain a first optimized model and its corresponding texture information and camera pose, including: When the first loss function does not satisfy the first optimization termination condition, the topological structure of the target reconstruction model and at least one of the geometric structure, camera pose and texture prediction network of the target reconstruction model are jointly optimized until the first loss function meets the first optimization termination condition to obtain the first optimized model and its corresponding texture information and camera pose.
8. The method according to claim 6, characterized in that Also includes: Collecting images of the target object at multiple viewing angles as sample images with texture information; According to the sample image and the target reconstruction model, the initial network is trained to output texture information based on the position coordinates on the target reconstruction model to obtain the texture prediction network.
9. The method according to claim 5, characterized in that If the topological structure and camera pose of the first optimization model are not adjustable, then according to a second loss function between the second rendered image and the original image, at least the geometric structure of the first optimization model is optimized to obtain a second optimization model and its corresponding texture information and camera pose, including: When the second loss function does not satisfy the second optimization termination condition, the geometric structure and texture information of the first optimization model are optimized to obtain the second optimization model and its texture information and camera pose.
10. A three-dimensional reconstruction method, characterized in that: include: Acquire an original image containing a target object, and perform three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model; Performing differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; The target reconstruction model is optimized by taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as an optimization target to obtain a target mesh model.
11. A model optimization method, characterized in that: include: Receiving an initial mesh model of a target object sent by a terminal device, wherein the initial mesh model is obtained by three-dimensionally reconstructing an original image containing the target object; Performing differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; Taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstructed model as an optimization target, optimizing the reconstructed model to obtain a target mesh model; The target grid model is sent to the terminal device so that the terminal device uses the target grid model.
12. A three-dimensional reconstruction method, characterized in that: include: Receiving an original image containing a target object sent by a terminal device, and performing three-dimensional reconstruction of the target object based on the original image to obtain an initial mesh model; Performing differentiable mesh reconstruction on the initial mesh model to obtain a target reconstructed model that can express the initial mesh model and has differentiable properties; Taking the loss function between the original image and the rendered image obtained by performing differentiable rendering on the target reconstruction model as an optimization target, optimizing the target reconstruction model to obtain a target mesh model; The target grid model is sent to the terminal device so that the terminal device uses the target grid model.
13. An electronic device, characterized in that: include: A memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method described in any one of claims 1-9, claim 10, claim 11 or claim 12.
14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 1 to 9, claim 10, claim 11 or claim 12.
15. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, performs the steps of the method according to any one of claims 1 to 9, claim 10, claim 11 or claim 12.
Citation Information
Patent Citations
Single-perspective reconstruction method based on multi-perspective supervision
CN107330973A
Incremental three-dimensional reconstruction method and device and computer equipment
CN114140510A
Single-view-angle blade three-dimensional reconstruction method based on deep learning
CN114241160A
Method for quickly reconstructing three-dimensional model with texture based on neural shader
CN115984441A
Rapid target three-dimensional reconstruction method and device
CN116310224A
Cited By
Webpage interface reconstruction method and system based on parametric modeling and continuous micro-optimization
CN121524461A
Lightweight processing method and system for three-dimensional BIM (Building Information Modeling) of frame structure building
CN121544847A
Three-dimensional model processing method and apparatus, and computer device, computer-readable storage medium and computer program product
WO2026108490A1
